Paper deep dive
FedPA-LoRA: Product-Aligned Framework for Mitigating Aggregation and Initialization Errors in Heterogeneous Federated LoRA
Juseok Jeon, Ramy E. Ali, Doyun Kwon, Myungbeom Her, Jinhwi Kim, Jinhyun So
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 8/22/2026, 2:02:56 AM
Summary
The paper introduces FedPA-LoRA, a federated learning framework for fine-tuning Large Language Models using Low-Rank Adaptation (LoRA). It addresses the tension between aggregation fidelity and local optimization continuity by preserving local factors across rounds and aligning them with a global reference in the product space. The server aggregates updates in the product space and reconstructs a rank-constrained global adapter efficiently. FedPA-LoRA supports heterogeneous client ranks and data heterogeneity, demonstrating superior performance over baselines.
Entities (9)
Relation Signals (7)
FedPA-LoRA → addresses → aggregation mismatch
confidence 95% · FedPA-LoRA... jointly addresses these limitations [aggregation and initialization errors]
FedPA-LoRA → addresses → factor-level initialization mismatch
confidence 95% · FedPA-LoRA... jointly addresses these limitations [aggregation and initialization errors]
FedPA-LoRA → improves → GLUE accuracy
confidence 95% · FedPA-LoRA consistently outperforms representative baselines... with up to a 6.82 percentage-point improvement in average GLUE accuracy
FedPA-LoRA → supports → heterogeneous client ranks
confidence 95% · This design supports client-specific computation and communication budgets... under heterogeneous client ranks
LoRA → enables → efficient federated fine-tuning
confidence 90% · Low-Rank Adaptation (LoRA) enables efficient federated fine-tuning of large language models
FedIT → uses → factor-wise averaging
confidence 90% · FedIT (31) directly averages B and A
FlexLoRA → uses → SVD-based reconstruction
confidence 90% · FlexLoRA (2) applies truncated singular value decomposition (SVD) to the aggregated product
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Low-Rank Adaptation (LoRA) enables efficient federated fine-tuning of large language models, but its factorized parameterization creates a tension between accurate aggregation of local updates and continuity of locally optimized factors. Factor-wise aggregation incurs aggregation mismatch but better preserves factor continuity, whereas product-space reconstruction reduces this mismatch at the cost of greater factor-level initialization mismatch from newly reconstructed factors. We propose FedPA-LoRA, a product-aligned federated LoRA framework that jointly addresses these limitations and provably converges under both homogeneous and heterogeneous client ranks. Each client preserves its local factors across communication rounds and aligns its product toward a rank-specific global reference, maintaining local optimization continuity while promoting global consistency under data heterogeneity. The server aggregates heterogeneous-rank updates in the common product space and efficiently reconstructs a rank-constrained global adapter without forming the dense aggregate. This design supports client-specific computation and communication budgets. Experiments on natural language understanding and generation tasks show that FedPA-LoRA consistently outperforms representative baselines across varying levels of data heterogeneity and homogeneous- and heterogeneous-rank settings, with up to a $6.82$ percentage-point improvement in average GLUE accuracy under heterogeneous client ranks.
Tags
Links
- Source: https://arxiv.org/abs/2608.15381v1
- Canonical: https://arxiv.org/abs/2608.15381v1
Trouble viewing inline? Open PDF directly →
Full Text
160,049 characters extracted from source content.
Expand or collapse full text
FedPA-LoRA: Product-Aligned Framework for Mitigating Aggregation and Initialization Errors in Heterogeneous Federated LoRA Juseok Jeon DGIST Daegu, Republic of Korea jooseuk@dgist.ac.kr Ramy E. Ali‡ Samsung San Diego, USA ramy.ali@ieee.org Doyun Kwon DGIST Daegu, Republic of Korea ehdbs810@dgist.ac.kr Myungbeom Her DGIST Daegu, Republic of Korea myungbeom.her@dgist.ac.kr Jinhwi Kim DGIST Daegu, Republic of Korea kjh2159@dgist.ac.kr Jinhyun So† DGIST Daegu, Republic of Korea jinhyun@dgist.ac.kr Thanks: † Corresponding author. ‡ The opinions expressed in this work are solely those of the author and do not represent the views of Samsung. Abstract Low-Rank Adaptation (LoRA) enables efficient federated fine-tuning of large language models, but its factorized parameterization creates a tension between accurate aggregation of local updates and continuity of locally optimized factors. Factor-wise aggregation incurs aggregation mismatch but better preserves factor continuity, whereas product-space reconstruction reduces this mismatch at the cost of greater factor-level initialization mismatch from newly reconstructed factors. We propose FedPA-LoRA, a product-aligned federated LoRA framework that jointly addresses these limitations and provably converges under both homogeneous and heterogeneous client ranks. Each client preserves its local factors across communication rounds and aligns its product toward a rank-specific global reference, maintaining local optimization continuity while promoting global consistency under data heterogeneity. The server aggregates heterogeneous-rank updates in the common product space and efficiently reconstructs a rank-constrained global adapter without forming the dense aggregate. This design supports client-specific computation and communication budgets. Experiments on natural language understanding and generation tasks show that FedPA-LoRA consistently outperforms representative baselines across varying levels of data heterogeneity and homogeneous- and heterogeneous-rank settings, with up to a 6.826.82 percentage-point improvement in average GLUE accuracy under heterogeneous client ranks. The code is available at FedPA-LoRA-Code. Keywords Federated Learning ⋅· LLMs ⋅· LoRA 1 Introduction Large language models (LLMs) owe much of their downstream performance to the scale and diversity of the data used to adapt them (15), yet a large share of that data sits on individual devices or within institutions that cannot release it externally for privacy or regulatory reasons (22). Federated learning (FL) addresses this constraint by coordinating model training across participants without ever transferring their raw data (20). Even so, fine-tuning the full parameter set in this setting remains costly: resource-limited clients cannot feasibly compute and transmit updates at the scale of the complete model every round (32; 27). Parameter-efficient fine-tuning (PEFT) alleviates these costs by training a compact set of additional parameters and leaving the pretrained backbone unchanged (17; 12). A prominent example is Low-Rank Adaptation (LoRA), which parameterizes each weight update Δ W as the product of two low-rank matrices B and A, i.e., Δ= W= B A (13). By substantially reducing the number of trainable and communicated parameters, LoRA has become a practical approach for federated fine-tuning of large models (1; 16). Despite these advantages, LoRA’s factorized parameterization complicates federated aggregation. Factor-wise averaging preserves a compact low-rank adapter, but the product of the averaged factors generally differs from the average of the locally optimized updates, resulting in an aggregation mismatch (2; 5; 30). Product-space aggregation avoids the cross-client terms introduced by factor-wise averaging by aggregating the local updates before low-rank reconstruction. However, the reconstructed global factors may differ substantially from the local factors, and repeatedly replacing the latter can disrupt factor-level continuity across communication rounds. Existing approaches, therefore, face a tension between global aggregation fidelity and local optimization continuity. Resource heterogeneity further complicates federated LoRA. Clients with different system capabilities may require different LoRA ranks, yielding incompatible factor dimensions and necessitating rank-aware aggregation and client-specific redistribution (6; 2; 33). Federated LoRA must, therefore, support heterogeneous adaptation capacities while maintaining global knowledge sharing and stable local optimization. Contributions. We propose FedPA-LoRA, a product-aligned federated LoRA framework that jointly preserves local optimization continuity, improves aggregation fidelity, and supports heterogeneous client resources. Each client retains its locally optimized factors and aligns their product with a rank-specific global reference, without overwriting the preserved factors. The server then aggregates heterogeneous-rank updates in the common product space and efficiently reconstructs an optimal rank-constrained global adapter. This formulation decouples the local rank rir_i, which controls computation and uplink cost, from the reference rank RiR_i, which controls downlink communication. Our contributions are summarized as follows. • We propose FedPA-LoRA to jointly address aggregation and factor-level initialization mismatches through product-space aggregation and local factor preservation. FedPA-LoRA uses global product-guided alignment to maintain global consistency under data heterogeneity and supports client-specific computation and communication budgets. • We prove that FedPA-LoRA converges to a stationary point of the product-guided local objective at the standard non-convex federated stochastic gradient descent (SGD) rate in the homogeneous setting, and at the same rate up to an additive term controlled by the reference-truncation error in the heterogeneous setting. • We demonstrate the superiority of FedPA-LoRA across diverse LLM fine-tuning tasks under both data and resource heterogeneity, achieving up to a 6.826.82 percentage-point improvement in accuracy over state-of-the-art baselines. 2 Related Work We now review the federated LoRA methods most closely related to FedPA-LoRA, and defer discussion of additional related methods to Appendix A. 2.1 Federated LoRA Federated LoRA methods primarily differ in how they aggregate and redistribute the two low-rank factors. (i) Factor-wise aggregation preserves a fixed-rank global adapter by aggregating the factors independently. FedIT (31) directly averages B and A, whereas FedRot-LoRA (30) aligns client factors before averaging to reduce cross-client subspace misalignment. Although alignment reduces aggregation error, the resulting update is still obtained by factor-wise averaging and therefore does not generally equal the average of the local products. (i) Product-space reconstruction approximates aggregated local updates with a low-rank global adapter. FlexLoRA (2) applies truncated singular value decomposition (SVD) to the aggregated product, while FedSRD (28) reconstructs the global update from sparsified client updates before decomposition. Although these methods improve aggregation fidelity, they require dense reconstruction and high-dimensional decomposition, and redistributing the reconstructed factors can disrupt local optimization continuity. LoRA-FAIR (3) instead optimizes a residual for one global factor to better approximate the aggregated update, avoiding high-dimensional SVD at the cost of iterative server optimization and additional hyperparameter tuning. FedPA-LoRA instead uses reduced QR factorization and performs SVD only on a small core matrix, reducing server-side computation and memory. (i) Partial training or sharing simplifies aggregation by restricting which factors are optimized or globally shared. FFA-LoRA (24) freezes A and trains only B, whereas RoLoRA (5) alternates the trainable factor across communication rounds. Although these approaches make aggregation linear, freezing one factor restricts joint optimization of the two factors and may slow convergence or limit adaptation capacity. FedSA-LoRA (10) instead trains both factors locally but aggregates only A, leading to an incomplete global adapter. (iv) High-communication aggregation achieves exact aggregation through additional communication. FedEx-LoRA (23) transmits the residual between the average of the local products and the factor-wise average and incorporates it into the shared pre-trained weights, eliminating aggregation mismatch. However, this modifies the shared backbone at every round and does not support heterogeneous client ranks. FLoRA (26) instead achieves exact aggregation by stacking local factors, increasing the global rank and communication overhead. These approaches therefore undermine the motivation for LoRA in bandwidth-constrained FL. 2.2 Heterogeneous Federated LoRA Resource heterogeneity motivates assigning client-specific LoRA ranks according to local computation and communication budgets. HetLoRA (6) zero-pads heterogeneous factors to a common rank for factor-wise aggregation and truncates the global factors for each client. However, independently averaging the padded factors still introduces aggregation mismatch. FlexLoRA (2) aggregates local products and applies SVD-based reconstruction to redistribute rank-specific factors. FLoRA instead exactly aggregates heterogeneous-rank updates by stacking local factors. However, its global rank grows with the sum of client ranks, increasing communication and adapter size, and its per-round factor reinitialization disrupts optimization continuity. Ravan (21) employs multiple fixed-basis heads and trains resource-dependent intermediate matrices, restricting local adaptation to subspaces determined at initialization. Fed-PLoRA (33) decomposes LoRA into parallel rank-one modules and allows each client to train a resource-dependent subset through Select-N-Fold, but distributes the complete collection of global modules to every client. Our work. Unlike the prior approaches, FedPA-LoRA jointly preserves local optimization continuity and improves aggregation fidelity by retaining client-specific factors while aggregating heterogeneous-rank updates in a common product space. It further decouples the local computation rank from the communication rank. 3 Motivating Example Federated LoRA exhibits a fundamental tension between accurate aggregation of local weight updates and continuity of locally optimized factors. We illustrate this trade-off using FedIT, FlexLoRA, and FedRot-LoRA as representative baselines and show that FedPA-LoRA jointly reduces aggregation mismatch and eliminates factor-level initialization mismatch. 3.1 Aggregation Mismatch We consider an FL setting with N clients. At communication round t, client i obtains a local LoRA update Δi(t)=i(t)i(t) W_i^(t)= B_i^(t) A_i^(t), where i(t)∈ℝd×r B_i^(t) ^d× r, i(t)∈ℝr×d A_i^(t) ^r× d, with d denoting the feature dimension and r≪dr d representing the LoRA rank. The ideal global update is the average of the locally optimized products, Δideal(t)=1N∑i=1Ni(t)i(t) W_ideal^(t)= 1N _i=1^N B_i^(t) A_i^(t), whose rank can be as large as min(Nr,d) (Nr,d). FedIT preserves rank r by independently averaging the local factors, g(t)=1N∑i(t) B_g^(t)= 1N _i B_i^(t) and g(t)=1N∑i(t) A_g^(t)= 1N _i A_i^(t). However, g(t)g(t)≠Δideal(t) B_g^(t) A_g^(t)≠ W_ideal^(t) in general, because the product of the averaged factors contains cross-client terms i(t)j(t) B_i^(t) A_j^(t) for i≠ji≠ j, which do not correspond to any locally optimized update. We refer to this discrepancy as the aggregation mismatch. FedRot-LoRA reduces this aggregation mismatch by applying a client-specific orthogonal rotation matrix i(t)∈ℝr×r R_i^(t) ^r× r to align the local factors before averaging. Specifically, it transforms the local factors as ~i(t)=i(t)i(t) B_i^(t)= B_i^(t) R_i^(t) and ~i(t)=(i(t))⊤i(t) A_i^(t)=( R_i^(t)) A_i^(t), where (i(t))⊤i(t)=r( R_i^(t)) R_i^(t)= I_r and det(i(t))>0 ( R_i^(t))>0. This transformation preserves each local product while improving consistency among the factors being aggregated. Nevertheless, the aligned factors are still averaged independently and therefore do not generally recover Δideal(t) W_ideal^(t) exactly. 3.2 Factor-Level Initialization Mismatch FlexLoRA avoids factor-wise aggregation by computing the rank-r truncated SVD of the ideal update, Δideal(t)≈r(t)r(t)(r(t))⊤ W_ideal^(t)≈ U_r^(t) _r^(t)( V_r^(t)) , where r(t),r(t)∈ℝd×r U_r^(t), V_r^(t) ^d× r have orthonormal columns and r(t)∈ℝr×r _r^(t) ^r× r is diagonal. It then reconstructs the global factors as g(t)=r(t)r(t) B_g^(t)= U_r^(t) _r^(t) and g(t)=(r(t))⊤ A_g^(t)=( V_r^(t)) . This yields the optimal rank-r approximation of the ideal update under the Frobenius norm. However, low-rank factorization is not unique. The SVD-based pair (g(t),g(t))( B_g^(t), A_g^(t)) is only one representative of an entire family of factor pairs that realize the same update. In particular, for any invertible matrix ∈ℝr×r Q ^r× r, the transformed factors g(t) B_g^(t) Q and −1g(t) Q^-1 A_g^(t) satisfy (g(t))(−1g(t))=g(t)g(t)( B_g^(t) Q)( Q^-1 A_g^(t))= B_g^(t) A_g^(t). Product-level equivalence therefore does not imply consistency between the corresponding factors. Replacing the locally optimized factors (i(t),i(t))( B_i^(t), A_i^(t)) with newly reconstructed global factors at the next round can disrupt local optimization continuity, resulting in a factor-level initialization mismatch. FedRot-LoRA provides an intermediate compromise. Its rotational alignment can reduce factor-wise aggregation error, but it modifies the local factors, and clients are still initialized with the aggregated global factors in the next round. It therefore preserves neither the exact average of the local products nor the locally optimized factors. 3.3 Empirical Observation We empirically examine the trade-off between aggregation fidelity and factor-level continuity using RoBERTa-Large (18) on MNLI with three clients, each using a common LoRA rank of r=4r=4. The local datasets are partitioned by a Dirichlet distribution with concentration parameter β=0.5β=0.5 to introduce data heterogeneity. We quantify the aggregation mismatch as the Frobenius distance between the aggregated global update and the average local update: Eagg(t)=‖g(t)g(t)−Δideal(t)‖F,E_agg^(t)= B_g^(t) A_g^(t)- W_ideal^(t) _F, (1) where Δideal(t)=1N∑i=1Ni(t)i(t) W_ideal^(t)= 1N _i=1^N B_i^(t) A_i^(t). To account for the different scales of the LoRA factors B and A, we measure factor-level initialization mismatch using a relative error. For each factor ∈, Y∈\ B, A\, we define Einit(t)()=1N∑i=1N‖i(t+1,0)−i(t)‖F‖i(t+1,0)‖F+ε,E_init^(t)( Y)= 1N _i=1^N Y_i^(t+1,0)- Y_i^(t) _F Y_i^(t+1,0) _F+ , (2) where ε>0 >0 avoids division by zero, i(t) Y_i^(t) denotes client i’s locally optimized factor at the end of round t, and i(t+1,0) Y_i^(t+1,0) its initialization at round t+1t+1. This metric measures the factor-level change at next-round initialization rather than natural differences among locally optimized factors caused by data heterogeneity. (a) Aggregation Error (b) Initialization Error of B (c) Initialization Error of A Figure 1: Aggregation error and relative factor-level initialization errors on MNLI with three clients. Panels (a)–(c) show the aggregation error and the relative initialization errors of B and A, respectively, for the query projection in layer 0 of RoBERTa-Large. The y-axes use logarithmic scales, with FedPA-LoRA’s zero initialization errors shown at the plotting floor. Fig. 1 reveals a clear trade-off between aggregation fidelity and factor-level continuity. FedRot-LoRA reduces the aggregation error of FedIT through factor alignment, while FlexLoRA further lowers it through optimal rank-r reconstruction of the averaged local products. FedPA-LoRA achieves the lowest observed error by combining optimal server-side reconstruction of the averaged products with product-guided alignment during local training, which keeps local products close to the global reference. The factor-level initialization errors exhibit the opposite trend. FedIT has the smallest mismatch among the baselines because it averages local factors without refactorization, whereas FlexLoRA has the largest due to SVD-based factor replacement. FedRot-LoRA lies between them because rotation modifies the factors less severely than refactorization. The larger errors for B than for A are consistent with prior findings that B captures more client-specific adaptation under non-IID data (10). FedPA-LoRA preserves local factors across rounds and therefore has zero initialization error, addressing both mismatches simultaneously. 4 Proposed Method: FedPA-LoRA We consider an FL system with N clients that collaboratively learn a global LoRA adapter over T communication rounds, where 0 W_0 denotes the frozen weight matrix of each adapted layer. At round t∈1,…,Tt∈\1,…,T\, client i performs τ local updates on its rank-rir_i factors, indexed by k=0,…,τ−1k=0,…,τ-1. The factors before the k-th update are denoted by (i(t,k),i(t,k))( B_i^(t,k), A_i^(t,k)), and the update produces (i(t,k+1),i(t,k+1))( B_i^(t,k+1), A_i^(t,k+1)). We denote the factors after τ local updates by (i(t),i(t)):=(i(t,τ),i(t,τ))( B_i^(t), A_i^(t)):=( B_i^(t,τ), A_i^(t,τ)). The server provides client i with a rank-RiR_i global LoRA reference that guides its locally preserved rank-rir_i factors through product-level alignment. Here, rir_i determines computation and uplink costs, whereas RiR_i determines downlink cost. The server then aggregates the local updates directly in the product space and efficiently reconstructs a low-rank global adapter without forming the dense aggregate, as illustrated in Fig. 2. Figure 2: Overview of FedPA-LoRA. Clients preserve local factors to avoid initialization mismatch and align their product i(t,k) X_i^(t,k) with a rank-RiR_i global reference. The server reconstructs the optimal rank-RgR_g adapter via reduced QR and core SVD without forming the dense aggregate. 4.1 Local Factor Preservation Conventional FL algorithms initialize local training from the global parameters at every communication round. In federated LoRA, however, replacing locally optimized factors with newly reconstructed global factors can introduce a factor-level initialization mismatch and disrupt optimization continuity. FedPA-LoRA instead preserves each client’s locally optimized factors across rounds. At initialization, the server constructs global LoRA factors (g(0),g(0))( B_g^(0), A_g^(0)) with rank rg=max1≤i≤Nrir_g= _1≤ i≤ Nr_i, where g(0) B_g^(0) is initialized to zero and g(0) A_g^(0) is initialized using Kaiming uniform initialization, following the default PEFT implementation (19). Client i obtains the rank-rir_i factors (g,i(0),g,i(0))( B_g,i^(0), A_g,i^(0)) by retaining the first rir_i columns of g(0) B_g^(0) and the first rir_i rows of g(0) A_g^(0), and initializes (i(0),i(0))←(g,i(0),g,i(0))( B_i^(0), A_i^(0))←( B_g,i^(0), A_g,i^(0)). At each subsequent round t, client i preserves its previously optimized factors by setting (i(t,0),i(t,0))←(i(t−1),i(t−1))( B_i^(t,0), A_i^(t,0))←( B_i^(t-1), A_i^(t-1)). 4.2 Global Product-Guided Alignment While local factor preservation maintains continuity, local training may still induce client drift under heterogeneous data. FedPA-LoRA therefore introduces product-level guidance without replacing the preserved factors. At round t, the server constructs a rank-RiR_i reference (g,i(t−1),g,i(t−1))( B_g,i^(t-1), A_g,i^(t-1)) for client i by retaining the first RiR_i columns of g(t−1) B_g^(t-1) and the first RiR_i rows of g(t−1) A_g^(t-1), where RiR_i is determined by its communication budget. Starting from (i(t,0),i(t,0))( B_i^(t,0), A_i^(t,0)), client i performs τ local updates to minimize ℒi(t)(i,i) _i^(t)( B_i, A_i) =fi(0+ii)+λ2‖ii−g,i(t−1)g,i(t−1)‖F2, =f_i ( W_0+ B_i A_i )+ λ2 \| B_i A_i- B_g,i^(t-1) A_g,i^(t-1) \|_F^2, (3) where 0 W_0 denotes the frozen pre-trained weight matrix, fif_i denotes the local task objective of client i and λ is a common regularization weight shared by all clients. The regularization term penalizes deviations of the local product from the global reference, thereby mitigating client drift while preserving local adaptation. By operating in the common product space, it accommodates different local and reference ranks. 4.3 Product-Space Aggregation After local training, each client sends its LoRA factors (i(t),i(t))( B_i^(t), A_i^(t)) to the server. Independently averaging these factors introduces an aggregation mismatch. We therefore aggregate the local updates directly in the product space as Δideal(t)=1N∑i=1Ni(t)i(t) W_ideal^(t)= 1N _i=1^N B_i^(t) A_i^(t). Since all products i(t)i(t) B_i^(t) A_i^(t) have the same dimensions as the adapted weight matrix, product-space aggregation naturally supports heterogeneous local ranks while avoiding factor-wise aggregation mismatch. Directly constructing and decomposing the dense aggregated update can incur substantial server-side computation and memory costs. To avoid this overhead, the server exploits the low-rank structure of the local updates. Denoting the total concatenated rank by rtot=∑i=1Nrir_tot= _i=1^Nr_i, we define cat(t)=1N[1(t),…,N(t)],cat(t)=1N[(1(t))⊤,…,(N(t))⊤]⊤. B_cat^(t)= 1 N [ B_1^(t),…, B_N^(t) ], A_cat^(t)= 1 N [( A_1^(t)) ,…,( A_N^(t)) ] . (4) These concatenated factors satisfy cat(t)cat(t)=Δideal(t) B_cat^(t) A_cat^(t)= W_ideal^(t) and thus represent the product-space aggregate without explicitly forming the dense update matrix. Under the typical LoRA setting rtot≪dr_tot d, the server computes the reduced QR factorizations cat(t)=B(t)B(t),(cat(t))⊤=A(t)A(t), B_cat^(t)= Q_B^(t) R_B^(t), ( A_cat^(t) ) = Q_A^(t) R_A^(t), (5) where B(t) Q_B^(t) and A(t) Q_A^(t) have orthonormal columns. The resulting core matrix (t)=B(t)(A(t))⊤ H^(t)= R_B^(t)( R_A^(t)) satisfies Δideal(t)=B(t)(t)(A(t))⊤ W_ideal^(t)= Q_B^(t) H^(t)( Q_A^(t)) . Setting Rg=max1≤i≤NRiR_g= _1≤ i≤ NR_i, the server computes SVDRg((t))=Rg(t)Rg(t)(Rg(t))⊤SVD_R_g( H^(t))= U_R_g^(t) _R_g^(t)( V_R_g^(t)) . The global LoRA factors are then reconstructed as g(t)=B(t)Rg(t)Rg(t),g(t)=(A(t)Rg(t))⊤. B_g^(t)= Q_B^(t) U_R_g^(t) _R_g^(t), A_g^(t)= ( Q_A^(t) V_R_g^(t) ) . (6) By the optimality of truncated SVD, the resulting global update Δg(t)=g(t)g(t) W_g^(t)= B_g^(t) A_g^(t) satisfies Δg(t)∈argminrank()≤Rg‖−Δideal(t)‖F. W_g^(t)∈ *arg\,min_rank( X)≤ R_g \| X- W_ideal^(t) \|_F. (7) Appendix G provides detailed complexity analyses together with empirical wall-clock and performance evaluations. Compared with dense product aggregation and SVD, whose dominant server-side complexity is (d3)O(d^3), the proposed reduced-QR and core-SVD reconstruction reduces the dominant complexity to (N2dr2)O(N^2dr^2) while recovering the same optimal rank-RgR_g approximation. A randomized extension further reduces this complexity to (Ndr2)O(Ndr^2) at the cost of replacing exact rank-constrained optimality with a probabilistic approximation guarantee, as detailed in Remark 2. 4.4 Convergence Analysis We first analyze FedPA-LoRA under homogeneous local and reference ranks, ri=Ri=r_i=R_i=r, with full client participation. For client i at local step k of communication round t, we define i(t,k)=i(t,k)i(t,k) X_i^(t,k)= B_i^(t,k) A_i^(t,k) and i(t,k)=0+i(t,k) W_i^(t,k)= W_0+ X_i^(t,k). The end-of-round iterates are denoted by i(t)=i(t,τ) X_i^(t)= X_i^(t,τ) and i(t)=i(t,τ) W_i^(t)= W_i^(t,τ), and the global product by g(t)=g(t)g(t) X_g^(t)= B_g^(t) A_g^(t). The round-wise joint objective is defined as Ψ(t)=1N∑i=1N[fi(i(t))+λ2‖i(t)−g(t)‖F2]. ^(t)= 1N _i=1^N [f_i ( W_i^(t) )+ λ2 \| X_i^(t)- X_g^(t) \|_F^2 ]. (8) Our analysis relies on Assumptions 1– 4. These assumptions are used to establish the stationarity guarantee in Theorem 1, while Assumption 4 is also used to derive the global-loss guarantee in Proposition 1. Assumption 1 (LsL_s-smoothness). For every client i and any ,′ W, W , ‖∇fi()−∇fi(′)‖F≤Ls‖−′‖F. \| _ Wf_i( W)- _ Wf_i( W ) \|_F≤ L_s \| W- W \|_F. (9) Assumption 2 (Stochastic task gradients). For every client i, communication round t, and local step k, the stochastic task gradient computed using mini-batch ξi(t,k) _i^(t,k) is unbiased: ξi(t,k)[∇fi(i(t,k),ξi(t,k))]=∇fi(i(t,k)).E_ _i^(t,k) [ _ Wf_i ( W_i^(t,k); _i^(t,k) ) ]= _ Wf_i ( W_i^(t,k) ). (10) Moreover, there exists a constant Gf>0G_f>0 such that ‖∇fi(i(t,k),ξi(t,k))‖F≤Gf. \| _ Wf_i ( W_i^(t,k); _i^(t,k) ) \|_F≤ G_f. (11) Assumption 3 (LoRA factor regularity). There exist constants CA,CB>0C_A,C_B>0 such that, for every client i, communication round t, and local step k, ‖i(t,k)‖F≤CA,‖i(t,k)‖F≤CB. \| A_i^(t,k) \|_F≤ C_A, \| B_i^(t,k) \|_F≤ C_B. (12) In addition, there exists a constant c>0c>0 such that ‖∇ℒi(t)(i(t,k))(i(t,k))⊤‖F2+‖(i(t,k))⊤∇ℒi(t)(i(t,k))‖F2≥c‖∇ℒi(t)(i(t,k))‖F2. \| _ WL_i^(t) ( W_i^(t,k) ) ( A_i^(t,k) ) \|_F^2+ \| ( B_i^(t,k) ) _ WL_i^(t) ( W_i^(t,k) ) \|_F^2≥ c \| _ WL_i^(t) ( W_i^(t,k) ) \|_F^2. (13) Remark 1. Eq. (13) is the gradient-preservation condition of 10, stated here in the equivalent norm form and in the weaker summed version: we require only that the two terms jointly dominate ‖∇ℒi(t)‖F2\| _ WL_i^(t)\|_F^2, rather than each separately. Assumption 4 (LcL_c-Lipschitz continuity). For every client i and any ,′ W, W , |fi()−fi(′)|≤Lc‖−′‖F. |f_i( W)-f_i( W ) |≤ L_c \| W- W \|_F. (14) Theorem 1 (Convergence of FedPA-LoRA). Under Assumptions 1– 4, suppose that Ψ⋆ is a uniform lower bound on the average task loss, i.e., 1N∑i=1Nfi()≥Ψ⋆ 1N _i=1^Nf_i( W)≥ for all W, and that [Ψ(0)]−Ψ⋆≤DE[ ^(0)]- ≤ D for some constant D>0D>0. Defining D~:=D+LcCACB D:=D+L_cC_AC_B, there exists a constant Mλ>0M_λ>0, depending only on LsL_s, λ, GfG_f, CAC_A, and CBC_B, and independent of T and τ, such that, for any learning rate 0<η≤10<η≤ 1, 1NTτ∑t=1T∑i=1N∑k=0τ−1[‖∇ℒi(t)(i(t,k))‖F2]≤D~cηTτ+Mληc. 1NTτ _t=1^T _i=1^N _k=0^τ-1E\! [ \| _ WL_i^(t) ( W_i^(t,k) ) \|_F^2 ]≤ Dcη Tτ+ M_ληc. (15) Consequently, when D~≤MλTτ D≤ M_λTτ, choosing η=D~/(MλTτ)η= D/(M_λTτ) yields an averaged stationarity rate of ((Tτ)−1/2)O((Tτ)^-1/2). Proposition 1 (Global LoRA Loss). Under Assumption 4, define F()=N−1∑i=1Nfi()F( W)=N^-1 _i=1^Nf_i( W) and g(t)=0+g(t) W_g^(t)= W_0+ X_g^(t). Then, F(g(t))≤Ψ(t)+Lc22λ.F ( W_g^(t) )≤ ^(t)+ L_c^22λ. (16) Thus, the global LoRA task loss is controlled by the joint objective up to an additive term depending on LcL_c and λ. The complete proofs for the homogeneous setting are provided in Appendix D.1. Appendix D.2 extends the analysis to heterogeneous local and reference ranks, introducing an additional term due to bounded reference-truncation error, while Appendix E extends the convergence result to partial client participation with K active clients per round. 5 Experiments We conduct experiments to answer four questions: (i) Does FedPA-LoRA consistently improve performance across various natural language tasks? (i) Is it robust to data and resource heterogeneity, client scale, and rank variation? (i) Do its core components contribute as intended? (iv) What efficiency trade-offs does it introduce? The main text presents the primary performance and design analyses. Appendix C provides complete experimental configurations and additional results, while Appendix G reports detailed complexity, server- and client-side runtime, and randomized-reconstruction evaluations. 5.1 Experimental Setup 5.1.1 Datasets and Models We evaluate RoBERTa-Large (18) on SST-2, QNLI, QQP, RTE, and MNLI from GLUE (25). For generation, we evaluate Llama 3-8B (9) on GSM8K (7) and HumanEval (4), using CodeSearchNet (14) for HumanEval fine-tuning. 5.1.2 Baselines For homogeneous ranks, we compare with FedIT (31), FlexLoRA (2), FFA-LoRA (24), RoLoRA (5), and FedRot-LoRA (30). FedPA-LoRA preserves local parameters across rounds and uploads only the LoRA factors. Accordingly, on GLUE, each client retains its local classifier head and evaluates the reconstructed global LoRA with that head. We therefore include the personalized methods FedDPA-LoRA (29) and FedSA-LoRA (10). For heterogeneous ranks, we compare with FlexLoRA, HetLoRA (6), Ravan (21), and Fed-PLoRA (33). 5.1.3 Implementation Details We implement all methods using FederatedScope-LLM (16) and report averages over three seeds. We use 20 local updates for 250 rounds on GLUE and 30 updates for 200 rounds on generation tasks. Implementation details are provided in Appendix C.1. 5.2 Natural Language Understanding Table 1: GLUE accuracy under homogeneous- and heterogeneous-rank settings with Dirichlet concentration parameter β=0.5β=0.5. The homogeneous setting uses N=10N=10 clients with rank r=4r=4, whereas the heterogeneous setting uses N=20N=20 clients divided into three resource groups. Setting Method SST-2 QNLI QQP RTE MNLI Average Homogeneous FedIT 0.956±0.0000.956±0.000 0.891±0.0460.891±0.046 0.861±0.0010.861±0.001 0.696±0.0180.696±0.018 0.858±0.0010.858±0.001 0.85240.8524 FlexLoRA 0.956±0.0010.956±0.001 0.642±0.1000.642±0.100 0.848±0.0100.848±0.010 0.742±0.0610.742±0.061 0.858±0.0020.858±0.002 0.80920.8092 FFA-LoRA 0.946±0.0010.946±0.001 0.890±0.0100.890±0.010 0.848±0.0020.848±0.002 0.611±0.0210.611±0.021 0.840±0.0010.840±0.001 0.82700.8270 RoLoRA 0.957±0.0020.957±0.002 0.882±0.0460.882±0.046 0.845±0.0240.845±0.024 0.794±0.0220.794±0.022 0.852±0.0020.852±0.002 0.86600.8660 FedRot-LoRA 0.954±0.0020.954±0.002 0.908±0.0040.908±0.004 0.849±0.0020.849±0.002 0.714±0.0380.714±0.038 0.858±0.0020.858±0.002 0.85660.8566 FedDPA-LoRA 0.957±0.0010.957±0.001 0.851±0.0620.851±0.062 0.811±0.0450.811±0.045 0.612±0.1050.612±0.105 0.879±0.0020.879±0.002 0.82200.8220 FedSA-LoRA 0.959±0.0020.959±0.002 0.905±0.0060.905±0.006 0.814±0.0250.814±0.025 0.766±0.0090.766±0.009 0.880±0.0010.880±0.001 0.86480.8648 FedPA-LoRA 0.969±0.0010.969 0.001 0.942±0.0030.942 0.003 0.884±0.0010.884 0.001 0.871±0.0050.871 0.005 0.907±0.0020.907 0.002 0.91460.9146 Heterogeneous FlexLoRA 0.850±0.0150.850±0.015 0.798±0.0980.798±0.098 0.761±0.0330.761±0.033 0.570±0.0110.570±0.011 0.832±0.0310.832±0.031 0.76220.7622 HetLoRA 0.957±0.0010.957±0.001 0.919±0.0030.919±0.003 0.850±0.0020.850±0.002 0.671±0.0390.671±0.039 0.875±0.0000.875±0.000 0.85440.8544 Ravan 0.953±0.0010.953±0.001 0.883±0.0400.883±0.040 0.851±0.0030.851±0.003 0.712±0.0160.712±0.016 0.866±0.0030.866±0.003 0.85300.8530 Fed-PLoRA 0.932±0.0270.932±0.027 0.712±0.1370.712±0.137 0.811±0.0060.811±0.006 0.588±0.0130.588±0.013 0.839±0.0040.839±0.004 0.77640.7764 FedPA-LoRA 0.966±0.0030.966 0.003 0.959±0.0010.959 0.001 0.917±0.0090.917 0.009 0.860±0.0100.860 0.010 0.911±0.0030.911 0.003 0.92260.9226 In the homogeneous-rank setting, FedPA-LoRA achieves the best performance on all five GLUE tasks, averaging 0.91460.9146 and outperforming RoLoRA by 4.864.86 percentage points (Table 1). This highlights the benefit of local factor preservation and product-space alignment. 5.2.1 Effect of Data Heterogeneity Table 2: MNLI accuracy across data heterogeneity levels β∈100,1,0.5,0.1β∈\100,1,0.5,0.1\ with N=3N=3 and LoRA rank r=4r=4. Method = β=100 = β=1 =0.5 β=0.5 =0.1 β=0.1 FedIT 0.873±0.0020.873±0.002 0.859±0.0010.859±0.001 0.869±0.0010.869±0.001 0.858±0.0040.858±0.004 FlexLoRA 0.868±0.0010.868±0.001 0.877±0.0020.877±0.002 0.869±0.0020.869±0.002 0.859±0.0110.859±0.011 FFA-LoRA 0.867±0.0020.867±0.002 0.856±0.0020.856±0.002 0.856±0.0020.856±0.002 0.824±0.0030.824±0.003 RoLoRA 0.867±0.0080.867±0.008 0.676±0.2520.676±0.252 0.862±0.0000.862±0.000 0.822±0.0080.822±0.008 FedRot-LoRA 0.874±0.0010.874±0.001 0.868±0.0020.868±0.002 0.868±0.0020.868±0.002 0.857±0.0050.857±0.005 FedDPA-LoRA 0.873±0.0030.873±0.003 0.702±0.2650.702±0.265 0.879±0.0020.879±0.002 0.686±0.2570.686±0.257 FedSA-LoRA 0.862±0.0010.862±0.001 0.865±0.0010.865±0.001 0.874±0.0020.874±0.002 0.863±0.0060.863±0.006 FedPA-LoRA 0.880±0.002 0.880±0.002 0.914±0.001 0.914±0.001 0.906±0.003 0.906±0.003 0.892±0.002 0.892±0.002 Table 2 reports MNLI accuracy under varying data heterogeneity. Larger β values produce more uniform client distributions, with β=100β=100 approximating IID data, while smaller values induce stronger non-IID label skew. Global federated LoRA methods remain competitive under relatively homogeneous data, where client updates are well aligned. As data heterogeneity increases, personalized methods become more effective by retaining client-specific information. In particular, FedDPA-LoRA and FedSA-LoRA outperform all global baselines at β=0.5β=0.5 and β=0.1β=0.1, respectively. However, the substantial variation in FedDPA-LoRA across different values of β indicates that personalization alone does not guarantee robust performance across data distributions. FedPA-LoRA achieves the highest accuracy across all evaluated levels of data heterogeneity. Local factor preservation retains knowledge learned from each client’s data, while product-guided regularization encourages consistency with the global product reference. By balancing client-specific knowledge with globally shared information, FedPA-LoRA maintains strong performance from near-IID to highly heterogeneous settings. 5.2.2 Effect of Number of Clients Table 3: MNLI accuracy under different numbers of clients N∈3,10,50N∈\3,10,50\ with LoRA rank r=4r=4 and Dirichlet concentration parameter β=0.5β=0.5. Method = N=3 = N=10 = N=50 FedIT 0.869±0.0010.869±0.001 0.858±0.0010.858±0.001 0.877±0.0010.877±0.001 FlexLoRA 0.869±0.0020.869±0.002 0.858±0.0020.858±0.002 0.740±0.1750.740±0.175 FFA-LoRA 0.856±0.0020.856±0.002 0.840±0.0010.840±0.001 0.860±0.0010.860±0.001 RoLoRA 0.862±0.0000.862±0.000 0.852±0.0020.852±0.002 0.882±0.0020.882±0.002 FedRot-LoRA 0.868±0.0020.868±0.002 0.858±0.0020.858±0.002 0.872±0.0010.872±0.001 FedDPA-LoRA 0.879±0.0020.879±0.002 0.879±0.0020.879±0.002 0.883±0.0140.883±0.014 FedSA-LoRA 0.874±0.0020.874±0.002 0.880±0.0010.880±0.001 0.891±0.0000.891±0.000 FedPA-LoRA 0.906±0.003 0.906±0.003 0.907±0.002 0.907±0.002 0.916±0.001 0.916±0.001 Table 3 shows MNLI accuracy as the number of clients increases from 33 to 5050. FedPA-LoRA consistently outperforms all baselines and maintains stable performance across the evaluated client scales. In particular, its accuracy slightly increases from 0.9060.906 with three clients to 0.9160.916 with fifty clients, demonstrating its robustness to an increasing number of participating clients. 5.2.3 Effect of Rank Figure 3: MNLI accuracy under different LoRA ranks r∈2,4,8,16,24r∈\2,4,8,16,24\ with β=0.5β=0.5 and N=3N=3. Figure 3 shows that FedPA-LoRA achieves the best performance across all evaluated ranks, with accuracy remaining stable between 0.9060.906 and 0.9100.910. In contrast, several baselines are sensitive to rank selection and exhibit noticeable fluctuations, with FedSA-LoRA dropping substantially at r=24r=24. These results indicate that FedPA-LoRA is robust to rank variation and performs well in both capacity-constrained low-rank and higher-rank settings. Detailed results are reported in Table 7 in Appendix C.3. 5.3 Natural Language Generation We evaluate both tasks under the homogeneous-rank setting with r=8r=8. GSM8K uses three IID clients, while HumanEval is fine-tuned on CodeSearchNet using six clients partitioned by programming language. FedPA-LoRA achieves the best performance on both tasks, reaching 0.46720.4672 accuracy on GSM8K and 0.41210.4121 pass@1 on HumanEval. Since neither task uses client-specific classifier heads, these gains indicate that local factor preservation and product-space aggregation effectively mitigate factor-level initialization and aggregation mismatch, respectively. Detailed results are provided in Table 8 of Appendix C.3. 5.4 In-Depth Analyses 5.4.1 Effect of Rank Heterogeneity We evaluate natural language understanding and generation under heterogeneous client resources using three resource groups. For FedPA-LoRA, we set Ri=riR_i=r_i for a controlled comparison. Detailed rank configurations are provided in Appendix C.1. (1) Natural Language Understanding. Under heterogeneous ranks, Table 1 shows that FedPA-LoRA achieves the best performance on all five GLUE tasks, attaining an average accuracy of 0.92260.9226 and outperforming HetLoRA by 6.826.82 percentage points. This demonstrates that FedPA-LoRA effectively accommodates heterogeneous client capacities while maintaining strong global performance. (2) Natural Language Generation. FedPA-LoRA also achieves the best performance on GSM8K and HumanEval, reaching 0.50180.5018 accuracy and 0.40480.4048 pass@1, respectively. It outperforms Ravan and Fed-PLoRA, which distribute their full sets of global modules, while transmitting only the global reference corresponding to each client’s assigned rank. Detailed results are provided in Table 9 of the Appendix C.3. 5.4.2 Effect of the Regularization Strength λ Figure 5 examines the sensitivity of FedPA-LoRA to the product-guided regularization strength λ. Under IID data, λ=0.01λ=0.01 yields substantially lower accuracy, indicating that weak guidance is insufficient to coordinate the locally preserved factors. Performance improves and remains stable for moderate values of λ, but declines at λ=10λ=10, suggesting that overly strong regularization can restrict local adaptation when client updates are already well aligned. A different trend emerges in the non-IID setting. With λ∈0.01,0.1λ∈\0.01,0.1\, FedPA-LoRA exhibits lower accuracy and greater variance due to insufficient global guidance. As λ increases, both accuracy and stability improve, remaining consistently strong for λ∈1,2,5,10λ∈\1,2,5,10\. This highlights the importance of product-guided regularization under data heterogeneity, where it mitigates client drift while preserving local adaptation. Appendix F further discusses the potential use of client-specific adaptive regularization weights λi _i to account for varying degrees of client drift. 5.4.3 Effect of the Communication Budget R Figure 5 shows that FedPA-LoRA maintains strong performance across downlink communication ranks R∈1,2,4,8R∈\1,2,4,8\, with even R=1R=1 providing effective global guidance. Under IID data, increasing R offers no consistent improvement, suggesting that a larger reference provides limited additional benefit when client updates are already well aligned. Under non-IID data, performance improves as R increases, indicating that richer global guidance is more useful under stronger data heterogeneity. Overall, FedPA-LoRA can reduce downlink communication by using a smaller reference rank while maintaining strong performance. (a) IID (b) Non-IID Figure 4: MNLI accuracy under different regularization strengths λ with N=3N=3 clients and r=R=4r=R=4. (a) IID with β=100β=100. (b) Non-IID with β=0.5β=0.5. Horizontal lines denote baseline means over three random seeds. (c) IID (d) Non-IID Figure 5: MNLI accuracy under different communication ranks R∈1,2,4,8R∈\1,2,4,8\ with local LoRA rank r=4r=4 and N=3N=3 clients. (a) IID with β=100β=100. (b) Non-IID with β=0.5β=0.5. 5.4.4 Ablation Study Table 4: Component ablation results on MNLI with N=3N=3 clients and Dirichlet concentration parameter β=0.5β=0.5. Method Local Local Regularization Accuracy Head Preservation FedIT × × × 0.869±0.0010.869±0.001 FedIT ✓ × × 0.869±0.0000.869±0.000 FlexLoRA × × × 0.869±0.0020.869±0.002 FlexLoRA ✓ × × 0.870±0.0010.870±0.001 Ours × × B A 0.867±0.0030.867±0.003 Ours ✓ × B A 0.877±0.0020.877±0.002 Ours × ✓ B A 0.871±0.0020.871±0.002 Ours ✓ ✓ B 0.875±0.0210.875±0.021 Ours ✓ ✓ A 0.879±0.0100.879±0.010 Ours ✓ ✓ , B, A 0.896±0.0080.896±0.008 Ours ✓ ✓ B A 0.906±0.0030.906 0.003 Table 4 shows that adding a local head alone provides negligible gains to FedIT and FlexLoRA. Within our framework, local preservation improves product-space regularization from 0.8670.867 to 0.8710.871. Combined with a local head, product-space regularization achieves the highest accuracy of 0.9060.906, outperforming regularization of B, A, or both factors separately. Since the local and reference products share the same weight-space dimensions, it also supports Ri≠riR_i≠ r_i and enables independent control of the downlink communication budget. With R=2R=2, FedPA-LoRA achieves 0.9030.903, still outperforming all factor-wise variants. The corresponding objectives are provided in Appendix C.2. 6 Conclusion We proposed FedPA-LoRA, a federated LoRA framework that preserves local factors while aggregating them in the product space, jointly mitigating aggregation and factor-level initialization mismatches. FedPA-LoRA provably converges under homogeneous and heterogeneous client ranks and supports client-specific computation and communication budgets. Experiments on natural language understanding and generation confirm consistent gains across data- and resource-heterogeneous settings. Future work will explore adaptive guidance and communication budgets at scale. References Babakniya et al. (2023) S. Babakniya, A. Elkordy, Y. Ezzeldin, Q. Liu, K. Song, M. EL-Khamy, and S. Avestimehr SLoRA: federated parameter efficient fine-tuning of language models. In International Workshop on Federated Learning in the Age of Foundation Models in Conjunction with NeurIPS 2023, External Links: Link Cited by: §1. Bai et al. (2024) J. Bai, D. Chen, B. Qian, L. Yao, and Y. Li Federated fine-tuning of large language models under heterogeneous tasks and client resources. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, External Links: Link Cited by: §1, §1, §2.1, §2.2, §5.1.2. Bian et al. (2025) J. Bian, L. Wang, L. Zhang, and J. Xu LoRA-FAIR: federated LoRA fine-tuning with aggregation and initialization refinement. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), p. 3737–3746. Cited by: §2.1. Chen et al. (2021) M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. D. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374. Cited by: §5.1.1. Chen et al. (2025) S. Chen, Y. Guo, Y. Ju, H. Dalal, Z. Zhu, and A. J. Khisti Robust federated finetuning of LLMs via alternating optimization of LoRA. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, External Links: Link Cited by: §1, §2.1, §5.1.2. Cho et al. (2024) Y. J. Cho, L. Liu, Z. Xu, A. Fahrezi, and G. Joshi Heterogeneous LoRA for federated fine-tuning of on-device foundation models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Y. Al-Onaizan, M. Bansal, and Y. Chen (Eds.), Miami, Florida, USA, p. 12903–12913. External Links: Link, Document Cited by: §1, §2.2, §5.1.2. Cobbe et al. (2021) K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168. Cited by: §5.1.1. Eckart and Young (1936) C. Eckart and G. Young The approximation of one matrix by another of lower rank. Psychometrika 1 (3), p. 211–218. Cited by: §D.2.1. Grattafiori et al. (2024) A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783. Cited by: §5.1.1. Guo et al. (2025) P. Guo, S. Zeng, Y. Wang, H. Fan, F. Wang, and L. Qu Selective aggregation for low-rank adaptation in federated learning. In The Thirteenth International Conference on Learning Representations, External Links: Link Cited by: Appendix A, §2.1, §3.3, §5.1.2, Remark 1. Halko et al. (2011) N. Halko, P. Martinsson, and J. Tropp Finding structure with randomness: probabilistic algorithms for constructing approximate matrix decompositions. SIAM Review 53 (2), p. 217–288. Cited by: Remark 2, Remark 2. Han et al. (2024) Z. Han, C. Gao, J. Liu, J. Zhang, and S. Q. Zhang Parameter-efficient fine-tuning for large models: a comprehensive survey. Transactions on Machine Learning Research. Note: External Links: ISSN 2835-8856, Link Cited by: §1. Hu et al. (2022) E. J. Hu, yelong shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen LoRA: low-rank adaptation of large language models. In International Conference on Learning Representations, External Links: Link Cited by: §1. Husain et al. (2019) H. Husain, H. Wu, T. Gazit, M. Allamanis, and M. Brockschmidt Codesearchnet challenge: evaluating the state of semantic code search. arXiv preprint arXiv:1909.09436. Cited by: §5.1.1. Kaplan et al. (2020) J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei Scaling laws for neural language models. arXiv preprint arXiv:2001.08361. Cited by: §1. Kuang et al. (2024) W. Kuang, B. Qian, Z. Li, D. Chen, D. Gao, X. Pan, Y. Xie, Y. Li, B. Ding, and J. Zhou Federatedscope-llm: a comprehensive package for fine-tuning large language models in federated learning. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, p. 5260–5271. Cited by: §C.1, §1, §5.1.3. Lialin et al. (2023) V. Lialin, V. Deshpande, X. Yao, and A. Rumshisky Scaling down to scale up: a guide to parameter-efficient fine-tuning. arXiv preprint arXiv:2303.15647. Cited by: §1. Liu et al. (2019) Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov Roberta: a robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. Cited by: §3.3, §5.1.1. Mangrulkar et al. (2022) S. Mangrulkar, S. Gugger, L. Debut, Y. Belkada, S. Paul, and B. Bossan PEFT: state-of-the-art parameter-efficient fine-tuning methods. Note: https://github.com/huggingface/peft Cited by: §4.1. McMahan et al. (2017) B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, p. 1273–1282. Cited by: §1. Raje et al. (2025) A. Raje, B. Askin, D. Jhunjhunwala, and G. Joshi Ravan: multi-head low-rank adaptation for federated fine-tuning. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, External Links: Link Cited by: §2.2, §5.1.2. Rieke et al. (2020) N. Rieke, J. Hancox, W. Li, F. Milletari, H. R. Roth, S. Albarqouni, S. Bakas, M. N. Galtier, B. A. Landman, K. Maier-Hein, et al. The future of digital health with federated learning. NPJ digital medicine 3 (1), p. 119. Cited by: §1. Singhal et al. (2025) R. Singhal, K. Ponkshe, and P. Vepakomma FedEx-LoRA: exact aggregation for federated and efficient fine-tuning of large language models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 1316–1336. Cited by: §2.1. Sun et al. (2024) Y. Sun, Z. Li, Y. Li, and B. Ding Improving LoRA in privacy-preserving federated learning. In International Conference on Learning Representations, B. Kim, Y. Yue, S. Chaudhuri, K. Fragkiadaki, M. Khan, and Y. Sun (Eds.), Vol. 2024, p. 17978–17994. External Links: Link Cited by: §2.1, §5.1.2. Wang et al. (2018) A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman GLUE: a multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, T. Linzen, G. Chrupała, and A. Alishahi (Eds.), Brussels, Belgium, p. 353–355. External Links: Link, Document Cited by: §5.1.1. Wang et al. (2024) Z. Wang, Z. Shen, Y. He, G. Sun, H. Wang, L. Lyu, and A. Li FLoRA: federated fine-tuning large language models with heterogeneous low-rank adaptations. In Proceedings of the 38th International Conference on Neural Information Processing Systems, NIPS ’24, Red Hook, NY, USA. External Links: ISBN 9798331314385 Cited by: §2.1. Xu et al. (2024) M. Xu, D. Cai, Y. Wu, X. Li, and S. Wang \fwdllm\: Efficient federated finetuning of large language models with perturbed inferences. In 2024 USENIX Annual Technical Conference (USENIX ATC 24), p. 579–596. Cited by: §1. Yan et al. (2026) G. Yan, L. Xie, Q. Shen, Y. Fang, and Z. Wu FedSRD: sparsify-reconstruct-decompose for communication-efficient federated large language models fine-tuning. In Proceedings of the ACM Web Conference 2026, W ’26, New York, NY, USA, p. 5087–5098. External Links: ISBN 9798400723070, Link, Document Cited by: §2.1. Yang et al. (2024) Y. Yang, G. Long, T. Shen, J. Jiang, and M. Blumenstein Dual-personalizing adapter for federated foundation models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, External Links: Link Cited by: Appendix A, §5.1.2. Zhang et al. (2026a) H. Zhang, D. Kim, S. Cha, and H. Vikalo FedRot-LoRA: mitigating rotational misalignment in federated LoRA. In Forty-third International Conference on Machine Learning, External Links: Link Cited by: §1, §2.1, §5.1.2. Zhang et al. (2024) J. Zhang, S. Vahidian, M. Kuo, C. Li, R. Zhang, T. Yu, G. Wang, and Y. Chen Towards building the federatedgpt: federated instruction tuning. In ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Vol. , p. 6915–6919. External Links: Document Cited by: §2.1, §5.1.2. Zhang et al. (2023) Z. Zhang, Y. Yang, Y. Dai, Q. Wang, Y. Yu, L. Qu, and Z. Xu Fedpetuning: when federated learning meets the parameter-efficient tuning methods of pre-trained language models. In Findings of the Association for Computational Linguistics: ACL 2023, p. 9963–9977. Cited by: §1. Zhang et al. (2026b) Z. Zhang, R. Hu, and J. Xu Heterogeneous federated fine-tuning with parallel one-rank adaptation. In The Fourteenth International Conference on Learning Representations, External Links: Link Cited by: §1, §2.2, §5.1.2. Appendices The appendices are organized as follows. A Appendix A provides additional discussion of related work, expanding on the personalized federated-LoRA baselines used in our experiments. B Appendix B presents the complete algorithmic pseudocode for FedPA-LoRA, formalizing the local preservation, alignment, and aggregation steps. C Appendix C reports the hyperparameter configurations used across all experiments, together with detailed results across different LoRA ranks and generation tasks. D Appendix D.1 gives the full proofs of Theorem 1 and Proposition 1 for the homogeneous setting. Appendix D.2 then extends this analysis to heterogeneous local and reference ranks, introducing the bounded reference-truncation error that appears in the resulting rate. E Appendix E discusses how our convergence results extend to the partial participation setting with K active clients per round. F Appendix F discusses three illustrative directions for adapting the regularization weight λi(t) _i^(t) using locally available task-loss and gradient information. G Appendix G examines the practical efficiency of FedPA-LoRA by connecting its theoretical server-side costs with empirical runtime and approximation quality, complemented by client-side wall-clock results. Appendix A Additional Discussion of Related Work FedDPA-LoRA [29] maintains a globally aggregated LoRA adapter alongside a personalized local adapter, and proposes two variants for initializing the latter: FedDPA-F re-initializes the personalized adapter from the freshly aggregated global adapter each round, whereas FedDPA-T instead re-initializes it from its own value at the end of the previous round, training it locally alongside the temporarily frozen global adapter. Only the global adapter is ever transmitted to the server, where it is aggregated through conventional factor-wise averaging, leaving it exposed to the same aggregation and factor-level initialization mismatches as FedIT. FedDPA-T is the only method in which a component genuinely continues its own trajectory across rounds rather than being replaced, but that component is the personalized adapter, which was never exposed to aggregation to begin with. The global adapter, which is what actually goes through aggregation each round, still gets replaced by a fresh average with no continuity at all. FedSA-LoRA [10] is grounded in an asymmetry analysis showing that A matrices consistently learn task-general structure while B matrices capture client-specific adaptation, the same finding Section 3 draws on to explain why B exhibits larger aggregation and initialization errors than A under naive averaging. FedDPA-LoRA and FedSA-LoRA therefore both respond to the client-specific character of part of the model by permanently excluding that part from federation, either an entire adapter or one LoRA factor, rather than by resolving aggregation and initialization mismatches within it. This raises a natural question for our work, FedPA-LoRA: if B is indeed more client-specific, why aggregate it at all? Local factor preservation answers this without withholding B from the global model: because B is never overwritten by the server and is only nudged toward the global reference through the regularizer in Eq. 3, its client-specific character is preserved across rounds in the same spirit as FedSA-LoRA’s design, while its information is still incorporated into the global adapter rather than withheld from it entirely. Appendix B FedPA-LoRA: Detailed Algorithm The FedPA-LoRA framework is summarized in Algorithm 1. Algorithm 1 FedPA-LoRA Input: Number of clients N, learning rate η, number of local updates τ, computation ranks rii=1N\r_i\_i=1^N, communication ranks Rii=1N\R_i\_i=1^N, rounds T, and regularization weight λ Output: Global LoRA factors (g(T),g(T))( B_g^(T), A_g^(T)) 1: rg←max1≤i≤Nrir_g← _1≤ i≤ Nr_i, Rg←max1≤i≤NRi R_g← _1≤ i≤ NR_i 2: Initialize (g(0),g(0))( B_g^(0), A_g^(0)) with rank rgr_g 3: for i=1,…,Ni=1,…,N do 4: Initialize (i(0),i(0))←(g,i(0),g,i(0))( B_i^(0), A_i^(0))←( B_g,i^(0), A_g,i^(0)), where (g,i(0),g,i(0))( B_g,i^(0), A_g,i^(0)) retains the first rir_i columns of g(0) B_g^(0) and the first rir_i rows of g(0) A_g^(0) 5: end for 6: for t=1,…,Tt=1,…,T do 7: for i=1,…,Ni=1,…,N in parallel do 8: Preserve the local factors (i(t,0),i(t,0))←(i(t−1),i(t−1))( B_i^(t,0), A_i^(t,0))←( B_i^(t-1), A_i^(t-1)) 9: Obtain the rank-RiR_i global reference (g,i(t−1),g,i(t−1))( B_g,i^(t-1), A_g,i^(t-1)) by retaining the first RiR_i columns of g(t−1) B_g^(t-1) and the first RiR_i rows of g(t−1) A_g^(t-1) 10: for k=0,…,τ−1k=0,…,τ-1 do 11: Update (i(t,k+1),i(t,k+1))( B_i^(t,k+1), A_i^(t,k+1)) from (i(t,k),i(t,k))( B_i^(t,k), A_i^(t,k)) with learning rate η by one optimization step on ℒi(t)(i,i)=fi(0+ii)+λ2‖ii−g,i(t−1)g,i(t−1)‖F2.L_i^(t)( B_i, A_i)=f_i ( W_0+ B_i A_i )+ λ2 \| B_i A_i- B_g,i^(t-1) A_g,i^(t-1) \|_F^2. 12: end for 13: Set (i(t),i(t))←(i(t,τ),i(t,τ))( B_i^(t), A_i^(t))←( B_i^(t,τ), A_i^(t,τ)) 14: Send (i(t),i(t))( B_i^(t), A_i^(t)) to the server 15: end for 16: Construct the concatenated factors cat(t)←1N[1(t),…,N(t)],cat(t)←1N[(1(t))⊤,…,(N(t))⊤]⊤. B_cat^(t)← 1 N[ B_1^(t),…, B_N^(t)], A_cat^(t)← 1 N[( A_1^(t)) ,…,( A_N^(t)) ] . 17: Compute the reduced QR factorizations (B(t),B(t))←QR(cat(t)),(A(t),A(t))←QR((cat(t))⊤).( Q_B^(t), R_B^(t)) ( B_cat^(t)), ( Q_A^(t), R_A^(t)) (( A_cat^(t)) ). 18: Form the core matrix (t)=B(t)(A(t))⊤ H^(t)= R_B^(t)( R_A^(t)) and compute its rank-RgR_g truncated SVD (Rg(t),Rg(t),Rg(t))←SVDRg((t)).( U_R_g^(t), _R_g^(t), V_R_g^(t)) _R_g( H^(t)). 19: Reconstruct the global LoRA factors g(t)←B(t)Rg(t)Rg(t),g(t)←(A(t)Rg(t))⊤. B_g^(t)← Q_B^(t) U_R_g^(t) _R_g^(t), A_g^(t)← ( Q_A^(t) V_R_g^(t) ) . 20: end for Appendix C Experimental Details and Extra Results C.1 Experimental Details All experiments are implemented using FederatedScope-LLM [16] and PyTorch, and conducted on Ubuntu 18.04.5 LTS with AMD EPYC 7543 processors, 1 TiB of system memory, and NVIDIA RTX A6000 or RTX 6000 Ada GPUs. We average results over three random seeds (00, 1313, 123123). Tables 5 and 6 report the configurations selected for the homogeneous- and heterogeneous-rank settings, respectively, based on validation performance. Table 5: Selected hyperparameters for experiments under homogeneous-rank settings. Experiment Dataset Clients (N) Rank (r) Data Distribution (β) Optimal Learning Rate η η Optimal λ λ FedIT FlexLoRA FFA- LoRA RoLoRA FedRot- LoRA FedDPA- LoRA FedSA- LoRA FedPA- LoRA FedRot- LoRA FedPA- LoRA Natural Language Understanding SST-2 1010 44 0.50.5 1e-2 1e-2 2e-2 1e-2 1e-2 5e-3 2e-2 2e-2 0.60.6 1.01.0 QNLI 1e-2 1e-2 2e-2 5e-3 5e-3 5e-3 1e-3 2e-2 0.20.2 2.02.0 QQP 1e-2 5e-3 2e-2 2e-2 5e-3 5e-3 2e-2 2e-2 0.60.6 2.02.0 RTE 1e-2 1e-2 2e-2 5e-3 1e-2 2e-2 1e-3 2e-2 0.40.4 1.01.0 MNLI 1e-2 5e-3 2e-2 5e-3 1e-2 1e-2 1e-2 2e-2 0.40.4 2.02.0 Data Heterogeneity MNLI 33 44 100100 1e-2 5e-3 2e-2 1e-2 1e-2 1e-2 5e-3 2e-2 0.80.8 1.01.0 11 5e-3 2e-2 2e-2 5e-3 1e-2 2e-2 5e-3 2e-2 0.80.8 1.01.0 0.10.1 1e-2 1e-2 2e-2 5e-3 1e-2 1e-2 1e-2 2e-2 0.40.4 1.01.0 Number of Clients MNLI 33 44 0.50.5 1e-2 1e-2 2e-2 1e-2 1e-2 1e-2 1e-2 2e-2 0.40.4 5.05.0 5050 2e-2 1e-2 2e-2 5e-3 1e-2 1e-2 1e-2 1e-2 0.40.4 5.05.0 Rank Number MNLI 33 22 0.50.5 1e-2 1e-2 2e-2 5e-3 1e-2 5e-3 5e-3 2e-2 0.40.4 5.05.0 88 5e-3 5e-3 1e-2 5e-3 1e-2 1e-2 1e-2 2e-2 0.40.4 2.02.0 1616 5e-3 1e-2 2e-2 5e-3 1e-2 5e-3 5e-3 2e-2 0.40.4 1.01.0 2424 1e-2 5e-3 2e-2 5e-3 1e-2 5e-3 5e-3 2e-2 0.40.4 2.02.0 Natural Language Generation GSM8K 33 88 IID 5e-3 5e-3 5e-3 1e-3 5e-3 – – 5e-3 0.20.2 1.01.0 HumanEval 66 88 Non-IID 5e-3 5e-3 5e-3 5e-3 5e-3 – – 5e-3 0.20.2 1.01.0 Table 6: Selected hyperparameters for experiments under heterogeneous-rank settings. Experiment Dataset Clients (N) Rank Configuration Data Distribution (β) Optimal Learning Rate η η Optimal λ λ FlexLoRA HetLoRA Ravan Fed- PLoRA FedPA- LoRA FedPA- LoRA Natural Language Understanding SST-2 2020 Other methods: ri∈2,4,16r_i∈\2,4,16\ Ravan: hi∈1,2,8h_i∈\1,2,8\, rh=64r_h=64 0.50.5 1e-2 2e-2 2e-2 1e-2 1e-2 2.02.0 QNLI 1e-2 1e-2 2e-2 1e-2 1e-2 2.02.0 QQP 1e-2 1e-2 1e-2 5e-3 2e-2 1.01.0 RTE 1e-2 1e-2 1e-2 1e-2 1e-2 1.01.0 MNLI 5e-3 2e-2 1e-2 5e-3 2e-2 1.01.0 Natural Language Generation GSM8K 33 Other methods: ri∈2,4,16r_i∈\2,4,16\ Ravan: hi∈1,2,8h_i∈\1,2,8\, rh=115r_h=115 IID 1e-3 1e-3 5e-3 5e-3 5e-3 0.10.1 HumanEval 66 Non-IID 1e-3 5e-3 5e-3 5e-3 5e-3 0.10.1 We search the learning rate over η∈5e-4,1e-3,5e-3,1e-2,2e-2η∈\ 5e-4, 1e-3, 5e-3, 1e-2, 2e-2\. For FedRot-LoRA, we use the alignment strength reported in the original paper when available and otherwise search λ∈0.2,0.4,0.6,0.8,1.0λ∈\0.2,0.4,0.6,0.8,1.0\. For FedPA-LoRA, we search the product-guided regularization strength over λ∈0.1,1.0,2.0,5.0,10.0λ∈\0.1,1.0,2.0,5.0,10.0\. The sensitivity study in Figure 5 additionally evaluates λ=0.01λ=0.01. In the heterogeneous-rank experiments, FlexLoRA, HetLoRA, and FedPA-LoRA use client ranks ri∈2,4,16r_i∈\2,4,16\, while Fed-PLoRA uses the corresponding numbers of parallel rank-one modules. Ravan uses hi∈1,2,8h_i∈\1,2,8\ active heads, with a per-head rank of rh=64r_h=64 for RoBERTa-Large and rh=115r_h=115 for Llama 3-8B. These configurations are chosen to approximately match the trainable parameter budgets of the corresponding resource groups across methods. All remaining training settings follow those described in the main text. C.2 Local Objectives of the Ablation Variants The ablation variants use the same task loss and server-side aggregation as FedPA-LoRA, but differ in the regularization space. Their local objectives are defined as ℒi,(t) _i, B^(t) =fi(0+ii)+λ2‖i−g,i(t−1)‖F2, =f_i( W_0+ B_i A_i)+ λ2 \| B_i- B_g,i^(t-1) \|_F^2, (17) ℒi,(t) _i, A^(t) =fi(0+ii)+λ2‖i−g,i(t−1)‖F2, =f_i( W_0+ B_i A_i)+ λ2 \| A_i- A_g,i^(t-1) \|_F^2, ℒi,,(t) _i, B, A^(t) =fi(0+ii)+λ2(‖i−g,i(t−1)‖F2+‖i−g,i(t−1)‖F2), =f_i( W_0+ B_i A_i)+ λ2 ( \| B_i- B_g,i^(t-1) \|_F^2+ \| A_i- A_g,i^(t-1) \|_F^2 ), ℒi,(t) _i, B A^(t) =fi(0+ii)+λ2‖ii−g,i(t−1)g,i(t−1)‖F2. =f_i( W_0+ B_i A_i)+ λ2 \| B_i A_i- B_g,i^(t-1) A_g,i^(t-1) \|_F^2. The first three variants regularize B, A, or both factors separately, whereas FedPA-LoRA regularizes their product directly. C.3 Extra Results We now report additional quantitative results, including detailed MNLI results across LoRA ranks underlying Figure 3, as well as generation-task performance on GSM8K and HumanEval under both homogeneous- and heterogeneous-rank settings. Table 7: MNLI accuracy under different LoRA ranks r∈2,4,8,16,24r∈\2,4,8,16,24\ with Dirichlet concentration parameter β=0.5β=0.5 and N=3N=3 clients. Method = r=2 = r=4 = r=8 = r=16 = r=24 FedIT 0.869±0.0020.869±0.002 0.869±0.0010.869±0.001 0.863±0.0010.863±0.001 0.865±0.0010.865±0.001 0.873±0.0000.873±0.000 FlexLoRA 0.876±0.0040.876±0.004 0.869±0.0020.869±0.002 0.866±0.0010.866±0.001 0.839±0.0320.839±0.032 0.868±0.0010.868±0.001 FFA-LoRA 0.837±0.0130.837±0.013 0.856±0.0020.856±0.002 0.845±0.0020.845±0.002 0.868±0.0010.868±0.001 0.871±0.0020.871±0.002 RoLoRA 0.849±0.0050.849±0.005 0.862±0.0000.862±0.000 0.856±0.0020.856±0.002 0.858±0.0020.858±0.002 0.857±0.0040.857±0.004 FedRot-LoRA 0.869±0.0010.869±0.001 0.868±0.0020.868±0.002 0.869±0.0020.869±0.002 0.870±0.0010.870±0.001 0.872±0.0020.872±0.002 FedDPA-LoRA 0.871±0.0010.871±0.001 0.879±0.0020.879±0.002 0.874±0.0050.874±0.005 0.877±0.0010.877±0.001 0.868±0.0120.868±0.012 FedSA-LoRA 0.859±0.0070.859±0.007 0.874±0.0020.874±0.002 0.878±0.0010.878±0.001 0.875±0.0020.875±0.002 0.713±0.2320.713±0.232 FedPA-LoRA 0.907±0.0020.907 0.002 0.906±0.0030.906 0.003 0.907±0.0010.907 0.001 0.908±0.0010.908 0.001 0.910±0.0030.910 0.003 Method GSM8K HumanEval (Acc.) (pass@1) FedIT 0.4316±0.0030.4316±0.003 0.4020±0.0020.4020±0.002 FlexLoRA 0.4460±0.0040.4460±0.004 0.3873±0.0100.3873±0.010 FFA-LoRA 0.4316±0.0030.4316±0.003 0.3902±0.0080.3902±0.008 RoLoRA 0.4233±0.0160.4233±0.016 0.3751±0.0030.3751±0.003 FedRot-LoRA 0.4412±0.0140.4412±0.014 0.4016±0.0070.4016±0.007 FedPA-LoRA 0.4672±0.0060.4672 0.006 0.4121±0.0020.4121 0.002 Table 8: Generative task performance under the homogeneous-rank setting with rank r=8r=8. GSM8K uses N=3N=3 clients with IID data, whereas HumanEval uses N=6N=6 clients with non-IID language-based partitions. Method GSM8K HumanEval (Acc.) (pass@1) FlexLoRA 0.4488±0.0130.4488±0.013 0.4012±0.0700.4012±0.070 HetLoRA 0.4230±0.0010.4230±0.001 0.3894±0.0070.3894±0.007 Ravan 0.4735±0.0050.4735±0.005 0.3959±0.0090.3959±0.009 Fed-PLoRA 0.4319±0.0180.4319±0.018 0.4016±0.0210.4016±0.021 FedPA-LoRA 0.5018±0.0170.5018 0.017 0.4048±0.0010.4048 0.001 Table 9: Generative task performance under the heterogeneous-rank setting. GSM8K uses N=3N=3 clients with IID data, whereas HumanEval uses N=6N=6 clients with non-IID language-based partitions. In both tasks, clients are divided into three resource groups. Appendix D Convergence Analysis D.1 Convergence Analysis: Homogeneous Setting We provide the complete proofs of Theorem 1 and Proposition 1. Throughout this section, we consider homogeneous local and reference ranks ri=Ri=r_i=R_i=r, full client participation, and λ>0λ>0. All one-step expectations are conditioned on the current iterates unless stated otherwise. D.1.1 Preliminaries We write i(t,k)=i(t,k)i(t,k) X_i^(t,k)= B_i^(t,k) A_i^(t,k) and i(t,k)=0+i(t,k) W_i^(t,k)= W_0+ X_i^(t,k) for the local LoRA update and model at local step k of communication round t, respectively. During communication round t, the global reference g(t−1)=g(t−1)g(t−1) X_g^(t-1)= B_g^(t-1) A_g^(t-1), reconstructed by the server in Eq. (6), remains fixed. The stochastic weight-space objective is ℒi(t)(,ξi(t,k))=fi(,ξi(t,k))+λ2‖−0−g(t−1)‖F2,L_i^(t)( W; _i^(t,k))=f_i( W; _i^(t,k))+ λ2 \| W- W_0- X_g^(t-1) \|_F^2, (18) and its deterministic counterpart is ℒi(t)()=fi()+λ2‖−0−g(t−1)‖F2.L_i^(t)( W)=f_i( W)+ λ2 \| W- W_0- X_g^(t-1) \|_F^2. (19) At the end of communication round t, i(t)=i(t,τ) X_i^(t)= X_i^(t,τ) and i(t)=i(t,τ) W_i^(t)= W_i^(t,τ). We define the round-wise joint objective as Ψ(t)=1N∑i=1N[fi(i(t))+λ2‖i(t)−g(t)‖F2]. ^(t)= 1N _i=1^N [f_i ( W_i^(t) )+ λ2 \| X_i^(t)- X_g^(t) \|_F^2 ]. (20) As assumed in Theorem 1, Ψ⋆ is a uniform lower bound on the average task loss satisfying 1N∑i=1Nfi()≥Ψ⋆ 1N _i=1^Nf_i( W)≥ for all W, and the initial objective gap is bounded as [Ψ(0)]−Ψ⋆≤DE[ ^(0)]- ≤ D for some constant D>0D>0. Since i(t,k)−0=i(t,k) W_i^(t,k)- W_0= X_i^(t,k), the stochastic weight-space gradient is ∇ℒi(t)(i(t,k),ξi(t,k))=∇fi(i(t,k),ξi(t,k))+λ(i(t,k)−g(t−1)). _ WL_i^(t) ( W_i^(t,k); _i^(t,k) )= _ Wf_i ( W_i^(t,k); _i^(t,k) )+λ ( X_i^(t,k)- X_g^(t-1) ). (21) Because the regularization gradient is deterministic conditioned on the current iterate, Assumption 2 implies ξi(t,k)[∇ℒi(t)(i(t,k),ξi(t,k))]=∇ℒi(t)(i(t,k)).E_ _i^(t,k) [ _ WL_i^(t) ( W_i^(t,k); _i^(t,k) ) ]= _ WL_i^(t) ( W_i^(t,k) ). (22) D.1.2 Smoothness For any W and ′ W , Assumption 1 gives ‖∇ℒi(t)()−∇ℒi(t)(′)‖F \| _ WL_i^(t)( W)- _ WL_i^(t)( W ) \|_F ≤‖∇fi()−∇fi(′)‖F+λ‖−′‖F ≤ \| _ Wf_i( W)- _ Wf_i( W ) \|_F+λ \| W- W \|_F (23) ≤(Ls+λ)‖−′‖F. ≤(L_s+λ) \| W- W \|_F. Thus, ℒi(t)L_i^(t) is (Ls+λ)(L_s+λ)-smooth and satisfies ℒi(t)(′)≤ℒi(t)()+⟨∇ℒi(t)(),′−⟩F+Ls+λ2‖′−‖F2.L_i^(t)( W ) _i^(t)( W)+ _ WL_i^(t)( W), W - W _F+ L_s+λ2 \| W - W \|_F^2. (24) D.1.3 Bounded Regularized Gradients Assumption 3 implies ‖i(t,k)‖F=‖i(t,k)i(t,k)‖F≤‖i(t,k)‖F‖i(t,k)‖F≤CACB. \| X_i^(t,k) \|_F= \| B_i^(t,k) A_i^(t,k) \|_F≤ \| B_i^(t,k) \|_F \| A_i^(t,k) \|_F≤ C_AC_B. (25) At initialization, all clients receive the same rank-r global factors, so that g(0)=i(1,0) X_g^(0)= X_i^(1,0) for every client i. Therefore, Assumption 3 gives ‖g(0)‖F=‖i(1,0)i(1,0)‖F≤‖i(1,0)‖F‖i(1,0)‖F≤CACB. \| X_g^(0) \|_F= \| B_i^(1,0) A_i^(1,0) \|_F≤ \| B_i^(1,0) \|_F \| A_i^(1,0) \|_F≤ C_AC_B. (26) We define ¯(t)=N−1∑i=1Ni(t) X^(t)=N^-1 _i=1^N X_i^(t). Since g(t) X_g^(t) is the rank-r truncated-SVD approximation of ¯(t) X^(t), ‖g(t)‖F≤‖¯(t)‖F≤1N∑i=1N‖i(t)‖F≤CACB. \| X_g^(t) \|_F≤ \| X^(t) \|_F≤ 1N _i=1^N \| X_i^(t) \|_F≤ C_AC_B. (27) Consequently, ‖i(t,k)−g(t−1)‖F≤‖i(t,k)‖F+‖g(t−1)‖F≤2CACB. \| X_i^(t,k)- X_g^(t-1) \|_F≤ \| X_i^(t,k) \|_F+ \| X_g^(t-1) \|_F≤ 2C_AC_B. (28) Under Assumption 2, the stochastic task gradient satisfies ‖∇fi(i(t,k),ξi(t,k))‖F≤Gf. \| _ Wf_i ( W_i^(t,k); _i^(t,k) ) \|_F≤ G_f. (29) Combining Eqs. (21), (28), and (29) gives ‖∇ℒi(t)(i(t,k),ξi(t,k))‖F≤Gf+2λCACB. \| _ WL_i^(t) ( W_i^(t,k); _i^(t,k) ) \|_F≤ G_f+2λ C_AC_B. (30) We define Gλ=Gf+2λCACB.G_λ=G_f+2λ C_AC_B. (31) Then, ‖∇ℒi(t)(i(t,k),ξi(t,k))‖F≤Gλ,‖∇ℒi(t)(i(t,k))‖F≤Gλ. \| _ WL_i^(t) ( W_i^(t,k); _i^(t,k) ) \|_F≤ G_λ, \| _ WL_i^(t) ( W_i^(t,k) ) \|_F≤ G_λ. (32) The second inequality follows from Eq. (22) and Jensen’s inequality. D.1.4 Proof of Theorem 1 One-Step Descent. Because i(t,k)=0+i(t,k)i(t,k) W_i^(t,k)= W_0+ B_i^(t,k) A_i^(t,k), the chain rule gives ∇ℒi(t)(i(t,k),ξi(t,k)) _ BL_i^(t)( W_i^(t,k); _i^(t,k)) =∇ℒi(t)(i(t,k),ξi(t,k))(i(t,k))⊤, = _ WL_i^(t)( W_i^(t,k); _i^(t,k))( A_i^(t,k)) , (33) ∇ℒi(t)(i(t,k),ξi(t,k)) _ AL_i^(t)( W_i^(t,k); _i^(t,k)) =(i(t,k))⊤∇ℒi(t)(i(t,k),ξi(t,k)). =( B_i^(t,k)) _ WL_i^(t)( W_i^(t,k); _i^(t,k)). The simultaneous stochastic-gradient updates are therefore i(t,k+1) B_i^(t,k+1) =i(t,k)−η∇ℒi(t)(i(t,k),ξi(t,k))(i(t,k))⊤, = B_i^(t,k)-η _ WL_i^(t)( W_i^(t,k); _i^(t,k))( A_i^(t,k)) , (34) i(t,k+1) A_i^(t,k+1) =i(t,k)−η(i(t,k))⊤∇ℒi(t)(i(t,k),ξi(t,k)). = A_i^(t,k)-η( B_i^(t,k)) _ WL_i^(t)( W_i^(t,k); _i^(t,k)). Since 0 W_0 is fixed, i(t,k+1)−i(t,k)=i(t,k+1)−i(t,k). W_i^(t,k+1)- W_i^(t,k)= X_i^(t,k+1)- X_i^(t,k). (35) Expanding the product of the updated factors gives i(t,k+1)−i(t,k)= X_i^(t,k+1)- X_i^(t,k)= −η∇ℒi(t)(i(t,k),ξi(t,k))(i(t,k))⊤i(t,k) -η _ WL_i^(t)( W_i^(t,k); _i^(t,k))( A_i^(t,k)) A_i^(t,k) (36) −ηi(t,k)(i(t,k))⊤∇ℒi(t)(i(t,k),ξi(t,k)) -η B_i^(t,k)( B_i^(t,k)) _ WL_i^(t)( W_i^(t,k); _i^(t,k)) +η2∇ℒi(t)(i(t,k),ξi(t,k))(i(t,k))⊤(i(t,k))⊤∇ℒi(t)(i(t,k),ξi(t,k)). +η^2 _ WL_i^(t)( W_i^(t,k); _i^(t,k))( A_i^(t,k)) ( B_i^(t,k)) _ WL_i^(t)( W_i^(t,k); _i^(t,k)). Applying Eq. (24) with =i(t,k) W= W_i^(t,k) and ′=i(t,k+1) W = W_i^(t,k+1) yields ℒi(t)(i(t,k+1))≤ℒi(t)(i(t,k))+⟨∇ℒi(t)(i(t,k)),i(t,k+1)−i(t,k)⟩F+Ls+λ2‖i(t,k+1)−i(t,k)‖F2.L_i^(t)( W_i^(t,k+1)) _i^(t)( W_i^(t,k))+ _ WL_i^(t)( W_i^(t,k)), W_i^(t,k+1)- W_i^(t,k) _F+ L_s+λ2 \| W_i^(t,k+1)- W_i^(t,k) \|_F^2. (37) Using Eq. (35), we substitute Eq. (36) into the inner-product and squared-increment terms of Eq. (37). Taking conditional expectation of the inner-product term gives ξi(t,k)[⟨∇ℒi(t)(i(t,k)),i(t,k+1)−i(t,k)⟩F] _ _i^(t,k) [ _ WL_i^(t)( W_i^(t,k)), W_i^(t,k+1)- W_i^(t,k) _F ] (38) = = −ηξi(t,k)[⟨∇ℒi(t)(i(t,k)),∇ℒi(t)(i(t,k),ξi(t,k))(i(t,k))⊤i(t,k)⟩F] - _ _i^(t,k) [ _ WL_i^(t)( W_i^(t,k)), _ WL_i^(t)( W_i^(t,k); _i^(t,k))( A_i^(t,k)) A_i^(t,k) _F ] −ηξi(t,k)[⟨∇ℒi(t)(i(t,k)),i(t,k)(i(t,k))⊤∇ℒi(t)(i(t,k),ξi(t,k))⟩F] - _ _i^(t,k) [ _ WL_i^(t)( W_i^(t,k)), B_i^(t,k)( B_i^(t,k)) _ WL_i^(t)( W_i^(t,k); _i^(t,k)) _F ] +η2ξi(t,k)[⟨∇ℒi(t)(i(t,k)),∇ℒi(t)(i(t,k),ξi(t,k))(i(t,k))⊤(i(t,k))⊤∇ℒi(t)(i(t,k),ξi(t,k))⟩F]. +η^2E_ _i^(t,k) [ _ WL_i^(t)( W_i^(t,k)), _ WL_i^(t)( W_i^(t,k); _i^(t,k))( A_i^(t,k)) ( B_i^(t,k)) _ WL_i^(t)( W_i^(t,k); _i^(t,k)) _F ]. Using Eq. (22) and standard properties of the Frobenius inner product, the first two terms reduce to −η‖∇ℒi(t)(i(t,k))(i(t,k))⊤‖F2−η‖(i(t,k))⊤∇ℒi(t)(i(t,k))‖F2.-η \| _ WL_i^(t)( W_i^(t,k))( A_i^(t,k)) \|_F^2-η \|( B_i^(t,k)) _ WL_i^(t)( W_i^(t,k)) \|_F^2. (39) Assumption 3, evaluated at the current iterate, gives ‖∇ℒi(t)(i(t,k))(i(t,k))⊤‖F2+‖(i(t,k))⊤∇ℒi(t)(i(t,k))‖F2≥c‖∇ℒi(t)(i(t,k))‖F2. \| _ WL_i^(t)( W_i^(t,k))( A_i^(t,k)) \|_F^2+ \|( B_i^(t,k)) _ WL_i^(t)( W_i^(t,k)) \|_F^2≥ c \| _ WL_i^(t)( W_i^(t,k)) \|_F^2. (40) Multiplying this inequality by −η<0-η<0 reverses its direction. Therefore, −η‖∇ℒi(t)(i(t,k))(i(t,k))⊤‖F2−η‖(i(t,k))⊤∇ℒi(t)(i(t,k))‖F2≤−cη‖∇ℒi(t)(i(t,k))‖F2.-η \| _ WL_i^(t)( W_i^(t,k))( A_i^(t,k)) \|_F^2-η \|( B_i^(t,k)) _ WL_i^(t)( W_i^(t,k)) \|_F^2≤-cη \| _ WL_i^(t)( W_i^(t,k)) \|_F^2. (41) For the remaining interaction term, the Cauchy–Schwarz inequality, submultiplicativity, and Eq. (32) give |⟨∇ℒi(t)(i(t,k)),∇ℒi(t)(i(t,k),ξi(t,k))(i(t,k))⊤(i(t,k))⊤∇ℒi(t)(i(t,k),ξi(t,k))⟩F| | _ WL_i^(t)( W_i^(t,k)), _ WL_i^(t)( W_i^(t,k); _i^(t,k))( A_i^(t,k)) ( B_i^(t,k)) _ WL_i^(t)( W_i^(t,k); _i^(t,k)) _F | (42) ≤‖∇ℒi(t)(i(t,k))‖F‖∇ℒi(t)(i(t,k),ξi(t,k))‖F2‖i(t,k)‖F‖i(t,k)‖F≤CACBGλ3. ≤ \| _ WL_i^(t)( W_i^(t,k)) \|_F \| _ WL_i^(t)( W_i^(t,k); _i^(t,k)) \|_F^2 \| A_i^(t,k) \|_F \| B_i^(t,k) \|_F≤ C_AC_BG_λ^3. Combining Eqs. (38), (41), and (42) yields ξi(t,k)[⟨∇ℒi(t)(i(t,k)),i(t,k+1)−i(t,k)⟩F]≤−cη‖∇ℒi(t)(i(t,k))‖F2+η2CACBGλ3.E_ _i^(t,k) [ _ WL_i^(t)( W_i^(t,k)), W_i^(t,k+1)- W_i^(t,k) _F ]≤-cη \| _ WL_i^(t)( W_i^(t,k)) \|_F^2+η^2C_AC_BG_λ^3. (43) We next bound the squared weight increment. By Eq. (35), ‖i(t,k+1)−i(t,k)‖F2=‖i(t,k+1)−i(t,k)‖F2. \| W_i^(t,k+1)- W_i^(t,k) \|_F^2= \| X_i^(t,k+1)- X_i^(t,k) \|_F^2. (44) Hence, it is sufficient to bound the three terms in Eq. (36). For the first term, repeated application of Frobenius-norm submultiplicativity gives ‖η∇ℒi(t)(i(t,k),ξi(t,k))(i(t,k))⊤i(t,k)‖F2 \|η _ WL_i^(t)( W_i^(t,k); _i^(t,k))( A_i^(t,k)) A_i^(t,k) \|_F^2 (45) ≤η2‖∇ℒi(t)(i(t,k),ξi(t,k))‖F2‖(i(t,k))⊤‖F2‖i(t,k)‖F2 ≤η^2 \| _ WL_i^(t)( W_i^(t,k); _i^(t,k)) \|_F^2 \|( A_i^(t,k)) \|_F^2 \| A_i^(t,k) \|_F^2 =η2‖∇ℒi(t)(i(t,k),ξi(t,k))‖F2‖i(t,k)‖F4≤η2CA4Gλ2. =η^2 \| _ WL_i^(t)( W_i^(t,k); _i^(t,k)) \|_F^2 \| A_i^(t,k) \|_F^4≤η^2C_A^4G_λ^2. Similarly, for the second term, ‖ηi(t,k)(i(t,k))⊤∇ℒi(t)(i(t,k),ξi(t,k))‖F2 \|η B_i^(t,k)( B_i^(t,k)) _ WL_i^(t)( W_i^(t,k); _i^(t,k)) \|_F^2 (46) ≤η2‖i(t,k)‖F2‖(i(t,k))⊤‖F2‖∇ℒi(t)(i(t,k),ξi(t,k))‖F2 ≤η^2 \| B_i^(t,k) \|_F^2 \|( B_i^(t,k)) \|_F^2 \| _ WL_i^(t)( W_i^(t,k); _i^(t,k)) \|_F^2 =η2‖i(t,k)‖F4‖∇ℒi(t)(i(t,k),ξi(t,k))‖F2≤η2CB4Gλ2. =η^2 \| B_i^(t,k) \|_F^4 \| _ WL_i^(t)( W_i^(t,k); _i^(t,k)) \|_F^2≤η^2C_B^4G_λ^2. For the second-order interaction term, ‖η2∇ℒi(t)(i(t,k),ξi(t,k))(i(t,k))⊤(i(t,k))⊤∇ℒi(t)(i(t,k),ξi(t,k))‖F2 \|η^2 _ WL_i^(t)( W_i^(t,k); _i^(t,k))( A_i^(t,k)) ( B_i^(t,k)) _ WL_i^(t)( W_i^(t,k); _i^(t,k)) \|_F^2 (47) ≤η4‖∇ℒi(t)(i(t,k),ξi(t,k))‖F4‖i(t,k)‖F2‖i(t,k)‖F2≤η4CA2CB2Gλ4. ≤η^4 \| _ WL_i^(t)( W_i^(t,k); _i^(t,k)) \|_F^4 \| A_i^(t,k) \|_F^2 \| B_i^(t,k) \|_F^2≤η^4C_A^2C_B^2G_λ^4. Applying ‖++‖F2≤3‖F2+3‖F2+3‖F2\| P+ Q+ R\|_F^2≤ 3\| P\|_F^2+3\| Q\|_F^2+3\| R\|_F^2 to Eq. (36), and then using Eqs. (45), (46), and (47), gives ξi(t,k)[‖i(t,k+1)−i(t,k)‖F2]≤3η2(CA4+CB4)Gλ2+3η4CA2CB2Gλ4.E_ _i^(t,k) [ \| W_i^(t,k+1)- W_i^(t,k) \|_F^2 ]≤ 3η^2(C_A^4+C_B^4)G_λ^2+3η^4C_A^2C_B^2G_λ^4. (48) For 0<η≤10<η≤ 1, we have η4≤η2η^4≤η^2, and hence ξi(t,k)[‖i(t,k+1)−i(t,k)‖F2]≤3η2[(CA4+CB4)Gλ2+CA2CB2Gλ4].E_ _i^(t,k) [ \| W_i^(t,k+1)- W_i^(t,k) \|_F^2 ]≤ 3η^2 [(C_A^4+C_B^4)G_λ^2+C_A^2C_B^2G_λ^4 ]. (49) Define Mλ=CACBGλ3+3(Ls+λ)2[(CA4+CB4)Gλ2+CA2CB2Gλ4].M_λ=C_AC_BG_λ^3+ 3(L_s+λ)2 [(C_A^4+C_B^4)G_λ^2+C_A^2C_B^2G_λ^4 ]. (50) Substituting Eqs. (43) and (49) into Eq. (37) gives ξi(t,k)[ℒi(t)(i(t,k+1))]≤ℒi(t)(i(t,k))−cη‖∇ℒi(t)(i(t,k))‖F2+Mλη2.E_ _i^(t,k) [L_i^(t)( W_i^(t,k+1)) ] _i^(t)( W_i^(t,k))-cη \| _ WL_i^(t)( W_i^(t,k)) \|_F^2+M_λη^2. (51) One-Round Descent. Applying the law of total expectation and summing Eq. (51) over k=0,…,τ−1k=0,…,τ-1 gives [ℒi(t)(i(t,τ))]≤[ℒi(t)(i(t,0))]−cη∑k=0τ−1[‖∇ℒi(t)(i(t,k))‖F2]+τMλη2.E [L_i^(t)( W_i^(t,τ)) ] [L_i^(t)( W_i^(t,0)) ]-cη _k=0^τ-1E [ \| _ WL_i^(t)( W_i^(t,k)) \|_F^2 ]+τ M_λη^2. (52) Local factor preservation implies i(t,0)=i(t−1),i(t,0)=i(t−1). X_i^(t,0)= X_i^(t-1), W_i^(t,0)= W_i^(t-1). (53) Using the definition of ℒi(t)L_i^(t), we therefore have ℒi(t)(i(t,0))=fi(i(t−1))+λ2‖i(t−1)−g(t−1)‖F2.L_i^(t)( W_i^(t,0))=f_i( W_i^(t-1))+ λ2 \| X_i^(t-1)- X_g^(t-1) \|_F^2. (54) Averaging this equality over the clients and using Eq. (20) gives 1N∑i=1Nℒi(t)(i(t,0))=Ψ(t−1). 1N _i=1^NL_i^(t)( W_i^(t,0))= ^(t-1). (55) At the end of local training, let i(t)=i(t,τ) X_i^(t)= X_i^(t,τ) and i(t)=i(t,τ) W_i^(t)= W_i^(t,τ). Then, ℒi(t)(i(t,τ))=fi(i(t))+λ2‖i(t)−g(t−1)‖F2.L_i^(t)( W_i^(t,τ))=f_i( W_i^(t))+ λ2 \| X_i^(t)- X_g^(t-1) \|_F^2. (56) We next compare the product-level regularization terms before and after server aggregation. Let ¯(t)=N−1∑i=1Ni(t) X^(t)=N^-1 _i=1^N X_i^(t). For any matrix X, the variance decomposition gives 1N∑i=1N‖i(t)−‖F2=1N∑i=1N‖i(t)−¯(t)‖F2+‖¯(t)−‖F2. 1N _i=1^N \| X_i^(t)- X \|_F^2= 1N _i=1^N \| X_i^(t)- X^(t) \|_F^2+ \| X^(t)- X \|_F^2. (57) The first term on the right-hand side is independent of X. Therefore, minimizing the left-hand side over matrices of rank at most r is equivalent to finding the best rank-r approximation of ¯(t) X^(t). By the truncated-SVD aggregation rule, g(t)∈argminrank()≤r1N∑i=1N‖i(t)−‖F2. X_g^(t)∈ *arg\,min_rank( X)≤ r 1N _i=1^N \| X_i^(t)- X \|_F^2. (58) The previous global product g(t−1) X_g^(t-1) is feasible because rank(g(t−1))≤rrank( X_g^(t-1))≤ r. Hence, the optimality of g(t) X_g^(t) gives 1N∑i=1N‖i(t)−g(t)‖F2≤1N∑i=1N‖i(t)−g(t−1)‖F2. 1N _i=1^N \| X_i^(t)- X_g^(t) \|_F^2≤ 1N _i=1^N \| X_i^(t)- X_g^(t-1) \|_F^2. (59) The task-loss terms fi(i(t))f_i( W_i^(t)) are unchanged by server aggregation. Combining this observation with Eq. (59) and Eq. (20) yields Ψ(t)≤1N∑i=1Nℒi(t)(i(t,τ)). ^(t)≤ 1N _i=1^NL_i^(t)( W_i^(t,τ)). (60) Averaging Eq. (52) over all clients and using Eqs. (55) and (60) gives [Ψ(t)]≤[Ψ(t−1)]−cηN∑i=1N∑k=0τ−1[‖∇ℒi(t)(i(t,k))‖F2]+τMλη2.E[ ^(t)] [ ^(t-1)]- cηN _i=1^N _k=0^τ-1E [ \| _ WL_i^(t)( W_i^(t,k)) \|_F^2 ]+τ M_λη^2. (61) Telescoping Over Communication Rounds. Rearranging Eq. (61) gives cηN∑i=1N∑k=0τ−1[‖∇ℒi(t)(i(t,k))‖F2]≤[Ψ(t−1)]−[Ψ(t)]+τMλη2. cηN _i=1^N _k=0^τ-1E [ \| _ WL_i^(t)( W_i^(t,k)) \|_F^2 ] [ ^(t-1)]-E[ ^(t)]+τ M_λη^2. (62) Summing this inequality over t=1,…,Tt=1,…,T telescopes the joint objective terms and gives cηN∑t=1T∑i=1N∑k=0τ−1[‖∇ℒi(t)(i(t,k))‖F2]≤[Ψ(0)]−[Ψ(T)]+TτMλη2. cηN _t=1^T _i=1^N _k=0^τ-1E [ \| _ WL_i^(t)( W_i^(t,k)) \|_F^2 ] [ ^(0) ]-E [ ^(T) ]+Tτ M_λη^2. (63) Since the regularization terms are nonnegative, Ψ(T)≥1N∑i=1Nfi(i(T)) ^(T)≥ 1N _i=1^Nf_i( W_i^(T)). By Assumption 4, fi(i(T))≥fi(0)−Lc‖i(T)−0‖Ff_i( W_i^(T))≥ f_i( W_0)-L_c\| W_i^(T)- W_0\|_F. Using i(T)−0=i(T) W_i^(T)- W_0= X_i^(T) and ‖i(T)‖F≤CACB\| X_i^(T)\|_F≤ C_AC_B from Eq. (25), we obtain Ψ(T) ^(T) ≥1N∑i=1Nfi(0)−LcN∑i=1N‖i(T)‖F ≥ 1N _i=1^Nf_i( W_0)- L_cN _i=1^N\| X_i^(T)\|_F ≥1N∑i=1Nfi(0)−LcCACB ≥ 1N _i=1^Nf_i( W_0)-L_cC_AC_B ≥Ψ⋆−LcCACB, ≥ -L_cC_AC_B, (64) where the last inequality follows by evaluating the assumed lower bound on the average task loss at the common model =0 W= W_0. Therefore, [Ψ(0)]−[Ψ(T)] [ ^(0)]-E[ ^(T)] ≤[Ψ(0)]−Ψ⋆+LcCACB [ ^(0)]- +L_cC_AC_B ≤D+LcCACB=D~, ≤ D+L_cC_AC_B= D, (65) with D~:=D+LcCACB D:=D+L_cC_AC_B. Substituting this inequality into Eq. (63) and dividing both sides by cηTτcη Tτ yields 1NTτ∑t=1T∑i=1N∑k=0τ−1[‖∇ℒi(t)(i(t,k))‖F2]≤D~cηTτ+Mληc. 1NTτ _t=1^T _i=1^N _k=0^τ-1E [ \| _ WL_i^(t)( W_i^(t,k)) \|_F^2 ]≤ Dcη Tτ+ M_ληc. (66) This proves Eq. (15) for 0<η≤10<η≤ 1. Finally, when D~≤MλTτ D≤ M_λTτ, choose η=D~MλTτ≤1.η= DM_λTτ≤ 1. (67) For this choice, D~cηTτ=Mληc=D~MλcTτ. Dcη Tτ= M_ληc= DM_λc Tτ. (68) Therefore, 1NTτ∑t=1T∑i=1N∑k=0τ−1[‖∇ℒi(t)(i(t,k))‖F2]≤2D~MλcTτ. 1NTτ _t=1^T _i=1^N _k=0^τ-1E [ \| _ WL_i^(t)( W_i^(t,k)) \|_F^2 ]≤ 2 DM_λc Tτ. (69) Therefore, FedPA-LoRA achieves the ((Tτ)−1/2)O((Tτ)^-1/2) averaged stationarity rate stated in Theorem 1. □ D.1.5 Proof of Proposition 1 Recall that i(t)=0+i(t) W_i^(t)= W_0+ X_i^(t) and g(t)=0+g(t) W_g^(t)= W_0+ X_g^(t). By Assumption 4, fi(g(t))≤fi(i(t))+Lc‖g(t)−i(t)‖F.f_i( W_g^(t))≤ f_i( W_i^(t))+L_c \| W_g^(t)- W_i^(t) \|_F. (70) Because the frozen pre-trained weight 0 W_0 is shared by the local and global models, g(t)−i(t)=g(t)−i(t). W_g^(t)- W_i^(t)= X_g^(t)- X_i^(t). (71) Substituting Eq. (71) into Eq. (70) gives fi(g(t))≤fi(i(t))+Lc‖g(t)−i(t)‖F.f_i( W_g^(t))≤ f_i( W_i^(t))+L_c \| X_g^(t)- X_i^(t) \|_F. (72) By Young’s inequality, for any a≥0a≥ 0 and λ>0λ>0, Lca≤λ2a2+Lc22λ.L_ca≤ λ2a^2+ L_c^22λ. (73) Applying Eq. (73) with a=‖i(t)−g(t)‖Fa=\| X_i^(t)- X_g^(t)\|_F yields Lc‖i(t)−g(t)‖F≤λ2‖i(t)−g(t)‖F2+Lc22λ.L_c \| X_i^(t)- X_g^(t) \|_F≤ λ2 \| X_i^(t)- X_g^(t) \|_F^2+ L_c^22λ. (74) Combining Eqs. (72) and (74) gives fi(g(t))≤fi(i(t))+λ2‖i(t)−g(t)‖F2+Lc22λ.f_i( W_g^(t))≤ f_i( W_i^(t))+ λ2 \| X_i^(t)- X_g^(t) \|_F^2+ L_c^22λ. (75) Averaging Eq. (75) over all clients gives 1N∑i=1Nfi(g(t))≤1N∑i=1N[fi(i(t))+λ2‖i(t)−g(t)‖F2]+Lc22λ. 1N _i=1^Nf_i( W_g^(t))≤ 1N _i=1^N [f_i( W_i^(t))+ λ2 \| X_i^(t)- X_g^(t) \|_F^2 ]+ L_c^22λ. (76) By the definitions of F()=N−1∑i=1Nfi()F( W)=N^-1 _i=1^Nf_i( W) and Ψ(t) ^(t) in Eq. (20), Eq. (76) becomes F(g(t))≤Ψ(t)+Lc22λ.F( W_g^(t))≤ ^(t)+ L_c^22λ. (77) This proves Proposition 1. □ D.2 Convergence Analysis: Heterogeneous Setting We extend the convergence guarantees of FedPA-LoRA to the heterogeneous setting where the local rank rir_i and reference rank RiR_i may differ across clients. As we will see, the server-side product-space aggregation and global product-guided alignment naturally accommodate such heterogeneity. D.2.1 Setup and Additional Assumptions For client i at round t, the preserved local LoRA product is i(t)=i(t)i(t) X_i^(t)= B_i^(t) A_i^(t) (rank≤rirank≤ r_i). The server first forms the average product ¯(t)=1N∑i=1Ni(t). X^(t)= 1N _i=1^N X_i^(t). The global update is obtained by projecting this average onto the set of matrices of rank at most Rg=maxiRiR_g= _iR_i as follows g(t)∈argminrank()≤Rg‖−¯(t)‖F. X_g^(t)∈ *arg\,min_rank( X)≤ R_g \| X- X^(t) \|_F. (78) The reference sent to client i for the next round is then the best rank-RiR_i approximation of the global product given as g,i(t)∈argminrank()≤Ri‖−g(t)‖F. X_g,i^(t)∈ *arg\,min_rank( X)≤ R_i \| X- X_g^(t) \|_F. (79) The reduced-QR and core-SVD procedure in Eqs. (5)–(6) computes these projections without ever forming dense matrices, as shown by the optimality property (7). In round t, client i minimizes the regularized local objective ℒi(t)()=fi()+λ2‖−0−g,i(t−1)‖F2,L_i^(t)( W)=f_i( W)+ λ2 \| W- W_0- X_g,i^(t-1) \|_F^2, (80) starting from the preserved product i(t,0)=0+i(t−1) W_i^(t,0)= W_0+ X_i^(t-1) and performing τ local factor SGD steps. Let i(t,k)=0+i(t,k)i(t,k) W_i^(t,k)= W_0+ B_i^(t,k) A_i^(t,k) be the weight matrix after k steps, with (i(t,0),i(t,0))=(i(t−1),i(t−1))( B_i^(t,0), A_i^(t,0))=( B_i^(t-1), A_i^(t-1)), and (i(t),i(t))=(i(t,τ),i(t,τ))( B_i^(t), A_i^(t))=( B_i^(t,τ), A_i^(t,τ)). The round-wise joint objective is defined with the client-specific references that were actually used during local training: Ψ(t)=1N∑i=1N[fi(i(t))+λ2‖i(t)−g,i(t−1)‖F2]. ^(t)= 1N _i=1^N [f_i( W_i^(t))+ λ2 \| X_i^(t)- X_g,i^(t-1) \|_F^2 ]. (81) For the analysis we also define the initial joint objective as Ψ(0)=1N∑i=1N[fi(i(0))+λ2‖i(0)−g(0)‖F2], ^(0)= 1N _i=1^N [f_i( W_i^(0))+ λ2 \| X_i^(0)- X_g^(0) \|_F^2 ], (82) where i(0)=0+i(0) W_i^(0)= W_0+ X_i^(0) are the initial local models. In addition to Assumptions 1, 2, and 4, we consider the following additional assumptions. Assumption 5 (LoRA factor regularity). There exist constants CA,CB>0C_A,C_B>0 such that for every client i, round t, and local step k, we have ‖i(t,k)‖F≤CA,‖i(t,k)‖F≤CB. \| A_i^(t,k) \|_F≤ C_A, \| B_i^(t,k) \|_F≤ C_B. (83) Furthermore, there exists c>0c>0 such that for any objective of the form ℒ()=fi()+λ2‖−0−‖F2L( W)=f_i( W)+ λ2\| W- W_0- Z\|_F^2 with ‖F≤Cg\| Z\|_F≤ C_g, where CgC_g is a uniform bound for the global product g(t) X_g^(t) and the client-specific references g,i(t) X_g,i^(t), the gradient of ℒL with respect to the LoRA factors satisfies ‖∇ℒ()⊤‖F2+‖⊤∇ℒ()‖F2≥c‖∇ℒ()‖F2, \| _ WL( W) A \|_F^2+ \| B _ WL( W) \|_F^2≥ c\, \| _ WL( W) \|_F^2, (84) where =0+ W= W_0+ B A and ‖F≤CA\| A\|_F≤ C_A, ‖F≤CB\| B\|_F≤ C_B. The constant c is independent of the specific , A, B and Z. Assumption 6 (Bounded truncation error). There exists δ≥0δ≥ 0 such that for every round t and every client i, ‖g(t)−g,i(t)‖F≤δ. \| X_g^(t)- X_g,i^(t) \|_F≤δ. (85) If the global product is approximately low-rank or the reference ranks RiR_i are chosen sufficiently large, the truncation error δ is small. In fact, the constant δ in Assumption 6 can be made explicit. Since g,i(t) X_g,i^(t) is the best rank-RiR_i approximation of g(t) X_g^(t), the Eckart–Young–Mirsky theorem [8] gives ‖g(t)−g,i(t)‖F2=∑k>Riσk(g(t))2, \| X_g^(t)- X_g,i^(t) \|_F^2= _k>R_i _k\! ( X_g^(t) )^2, (86) where σk(g(t)) _k( X_g^(t)) are the singular values of g(t) X_g^(t) in non-increasing order. Thus we may take δ=maxt∈0,…,T−1maxi∈1,…,N∑k>Riσk(g(t))2.δ= _t∈\0,…,T-1\ _i∈\1,…,N\ _k>R_i _k\! ( X_g^(t) )^2. (87) When miniRi≥rank(g(t)) _iR_i ( X_g^(t)) for all t, the tail sums vanish and δ=0δ=0, so the truncation-induced term in the heterogeneous bound vanishes, as in the homogeneous setting. To handle the mismatch between the per-client truncated references and the global product, we introduce the ideal local objective that uses the full global product as a reference: ℒiideal,(t)()=fi()+λ2‖−0−g(t−1)‖F2.L_i^ideal,(t)( W)=f_i( W)+ λ2 \| W- W_0- X_g^(t-1) \|_F^2. (88) Let i(t−1)=g,i(t−1)−g(t−1) _i^(t-1)= X_g,i^(t-1)- X_g^(t-1). By Assumption 6, we have ‖i(t−1)‖F≤δ\| _i^(t-1)\|_F≤δ. Taking the gradients of both objectives, we obtain ∇ℒi(t)() _ WL_i^(t)( W) =∇fi()+λ(−0−g,i(t−1)), = _ Wf_i( W)+λ ( W- W_0- X_g,i^(t-1) ), ∇ℒiideal,(t)() _ WL_i^ideal,(t)( W) =∇fi()+λ(−0−g(t−1)). = _ Wf_i( W)+λ ( W- W_0- X_g^(t-1) ). (89) Hence, we have ∇ℒi(t)()=∇ℒiideal,(t)()−λi(t−1), _ WL_i^(t)( W)= _ WL_i^ideal,(t)( W)-λ _i^(t-1), (90) a relation that will be used repeatedly in the analysis. D.2.2 Convergence Result Theorem 2 (FedPA-LoRA, heterogeneous ranks). Under Assumptions 1, 2, 4, 5, and 6, suppose that Ψ⋆ is a uniform lower bound on the average task loss, i.e., 1N∑i=1Nfi()≥Ψ⋆ 1N _i=1^Nf_i( W)≥ for all W, and that [Ψ(0)]−Ψ⋆≤DE[ ^(0)]- ≤ D for some constant D>0D>0. Defining D~:=D+LcCACB D:=D+L_cC_AC_B, there exist constants M¯λ>0 M_λ>0 and M~λ>0 M_λ>0, both independent of T and τ, such that for any 0<η≤10<η≤ 1, 1NTτ∑t=1T∑i=1N∑k=0τ−1[‖∇ℒi(t)(i(t,k))‖F2]≤8D~cηTτ+M¯ληc+M~λλδ2c. 1NTτ _t=1^T _i=1^N _k=0^τ-1E [ \| _ WL_i^(t)( W_i^(t,k)) \|_F^2 ]≤ 8 Dcη Tτ+ M_ληc+ M_λδ^2c. (91) When D~≤M¯λTτ/8 D≤ M_λTτ/8, so that η≤1η≤ 1, choosing η=8D~/(M¯λTτ)η= 8 D/( M_λTτ) yields 1NTτ∑t=1T∑i=1N∑k=0τ−1[‖∇ℒi(t)(i(t,k))‖F2]≤42D~M¯λcTτ+M~λλδ2c. 1NTτ _t=1^T _i=1^N _k=0^τ-1E [ \| _ WL_i^(t)( W_i^(t,k)) \|_F^2 ]≤ 4 2 D M_λc Tτ+ M_λδ^2c. (92) Thus, up to the additive constant governed by δ, FedPA-LoRA retains the ((Tτ)−1/2)O ((Tτ)^-1/2 ) averaged stationarity rate of the homogeneous case. Proposition 2 (Global LoRA loss). Under Assumption 4, for any round t≥1t≥ 1, the global model from the previous round, g(t−1)=0+g(t−1) W_g^(t-1)= W_0+ X_g^(t-1), satisfies F(g(t−1))≤Ψ(t)+Lc22λ+Lcδ,F()=1N∑i=1Nfi().F( W_g^(t-1))≤ ^(t)+ L_c^22λ+L_cδ, F( W)= 1N _i=1^Nf_i( W). (93) D.2.3 Preliminary Lemmas We begin with two intermediate lemmas that we build on to prove Theorem 2. Lemma 1 (Boundedness and smoothness). There exist constants CX,Cg,L,G>0C_X,C_g,L,G>0 such that for all rounds t, clients i, and local steps k, the following properties hold. (i) Bounded products. ‖i(t,k)‖F≤CX\| X_i^(t,k)\|_F≤ C_X, ‖g,i(t)‖F≤Cg\| X_g,i^(t)\|_F≤ C_g, ‖g(t)‖F≤Cg\| X_g^(t)\|_F≤ C_g. (i) Smoothness. The objective ℒi(t)L_i^(t) is L-smooth in W: for any ,′ W, W , ‖∇ℒi(t)()−∇ℒi(t)(′)‖F≤L‖−′‖F. \| _ WL_i^(t)( W)- _ WL_i^(t)( W ) \|_F≤ L\,\| W- W \|_F. (94) The ideal objective ℒiideal,(t)L_i^ideal,(t) is also L-smooth in W: for any ,′ W, W , ‖∇ℒiideal,(t)()−∇ℒiideal,(t)(′)‖F≤L‖−′‖F, \| _ WL_i^ideal,(t)( W)- _ WL_i^ideal,(t)( W ) \|_F≤ L\,\| W- W \|_F, (95) where L=Ls+λL=L_s+λ. (i) Bounded factor gradients. For any mini-batch ξ, the stochastic gradients of ℒi(t)L_i^(t) with respect to the LoRA factors, ∇~ℒi(t) ∇_ BL_i^(t) and ∇~ℒi(t) ∇_ AL_i^(t), satisfy ‖∇~ℒi(t)‖F2+‖∇~ℒi(t)‖F2≤G2. \| ∇_ BL_i^(t)\|_F^2+\| ∇_ AL_i^(t)\|_F^2≤ G^2. (96) Proof. Bounded products. From the factor bounds (Assumption 5), we have for any =0+ W= W_0+ B A with ‖F≤CA\| A\|_F≤ C_A and ‖F≤CB\| B\|_F≤ C_B, ‖F=‖‖F≤‖F‖F≤CACB=:CX.\| X\|_F=\| B A\|_F≤\| B\|_F\| A\|_F≤ C_AC_B=:C_X. (97) Thus, all local products are bounded by CXC_X. For the global products g(t) X_g^(t) (with t≥1t≥ 1), by Eq. (78), g(t) X_g^(t) is the best rank-RgR_g approximation of the average product ¯(t) X^(t). Since the Frobenius norm of a truncated SVD does not exceed the norm of the original matrix (indeed, for any matrix A, ‖F2=∑kσk2()\| A\|_F^2= _k _k^2( A), and truncating the SVD removes only nonnegative terms from this sum), we have ‖g(t)‖F≤‖¯(t)‖F≤1N∑i=1N‖i(t)‖F≤CX.\| X_g^(t)\|_F≤\| X^(t)\|_F≤ 1N _i=1^N\| X_i^(t)\|_F≤ C_X. (98) For the references g,i(t) X_g,i^(t), by Eq. (79), each is the best rank-RiR_i approximation of g(t) X_g^(t). Again, truncation cannot increase the Frobenius norm, so ‖g,i(t)‖F≤‖g(t)‖F≤CX.\| X_g,i^(t)\|_F≤\| X_g^(t)\|_F≤ C_X. (99) At initialization (t=0t=0), the global factors g(0) B_g^(0) and g(0) A_g^(0) are constructed with rank rg=maxirir_g= _ir_i. The client with rank rgr_g receives the full global factors, so by Assumption 5, ‖g(0)‖F≤CB\| B_g^(0)\|_F≤ C_B and ‖g(0)‖F≤CA\| A_g^(0)\|_F≤ C_A. The sub-blocks sent to other clients are subsets of these factors, so their norms are also bounded. Hence g(0) X_g^(0) and all references g,i(0) X_g,i^(0) have Frobenius norm at most CXC_X. Therefore, for all rounds t≥0t≥ 0 and clients i, ‖i(t,k)‖F≤CX,‖g(t)‖F≤CX,‖g,i(t)‖F≤CX.\| X_i^(t,k)\|_F≤ C_X, \| X_g^(t)\|_F≤ C_X, \| X_g,i^(t)\|_F≤ C_X. (100) We set Cg=CX=CACBC_g=C_X=C_AC_B, because the rank-RgR_g projection does not increase the Frobenius norm, and the references g,i(t) X_g,i^(t) are truncations of g(t) X_g^(t); the initial global product and round-one references are also bounded by CXC_X via the same factor bounds. Smoothness. The local objective ℒi(t)()=fi()+λ2‖−0−g,i(t−1)‖F2L_i^(t)( W)=f_i( W)+ λ2\| W- W_0- X_g,i^(t-1)\|_F^2 is LsL_s-smooth in fif_i and quadratic in the regularizer, hence L=(Ls+λ)L=(L_s+λ)-smooth. The ideal objective ℒiideal,(t)L_i^ideal,(t) differs only by the reference matrix, which does not affect the Hessian, so it is also L-smooth. Bounded factor gradients. For a mini-batch ξ, the stochastic gradient of ℒi(t)L_i^(t) with respect to W is ∇~ℒi(t)(,ξ)=∇~fi(,ξ)+λ(−0−g,i(t−1)), ∇_ WL_i^(t)( W;ξ)= ∇_ Wf_i( W;ξ)+λ( W- W_0- X_g,i^(t-1)), (101) where ∇~fi ∇_ Wf_i is bounded by GfG_f (Assumption 2). Moreover, ‖−0−g,i(t−1)‖F≤‖‖F+‖g,i(t−1)‖F≤CX+Cg.\| W- W_0- X_g,i^(t-1)\|_F≤\| B A\|_F+\| X_g,i^(t-1)\|_F≤ C_X+C_g. (102) Thus ‖∇~ℒi(t)‖F≤Gf+λ(CX+Cg)\| ∇_ WL_i^(t)\|_F≤ G_f+λ(C_X+C_g). By the chain rule, the stochastic factor gradients are ∇~ℒi(t)=∇~ℒi(t)⊤ ∇_ BL_i^(t)= ∇_ WL_i^(t) A and ∇~ℒi(t)=⊤∇~ℒi(t) ∇_ AL_i^(t)= B ∇_ WL_i^(t). Using the factor bounds, ‖∇~ℒi(t)‖F≤(Gf+λ(CX+Cg))CA,‖∇~ℒi(t)‖F≤(Gf+λ(CX+Cg))CB.\| ∇_ BL_i^(t)\|_F≤ (G_f+λ(C_X+C_g) )C_A, \| ∇_ AL_i^(t)\|_F≤ (G_f+λ(C_X+C_g) )C_B. (103) Therefore, ‖∇~ℒi(t)‖F2+‖∇~ℒi(t)‖F2≤2(Gf+λ(CX+Cg))2(CA2+CB2)=:G2.\| ∇_ BL_i^(t)\|_F^2+\| ∇_ AL_i^(t)\|_F^2≤ 2 (G_f+λ(C_X+C_g) )^2(C_A^2+C_B^2)=:G^2. (104) This provides the uniform bound on the stochastic factor gradients. ∎ Lemma 2 (Weight-space descent). Consider a single local SGD step starting from (,)( B, A) with =0+ W= W_0+ B A, using a mini-batch ξ to update the factors on the actual objective ℒi(t)L_i^(t): ′=−η∇~ℒi(t)(,,ξ),′=−η∇~ℒi(t)(,,ξ), B = B-η\, ∇_ BL_i^(t)( B, A;ξ), A = A-η\, ∇_ AL_i^(t)( B, A;ξ), (105) and let ′=0+′′ W = W_0+ B A . Then, there exist constants C1,C2>0C_1,C_2>0 such that for any 0<η≤10<η≤ 1, we have ξ[ℒiideal,(t)(′)]≤ℒiideal,(t)()−ηc4‖∇ℒiideal,(t)()‖F2+ηλ2δ2C1+η2C2.E_ξ [L_i^ideal,(t)( W ) ]≤\ L_i^ideal,(t)( W)- η c4 \| _ WL_i^ideal,(t)( W) \|_F^2+ηλ^2δ^2C_1+η^2C_2. (106) Proof. By the L-smoothness of ℒiideal,(t)L_i^ideal,(t) in W (Lemma 1), we have ℒiideal,(t)(′)≤ℒiideal,(t)()+⟨∇ℒiideal,(t),Δ⟩+L2‖Δ‖F2,L_i^ideal,(t)( W ) _i^ideal,(t)( W)+ _ WL_i^ideal,(t),\, W + L2\| W\|_F^2, (107) where Δ=′− W= W - W. We now bound the two terms on the right-hand side separately. Step 1: Expected inner product. Expanding the product of the updated LoRA factors gives the stochastic weight change Δ=−η(B+A)+η2BA, W=-η( g_B A+ B g_A)+η^2\, g_B g_A, (108) where B=∇~ℒi(t) g_B= ∇_ BL_i^(t) and A=∇~ℒi(t) g_A= ∇_ AL_i^(t) are the stochastic gradients with respect to the LoRA factors. Taking expectation over the mini-batch ξ, and using the unbiasedness of the stochastic gradients ([B]=∇ℒi(t)=:BE[ g_B]= _ BL_i^(t)=: G_B and [A]=∇ℒi(t)=:AE[ g_A]= _ AL_i^(t)=: G_A), yields [Δ]=−η(B+A)+η2[BA].E[ W]=-η( G_B A+ B G_A)+η^2\,E[ g_B g_A]. (109) Using the chain rule and the gradient bias relation ∇ℒi(t)=∇ℒiideal,(t)−λ _ WL_i^(t)= _ WL_i^ideal,(t)-λ (with ‖F≤δ\| \|_F≤δ), we expand the inner product: ⟨∇ℒiideal,(t),B+A⟩ _ WL_i^ideal,(t), G_B A+ B G_A =‖∇ℒiideal,(t)⊤‖F2+‖⊤∇ℒiideal,(t)‖F2 = \| _ WL_i^ideal,(t) A \|_F^2+ \| B _ WL_i^ideal,(t) \|_F^2 −λ⟨∇ℒiideal,(t),⊤+⊤⟩. -λ _ WL_i^ideal,(t), A A+ B B . By Cauchy–Schwarz, submultiplicativity, and the factor bounds, |⟨∇ℒiideal,(t),⊤+⊤⟩|≤δ(CA2+CB2)‖∇ℒiideal,(t)‖F. | _ WL_i^ideal,(t), A A+ B B |≤δ(C_A^2+C_B^2)\,\| _ WL_i^ideal,(t)\|_F. (110) Applying Young’s inequality in the form ab≤ϵ2a2+12ϵb2ab≤ ε2a^2+ 12εb^2, with a=‖∇ℒiideal,(t)‖Fa=\| _ WL_i^ideal,(t)\|_F, b=λδ(CA2+CB2)b=λδ(C_A^2+C_B^2), and choosing ϵ=cε=c (so that the coefficient of ‖∇‖2\|∇\|^2 becomes c/2c/2), gives λδ(CA2+CB2)‖∇ℒiideal,(t)‖F≤c2‖∇ℒiideal,(t)‖F2+λ2δ2(CA2+CB2)22c.λδ(C_A^2+C_B^2)\| _ WL_i^ideal,(t)\|_F≤ c2\| _ WL_i^ideal,(t)\|_F^2+ λ^2δ^2(C_A^2+C_B^2)^22c. (111) Thus, the perturbation contributes exactly −c2‖∇ℒiideal,(t)‖F2- c2\| _ WL_i^ideal,(t)\|_F^2 plus a term controlled by C1=(CA2+CB2)22cC_1= (C_A^2+C_B^2)^22c. Next, the η2η^2 term from the stochastic interaction is bounded using Cauchy–Schwarz and the bound [‖BA‖F]≤[‖B‖F2][‖A‖F2]≤G2E[\| g_B g_A\|_F]≤ E[\| g_B\|_F^2]\,E[\| g_A\|_F^2]≤ G^2 from Lemma 1(i): |⟨∇ℒiideal,(t),η2[BA]⟩|≤η2G2‖∇ℒiideal,(t)‖F. | _ WL_i^ideal,(t),\,η^2E[ g_B g_A] |≤η^2G^2\| _ WL_i^ideal,(t)\|_F. (112) By AM–GM, we have η2G2‖∇ℒiideal,(t)‖F≤ηc4‖∇ℒiideal,(t)‖F2+η3G4c≤ηc4‖∇ℒiideal,(t)‖F2+η2G4c, η^2G^2\,\| _ WL_i^ideal,(t)\|_F≤ η c4\| _ WL_i^ideal,(t)\|_F^2+ η^3G^4c≤ η c4\| _ WL_i^ideal,(t)\|_F^2+ η^2G^4c, (113) where the last inequality uses η≤1η≤ 1. Combining these results, the expected inner product is bounded by ξ[⟨∇ℒiideal,(t),Δ⟩]≤−ηc4‖∇ℒiideal,(t)‖F2+ηλ2δ2C1+η2G4c.E_ξ [ _ WL_i^ideal,(t), W ]≤- η c4\| _ WL_i^ideal,(t)\|_F^2+ηλ^2δ^2C_1+η^2 G^4c. (114) Step 2: Expected squared increment. Expanding ‖Δ‖F2\| W\|_F^2 and using ‖a+b‖F2≤2‖a‖F2+2‖b‖F2\|a+b\|_F^2≤ 2\|a\|_F^2+2\|b\|_F^2, we obtain ‖Δ‖F2≤2η2‖B+A‖F2+2η4‖BA‖F2≤4η2(‖B‖F2‖F2+‖F2‖A‖F2)+2η4G4. \| W\|_F^2≤ 2η^2\| g_B A+ B g_A\|_F^2+2η^4\| g_B g_A\|_F^2≤ 4η^2 (\| g_B\|_F^2\| A\|_F^2+\| B\|_F^2\| g_A\|_F^2 )+2η^4G^4. Taking expectation and using [‖B‖F2],[‖A‖F2]≤G2E[\| g_B\|_F^2],E[\| g_A\|_F^2]≤ G^2, together with the factor bounds ‖F≤CA\| A\|_F≤ C_A, ‖F≤CB\| B\|_F≤ C_B, gives L2ξ[‖Δ‖F2]≤η2L2(4G2(CA2+CB2)+2G4). L2E_ξ[\| W\|_F^2]≤η^2 L2 (4G^2(C_A^2+C_B^2)+2G^4 ). (115) Step 3: Combine and conclude. Merging the η2G4cη^2 G^4c term from Step 1 with the η2η^2 term from Step 2, we define C2=G4c+L2(4G2(CA2+CB2)+2G4).C_2= G^4c+ L2 (4G^2(C_A^2+C_B^2)+2G^4 ). (116) Substituting the bounds from Steps 1 and 2 into the original smoothness expansion yields the claimed inequality (106). ∎ D.2.4 Proof of Theorem 2 This proof follows the same reasoning as in the homogeneous case, but uses the truncated references. Proof. Define the ideal local objective ℒiideal,(t)()=fi()+λ2‖−0−g(t−1)‖F2,L_i^ideal,(t)( W)=f_i( W)+ λ2 \| W- W_0- X_g^(t-1) \|_F^2, (117) and recall that from (90) their gradients differ by −λi(t−1)-λ _i^(t-1) with ‖i(t−1)‖F≤δ\| _i^(t-1)\|_F≤δ. Apply Lemma 2 to each local step k=0,…,τ−1k=0,…,τ-1 of client i in round t. For each step, take the conditional expectation given the state at the beginning of that step. This yields ξi(t,k)[ℒiideal,(t)(i(t,k+1))]≤ℒiideal,(t)(i(t,k))−ηc4‖∇ℒiideal,(t)(i(t,k))‖F2+ηλ2δ2C1+η2C2.E_ _i^(t,k) [L_i^ideal,(t)( W_i^(t,k+1)) ] _i^ideal,(t)( W_i^(t,k))- η c4 \| _ WL_i^ideal,(t)( W_i^(t,k)) \|_F^2+ηλ^2δ^2C_1+η^2C_2. (118) Summing this inequality over k=0,…,τ−1k=0,…,τ-1 and taking the total expectation over all mini-batch noise, we obtain [ℒiideal,(t)(i(t))]≤[ℒiideal,(t)(i(t−1))]−ηc4∑k=0τ−1[‖∇ℒiideal,(t)(i(t,k))‖F2]+τηλ2δ2C1+τη2C2, [L_i^ideal,(t)( W_i^(t)) ] [L_i^ideal,(t)( W_i^(t-1)) ]- η c4 _k=0^τ-1E [ \| _ WL_i^ideal,(t)( W_i^(t,k)) \|_F^2 ]+τηλ^2δ^2C_1+τη^2C_2, (119) where the first term on the right is obtained from the initial condition i(t,0)=i(t−1) W_i^(t,0)= W_i^(t-1) (local factor preservation). Now introduce the auxiliary joint objective based on the full global product: Φ(t)=1N∑i=1N[fi(i(t))+λ2‖i(t)−g(t)‖F2]. ^(t)= 1N _i=1^N [f_i( W_i^(t))+ λ2 \| X_i^(t)- X_g^(t) \|_F^2 ]. (120) We first record an identity relating Φ(t−1) ^(t-1) to ℒiideal,(t)L_i^ideal,(t): since i(t−1)=0+i(t−1) W_i^(t-1)= W_0+ X_i^(t-1), ℒiideal,(t)(i(t−1))=fi(i(t−1))+λ2‖i(t−1)−g(t−1)‖F2,L_i^ideal,(t)( W_i^(t-1))=f_i( W_i^(t-1))+ λ2 \| X_i^(t-1)- X_g^(t-1) \|_F^2, (121) which is exactly the i-th term of Φ(t−1) ^(t-1); averaging over i gives Φ(t−1)=1N∑i=1Nℒiideal,(t)(i(t−1)). ^(t-1)= 1N _i=1^NL_i^ideal,(t)( W_i^(t-1)). (122) By definition (78), g(t) X_g^(t) is the best rank-RgR_g approximation of ¯(t) X^(t). Hence, for any matrix Z with rank()≤Rgrank( Z)≤ R_g, ∑i=1N‖i(t)−g(t)‖F2≤∑i=1N‖i(t)−‖F2. _i=1^N \| X_i^(t)- X_g^(t) \|_F^2≤ _i=1^N \| X_i^(t)- Z \|_F^2. (123) Choosing =g(t−1) Z= X_g^(t-1) (which has rank ≤Rg≤ R_g) gives 1N∑i=1N‖i(t)−g(t)‖F2≤1N∑i=1N‖i(t)−g(t−1)‖F2. 1N _i=1^N \| X_i^(t)- X_g^(t) \|_F^2≤ 1N _i=1^N \| X_i^(t)- X_g^(t-1) \|_F^2. (124) Recalling Φ(t) ^(t) from (120) and using ℒiideal,(t)(i(t))=fi(i(t))+λ2‖i(t)−g(t−1)‖F2, _i^ideal,(t)( W_i^(t))=f_i( W_i^(t))+ λ2 \| X_i^(t)- X_g^(t-1) \|_F^2, (125) which follows from i(t)=0+i(t) W_i^(t)= W_0+ X_i^(t), substituting the projection inequality into the definition of Φ(t) ^(t) gives Φ(t)≤1N∑i=1Nℒiideal,(t)(i(t)). ^(t)≤ 1N _i=1^NL_i^ideal,(t)( W_i^(t)). (126) Separately, averaging (119) over i=1,…,Ni=1,…,N, taking total expectation over all randomness, and rewriting the right-hand side using the identity (122), we obtain 1N∑i=1N[ℒiideal,(t)(i(t))]≤[Φ(t−1)]−ηc4N∑i=1N∑k=0τ−1[‖∇ℒiideal,(t)(i(t,k))‖F2]+τηλ2δ2C1+τη2C2. 1N _i=1^NE [L_i^ideal,(t)( W_i^(t)) ] [ ^(t-1)]- η c4N _i=1^N _k=0^τ-1E [ \| _ WL_i^ideal,(t)( W_i^(t,k)) \|_F^2 ]+τηλ^2δ^2C_1+τη^2C_2. (127) Taking expectations in (126) and combining with (127) gives the round-wise descent for Φ : [Φ(t)] [ ^(t)] ≤[Φ(t−1)]−ηc4N∑i=1N∑k=0τ−1[‖∇ℒiideal,(t)(i(t,k))‖F2]+τηλ2δ2C1+τη2C2. [ ^(t-1)]- η c4N _i=1^N _k=0^τ-1E [ \| _ WL_i^ideal,(t)( W_i^(t,k)) \|_F^2 ]+τηλ^2δ^2C_1+τη^2C_2. (128) Summing (128) for t=1,…,Tt=1,…,T and rearranging, ηc4N∑t=1T∑i=1N∑k=0τ−1[‖∇ℒiideal,(t)(i(t,k))‖F2]≤[Φ(0)]−[Φ(T)]+Tτηλ2δ2C1+Tτη2C2. η c4N _t=1^T _i=1^N _k=0^τ-1E [ \| _ WL_i^ideal,(t)( W_i^(t,k)) \|_F^2 ] [ ^(0)]-E[ ^(T)]+Tτηλ^2δ^2C_1+Tτη^2C_2. (129) Following the same Lipschitz-based derivation used to obtain Eq. (64) and using ‖i(T)‖F≤CACB\| X_i^(T)\|_F≤ C_AC_B from Lemma 1, we have Φ(T)≥Ψ⋆−LcCACB ^(T)≥ -L_cC_AC_B. Since [Φ(0)]=[Ψ(0)]E[ ^(0)]=E[ ^(0)] and [Ψ(0)]−Ψ⋆≤DE[ ^(0)]- ≤ D, it follows that [Φ(0)]−[Φ(T)]≤D+LcCACB=D~E[ ^(0)]-E[ ^(T)]≤ D+L_cC_AC_B= D. Dividing by ηcTτ/4η cTτ/4 yields 1NTτ∑t=1T∑i=1N∑k=0τ−1[‖∇ℒiideal,(t)(i(t,k))‖F2]≤4D~cηTτ+4λ2δ2C1c+4ηC2c. 1NTτ _t=1^T _i=1^N _k=0^τ-1E [ \| _ WL_i^ideal,(t)( W_i^(t,k)) \|_F^2 ]≤ 4 Dcη Tτ+ 4λ^2δ^2C_1c+ 4η C_2c. (130) Finally, we relate the gradient of the actual local objective to the ideal one. From (90), ‖∇ℒi(t)()‖F2≤2‖∇ℒiideal,(t)()‖F2+2λ2δ2. \| _ WL_i^(t)( W) \|_F^2≤ 2 \| _ WL_i^ideal,(t)( W) \|_F^2+2λ^2δ^2. (131) Averaging this inequality over all rounds and local steps and substituting (130) gives 1NTτ∑t=1T∑i=1N∑k=0τ−1[‖∇ℒi(t)(i(t,k))‖F2]≤8D~cηTτ+8λ2δ2C1c+8ηC2c+2λ2δ2. 1NTτ _t=1^T _i=1^N _k=0^τ-1E [ \| _ WL_i^(t)( W_i^(t,k)) \|_F^2 ]≤ 8 Dcη Tτ+ 8λ^2δ^2C_1c+ 8η C_2c+2λ^2δ^2. (132) Set M¯λ=8C2 M_λ=8C_2 and M~λ=λ(8C1+2c) M_λ=λ(8C_1+2c); the three η- and δ-dependent terms then combine into M¯ληc+M~λλδ2c M_ληc+ M_λδ^2c, which establishes (91). Choosing η=8D~/(M¯λTτ)η= 8 D/( M_λTτ) yields (92). ∎ D.2.5 Proof of Proposition 2 Proof. For each client i, the LcL_c-Lipschitz continuity of fif_i gives fi(g(t−1))≤fi(i(t))+Lc‖g(t−1)−i(t)‖F=fi(i(t))+Lc‖g(t−1)−i(t)‖F. f_i( W_g^(t-1))≤ f_i( W_i^(t))+L_c \| W_g^(t-1)- W_i^(t) \|_F=f_i( W_i^(t))+L_c \| X_g^(t-1)- X_i^(t) \|_F. (133) By the triangle inequality, ‖g(t−1)−i(t)‖F≤‖i(t)−g,i(t−1)‖F+‖g,i(t−1)−g(t−1)‖F. \| X_g^(t-1)- X_i^(t)\|_F≤\| X_i^(t)- X_g,i^(t-1)\|_F+\| X_g,i^(t-1)- X_g^(t-1)\|_F. (134) The second term is exactly the truncation error from Assumption 6: ‖g,i(t−1)−g(t−1)‖F≤δ.\| X_g,i^(t-1)- X_g^(t-1)\|_F≤δ. (135) Summing the inequality for fif_i over i=1,…,Ni=1,…,N and dividing by N yields F(g(t−1))≤1N∑i=1Nfi(i(t))+Lc1N∑i=1N‖i(t)−g,i(t−1)‖F+Lcδ. F( W_g^(t-1))≤ 1N _i=1^Nf_i( W_i^(t))+L_c\, 1N _i=1^N\| X_i^(t)- X_g,i^(t-1)\|_F+L_cδ. (136) Now apply Young’s inequality: Lca≤λ2a2+Lc22λL_ca≤ λ2a^2+ L_c^22λ with a=‖i(t)−g,i(t−1)‖Fa=\| X_i^(t)- X_g,i^(t-1)\|_F, which gives Lc1N∑i=1Nai≤λ21N∑i=1Nai2+Lc22λ. L_c\, 1N _i=1^Na_i≤ λ2\, 1N _i=1^Na_i^2+ L_c^22λ. (137) Therefore, F(g(t−1)) F( W_g^(t-1)) ≤1N∑i=1Nfi(i(t))+λ21N∑i=1N‖i(t)−g,i(t−1)‖F2+Lc22λ+Lcδ ≤ 1N _i=1^Nf_i( W_i^(t))+ λ2\, 1N _i=1^N \| X_i^(t)- X_g,i^(t-1) \|_F^2+ L_c^22λ+L_cδ (138) =Ψ(t)+Lc22λ+Lcδ, = ^(t)+ L_c^22λ+L_cδ, (139) where the last equality follows from the definition of Ψ(t) ^(t) in (81). ∎ Appendix E Convergence with Partial Participation Our earlier analysis assumed all N clients participate every round. We now extend to the setting where only a uniformly sampled subset t⊆1,…,NS_t \1,…,N\ of size K is active at round t. In Theorem 3, we provide a convergence guarantee for this partial participation setting. Theorem 3 (Convergence under partial participation). Assume the following: 1. Each local loss fif_i is LsL_s-smooth and LcL_c-Lipschitz, and its stochastic gradients are unbiased with norm bounded by GfG_f (Assumptions 1, 2, and 4). 2. The LoRA factors satisfy ‖i(t,k)‖F≤CA\| A_i^(t,k)\|_F≤ C_A and ‖i(t,k)‖F≤CB\| B_i^(t,k)\|_F≤ C_B, and the gradient preservation property holds with constant c>0c>0 (Assumption 3). 3. Ψ⋆ is a uniform lower bound on the average task loss, i.e., 1N∑i=1Nfi()≥Ψ⋆ 1N _i=1^Nf_i( W)≥ for all W, and the initial joint objective gap satisfies [Ψ(0)]−Ψ⋆≤DE[ ^(0)]- ≤ D for some constant D>0D>0. Define D~:=D+LcCACB D:=D+L_cC_AC_B and CX:=CACBC_X:=C_AC_B, and let MλM_λ denote the per-step descent-error constant from the homogeneous proof of Theorem 1. Suppose that at each round t, a uniformly sampled subset t⊆1,…,NS_t \1,…,N\ of size K, drawn independently of the mini-batch noise and of the past, is active. Clients not in tS_t perform no local updates at round t and retain their factors, so i(t)=i(t−1) W_i^(t)= W_i^(t-1) and i(t)=i(t−1) X_i^(t)= X_i^(t-1) for i∉ti _t. The server reconstructs the global update g(t) X_g^(t) as the best rank-r approximation of the sampled average ^(t)=1K∑i∈ti(t) X^(t)= 1K _i _t X_i^(t). For i∉ti _t, the iterates i(t,k) W_i^(t,k) in Eq. (140) denote the virtual local trajectory that client i would follow if it performed the round-t local updates; this is well defined because the sampling is independent of the mini-batch noise. Then, with B=6λCX2B=6λ C_X^2, for any 0<η≤10<η≤ 1, 1NTτ∑t=1T∑i=1N∑k=0τ−1[‖∇ℒi(t)(i(t,k))‖F2]≤D~NcηKTτ+Mληc+NBcηKτ(1−KN). 1NTτ _t=1^T _i=1^N _k=0^τ-1E [ \| _ WL_i^(t)( W_i^(t,k)) \|_F^2 ]≤ DNcη KTτ+ M_ληc+ NBcη Kτ (1- KN ). (140) When K=NK=N, the third term vanishes, and the bound reduces exactly to the full-participation bound D~cηTτ+Mληc Dcη Tτ+ M_ληc. Proof. Throughout the proof, expectations at round t are taken conditionally on the state at the start of round t; the total-expectation statements used for telescoping follow by the tower property. Recall the joint objective Ψ(t)=1N∑i=1N[fi(i(t))+λ2‖i(t)−g(t)‖F2], ^(t)= 1N _i=1^N [f_i( W_i^(t))+ λ2 \| X_i^(t)- X_g^(t) \|_F^2 ], (141) and define the sampled joint objective Ψt(t)=1K∑i∈t[fi(i(t))+λ2‖i(t)−g(t)‖F2]. _S_t^(t)= 1K _i _t [f_i( W_i^(t))+ λ2 \| X_i^(t)- X_g^(t) \|_F^2 ]. (142) Since g(t) X_g^(t) is optimal for the sampled clients and the previous global product g(t−1) X_g^(t-1), of rank at most r, is feasible in the sampled rank-constrained consensus problem, the same one-round descent proof applies to Ψt(t) _S_t^(t). Taking expectations over the mini-batch noise and the random sampling tS_t, and using the unbiasedness of the gradient estimates and of the sampling, we obtain [Ψt(t)]≤Ψ(t−1)−cηN∑i=1N∑k=0τ−1[‖∇ℒi(t)(i(t,k))‖F2]+τMλη2.E[ _S_t^(t)]≤ ^(t-1)- cηN _i=1^N _k=0^τ-1E [ \| _ WL_i^(t)( W_i^(t,k)) \|_F^2 ]+τ M_λη^2. (143) Now relate Ψ(t) ^(t) to Ψt(t) _S_t^(t). For any realization of tS_t, we have Ψ(t)=KNΨt(t)+1N∑i∉t[fi(i(t))+λ2‖i(t)−g(t)‖F2], ^(t)= KN _S_t^(t)+ 1N _i _t [f_i( W_i^(t))+ λ2 \| X_i^(t)- X_g^(t) \|_F^2 ], (144) Ψ(t−1)=KNΨt(t−1)+1N∑i∉t[fi(i(t−1))+λ2‖i(t−1)−g(t−1)‖F2]. ^(t-1)= KN _S_t^(t-1)+ 1N _i _t [f_i( W_i^(t-1))+ λ2 \| X_i^(t-1)- X_g^(t-1) \|_F^2 ]. (145) Subtracting the two decompositions, using that, by assumption, non-participating clients (i∉ti _t) keep i(t)=i(t−1) W_i^(t)= W_i^(t-1) and i(t)=i(t−1) X_i^(t)= X_i^(t-1) so that their task-loss terms cancel, and taking expectations gives [Ψ(t)−Ψ(t−1)]=KN[Ψt(t)−Ψt(t−1)]+[1N∑i∉tλ2(‖i(t−1)−g(t)‖F2−‖i(t−1)−g(t−1)‖F2)]. [ ^(t)- ^(t-1)]= KN\,E [ _S_t^(t)- _S_t^(t-1) ]+E [ 1N _i _t λ2 ( \| X_i^(t-1)- X_g^(t) \|_F^2- \| X_i^(t-1)- X_g^(t-1) \|_F^2 ) ]. (146) Let Δg(t)=g(t)−g(t−1) _g^(t)= X_g^(t)- X_g^(t-1). Then λ2(‖i(t−1)−g(t)‖F2−‖i(t−1)−g(t−1)‖F2) λ2 (\| X_i^(t-1)- X_g^(t)\|_F^2-\| X_i^(t-1)- X_g^(t-1)\|_F^2 ) =λ2(−2⟨i(t−1)−g(t−1),Δg(t)⟩+‖Δg(t)‖F2) = λ2 (-2 X_i^(t-1)- X_g^(t-1), _g^(t) +\| _g^(t)\|_F^2 ) ≤λ2(‖i(t−1)−g(t−1)‖F2+‖Δg(t)‖F2+‖Δg(t)‖F2) ≤ λ2 (\| X_i^(t-1)- X_g^(t-1)\|_F^2+\| _g^(t)\|_F^2+\| _g^(t)\|_F^2 ) =λ2‖i(t−1)−g(t−1)‖F2+λ‖Δg(t)‖F2, = λ2\| X_i^(t-1)- X_g^(t-1)\|_F^2+λ\| _g^(t)\|_F^2, (147) where the inequality is the elementary bound −2⟨,⟩≤‖F2+‖F2-2 u, v ≤\| u\|_F^2+\| v\|_F^2. Since truncating singular values cannot increase the Frobenius norm, ‖g(t)‖F≤‖^(t)‖F≤1K∑i∈t‖i(t)‖F≤CX,\| X_g^(t)\|_F≤\| X^(t)\|_F≤ 1K _i _t\| X_i^(t)\|_F≤ C_X, (148) and the same bound holds for g(t−1) X_g^(t-1); at initialization, ‖g(0)‖F≤CX\| X_g^(0)\|_F≤ C_X because all clients start from the same rank-r factors. Hence, ‖Δg(t)‖F≤‖g(t)‖F+‖g(t−1)‖F≤2CX,λ‖Δg(t)‖F2≤4λCX2.\| _g^(t)\|_F≤\| X_g^(t)\|_F+\| X_g^(t-1)\|_F≤ 2C_X, λ\| _g^(t)\|_F^2≤ 4λ C_X^2. (149) Therefore, for each non-participating client, the summand in the last term of Eq. (146) is at most λ2‖i(t−1)−g(t−1)‖F2+4λCX2. λ2 \| X_i^(t-1)- X_g^(t-1) \|_F^2+4λ C_X^2. (150) Averaging over the N−KN-K non-participating clients and taking expectations, the last term of Eq. (146) is bounded by [1N∑i∉tλ2‖i(t−1)−g(t−1)‖F2]+(1−KN)4λCX2.E [ 1N _i _t λ2 \| X_i^(t-1)- X_g^(t-1) \|_F^2 ]+ (1- KN )4λ C_X^2. (151) The first expectation equals (1−KN)R(t−1) (1- KN )R^(t-1), where R(t−1)=1N∑i=1Nλ2‖i(t−1)−g(t−1)‖F2R^(t-1)= 1N _i=1^N λ2\| X_i^(t-1)- X_g^(t-1)\|_F^2 is the regularization part of Ψ(t−1) ^(t-1). Since R(t−1)≤2λCX2R^(t-1)≤ 2λ C_X^2 (using the same product bound), the last term of Eq. (146) is at most (1−KN)2λCX2+(1−KN)4λCX2=(1−KN)6λCX2. (1- KN )2λ C_X^2+ (1- KN )4λ C_X^2= (1- KN )6λ C_X^2. (152) Substituting Eq. (143) into Eq. (146), using [Ψt(t−1)]=Ψ(t−1)E[ _S_t^(t-1)]= ^(t-1) (unbiasedness of uniform sampling) together with the excess bound above, we obtain [Ψ(t)]≤Ψ(t−1)−cηKN2∑i=1N∑k=0τ−1[‖∇ℒi(t)(i(t,k))‖F2]+KNτMλη2+(1−KN)6λCX2.E[ ^(t)]≤ ^(t-1)- cη KN^2 _i=1^N _k=0^τ-1E [ \| _ WL_i^(t)( W_i^(t,k)) \|_F^2 ]+ KNτ M_λη^2+ (1- KN )6λ C_X^2. (153) Following the same Lipschitz-based derivation used to obtain Eq. (64) and using ‖i(T)‖F≤CACB\| X_i^(T)\|_F≤ C_AC_B, we have Ψ(T)≥Ψ⋆−LcCACB ^(T)≥ -L_cC_AC_B. Hence, [Ψ(0)]−[Ψ(T)]≤D+LcCACB=D~E[ ^(0)]-E[ ^(T)]≤ D+L_cC_AC_B= D. Taking total expectations, telescoping over t=1,…,Tt=1,…,T, and dividing by cηKTτ/Ncη KTτ/N yields the bound in Eq. (140). This completes the proof. ∎ When K=NK=N, the bias term vanishes, recovering the full-participation rate. For K<NK<N, the bound implies convergence to a neighborhood of stationarity, with radius governed by K/NK/N and B. A refined analysis exploiting the concentration of uniform client sampling, rather than the worst-case reference-shift bound used here, may tighten or remove this term; we leave this to future work. Appendix F Adaptive Regularization Strength The regularization weight λ in Eq. (3) is shared across clients and communication rounds, and is selected offline through grid search. However, Figure 5 suggests that an appropriate value may vary with data heterogeneity. Near-IID settings may require weaker regularization than highly non-IID settings, whereas stronger regularization may be needed under non-IID data to control client drift. This motivates client- and round-specific weights λi(t) _i^(t) determined from quantities available locally at each client. We discuss three possible directions as illustrative alternatives rather than a definitive adaptive rule. F.1 Task-loss-based adaptation The local task loss reflects how strongly a client still needs to adapt to its data. One possible rule is λi,loss(t)=λ01+αfi(i(t,0),ξi(t,0)), _i,loss^(t)= _01+α f_i ( W_i^(t,0); _i^(t,0) ), (154) where ξi(t,0) _i^(t,0) denotes the first mini-batch sampled at client i in round t, λ0>0 _0>0 is a base regularization weight and α>0α>0 controls its sensitivity to the initial local loss. Since the task loss is nonnegative, the resulting weight naturally satisfies 0<λi,loss(t)≤λ00< _i,loss^(t)≤ _0. A large task loss weakens the global pull and prioritizes local adaptation, whereas a smaller loss restores stronger drift control. This quantity is readily available during training, although its behavior may depend on the loss scale and the sampled mini-batch. F.2 Gradient-aware drift adaptation The task-gradient magnitude provides a direct estimate of the local update strength. In LoRA, the trainable variables are the factors i B_i and i A_i, whose stochastic gradients on the initial mini-batch ξi(t,0) _i^(t,0) satisfy ∇ifi=∇fii⊤ _ B_if_i= _ Wf_i A_i and ∇ifi=i⊤∇fi _ A_if_i= B_i _ Wf_i. Because these gradients are scaled by the complementary factors, we use the normalized surrogate g^f,i(t)=‖∇ifi(i(t,0),ξi(t,0))‖F2(‖i(t,0)‖F+ε)2+‖∇ifi(i(t,0),ξi(t,0))‖F2(‖i(t,0)‖F+ε)2, g_f,i^(t)= \| _ B_if_i ( W_i^(t,0); _i^(t,0) ) \|_F^2 (\| A_i^(t,0)\|_F+ )^2+ \| _ A_if_i ( W_i^(t,0); _i^(t,0) ) \|_F^2 (\| B_i^(t,0)\|_F+ )^2, (155) where ε>0 >0 is a small constant for numerical stability. Combining this estimate with the current product mismatch yields λi,grad(t)=max(λmin,min(αg^f,i(t)‖i(t−1)−g,i(t−1)‖F+ε,λmax)), _i,grad^(t)= ( _ , (α g_f,i^(t) \| X_i^(t-1)- X_g,i^(t-1) \|_F+ , _ ) ), (156) where i(t−1)=i(t−1)i(t−1) X_i^(t-1)= B_i^(t-1) A_i^(t-1), g,i(t−1) X_g,i^(t-1) denotes the rank-RiR_i global reference product used in Eq. (3), α>0α>0 is a scaling factor controlling the influence of the gradient surrogate, and λmax>λmin≥0 _ > _ ≥ 0 are clipping bounds that confine the adaptive weight to a prescribed safe range. This rule adjusts the regularization strength according to the estimated local update magnitude relative to the client’s existing deviation from the global reference. Its robustness may still be limited by small factor norms and variability in the sampled mini-batch. F.3 Gradient-direction-based adaptation A third possibility is to consider the directional relationship between the task and regularization gradients. Let f,i(t) g_f,i^(t) and r,i(t) g_r,i^(t) denote the task and product-regularization factor gradients, respectively, concatenated across all LoRA layers after normalization by the corresponding complementary factor norms. We define λi,dir(t)=λ0(1−⟨f,i(t),r,i(t)⟩‖f,i(t)‖2‖r,i(t)‖2+ε). _i,dir^(t)= _0 (1- g_f,i^(t), g_r,i^(t) \| g_f,i^(t) \|_2 \| g_r,i^(t) \|_2+ ). (157) The cosine similarity measures the directional agreement between task optimization and product-space regularization. A larger value indicates that the two objectives favor similar updates, leading to a smaller regularization weight, whereas a smaller or negative value increases the weight to place greater emphasis on limiting client drift. With the stabilizing term ε in the denominator, the resulting weight stays strictly within the range (0,2λ0)(0,2 _0) and avoids division by zero. Overall, these alternatives provide initial directions for adaptive regularization. They introduce global hyperparameters, such as λ0 _0 and α, while dynamically adjusting the effective regularization strength based on the local state of each client, such as its current task loss or gradient magnitude. For example, in Eq. (154), a larger local loss leads to a smaller regularization weight, allowing greater local adaptation, whereas a smaller loss results in stronger global guidance. Such client-specific adaptation may reduce the need for heterogeneity-specific tuning of the regularization strength. In our preliminary exploration, we use fixed scale constants, e.g., λ0=1 _0=1, α=1α=1, λmin=0 _ =0, and λmax=10 _ =10, across the considered heterogeneity levels. The clipping bounds are used as safety limits based on preliminary experiments. Whether this fixed parameterization transfers robustly across different heterogeneous settings remains an empirical question for future work. Appendix G Complexity Analysis We first analyze the per-layer server-side complexity of FedPA-LoRA and representative federated LoRA methods under the homogeneous-rank setting. We consider N participating clients, a d×d× d weight matrix, and a common local and global LoRA rank r=Rgr=R_g. We denote the total concatenated rank by rtot=Nr_tot=Nr and focus on the typical low-rank regime rtot≪dr_tot d. The memory analysis assumes that all client uploads are retained during server aggregation. We then complement the server-side analysis with client-side wall-clock measurements that include local optimization and method-specific operations such as factor alignment, dual-adapter training, and product-space regularization. G.1 Server-Side Complexity Analysis G.1.1 Server-Side Asymptotic Complexity Table 10: Dominant per-layer server-side complexity under the homogeneous-rank setting with Nr≪dNr d. The randomized complexity assumes a sketch dimension ℓ=Rg+p=(r) =R_g+p=O(r). Method Client Upload Server Operation Computation Memory FedIT i,i B_i, A_i Two-Factor Averaging (Ndr)O(Ndr) (Ndr)O(Ndr) FedRot-LoRA ~i,~i B_i, A_i Two-Factor Averaging (Ndr)O(Ndr) (Ndr)O(Ndr) FedDPA-LoRA g,i,g,i B_g,i, A_g,i Two-Factor Averaging (Ndr)O(Ndr) (Ndr)O(Ndr) FFA-LoRA i B_i Single-Factor Averaging (Ndr)O(Ndr) (Ndr)O(Ndr) RoLoRA i B_i or i A_i Single-Factor Averaging (Ndr)O(Ndr) (Ndr)O(Ndr) FedSA-LoRA i A_i Single-Factor Averaging (Ndr)O(Ndr) (Ndr)O(Ndr) FlexLoRA i,i B_i, A_i Dense Product Aggregation and SVD (d3)O(d^3) (d2)O(d^2) FedPA-LoRA i,i B_i, A_i Reduced QR and Core SVD (N2dr2)O(N^2dr^2) (Ndr)O(Ndr) FedPA-LoRA (Randomized) i,i B_i, A_i Factored Randomized Reconstruction (Ndr2)O(Ndr^2) (Ndr)O(Ndr) Table 10 compares the dominant per-layer server-side computation and memory complexities. We include both the exact reduced-QR reconstruction of FedPA-LoRA and its randomized extension with sketch dimension ℓ=Rg+p =R_g+p. FedIT, FedRot-LoRA, and FedDPA-LoRA average both LoRA factors, whereas FFA-LoRA, FedSA-LoRA, and RoLoRA average a single factor per round. All these factor-wise methods require (Ndr)O(Ndr) server-side computation. FlexLoRA instead constructs the dense d×d× d product-space aggregate and performs SVD, requiring (Nd2r+d3)O(Nd^2r+d^3) computation and (d2+Ndr)O(d^2+Ndr) memory. In contrast, FedPA-LoRA applies reduced QR factorizations to the d×Nrd× Nr concatenated factors and performs SVD only on the resulting Nr×NrNr× Nr core matrix. Its full computation and memory complexities are (N2dr2+N3r3+Ndr2)O(N^2dr^2+N^3r^3+Ndr^2) and (Ndr+N2r2)O(Ndr+N^2r^2), respectively. Under Nr≪dNr d, the dominant costs reduce to (d3)O(d^3) and (d2)O(d^2) for FlexLoRA, and (N2dr2)O(N^2dr^2) and (Ndr)O(Ndr) for FedPA-LoRA. Thus, FedPA-LoRA avoids constructing and decomposing the dense aggregate while recovering the same optimal rank-r approximation as truncated SVD of the dense product-space aggregate. The exact reconstruction retains a quadratic dependence on N through rtot=Nr_tot=Nr. To reduce this dependence, we further consider a randomized extension with sketch dimension ℓ=Rg+p =R_g+p. For a general ℓ , its computation and memory complexities are (Ndrℓ+dℓ2)O(Ndr +d ^2) and (Ndr+dℓ+Nrℓ)O(Ndr+d +Nr ), respectively. When ℓ=(r) =O(r), these reduce to the dominant terms (Ndr2)O(Ndr^2) and (Ndr)O(Ndr). The randomized extension therefore changes the dependence on the number of clients from quadratic to linear while allowing the sketch dimension to control the trade-off between computational cost and reconstruction accuracy. G.1.2 Randomized Server-Side Reconstruction We now provide a lightweight server-side reconstruction to further reduce the quadratic dependence on the number of clients N established in Appendix G.1.1. Remark 2 (Randomized Server-Side Reconstruction). The proposed server-side reconstruction can be generalized using a randomized sketch applied directly to Δideal(t)=cat(t)cat(t) W_ideal^(t)= B_cat^(t) A_cat^(t) through its factored form. Specifically, by drawing a Gaussian test matrix (t)∈ℝd×(Rg+p) ^(t) ^d×(R_g+p), the server forms (t)=cat(t)(cat(t)(t)) Y^(t)= B_cat^(t)( A_cat^(t) ^(t)) and applies the randomized range finder of [11], followed by the rank-RgR_g truncated-SVD reconstruction in the sampled subspace. When p is a small oversampling parameter and Rg+p=(r)R_g+p=O(r), this procedure avoids both the dense d×d× d aggregate and the full rtotr_tot-dimensional basis. Its dominant computation is reduced from (N2dr2)O(N^2dr^2) to (Ndr2)O(Ndr^2), at the cost of replacing the exact optimality guarantee in Eq. (7) with the following in-expectation bound [11, Theorem 1.1]: ‖g(t)g(t)−Δideal(t)‖2≤[2+4Rg+p−1d]σRg+1(Δideal(t)),E \| B_g^(t) A_g^(t)- W_ideal^(t) \|_2≤ [2+4 R_g+pp-1 d ] _R_g+1 ( W_ideal^(t) ), where σRg+1(⋅) _R_g+1(·) denotes the (Rg+1)(R_g+1)-th singular value and p≥2p≥ 2. At the other extreme, setting Rg+p=rtotR_g+p=r_tot captures the column space of Δideal(t) W_ideal^(t) almost surely in exact arithmetic. The resulting reconstruction is therefore an optimal rank-RgR_g approximation equivalent to that obtained by the reduced-QR and core-SVD procedure, while its dominant computation becomes (drtot2)=(N2dr2)O(dr_tot^2)=O(N^2dr^2). Thus, the sketch size Rg+pR_g+p generalizes the proposed reconstruction from a lower-cost randomized approximation to the original exact reconstruction. We empirically examine this trade-off below through wall-clock measurements and downstream performance under different sketch dimensions. G.1.3 Server-Side Wall-Clock Evaluation To complement the asymptotic complexity analysis, we measure the average server-side aggregation time per communication round for RoBERTa-Large and Llama 3-8B under the homogeneous-rank setting. RoBERTa-Large uses N=10N=10 and Rg=4R_g=4, while Llama 3-8B uses N=6N=6 and Rg=8R_g=8. All methods are evaluated in the same environment. As shown in Table 11, factor-wise averaging incurs the lowest server-side cost, whereas dense product aggregation and SVD become substantially more expensive as the model dimension increases. FedPA-LoRA avoids the dense d×d× d decomposition by performing SVD only on the smaller core matrix. Randomized reconstruction further provides lower-cost operating points through the sketch dimension ℓ , with its wall-clock time generally increasing as ℓ approaches the full concatenated rank rtotr_tot. We additionally include p=0p=0 as an empirical minimum-sketch configuration in the wall-clock evaluation. Although the randomized reconstruction remains computationally well defined in this setting, the expectation bound in Remark 2 does not apply, as it requires p≥2p≥ 2. Table 11: Average per-round server-side aggregation wall-clock time under the homogeneous-rank setting. For randomized reconstruction, the sketch dimension is ℓ=Rg+p =R_g+p. RoBERTa-Large uses N=10N=10 clients and Rg=4R_g=4, yielding rtot=NRg=40r_tot=NR_g=40, whereas Llama 3-8B uses N=6N=6 clients and Rg=8R_g=8, yielding rtot=48r_tot=48. Server Operation RoBERTa-Large (N=10,Rg=4)(N=10,\;R_g=4) Llama 3-8B (N=6,Rg=8)(N=6,\;R_g=8) Sketch Dim. ℓ Time (s) Sketch Dim. ℓ Time (s) Two-Factor Averaging – 0.02980.0298 – 0.01990.0199 Single-Factor Averaging – 0.02060.0206 – 0.01090.0109 Dense Product Aggregation and SVD – 4.73404.7340 – 152.3957152.3957 Reduced QR and Core SVD – 0.08400.0840 – 0.11690.1169 Randomized Reconstruction (p=0p=0) 44 0.04730.0473 88 0.05960.0596 Randomized Reconstruction (p=8p=8) 1212 0.04930.0493 1616 0.06380.0638 Randomized Reconstruction (p=16p=16) 2020 0.05290.0529 2424 0.07060.0706 Randomized Reconstruction (p=32p=32) 3636 0.06150.0615 4040 0.09010.0901 G.1.4 Effect of the Sketch Dimension We further evaluate how the sketch dimension affects downstream performance. Experiments are conducted on MNLI under the homogeneous-rank setting with N=3N=3 clients, r=Rg=4r=R_g=4, and Dirichlet concentration parameter β=0.5β=0.5. We consider ℓ=Rg+p∈4,5,6,8,12 =R_g+p∈\4,5,6,8,12\, corresponding to p∈0,1,2,4,8p∈\0,1,2,4,8\. The largest sketch dimension ℓ=12=rtot =12=r_tot uses the full concatenated rank. Although the bound in Remark 2 requires p≥2p≥ 2, we also evaluate p∈0,1p∈\0,1\ empirically to assess further cost reductions without claiming the stated theoretical guarantee. Figure 6: MNLI accuracy of randomized server-side reconstruction under different sketch dimensions ℓ=Rg+p∈4,5,6,8,12 =R_g+p∈\4,5,6,8,12\. Experiments use the homogeneous-rank setting with N=3N=3 clients, r=Rg=4r=R_g=4, and β=0.5β=0.5. The corresponding oversampling parameters are p∈0,1,2,4,8p∈\0,1,2,4,8\, and ℓ=12=rtot =12=r_tot uses the full concatenated rank. Horizontal dashed lines denote baseline accuracies averaged over three random seeds. As shown in Figure 6, FedPA-LoRA remains robust to randomized approximation across different sketch dimensions, preserving downstream accuracy while reducing server-side computation. G.2 Client-Side Wall-Clock Evaluation We measure the client-side wall-clock time of representative local update strategies on RoBERTa-Large and Llama 3-8B. The time is first averaged across participating clients within each communication round and then across rounds. The evaluated strategies include single-factor and full LoRA updates, post-training alignment in FedRot-LoRA, joint optimization of global and local adapters in FedDPA-LoRA, and product-space regularization in FedPA-LoRA. Table 12: Average client-side wall-clock time in seconds, first averaged across participating clients within each communication round and then across rounds. The first round is excluded to avoid one-time initialization overhead. Local Update Strategy Representative Methods RoBERTa-Large Llama 3-8B Single-Factor LoRA Update FFA-LoRA, RoLoRA, FedSA-LoRA 9.13889.1388 3.80153.8015 Full LoRA Update FedIT, FlexLoRA 9.20539.2053 3.83243.8324 LoRA Update + Alignment FedRot-LoRA 9.21219.2121 3.93233.9323 Global and Local LoRA Update FedDPA-LoRA 10.343110.3431 3.83713.8371 Product-Regularized LoRA Update FedPA-LoRA 9.46039.4603 5.13435.1343 As shown in Table 12, single-factor LoRA updates incur the lowest client-side cost for both models. On RoBERTa-Large, FedRot-LoRA introduces negligible overhead over a full LoRA update, requiring 9.21219.2121 and 9.20539.2053 seconds, respectively. FedDPA-LoRA is the most expensive at 10.343110.3431 seconds because it optimizes both global and local adapters. FedPA-LoRA requires 9.46039.4603 seconds, corresponding to a 2.8%2.8\% overhead over the full LoRA update while remaining less expensive than FedDPA-LoRA. On Llama 3-8B, FedRot-LoRA and FedDPA-LoRA require 3.93233.9323 and 3.83713.8371 seconds, respectively, exhibiting comparable client-side costs despite their different update procedures. FedPA-LoRA requires 5.13435.1343 seconds, representing a 34.0%34.0\% increase over the full LoRA update but an absolute overhead of only about 1.301.30 seconds per client and round. Thus, product-space regularization introduces additional client-side computation, particularly in the larger-model setting, which represents a limitation of FedPA-LoRA despite its substantial performance improvements.