Paper deep dive
Rethinking Factor Sharing in Federated LoRA: A Rank-Aware Adaptive Approach
Xinyi Xu, Bingnan Xiao, Shuang Qin, Gang Feng, Tony Q. S. Quek
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Low-rank adaptation (LoRA) represents large language model (LLM) updates with two compact matrix factors, i.e., $A$ and $B$, providing an efficient way to fine-tune large models in federated learning paradigm. Inspired by the asymmetric roles of the LoRA factors, we study whether $A$ should be shared across clients while $B$ remains client-specific (Share-A/Local-B), or whether $B$ should instead be shared while $A$ remains client-specific (Share-B/Local-A). With a least-squares surrogate, we reveal that Share-A/Local-B requires the client-specific LoRA update matrices to use a common rank-$r$ input-side space, whereas Share-B/Local-A requires a common rank-$r$ output-side space. The two strategies therefore incur different projection residuals, indicating that the preferred strategy is the one with the smaller aggregate residual across clients. With this insight, we propose Federated Adaptive Factor Sharing Low-Rank Adaptation (FedAS-LoRA), which selects the sharing side before training to enhance fine-tuning performance. To enable adaptive factor selection before training, we design a Rank-Aware Shared-Subspace Sufficiency (RSS) metric, which effectively assesses whether a shared rank-$r$ input subspace is sufficient for the local data distributions using representations extracted from a frozen LLM backbone. Experiments across different tasks, data distributions, LoRA ranks, and participation settings confirm the effectiveness of RSS and the superior performance of FedAS-LoRA.
Tags
Links
- Source: https://arxiv.org/abs/2608.09742v1
- Canonical: https://arxiv.org/abs/2608.09742v1
Trouble viewing inline? Open PDF directly →
Full Text
106,368 characters extracted from source content.
Expand or collapse full text
Rethinking Factor Sharing in Federated LoRA: A Rank-Aware Adaptive Approach Xinyi Xu1,* Bingnan Xiao2,* Shuang Qin1,† Gang Feng1 Tony Q.S. Quek3 1University of Electronic Science and Technology of China 2Fudan University 3Singapore University of Technology and Design *Equal contribution. †Corresponding author. xinyixu@std.uestc.edu.cn, 22110720061@m.fudan.edu.cn, blueqs@uestc.edu.cn fenggang@uestc.edu.cn, tonyquek@sutd.edu.sg Abstract Low-rank adaptation (LoRA) represents large language model (LLM) updates with two compact matrix factors, i.e., A and B, providing an efficient way to fine-tune large models in federated learning paradigm. Inspired by the asymmetric roles of the LoRA factors, we study whether A should be shared across clients while B remains client-specific (Share-A/Local-B), or whether B should instead be shared while A remains client-specific (Share-B/Local-A). With a least-squares surrogate, we reveal that Share-A/Local-B requires the client-specific LoRA update matrices to use a common rank-r input-side space, whereas Share-B/Local-A requires a common rank-r output-side space. The two strategies therefore incur different projection residuals, indicating that the preferred strategy is the one with the smaller aggregate residual across clients. With this insight, we propose Federated Adaptive Factor Sharing Low-Rank Adaptation (FedAS-LoRA), which selects the sharing side before training to enhance fine-tuning performance. To enable adaptive factor selection before training, we design a Rank-Aware Shared-Subspace Sufficiency (RSS) metric, which effectively assesses whether a shared rank-r input subspace is sufficient for the local data distributions using representations extracted from a frozen LLM backbone. Experiments across different tasks, data distributions, LoRA ranks, and participation settings confirm the effectiveness of RSS and the superior performance of FedAS-LoRA. I Introduction Large language models (LLMs) have demonstrated strong capabilities in language understanding and generation, supporting a wide range of natural language processing applications Brown et al. (2020); Touvron et al. (2023). For pretrained LLMs, task-specific adaptations are needed before deployment to meet the requirements of different scenarios Houlsby et al. (2019); Li and Liang (2021); Hu et al. (2022). In many practical settings, the data required for fine-tuning are distributed across clients or organizations and cannot be centralized due to privacy, ownership, or regulatory constraints. Federated learning (FL) enables multiple clients to collaboratively fine-tune a model without sharing their private data McMahan et al. (2017); Li et al. (2020a). Nevertheless, full-model fine-tuning in FL incurs substantial computation, storage, and communication costs. Parameter-efficient fine-tuning provides a practical alternative by updating only a small set of additional parameters He et al. (2022). In particular, low-rank adaptation (LoRA) keeps the pretrained LLM backbone frozen and represents each model update by two trainable low-rank factors, making it well suited to resource-constrained federated adaptation. One critical challenge of federated LoRA lies in aggregation mismatch. For an FL system with N clients, let the LoRA update of client i be ΔWi=BiAi W_i=B_iA_i. Independently averaging the two factors BiB_i and AiA_i at the server yields (1N∑i=1NBi)(1N∑i=1NAi)≠1N∑i=1NBiAi, ( 1N _i=1^NB_i ) ( 1N _i=1^NA_i )≠ 1N _i=1^NB_iA_i, (1) which differs from directly averaging the client model updates. Several studies have explored methods to address this mismatch Bian et al. (2025); Wang et al. (2026); Chen et al. (2026). One representative line of work assigns asymmetric training and aggregation roles to the two LoRA factors. For example, FFA-LoRA fixes a common randomly initialized factor A0A_0, and only trains and aggregates BiB_i Sun et al. (2024b). FedSA-LoRA instead keeps both factors trainable, aggregates AiA_i, and retains BiB_i locally for personalized adaptation Guo et al. (2025). Although such methods adopt different factor-handling strategies, they hard-code the roles of A and B before training and apply the same assignment across diverse system settings. It remains unclear whether this hard-coded assignment is consistently superior, prompting a reconsideration of the roles of the LoRA factors B and A in federated settings. Figure 1: Accuracy gap between Share-A/Local-B and Share-B/Local-A on the MNLI-m task under label-balanced input skew. The federated system includes 3 clients with full participation and 500 training rounds. To examine this question, we compare Share-A/Local-B and Share-B/Local-A under a label-balanced input-skew partition. As shown in Fig. 1, Share-B/Local-A outperforms Share-A/Local-B, and the performance gap varies with LoRA rank. This observation does not imply that Share-B/Local-A is always preferable. Instead, it means that fixing a sharing side is unsuitable for all federated LoRA settings. We further analyze the structural asymmetry of the two LoRA factors through a least-squares surrogate. The resultant projection characterization shows that sharing A imposes a common rank-r input-side representation across clients, whereas sharing B imposes a common rank-r output-side representation. Thus, the hard-coded sharing side assignment is not globally optimal, since the appropriate shared factor depends on whether a common rank-r input-side or output-side space yields a smaller aggregate projection residual across clients. Figure 2: Illustration of (a) FFA-LoRA, (b) FedSA-LoRA, and (c) FedAS-LoRA. In FFA-LoRA, A is fixed after initialization, while B is trainable and shared with the server for aggregation. In FedSA-LoRA, both A and B are trainable, while A is shared and B is retained locally. In FedAS-LoRA, the server uses RSS to determine the sharing strategy before training. For either strategy, both factors are updated locally; each client uploads only the shared factor and retains the other factor locally. Based on this insight, we propose Federated Adaptive Factor Sharing Low-Rank Adaptation (FedAS-LoRA) framework, as illustrated in Fig. 2. FedAS-LoRA adaptively selects the shared factor before training instead of hard-coding the same policy. To guide this selection, we design a Rank-Aware Shared-Subspace Sufficiency (RSS) metric using sequence-level representations extracted from a frozen LLM backbone. RSS compares the second-order variation captured by a global rank-r input subspace with that captured by the corresponding client-specific subspaces. When the RSS deficit exceeds the calibrated threshold, FedAS-LoRA treats the global rank-r input subspace as insufficient and selects Share-B/Local-A; otherwise, it selects Share-A/Local-B. We further establish that FedAS-LoRA converges to a stationary neighborhood under arbitrary client participation. Our main contributions are summarized as follows: • We provide the first projection-based comparison between Share-A/Local-B and Share-B/Local-A in federated LoRA. We reveal that Share-A/Local-B and Share-B/Local-A yield input-side and output-side projection residuals, respectively, which explains why fixed sharing strategies are not suitable for all federated settings. • Based on this insight, we design an RSS metric to efficiently determine the shared factor before training. With RSS, we propose FedAS-LoRA, which adapts different federated settings to enhance fine-tuning performance. • We prove the convergence of FedAS-LoRA under arbitrary client participation, extending the analysis beyond the full-participation setting. For either Share-A/Local-B or Share-B/Local-A strategy, we prove that the corresponding iterates converge to a stationary neighborhood. • Extensive experiments on diverse natural language tasks demonstrate the superiority of FedAS-LoRA over other methods, and validate the effectiveness of RSS-based adaptive sharing side selection. I Related Work I-A Federated Parameter-Efficient Fine-Tuning Parameter-efficient fine-tuning (PEFT) reduces the computation, memory, and communication costs of adapting pretrained models in federated learning. Existing studies apply prompt tuning and other compact trainable modules while keeping the pretrained backbone frozen Che et al. (2023); Sun et al. (2024a); Guo et al. (2024). Low-rank adaptation (LoRA) represents each weight update with two low-rank factors, making it suitable for federated fine-tuning with limited client resources. Existing federated LoRA methods mainly address personalization, resource heterogeneity, and aggregation error Yang et al. (2025); Koo et al. (2025); Byun and Lee (2025); Shen et al. (2025). FedDPA maintains global and personalized adapters to capture shared and client-specific knowledge Yang et al. (2024). HetLoRA supports resource-heterogeneous clients through rank self-pruning and sparsity-weighted aggregation Cho et al. (2024), while FlexLoRA and FLoRA aggregate LoRA modules with different ranks through reconstruction or stacking Bai et al. (2024); Wang et al. (2024). FedEx-LoRA instead corrects the mismatch between factor-wise averaging and direct averaging of the resulting updates Singhal et al. (2025). These methods determine shared and personalized components at the adapter level, or improve the aggregation of heterogeneous LoRA updates. I-B LoRA Factor Asymmetry and Factor-Wise Federated Aggregation The two LoRA factors can exhibit different optimization and representation behaviors. LoRA-FA freezes A and trains only B to reduce the memory cost of fine-tuning Zhang et al. (2023), whereas LoRA+ assigns different learning rates to the two factors Hayou et al. (2024). Zhu et al. interpret A as extracting input features and B as mapping these features to the output space Zhu et al. (2024). HydraLoRA further adopts an asymmetric architecture with a shared A and multiple B experts Tian et al. (2024). These studies establish factor asymmetry in centralized fine-tuning, but do not consider the division of shared and client-specific factors in federated learning. Factor-wise handling has also been explored in federated LoRA Chen et al. (2026); Wang et al. (2026); Xu et al. (2025). FFA-LoRA fixes a common randomly initialized A and trains and aggregates B Sun et al. (2024b). FedSA-LoRA trains both factors, shares A, and retains B locally for personalization Guo et al. (2025). RoLoRA alternates the optimization of the two factors during federated training Chen et al. (2025), while FedRot-LoRA aligns client factors through orthogonal transformations before aggregation Zhang et al. (2026). Although these methods assign different training or aggregation roles to the two factors, the sharing side is either predetermined or not explicitly considered. Which LoRA factor should be shared across clients and which should remain client-specific remains underexplored. I Preliminaries and Motivation This section formulates the federated LoRA factor-sharing problem and proves that a fixed sharing side is insufficient. We first define a general federated factor-sharing model covering Share-A, Share-B, and full-factor sharing, followed by a motivating example to illustrate that Share-B/Local-A can outperform Share-A/Local-B. Based on a least-squares surrogate, we characterize the structural asymmetry between the two LoRA factors in federated settings. I-A Factor Sharing in Federated LoRA Consider a federated fine-tuning system with a server and N clients, indexed by =1,2,…,NN=\1,2,…,N\. Client i owns a local dataset iD_i and optimizes a local objective fif_i. For a target linear layer with frozen pretrained weight W0∈ℝdout×dinW_0 ^d_out× d_in, the LoRA update of each client i is ΔWi=BiAi W_i=B_iA_i. Here, Ai∈ℝr×dinA_i ^r× d_in, Bi∈ℝdout×rB_i ^d_out× r, and r≪mindin,doutr \d_in,d_out\ denotes the LoRA rank. Given an input representation x∈ℝdinx ^d_in, AiA_i maps x to an r-dimensional latent representation, while BiB_i maps this representation to the output space. For federated LoRA settings, we consider a fixed shared factor type Q∈A,BQ∈\A,B\. Let Qi=AiQ_i=A_i when Q=AQ=A and Qi=BiQ_i=B_i when Q=BQ=B for brevity. The corresponding personalized federated objective is minAi,Bii=1N _\A_i,B_i\_i=1^N 1N∑i=1Nfi(BiAi) 1N _i=1^Nf_i(B_iA_i) (2) s.t. .t. Qi=Qj,∀i,j∈. Q_i=Q_j,\ ∀ i,j . Here, fi(BiAi)f_i(B_iA_i) denotes the local loss evaluated at W0+BiAiW_0+B_iA_i. When Q=AQ=A, the constraint shares A across clients while leaving BiB_i client-specific, yielding Share-A/Local-B. When Q=BQ=B, it shares B while leaving AiA_i client-specific, yielding Share-B/Local-A. At training round t, let QtQ_t be the shared factor maintained by the server and Qt,ieQ_t,i^e its local copy at participating client i after e local updates. In each training round t, the following steps are executed: • Client selection and downlink transmission: The server selects an arbitrary subset of t⊆S_t with StS_t clients, and broadcasts QtQ_t, Q∈A,BQ∈\A,B\, to clients i∈ti _t. • Local update: Each selected client initializes the local copy of every shared factor as Qt,i0=QtQ_t,i^0=Q_t, and performs E local update steps with local learning rate ηℓ _ on its local dataset iD_i. During local training, the client updates all trainable LoRA factors in its local model, while keeping non-shared factors client-local. • Uplink Aggregation: After local training, each selected client uploads the update of the shared factor, ΔQt,i=Qt,iE−Qt,Q∈A,B Q_t,i\!=\!Q_t,i^E\!-\!Q_t,Q∈\A,B\ to the server for aggregation: Qt+1=Qt+ηg1St∑i∈tΔQt,i,Q_t+1=Q_t+ _g 1S_t _i _t Q_t,i, (3) where ηg _g is the server learning rate. The sharing strategy determines which side of the LoRA adaptation is exposed to cross-client averaging and which side remains client-specific. I-B Observation: Fixed Sharing Can Be Suboptimal One representative personalized federated LoRA design adopts Share-A/Local-B, i.e., sharing factor A while keeping Bi,∀i∈B_i,∀ i client-specific in each training round Guo et al. (2025). However, it remains unclear whether sharing A is always preferable to sharing B. To answer this question, we compare Share-A/Local-B and Share-B/Local-A on MNLI-m Wang et al. (2018) using a label-balanced input-skew partition. To create input skew while preserving label balance, we construct a three-client partition based on the genre annotations provided by MNLI Wang et al. (2018). Each client is assigned premise–hypothesis pairs from one of three genres: telephone, government, and fiction. The partition is constructed such that the clients have nearly identical distributions over the three MNLI labels. Thus, the clients share the same prediction task and similar label distributions, while their inputs originate from different textual genres. This setting introduces genre-based input heterogeneity while controlling for label skew. Fig. 1 compares their test accuracies for r∈2,4,8,16r∈\2,4,8,16\. It is noted that Share-B/Local-A achieves better performance for all rank settings, while the accuracy advantage varies with the LoRA rank r, indicating that the performance difference between the two sharing strategies depends on the LoRA rank r. Fig. 1 demonstrates that Share-A/Local-B is not uniformly optimal across different FL settings. This observation motivates the following question: For a given LoRA rank r and client data distributions, what determines which factor should be shared and which should remain client-specific? We next examine this question by analyzing the structural asymmetry between the two LoRA factors. I-C Structural Asymmetry Between LoRA Factors For Share-A/Local-B and Share-B/Local-A in federated LoRA, although both strategies parameterize the update as BiAiB_iA_i, the two factors act on different sides of this product. AiA_i first maps the input into an r-dimensional representation, whereas BiB_i maps that representation to the output. Sharing A therefore enforces a common input representation, while sharing B enforces a common set of output directions. We clarify the difference with a single-layer least-squares surrogate. For each client i∈i , let Δi∈ℝdout×din _i ^d_out× d_in denote its desired adaptation. The error of a rank-r LoRA update BiAiB_iA_i is then formulated as ℒi(Bi,Ai)=‖(Δi−BiAi)Σi1/2‖F2, _i(B_i,A_i)=\|( _i-B_iA_i) _i^1/2\|_F^2, (4) where Σi=[xixi⊤]≻0 _i=E[x_ix_i ] 0 with xi∈ℝdinx_i ^d_in denoting the input feature. Since each client uploads only the shared factor, the corresponding optimization problems can be formulated as ℰA=minA,Bii1N∑i=1N‖(Δi−BiA)Σi1/2‖F2E_A= _A,\B_i\_i 1N _i=1^N\|( _i-B_iA) _i^1/2\|_F^2 and ℰB=minB,Aii1N∑i=1N‖(Δi−BAi)Σi1/2‖F2E_B= _B,\A_i\_i 1N _i=1^N\|( _i-BA_i) _i^1/2\|_F^2. Here, ℰAE_A corresponds to Share-A/Local-B and ℰBE_B to Share-B/Local-A, respectively. Based on the formulations of ℰAE_A and ℰBE_B, we derive the following theorem to reveal the differences between Share-A/Local-B and Share-B/Local-A. Theorem 1. Assume Σi≻0,∀i∈ _i 0,∀ i . Minimizing ℰAE_A over Bii\B_i\_i and ℰBE_B over Aii\A_i\_i yields the following equivalent problems over the shared factors A and B, respectively: ℰA _A =minA1N∑i=1N‖ΔiΣi1/2(I−Πi,A)‖F2, = _A 1N _i=1^N \| _i _i^1/2(I- _i,A) \|_F^2, (5) ℰB _B =minB1N∑i=1N‖(I−ΠB)ΔiΣi1/2‖F2. = _B 1N _i=1^N \|(I- _B) _i _i^1/2 \|_F^2. (6) where Πi,A=(AΣi1/2)⊤(AΣiA⊤)†(AΣi1/2) _i,A=(A _i^1/2) (A _iA ) (A _i^1/2), and ΠB=B(B⊤B)†B⊤ _B=B(B B) B . From Theorem 1, it is noted that Πi,A _i,A keeps the part that lies in the row space of AΣi1/2A _i^1/2, while I−Πi,AI- _i,A keeps the part outside this space. Similarly, ΠB _B keeps the part that lies in the column space of B, while I−ΠBI- _B keeps the part outside this space. For Share-A/Local-B, with optimal Bi,∗i\B_i,*\_i, ℰAE_A is dominated by ‖ΔiΣi1/2(I−Πi,A)‖F2,∀i\| _i _i^1/2(I- _i,A)\|_F^2,∀ i, which measures the residual not covered by the shared input-side space induced by A. For Share-B/Local-A, with optimal Ai,∗i\A_i,*\_i, ‖(I−ΠB)ΔiΣi1/2‖F2\|(I- _B) _i _i^1/2\|_F^2 in ℰBE_B denotes the parts not covered by the shared output-side space induced by B. Therefore, Theorem 1 shows a structural asymmetry between the two sharing strategies: sharing A constrains the input-side space used by all clients, whereas sharing B constrains the output-side space used by all clients. The structural asymmetry between (5) and (6) explains why the same LoRA rank r may lead to different preferred sharing strategies. For fixed Δi,Σii=1N\ _i, _i\_i=1^N, r determines the dimension of the shared input-side or output-side space. Increasing r relaxes the rank constraint in both (5) and (6), so the optimal value of neither residual can increase. Therefore, neither ℰAE_A nor ℰBE_B can increase with r. Since the two residuals measure projection errors on different sides of ΔiΣi1/2 _i _i^1/2, the difference ℰA−ℰBE_A-E_B is not generally invariant to r. Consequently, the preferred sharing strategy under the least-squares surrogate may depend on the target LoRA rank. The sharing strategy with the smaller residual is preferred, while the other factor remains client-specific. This leads to a design principle: share the side on which the clients have a common rank-r structure, and keep the other side local. Since both ℰAE_A and ℰBE_B depend on Δi _i and Σi _i, which are unavailable before fine-tuning, in the next section, we design a training-free metric to effectively choose the sharing side before training. IV Method In this section, we design a rank-aware shared-subspace sufficiency metric, RSS, to determine the shared factor before training, and prove the convergence of FedAS-LoRA under arbitrary LoRA factor selection and client participation. IV-A RSS-Based Sharing-Side Selection The goal of RSS is to estimate the preferred sharing factor to enhance federated fine-tuning performance. Theorem 1 shows that this selection is determined by comparing ℰAE_A and ℰBE_B associated with common rank-r input- and output-side spaces, respectively. Since both ℰAE_A and ℰBE_B depend on Δi _i and Σi _i, which are unavailable before training, RSS utilizes sequence-level representations extracted from a frozen LLM backbone to evaluate whether the shared rank-r input subspace is sufficient for the local data distributions. Specifically, it compares the projected global-mean-centered second moment captured by a global rank-r subspace with that retained by client-specific rank-r subspaces. Based on local datasets iD_i with size |i|=Di|D_i|=D_i, we define the RSS score as RSS(r)=∑i=1NpiTr(Ug,r⊤SiUg,r)Tr(Ui,r⊤SiUi,r)+ϵ, (r)= _i=1^Np_i Tr (U_g,r S_iU_g,r )Tr (U_i,r S_iU_i,r )+ε, (7) where pi=Di∑i=1NDip_i= D_i _i=1^ND_i, and ϵ>0ε>0 is a small constant for numerical stability. SiS_i represents the centered covariance matrix of client i with Ui,rU_i,r containing its top r eigenvectors, and Ug,rU_g,r contains the top r eigenvectors of Sg=∑i=1NpiSiS_g= _i=1^Np_iS_i. RSS(r)→1RSS(r)→ 1 indicates that the global rank-r subspace preserves as much client-specific representation energy as the locally optimal rank-r subspaces. We further construct a threshold τ(r)τ(r) for the RSS deficit Δ(r)=1−RSS(r) (r)=1-RSS(r), accounting for the errors in randomly reassigning samples across clients and estimating local subspaces from finite data samples. τ(r)τ(r) is formulated as τ(r)=maxτnull(r),τboot(r),τ(r)= \ _null(r), _boot(r) \, (8) where τnull(r) _null(r) is the 9595th percentile of the RSS deficits obtained by randomly reassigning samples across clients while preserving each client’s sample size, and τboot(r) _boot(r) is the empirical 9595th percentile of the representation-energy losses obtained after re-estimating each client’s top-r subspace from bootstrap resamples of its local data. Thus, Δ(r)>τ(r) (r)>τ(r) only when the deficit Δ(r) (r) exceeds both thresholds, and the sharing-side strategy ℳ(r)M(r) is ℳ(r)=Share-B/Local-A,ifΔ(r)>τ(r),Share-A/Local-B,ifΔ(r)≤τ(r).M(r)= casesShare-B/Local-A,if\ (r)\!>\!τ(r),\\ Share-A/Local-B,if\ (r)\!≤\!τ(r). cases (9) From (9), we can see that when Δ(r)>τ(r) (r)>τ(r), the loss in representation energy from using the global rank-r subspace instead of the client-specific rank-r subspaces exceeds both calibration thresholds. Since sharing A requires all clients to use a common input-side subspace, FedAS-LoRA selects Share-B/Local-A in this case. When Δ(r)≤τ(r) (r)≤τ(r), the observed energy loss does not exceed the calibrated threshold, which leads to Share-A/Local-B. Before training starts, RSS is computed with representations extracted by a frozen LLM backbone. It compares the representation energy retained by the global rank-r subspace with that retained by the client-specific rank-r subspaces. Together with the calibrated threshold, such comparison is used in (9) to decide whether A is shared or kept client-specific. The details of the client and global subspaces, the two calibration terms, and the complete RSS workflow are provided in Appendix. IV-B Convergence Analysis Based on the designed RSS to determine the shared LoRA factor, we establish the convergence of FedAS-LoRA under arbitrary client subset tS_t with |t|=St|S_t|=S_t, ∀t∀ t. This starts with the following assumptions. Assumption 1 (Smoothness). For each client i∈i , the local objective fif_i is L-smooth. For ∀ U1,U2∈ℝdout×dinU_1,U_2 ^d_out× d_in, fi(U2)≤fi(U1)+⟨∇fi(U1),U2−U1⟩F+L2‖U2−U1‖F2.f_i(U_2)≤ f_i(U_1)+ ∇ f_i(U_1),U_2-U_1 _F+ L2\|U_2-U_1\|_F^2. (10) Assumption 2 (Unbiased and Bounded Stochastic Gradients). For any client i, let Gt,ie=∇fi(Ut,ie;ξt,ie)G_t,i^e=∇ f_i(U_t,i^e; _t,i^e) and G¯t,ie=∇fi(Ut,ie) G_t,i^e=∇ f_i(U_t,i^e), respectively. The stochastic gradient Gt,ieG_t,i^e is unbiased and bounded with Gmax>0G_ >0, i.e., [Gt,ie∣Ut,ie]=G¯t,ieand‖Gt,ie‖F≤Gmax.E [G_t,i^e U_t,i^e ]= G_t,i^e\ and\ \|G_t,i^e\|_F≤ G_ . (11) Assumption 3 (LoRA Parameter Bounds and Alignment). There exist constants CA,CB>0C_A,C_B>0 and cA,cB>0c_A,c_B>0 such that ‖At,ie‖F≤CAand‖Bt,ie‖F≤CB. \|A_t,i^e\|_F≤ C_A\ and\ \|B_t,i^e\|_F≤ C_B. (12) ⟨At,ie⊤At,ie,G¯t,ie⊤G¯t,ie⟩F≥cA‖G¯t,ie‖F2, A_t,i^e A_t,i^e, G_t,i^e G_t,i^e _F≥ c_A\| G_t,i^e\|_F^2, ⟨Bt,ieBt,ie⊤,G¯t,ieG¯t,ie⊤⟩F≥cB‖G¯t,ie‖F2. B_t,i^eB_t,i^e , G_t,i^e G_t,i^e _F≥ c_B\| G_t,i^e\|_F^2. (13) Assumptions 1 and 2 are standard in federated optimization Li et al. (2020b); Yu et al. (2019); Reddi et al. (2021). Assumption 3 characterizes the bilinear LoRA parameterization, as widely used in federated LoRA analysis Guo et al. (2025); Chen et al. (2025); Park and Klabjan (2025). Specifically, the alignment inequalities in Assumption 3 hold when the nonzero singular values of At,ieA_t,i^e and Bt,ieB_t,i^e are uniformly bounded away from zero and the realized gradient has non-vanishing projections onto their induced row and column spaces, respectively. The norm bounds keep the factor iterates in a bounded region. By the chain rule for U=BAU=BA, the gradient G¯t,ie G_t,i^e with respect to U induces the gradients G¯t,ieAt,ie⊤ G_t,i^eA_t,i^e and Bt,ie⊤G¯t,ieB_t,i^e G_t,i^e with respect to Bt,ieB_t,i^e and At,ieA_t,i^e, respectively. Since the alignment inequalities are imposed only along the realized directions of G¯t,ie G_t,i^e, they do not impose a global full-rank condition on the low-rank factors. The following result applies to any realized participation sequence that satisfies the gradient-mass coverage condition, as specified in the Appendix. Theorem 2. Let F⋆F_ be a lower bound of the aggregate objective. For both Share-A/Local-B and Share-B/Local-A sharing strategies, the iterates of FedAS-LoRA satisfy 1NT∑t=0T−1∑i=1N[‖∇fi(Ut,i)‖F2]≤F0−F⋆ηℓcT 1NT _t=0^T-1 _i=1^NE\! [\|∇ f_i(U_t,i)\|_F^2 ]≤ F_0-F_ _ cT +1ηℓcT∑t=0T−1[StNℛsel+(1−StN)ℛuns], + 1 _ cT _t=0^T-1 [ S_tNR_sel+ (1- S_tN )R_uns ], (14) where c is a positive constant, C⋆:=maxCA,CBC_ := \C_A,C_B\, ℛsel=Eηℓ2CACBGmax3+3LE2ηℓ2(CA4+CB4)Gmax2+3LE2⋅ηℓ4CA2CB2Gmax4+ηℓ2Gmax2+(12ηℓ+L2)(1+ηg)2ηℓ2E2C⋆4Gmax2R_sel\!=\!E _ ^2C_AC_BG_ ^3\!+\! 3LE2 _ ^2(C_A^4\!+\!C_B^4)G_ ^2\!+\! 3LE2· _ ^4C_A^2C_B^2G_ ^4\!+\! _ 2G_ ^2\!+\!( 12 _ \!+\! L2)(1\!+\! _g)^2 _ ^2E^2C_ ^4G_ ^2, and ℛuns=(12ρηℓ+L2)ηg2ηℓ2E2C⋆4Gmax2R_uns=( 12ρ _ + L2) _g^2 _ ^2E^2C_ ^4G_ ^2. The proof of Theorem 2 is provided in the Appendix. Theorem 2 shows that FedAS-LoRA reaches a stationary neighborhood under arbitrary client participation. The first term on the right-hand side (RHS) of (14) decreases as O(1/T)O(1/T), while the second term serves as the non-vanishing error induced by stochastic local updates, multiple local steps, and client sampling drift. Method MNLI-m MNLI-m SST-2 QNLI QQP RTE Avg. LoRA LoRA 88.36 ± 0.08 88.65 ± 0.05 95.19 ± 0.09 90.80 ± 0.89 85.42 ± 1.27 85.84 ± 0.36 89.04 FFA-LoRA 86.46 ± 0.04 87.47 ± 0.09 95.53 ± 0.03 89.24 ± 0.84 86.38 ± 0.53 87.05 ± 0.07 88.69 FedDPA-LoRA 88.77 ± 0.06 88.02 ± 0.10 96.10 ± 0.05 89.92 ± 0.59 86.60 ± 0.81 87.36 ± 0.15 89.46 FedSA-LoRA 89.77 ± 0.02 87.80 ± 0.09 95.72 ± 0.03 91.13 ± 0.56 86.87 ± 0.64 87.75 ± 0.02 89.84 FedAS-LoRA (Ours) 89.95 ± 0.06 88.86 ± 0.49 97.17 ± 0.19 92.94 ± 0.39 87.95 ± 0.56 87.77 ± 0.04 90.77 rsLoRA rsLoRA 84.46 ± 0.27 86.99 ± 0.13 96.49 ± 0.09 89.29 ± 1.44 86.11 ± 0.87 86.42 ± 0.22 88.29 FFA-rsLoRA 89.10 ± 0.12 87.85 ± 0.25 96.10 ± 0.08 89.13 ± 0.58 86.02 ± 0.34 86.98 ± 0.09 89.20 FedDPA-rsLoRA 89.36 ± 0.19 88.67 ± 0.29 96.79 ± 0.10 90.89 ± 1.17 86.68 ± 0.02 86.71 ± 0.28 89.85 FedSA-rsLoRA 90.28 ± 0.09 88.13 ± 0.12 96.75 ± 0.06 91.34 ± 0.67 86.65 ± 0.28 87.52 ± 0.07 90.12 FedAS-rsLoRA (Ours) 90.61 ± 0.10 89.79 ± 0.23 97.02 ± 0.07 91.90 ± 0.45 87.11 ± 0.11 87.58 ± 0.08 90.67 VeRA VeRA 89.57 ± 0.02 87.79 ± 0.11 94.33 ± 0.17 91.94 ± 0.52 88.67 ± 0.48 85.63 ± 0.20 89.66 FFA-VeRA 87.06 ± 0.13 86.72 ± 0.41 93.12 ± 0.35 90.59 ± 0.11 87.22 ± 0.04 85.15 ± 0.29 88.31 FedDPA-VeRA 87.98 ± 0.46 86.78 ± 0.36 94.39 ± 0.48 91.56 ± 0.31 88.71 ± 0.35 86.24 ± 0.06 89.28 FedSA-VeRA 89.73 ± 0.16 87.26 ± 0.03 94.95 ± 0.42 92.64 ± 0.35 89.05 ± 0.11 87.24 ± 0.02 90.15 FedAS-VeRA (Ours) 90.28 ± 0.54 88.38 ± 0.32 95.87 ± 0.12 93.08 ± 0.11 89.26 ± 0.22 87.22 ± 0.03 90.68 TABLE I: Performance of different methods on the GLUE benchmark. MNLI-m denotes MNLI with matched test sets, and MNLI-m denotes MNLI with mismatched test sets. The proof covers both sharing strategies since their local updates are identical: each selected client updates both A and B before uploading only the shared factor. The only difference lies in the uploaded factor update, leading to a different drift bound, i.e., ‖Qt,kE−Qt‖F≤ηℓECQGmax\|Q_t,k^E-Q_t\|_F≤ _ EC_QG_ , where Q∈A,BQ∈\A,B\. V Experiments In this section, we systematically evaluate the performance of FedAS-LoRA on two types of tasks: natural language understanding and natural language generation. V-A Experimental Setup Datasets and implementation. For natural language understanding tasks, we evaluate FedAS-LoRA with RoBERTa-large Liu et al. (2019) on the GLUE benchmark Wang et al. (2018), including MNLI, SST-2, QNLI, QQP, and RTE. For natural language generation tasks, we evaluate FedAS-LoRA on the GSM8K dataset Cobbe et al. (2021) with LLaMA3-8B Meta AI (2024). For natural language understanding tasks, consistent with FedSA-LoRA Guo et al. (2025), we use the pre-trained RoBERTa-large model with 355M parameters Liu et al. (2019) from the HuggingFace Transformers library Wolf et al. (2020) as the backbone. LoRA modules are inserted into the query and value projection matrices of each attention layer. The default LoRA rank is set to r=8r=8 with the scaling factor α=16α=16. For LoRA variants, rsLoRA adopts the same training configuration except for the rank-stabilized scaling rule Kalajdzievski (2023). We use SGD for all LoRA- and rsLoRA-based methods. For VeRA Kopiczko et al. (2024), we set r=256r=256 and use AdamW optimizer Loshchilov and Hutter (2019) with separate learning rates for the classification head and the adapted layers. Across all methods, we use a batch size of 128, E=10E=10, and T=500T=500. The local learning rate ηl _l is tuned from 0.001,0.002,0.005,0.01,0.02,0.05\0.001,0.002,0.005,0.01,0.02,0.05\, and the global learning rate ηg _g is set to 1. Experiments are conducted on NVIDIA RTX 4090 GPUs. All results are reported as mean ± standard deviation over three independent runs. See the Appendix for more experimental details. Benchmarks. We compare FedAS-LoRA with representative federated LoRA methods and fixed factor-sharing policies. LoRA denotes the standard federated implementation that trains and aggregates both factors AiA_i and BiB_i. FFA-LoRA Sun et al. (2024b) fixes a common randomly initialized factor A and only trains and aggregates BiB_i. FedDPA-LoRA Yang et al. (2024) maintains global and personalized LoRA modules to capture shared and client-specific knowledge. FedSA-LoRA Guo et al. (2025) trains both factors, aggregates AiA_i, and retains BiB_i locally, i.e., Share-A/Local-B. QNLI SST-2 MNLI-m LoRA 89.11 ± 0.41 96.74 ± 0.67 86.87 ± 0.35 FFA-LoRA 89.78 ± 0.68 96.51 ± 0.37 87.16 ± 0.24 FedDPA-LoRA 90.26 ± 0.45 96.79 ± 0.21 87.90 ± 0.20 FedSA-LoRA 90.34 ± 0.67 96.72 ± 0.44 88.16 ± 0.27 FedAS-LoRA (Ours) 91.04 ± 0.34 97.64 ± 0.26 88.67 ± 0.19 LoRA 84.15 ± 0.52 95.79 ± 0.41 83.07 ± 0.26 FFA-LoRA 84.48 ± 0.63 95.83 ± 0.51 84.25 ± 0.42 FedDPA-LoRA 85.45 ± 0.77 95.38 ± 0.59 85.80 ± 0.54 FedSA-LoRA 86.11 ± 0.50 96.50 ± 0.62 86.27 ± 0.43 FedAS-LoRA (Ours) 86.64 ± 0.41 97.58 ± 0.30 87.52 ± 0.28 TABLE I: Performance on QNLI, SST-2, and MNLI-m with N=10N=10 clients under uniform and non-uniform sampling. The upper block uses a uniform per-round sampling rate of 0.30.3, while the lower block uses independent client-specific Bernoulli sampling probabilities linearly spaced from 0.10.1 to 0.50.5. V-B Main Results For natural language understanding tasks, consistent with FFA-LoRA Sun et al. (2024b), we randomly split the data across three clients for federated learning. We consider a non-IID setting using a Dirichlet distribution with α=0.5α=0.5, i.e., Dir(0.5). From Table I, we observe that FedAS-LoRA, FedAS-rsLoRA, and FedAS-VeRA consistently outperform the compared methods in terms of average accuracy, demonstrating the effectiveness of the proposed method. FedAS-LoRA achieves an average accuracy of 90.77%90.77\%, which is 0.93 percentage points higher than that of FedSA-LoRA. These results indicate that applying the same fixed factor-sharing policy across tasks and adaptation variants can lead to suboptimal performance, supporting our design of selecting the sharing side according to the specific federated setting. For natural language generation, we evaluate LLaMA3-8B Meta AI (2024) on GSM8K using the HuggingFace Transformers library Wolf et al. (2020). Following FederatedScope-LLM Kuang et al. (2024), we split the data across three clients under an IID distribution and adopt the same optimization settings. LoRA, FFA-LoRA, and FedAS-LoRA achieve accuracies of 55.14%, 54.51%, and 56.28%, respectively, with generated examples in the Appendix. Method IID Dir(1)(1) Dir(0.5)(0.5) LoRA 90.85 ± 0.08 90.96 ± 0.46 90.80 ± 0.89 FFA-LoRA 89.51 ± 0.25 90.39 ± 0.42 89.24 ± 0.44 FedSA-LoRA 91.09 ± 0.19 90.89 ± 0.47 91.13 ± 0.57 FedAS-LoRA (Ours) 91.12 ± 0.22 91.58 ± 0.11 92.94 ± 0.39 LoRA 95.27 ± 0.02 95.41 ± 0.04 95.19 ± 0.09 FFA-LoRA 95.45 ± 0.03 95.70 ± 0.10 95.53 ± 0.03 FedSA-LoRA 96.09 ± 0.04 96.56 ± 0.04 95.72 ± 0.03 FedAS-LoRA (Ours) 96.10 ± 0.02 96.79 ± 0.04 97.17 ± 0.19 LoRA 88.30 ± 0.02 87.78 ± 0.05 88.36 ± 0.08 FFA-LoRA 88.69 ± 0.03 88.90 ± 0.06 86.46 ± 0.04 FedSA-LoRA 89.41 ± 0.05 89.01 ± 0.04 89.77 ± 0.02 FedAS-LoRA (Ours) 89.43 ± 0.04 89.02 ± 0.03 89.95 ± 0.06 TABLE I: Performance comparison on the QNLI, SST-2, and MNLI-m tasks with various degrees of data heterogeneity. From top to bottom, the three blocks correspond to QNLI, SST-2, and MNLI-m, respectively. V-C In-Depth Analyses We proceed to utilize LoRA-based methods to conduct analyses on the natural language understanding tasks of QNLI, SST-2, and MNLI-m to assess the impact of client sampling, data heterogeneity, and LoRA rank on model performance. Detailed comparisons of RSS decisions with Share-A/Local-B and Share-B/Local-A strategies, together with the communication cost analysis, are provided in the Appendix. V-C1 Effect of Client Sampling We consider 1010 clients under both uniform and non-uniform sampling settings. For uniform sampling, we apply a sampling rate of 0.3 in each round. For non-uniform sampling, client i participates independently according to a Bernoulli distribution with probability of 0.1+0.4(i−1)/(N−1)0.1+0.4(i-1)/(N-1), i.e., the participation probabilities increase linearly from 0.10.1 to 0.50.5. As shown in Table I, FedAS-LoRA consistently outperforms FedSA-LoRA across all evaluated tasks under both uniform and non-uniform sampling. Additional scalability results with N=50N=50 clients are provided in the Appendix. V-C2 Effect of Data Heterogeneity We consider the performance of FedAS-LoRA under different data heterogeneity levels by examining IID data distributions, Dir(1), and Dir(0.5) under full client participation. Table I shows that FedAS-LoRA outperforms the compared baselines across the evaluated IID and Dirichlet partitions. Additional results under structured input-skew partitions are provided in Appendix. V-C3 Effect of LoRA Rank We also explore the model performance over varying LoRA ranks r∈2,4,8,16r∈\2,4,8,16\. The corresponding results are reported in Table IV. It is noted that FedAS-LoRA maintains competitive performance across the evaluated ranks. These results indicate that no sharing factor policy is consistently preferable across different ranks, demonstrating the effectiveness of the rank-aware sharing-side selection in FedAS-LoRA. Method QNLI SST-2 MNLI-m LoRA 87.12 ± 0.16 95.41 ± 0.19 88.48 ± 0.27 FFA-LoRA 86.45 ± 0.83 94.90 ± 0.22 88.61 ± 0.09 FedSA-LoRA 89.42 ± 0.29 96.56 ± 0.41 89.61 ± 0.23 FedAS-LoRA (Ours) 89.46 ± 0.27 96.79 ± 0.21 89.63 ± 0.15 LoRA 90.25 ± 0.32 95.67 ± 0.48 87.38 ± 0.35 FFA-LoRA 91.05 ± 0.21 94.78 ± 0.49 90.62 ± 0.46 FedSA-LoRA 92.01 ± 0.35 96.30 ± 0.20 90.01 ± 0.39 FedAS-LoRA (Ours) 92.80 ± 0.23 96.33 ± 0.14 90.28 ± 0.25 LoRA 90.80 ± 0.29 95.19 ± 0.09 88.36 ± 0.08 FFA-LoRA 89.24 ± 0.24 95.53 ± 0.03 86.46 ± 0.04 FedSA-LoRA 91.13 ± 0.27 95.72 ± 0.03 89.77 ± 0.02 FedAS-LoRA (Ours) 92.94 ± 0.39 97.17 ± 0.19 89.95 ± 0.06 LoRA 88.47 ± 0.52 94.95 ± 0.19 88.75 ± 0.27 FFA-LoRA 88.25 ± 0.55 94.92 ± 0.16 88.23 ± 0.13 FedSA-LoRA 89.12 ± 0.39 95.73 ± 0.18 89.01 ± 0.17 FedAS-LoRA (Ours) 89.56 ± 0.19 95.72 ± 0.16 89.73 ± 0.07 TABLE IV: Test accuracy on QNLI, SST-2, and MNLI-m with different LoRA ranks r. From top to bottom, the four blocks correspond to r=2r=2, r=4r=4, r=8r=8, and r=16r=16, respectively. VI Conclusion In this paper, we studied which LoRA factor should be shared across clients and which should remain local during federated fine-tuning. Our investigations have shown that Share-A/Local-B and Share-B/Local-A induce input-side and output-side projection residuals, respectively. To address the structural asymmetry, we proposed FedAS-LoRA, which selects the LoRA sharing factor before training and keeps the other factor client-specific. We designed a training-free RSS metric that compares the energy captured by global and local rank-r subspaces to effectively determine the sharing factor. We established the convergence guarantees for both sharing strategies under arbitrary client participation. Experimental results demonstrated that the preferred sharing side can change with the data distribution, adapter rank, and participation pattern, while FedAS-LoRA achieves superior performance across the evaluated settings. References J. Bai, D. Chen, B. Qian, L. Yao, and Y. Li (2024) Federated fine-tuning of large language models under heterogeneous tasks and client resources. In Advances in Neural Information Processing Systems, Vol. 37, p. 14457–14483. External Links: Document, Link Cited by: §I-A. J. Bian, L. Wang, L. Zhang, and J. Xu (2025) LoRA-FAIR: federated LoRA fine-tuning with aggregation and initialization refinement. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 3737–3746. Cited by: §I. T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei (2020) Language models are few-shot learners. In Advances in Neural Information Processing Systems, Vol. 33, p. 1877–1901. Cited by: §I. Y. Byun and J. Lee (2025) Towards federated low-rank adaptation of language models with rank heterogeneity. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers), Albuquerque, New Mexico, p. 356–362. External Links: Document Cited by: §I-A. T. Che, J. Liu, Y. Zhou, J. Ren, J. Zhou, V. Sheng, H. Dai, and D. Dou (2023) Federated learning of large language models with parameter-efficient prompt tuning and adaptive optimization. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Singapore, p. 7871–7888. External Links: Document, Link Cited by: §I-A. S. Chen, Y. Guo, Y. Ju, H. Dalal, Z. Zhu, and A. Khisti (2025) Robust federated finetuning of LLMs via alternating optimization of LoRA. In Advances in Neural Information Processing Systems, Vol. 38. External Links: Link Cited by: §I-B, §IV-B. Y. Chen, R. Li, J. Shao, L. Sun, C. Zhang, J. Zhang, and X. Li (2026) Federated LoRA fine-tuning of LLMs with only transmitting matrix A or B. IEEE Transactions on Mobile Computing, p. 1–17. Note: Early Access, Art. no. 11535031 External Links: Document Cited by: §I, §I-B. Y. J. Cho, L. Liu, Z. Xu, A. Fahrezi, and G. Joshi (2024) Heterogeneous LoRA for federated fine-tuning of on-device foundation models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Miami, Florida, USA, p. 12903–12913. External Links: Document, Link Cited by: §I-A. K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, C. Hesse, and J. Schulman (2021) Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168. External Links: 2110.14168, Document, Link Cited by: §V-A. P. Guo, S. Zeng, Y. Wang, H. Fan, F. Wang, and L. Qu (2025) Selective aggregation for low-rank adaptation in federated learning. In International Conference on Learning Representations, External Links: Link Cited by: §I, §I-B, §I-B, §IV-B, §V-A, §V-A. T. Guo, S. Guo, J. Wang, X. Tang, and W. Xu (2024) PromptFL: let federated participants cooperatively learn prompts instead of models—federated learning in age of foundation model. IEEE Transactions on Mobile Computing 23 (5), p. 5179–5194. External Links: Document Cited by: §I-A. S. Hayou, N. Ghosh, and B. Yu (2024) LoRA+: efficient low rank adaptation of large models. In Proceedings of the 41st International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 235, p. 17783–17806. External Links: Link Cited by: §I-B. J. He, C. Zhou, X. Ma, T. Berg-Kirkpatrick, and G. Neubig (2022) Towards a unified view of parameter-efficient transfer learning. In International Conference on Learning Representations, External Links: Link Cited by: §I. N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. de Laroussilhe, A. Gesmundo, M. Attariyan, and S. Gelly (2019) Parameter-efficient transfer learning for NLP. In Proceedings of the 36th International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 97, p. 2790–2799. Cited by: §I. E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen (2022) LoRA: low-rank adaptation of large language models. In International Conference on Learning Representations, External Links: Link Cited by: §I. D. Kalajdzievski (2023) A rank stabilization scaling factor for fine-tuning with LoRA. arXiv preprint arXiv:2312.03732. External Links: 2312.03732, Document, Link Cited by: §V-A. J. Koo, M. Jang, and J. Ok (2025) Towards robust and efficient federated low-rank adaptation with heterogeneous clients. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Vienna, Austria, p. 416–429. External Links: Document Cited by: §I-A. D. J. Kopiczko, T. Blankevoort, and Y. M. Asano (2024) VeRA: vector-based random matrix adaptation. In International Conference on Learning Representations, External Links: Link Cited by: §V-A. W. Kuang, B. Qian, Z. Li, D. Chen, D. Gao, X. Pan, Y. Xie, Y. Li, B. Ding, and J. Zhou (2024) FederatedScope-LLM: a comprehensive package for fine-tuning large language models in federated learning. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, New York, NY, USA, p. 5260–5271. External Links: Document, Link Cited by: §V-B. T. Li, A. K. Sahu, A. Talwalkar, and V. Smith (2020a) Federated learning: challenges, methods, and future directions. IEEE Signal Processing Magazine 37 (3), p. 50–60. External Links: Document Cited by: §I. X. Li, K. Huang, W. Yang, S. Wang, and Z. Zhang (2020b) On the convergence of FedAvg on non-IID data. In International Conference on Learning Representations, External Links: Link Cited by: §IV-B. X. L. Li and P. Liang (2021) Prefix-Tuning: optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Online, p. 4582–4597. External Links: Document Cited by: §I. Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov (2019) RoBERTa: a robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692. External Links: 1907.11692, Document, Link Cited by: §V-A, §V-A. I. Loshchilov and F. Hutter (2019) Decoupled weight decay regularization. In International Conference on Learning Representations, External Links: Link Cited by: §V-A. B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y. Arcas (2017) Communication-efficient learning of deep networks from decentralized data. In Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, Proceedings of Machine Learning Research, Vol. 54, p. 1273–1282. Cited by: §I. Meta AI (2024) Introducing Meta Llama 3: the most capable openly available LLM to date. Note: https://ai.meta.com/blog/meta-llama-3/Published April 18, 2024 Cited by: §V-A, §V-B. H. Park and D. Klabjan (2025) Communication-efficient federated low-rank update algorithm and its connection to implicit regularization. In International Conference on Learning Representations, External Links: Link Cited by: §IV-B. S. J. Reddi, Z. Charles, M. Zaheer, Z. Garrett, K. Rush, J. Konečný, S. Kumar, and H. B. McMahan (2021) Adaptive federated optimization. In International Conference on Learning Representations, External Links: Link Cited by: §IV-B. Z. Shen, T. Xu, H. Wang, J. Li, and M. Pan (2025) PFedGPT: hierarchically optimizing LoRA aggregation weights for personalized federated GPT models. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, Suzhou, China, p. 4766–4778. External Links: Document Cited by: §I-A. R. Singhal, K. Ponkshe, and P. Vepakomma (2025) FedEx-LoRA: exact aggregation for federated and efficient fine-tuning of large language models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Vienna, Austria, p. 1316–1336. External Links: Document, Link Cited by: §I-A. J. Sun, Z. Xu, H. Yin, D. Yang, D. Xu, Y. Liu, Z. Du, Y. Chen, and H. R. Roth (2024a) FedBPT: efficient federated black-box prompt tuning for large language models. In Proceedings of the 41st International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 235, p. 47159–47173. External Links: Link Cited by: §I-A. Y. Sun, Z. Li, Y. Li, and B. Ding (2024b) Improving LoRA in privacy-preserving federated learning. In International Conference on Learning Representations, External Links: Link Cited by: §I, §I-B, §V-A, §V-B. C. Tian, Z. Shi, Z. Guo, L. Li, and C. Xu (2024) HydraLoRA: an asymmetric LoRA architecture for efficient fine-tuning. In Advances in Neural Information Processing Systems, Vol. 37, p. 9565–9584. External Links: Document, Link Cited by: §I-B. H. Touvron, T. Lavril, G. Izacard, X. Martinet, M. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample (2023) LLaMA: open and efficient foundation language models. arXiv preprint arXiv:2302.13971. External Links: Document Cited by: §I. A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman (2018) GLUE: a multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, Brussels, Belgium, p. 353–355. External Links: Document, Link Cited by: §I-B, §I-B, §V-A. H. Wang, X. Wang, Y. Li, J. Chen, J. Zhang, N. Yan, K. He, and W. Wang (2026) Federated LoRA fine-tuning with pipelined error-mitigated aggregation and matrix-wise freezing. In Findings of the Association for Computational Linguistics: ACL 2026, San Diego, California, United States, p. 5749–5762. External Links: Document, Link Cited by: §I, §I-B. Z. Wang, Z. Shen, Y. He, G. Sun, H. Wang, L. Lyu, and A. Li (2024) FLoRA: federated fine-tuning large language models with heterogeneous low-rank adaptations. In Advances in Neural Information Processing Systems, Vol. 37, p. 22513–22533. External Links: Document, Link Cited by: §I-A. A. Williams, N. Nangia, and S. R. Bowman (2018) A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), New Orleans, Louisiana, p. 1112–1122. External Links: Document Cited by: Results under Structured Input Skew. T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, J. Davison, S. Shleifer, P. von Platen, C. Ma, Y. Jernite, J. Plu, C. Xu, T. Le Scao, S. Gugger, M. Drame, Q. Lhoest, and A. Rush (2020) Transformers: state-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, Online, p. 38–45. External Links: Document, Link Cited by: §V-A, §V-B. B. Xu, H. Mei, Z. Bai, J. Gong, R. Yan, G. Xie, Y. Yao, B. Fernando, and X. Shu (2025) You only communicate once: one-shot federated low-rank adaptation of MLLM. In Advances in Neural Information Processing Systems, Vol. 38. Cited by: §I-B. Y. Yang, G. Long, Q. Lu, L. Zhu, J. Jiang, and C. Zhang (2025) Federated low-rank adaptation for foundation models: a survey. In Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence, p. 10779–10787. Note: Survey Track External Links: Document Cited by: §I-A. Y. Yang, G. Long, T. Shen, J. Jiang, and M. Blumenstein (2024) Dual-personalizing adapter for federated foundation models. In Advances in Neural Information Processing Systems, Vol. 37, p. 39409–39433. External Links: Document, Link Cited by: §I-A, §V-A. H. Yu, S. Yang, and S. Zhu (2019) Parallel restarted SGD with faster convergence and less communication: demystifying why model averaging works for deep learning. Proceedings of the AAAI Conference on Artificial Intelligence 33 (01), p. 5693–5700. External Links: Document, Link Cited by: §IV-B. H. Zhang, D. Kim, S. Cha, and H. Vikalo (2026) FedRot-LoRA: mitigating rotational misalignment in federated LoRA. In Forty-third International Conference on Machine Learning, External Links: Link Cited by: §I-B. L. Zhang, L. Zhang, S. Shi, X. Chu, and B. Li (2023) LoRA-FA: efficient and effective low rank representation fine-tuning. arXiv preprint arXiv:2308.03303. External Links: Document, Link Cited by: §I-B. J. Zhu, K. Greenewald, K. Nadjahi, H. Sáez De Ocáriz Borde, R. B. Gabrielsson, L. Choshen, M. Ghassemi, M. Yurochkin, and J. Solomon (2024) Asymmetry in low-rank adapters of foundation models. In Proceedings of the 41st International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 235, p. 62369–62385. External Links: Link Cited by: §I-B. Appendix A Training Process of FedAS-LoRA Algorithm 1 summarizes the training flow of FedAS-LoRA. Before federated training, the server computes RSS from the sequence-level representations extracted by the frozen LLM backbone and determines the sharing policy ℳ(r)M(r) according to (9). The selected policy remains fixed throughout training. Specifically, Share-A/Local-B sets Q=AQ=A, whereas Share-B/Local-A sets Q=BQ=B. At each training round t, the server selects a nonempty client subset t⊆S_t and sends the current shared factor QtQ_t to the selected clients. Each selected client initializes its shared-factor copy with QtQ_t, restores its retained factor, and updates both LoRA factors for E local steps. It then uploads only the shared-factor increment ΔQt,i=Qt,iE−Qt Q_t,i=Q_t,i^E-Q_t and retains the updated nonshared factor locally. The server aggregates the received increments according to (3), while the client-local factors of unselected clients remain unchanged. For clarity, Algorithm 1 is written for one LoRA-adapted layer. Algorithm 1 FedAS-LoRA 1: Input: Client datasets ii=1N\D_i\_i=1^N, frozen LLM backbone, rank r, initial LoRA factors, rounds T, local steps E, local learning rate ηℓ _ , and server learning rate ηg _g. 2: At the server, compute RSS(r)RSS(r), Δ(r)=1−RSS(r) (r)=1-RSS(r), and τ(r)τ(r), and determine ℳ(r)M(r) according to (9). 3: if ℳ(r)=Share-A/Local-BM(r)=Share-A/Local-B then 4: Set Q=AQ=A; the server maintains Qt=AtQ_t=A_t, and client i retains Bt,iB_t,i locally. 5: else 6: Set Q=BQ=B; the server maintains Qt=BtQ_t=B_t, and client i retains At,iA_t,i locally. 7: end if 8: for t=0,1,…,T−1t=0,1,…,T-1 do 9: The server selects a nonempty subset t⊆S_t with St=|t|S_t=|S_t| and sends QtQ_t to each client i∈ti _t. 10: for each client i∈ti _t in parallel do 11: Set Qt,i0=QtQ_t,i^0=Q_t and initialize the nonshared factor from client i’s retained local state. 12: for e=0,1,…,E−1e=0,1,…,E-1 do 13: Sample a mini-batch ξt,ie _t,i^e and set Ut,ie=Bt,ieAt,ieU_t,i^e=B_t,i^eA_t,i^e. 14: Compute Gt,ie=∇fi(Ut,ie;ξt,ie)G_t,i^e=∇ f_i(U_t,i^e; _t,i^e). 15: Update Bt,ie+1=Bt,ie−ηℓGt,ieAt,ie⊤B_t,i^e+1=B_t,i^e- _ G_t,i^eA_t,i^e . 16: Update At,ie+1=At,ie−ηℓBt,ie⊤Gt,ieA_t,i^e+1=A_t,i^e- _ B_t,i^e G_t,i^e. 17: end for 18: Upload ΔQt,i=Qt,iE−Qt Q_t,i=Q_t,i^E-Q_t; retain Bt,iEB_t,i^E if Q=AQ=A, or At,iEA_t,i^E if Q=BQ=B. 19: end for 20: The server updates Qt+1=Qt+ηgSt−1∑i∈tΔQt,iQ_t+1=Q_t+ _gS_t^-1 _i _t Q_t,i. 21: Each client j∉tj _t keeps its client-local factor unchanged. 22: end for 23: return The shared factor QTQ_T and the retained client-local factors. Appendix B Design Details of the RSS Metric Algorithm 2 RSS Computation Workflow 1: Input: Client datasets ii=1N\D_i\_i=1^N, frozen LLM backbone, rank r, and ϵ>0ε>0. 2: Extract the sequence-level representation h(x)h(x) for every sample using the frozen LLM backbone. 3: Compute the global representation mean h¯ h from all client samples. 4: for each client i∈i do 5: Compute SiS_i according to (16). 6: end for 7: Compute RSS(r)RSS(r) according to (7), and set Δ(r)=1−RSS(r) (r)=1-RSS(r). 8: for each random reassignment indexed by b do 9: Randomly reassign the samples across clients while preserving the client sizes Dii=1N\D_i\_i=1^N. 10: Recompute the client and global rank-r subspaces from the reassigned datasets. 11: Compute RSSnull(b)(r)RSS_null^(b)(r) using (7), and set Δnull(b)(r)=1−RSSnull(b)(r) _null^(b)(r)=1-RSS_null^(b)(r). 12: end for 13: Compute τnull(r) _null(r) according to (18). 14: for each bootstrap replicate indexed by b do 15: for each client i∈i do 16: Draw DiD_i samples with replacement from iD_i. 17: Using the same global mean h¯ h, recompute the client matrix and let Ui,r(b)U_i,r^(b) contain its top r orthonormal eigenvectors. 18: end for 19: Compute eboot(b)(r)e_boot^(b)(r) according to (19). 20: end for 21: Compute τboot(r) _boot(r) according to (20). 22: Set τ(r)=maxτnull(r),τboot(r)τ(r)= \ _null(r), _boot(r)\. 23: if Δ(r)>τ(r) (r)>τ(r) then 24: Set ℳ(r)=Share-B/Local-AM(r)=Share-B/Local-A. 25: else 26: Set ℳ(r)=Share-A/Local-BM(r)=Share-A/Local-B. 27: end if 28: return RSS(r)RSS(r), Δ(r) (r), τ(r)τ(r), and ℳ(r)M(r). This section provides the construction of the client and global subspaces in (7), followed by the two calibration terms in (8). Client and global subspaces. Let h(x)∈ℝdhh(x) ^d_h denote the sequence-level representation of input x extracted from the frozen LLM backbone. Given the client datasets ii=1N\D_i\_i=1^N with Di=|i|D_i=|D_i|, the global representation mean is h¯=1∑j=1NDj∑j=1N∑(x,y)∈jh(x). h= 1 _j=1^ND_j _j=1^N _(x,y) _jh(x). (15) For each client i∈i , we then define Si=1Di∑(x,y)∈i(h(x)−h¯)(h(x)−h¯)⊤. S_i= 1D_i _(x,y) _i (h(x)- h ) (h(x)- h ) . (16) All clients are centered using the same global mean h¯ h. Hence, SiS_i reflects both the variation of client i’s representations and the difference between its representation mean and the global mean. Let Ui,r∈ℝdh×rU_i,r ^d_h× r contain the top r orthonormal eigenvectors of SiS_i. The global matrix is Sg=∑i=1NpiSi=∑i=1NDi∑i=1NDiSiS_g= _i=1^Np_iS_i= _i=1^N D_i _i=1^ND_iS_i, with Ug,r∈ℝdh×rU_g,r ^d_h× r containing its top r orthonormal eigenvectors. By the Rayleigh–Ritz principle, Tr(Ui,r⊤SiUi,r)Tr(U_i,r S_iU_i,r) is the maximum representation energy that a rank-r orthonormal subspace can retain for client i. In contrast, Tr(Ug,r⊤SiUg,r)Tr(U_g,r S_iU_g,r) is the energy retained for the same client by the global rank-r subspace. Their ratio in (7) therefore measures how much of client i’s locally attainable rank-r energy is preserved by the global subspace. RSS averages these ratios across clients using pii=1N\p_i\_i=1^N. Random-split calibration. A nonzero deficit may arise even when samples are randomly assigned to clients. To measure this variation, we randomly reassign the samples while preserving the client sizes Dii=1N\D_i\_i=1^N. For the b-th reassignment, we recompute the client and global subspaces SiS_i and SgS_g. Let RSSnull(b)(r)RSS_null^(b)(r) denote the resulting RSS score and define Δnull(b)(r)=1−RSSnull(b)(r). _null^(b)(r)=1-RSS_null^(b)(r). (17) The random-split calibration term is τnull(r)=Q0.95(Δnull(b)(r)b), _null(r)=Q_0.95 ( \ _null^(b)(r) \_b ), (18) where Q0.95Q_0.95 denotes the empirical 9595th percentile. Thus, τnull(r) _null(r) measures the RSS deficit that can be produced by random reassignment under the same client sizes. Bootstrap calibration. The estimated local subspaces can also vary because each client has finitely many samples. For the b-th bootstrap replicate, client i draws DiD_i samples with replacement from iD_i. Using the same global mean h¯ h, we recompute the client matrix and let Ui,r(b)U_i,r^(b) contain its top r orthonormal eigenvectors. The corresponding loss of representation energy is eboot(b)(r)=∑i=1Npi[1−Tr(Ui,r(b)⊤SiUi,r(b))Tr(Ui,r⊤SiUi,r)+ϵ]+, e_boot^(b)(r)= _i=1^Np_i [1- Tr (U_i,r^(b) S_iU_i,r^(b) )Tr (U_i,r S_iU_i,r )+ε ]_+, (19) where [z]+=maxz,0[z]_+= \z,0\. The bootstrap calibration term is τboot(r)=Q0.95(eboot(b)(r)b). _boot(r)=Q_0.95 ( \e_boot^(b)(r) \_b ). (20) Here, τboot(r) _boot(r) measures the loss caused by re-estimating the client-specific rank-r subspaces from finite samples. The final threshold in (8) is the larger of τnull(r) _null(r) and τboot(r) _boot(r). Therefore, Δ(r)>τ(r) (r)>τ(r) only when the observed RSS deficit exceeds both the random-split variation and the finite-sample subspace error. The sharing policy is then determined by (9). Algorithm 2 summarizes the computation process of RSS, whose quantities are computed before federated optimization without gradients, LoRA warm-up, or trained LoRA factors. Relation to Theorem 1. Theorem 1 shows that, under the least-squares surrogate, the preferred sharing strategy is determined by comparing the aggregate input-side and output-side projection residuals, ℰAE_A and ℰBE_B. Directly evaluating these residuals before training is infeasible because they depend on the client-specific desired adaptations Δi _i, which are unavailable at that stage. RSS therefore does not directly estimate ℰAE_A, ℰBE_B, or their difference. Instead, it evaluates whether the common rank-r input-side structure required by Share-A/Local-B is sufficient for the local data distributions, using sequence-level representations extracted from the frozen LLM backbone. When Δ(r)>τ(r) (r)>τ(r), the observed RSS deficit exceeds both the random-split variation and the finite-sample subspace error, indicating that the global rank-r input subspace is insufficient. FedAS-LoRA therefore selects Share-B/Local-A. Otherwise, the observed deficit does not exceed the calibrated threshold, and Share-A/Local-B is selected. Thus, RSS serves as a training-free sharing-side selector motivated by Theorem 1, rather than a direct estimator of the two projection residuals. Its agreement with the empirically better fixed sharing strategy is reported in Table VIII. Appendix C Proof of Theorem 1 For the Share-A/Local-B case, fix A∈ℝr×dinA ^r× d_in, and let Yi=ΔiΣi1/2Y_i= _i _i^1/2 and Xi=AΣi1/2X_i=A _i^1/2. Optimizing the client-local factor BiB_i gives the least-squares problem minBi‖Yi−BiXi‖F2 _B_i\|Y_i-B_iX_i\|_F^2, and the corresponding solution is given by Bi⋆(A) B_i (A) =YiXi⊤(XiXi⊤)†=ΔiΣiA⊤(AΣiA⊤)†. =Y_iX_i (X_iX_i ) = _i _iA (A _iA ) . (21) With the definition of Πi,A _i,A in Theorem 1, we have Bi⋆(A)Xi=YiΠi,AB_i (A)X_i=Y_i _i,A. Therefore, minBi‖(Δi−BiA)Σi1/2‖F2=‖Yi(I−Πi,A)‖F2 _B_i\|( _i-B_iA) _i^1/2\|_F^2=\|Y_i(I- _i,A)\|_F^2 =‖ΔiΣi1/2(I−Πi,A)‖F2. = \| _i _i^1/2(I- _i,A) \|_F^2. (22) Averaging over the clients and minimizing over the shared factor A yields (5). For Share-B/Local-A, fix B∈ℝdout×rB ^d_out× r and again let Yi=ΔiΣi1/2Y_i= _i _i^1/2. Since Σi≻0 _i 0, the change of variables Ci=AiΣi1/2C_i=A_i _i^1/2 is bijective. The local problem can therefore be written as minCi‖Yi−BCi‖F2 _C_i\|Y_i-BC_i\|_F^2. Taking Ci⋆=B†YiC_i =B Y_i gives Ai⋆(B)=Ci⋆Σi−1/2=B†Δi. A_i (B)=C_i _i^-1/2=B _i. (23) The fitted value satisfies BCi⋆=BB†Yi=ΠBYiBC_i =B Y_i= _BY_i, where ΠB _B is the orthogonal projector onto the column space of B. It follows that minAi‖(Δi−BAi)Σi1/2‖F2=‖(I−ΠB)Yi‖F2 _A_i\|( _i-BA_i) _i^1/2\|_F^2=\|(I- _B)Y_i\|_F^2 =‖(I−ΠB)ΔiΣi1/2‖F2. = \|(I- _B) _i _i^1/2 \|_F^2. (24) Averaging over the clients and minimizing over the shared factor B yields (6), which completes the proof. Appendix D Proof of Theorem 2 For client i at local step e of round t, let Ut,ie=Bt,ieAt,ieU_t,i^e=B_t,i^eA_t,i^e and Ut,i=Ut,i0U_t,i=U_t,i^0. We define the aggregate objective as Ft=1N∑i=1Nfi(Ut,i)F_t= 1N _i=1^Nf_i(U_t,i) and recall that C⋆=maxCA,CBC_ = \C_A,C_B\. By Assumption 2 and Jensen’s inequality, we have ‖G¯t,ie‖F≤[‖Gt,ie‖F∣Ut,ie]≤Gmax. \| G_t,i^e\|_F [\|G_t,i^e\|_F U_t,i^e ]≤ G_ . (25) During local training, both LoRA factors are updated according to Bt,ie+1=Bt,ie−ηℓGt,ieAt,ie⊤,At,ie+1=At,ie−ηℓBt,ie⊤Gt,ieB_t,i^e+1=B_t,i^e- _ G_t,i^eA_t,i^e ,A_t,i^e+1=A_t,i^e- _ B_t,i^e G_t,i^e. Thus, we have Ut,ie+1−Ut,ie U_t,i^e+1-U_t,i^e =−ηℓBt,ieBt,ie⊤Gt,ie−ηℓGt,ieAt,ie⊤At,ie =- _ B_t,i^eB_t,i^e G_t,i^e- _ G_t,i^eA_t,i^e A_t,i^e +ηℓ2Gt,ieAt,ie⊤Bt,ie⊤Gt,ie. + _ ^2G_t,i^eA_t,i^e B_t,i^e G_t,i^e. (26) Applying Assumption 1 to the increment (26) gives fi(Ut,ie+1)−fi(Ut,ie) f_i(U_t,i^e+1)-f_i(U_t,i^e) ≤⟨G¯t,ie,Ut,ie+1−Ut,ie⟩F ≤ G_t,i^e,U_t,i^e+1-U_t,i^e _F +L2‖Ut,ie+1−Ut,ie‖F2. + L2\|U_t,i^e+1-U_t,i^e\|_F^2. (27) For the first term on the RHS of (27), based on the unbiasedness of Gt,ieG_t,i^e and Assumption 3, we have −ηℓ[⟨G¯t,ie,Bt,ieBt,ie⊤Gt,ie⟩F]≤−ηℓcB‖G¯t,ie‖F2, - _ E [ G_t,i^e,B_t,i^eB_t,i^e G_t,i^e _F ]\!≤\!- _ c_B\| G_t,i^e\|_F^2, −ηℓ[⟨G¯t,ie,Gt,ieAt,ie⊤At,ie⟩F]≤−ηℓcA‖G¯t,ie‖F2. - _ E [ G_t,i^e,G_t,i^eA_t,i^e A_t,i^e _F ]\!≤\!- _ c_A\| G_t,i^e\|_F^2. (28) For the ηℓ2Gt,ieAt,ie⊤Bt,ie⊤Gt,ie _ ^2G_t,i^eA_t,i^e B_t,i^e G_t,i^e term in (26), we have ηℓ2|⟨G¯t,ie,Gt,ieAt,ie⊤Bt,ie⊤Gt,ie⟩F|≤ηℓ2‖G¯t,ie‖F‖Gt,ie‖F2 _ ^2 | G_t,i^e,G_t,i^eA_t,i^e B_t,i^e G_t,i^e _F |≤ _ ^2\| G_t,i^e\|_F\|G_t,i^e\|_F^2 ‖At,ie‖F‖Bt,ie‖F≤ηℓ2CACBGmax3. \|A_t,i^e\|_F\|B_t,i^e\|_F≤ _ ^2C_AC_BG_ ^3. (29) where the first inequality is due to the Cauchy–Schwarz inequality, and the second inequality comes from Assumptions 2 and 3. With (28), (Appendix D Proof of Theorem 2) and the Cauchy–Schwarz inequality, we obtain ‖Ut,ie+1−Ut,ie‖F2 \|U_t,i^e+1-U_t,i^e\|_F^2 ≤3ηℓ2(CA4+CB4)Gmax2 ≤ 3 _ ^2(C_A^4+C_B^4)G_ ^2 +3ηℓ4CA2CB2Gmax4. +3 _ ^4C_A^2C_B^2G_ ^4. (30) Combining the upper bounds above with (27), the second term on the RHS of (27) is bounded, as given by [fi(Ut,ie+1)−fi(Ut,ie)]≤−ηℓ(cA+cB)[‖G¯t,ie‖F2] \!\! [f_i(U_t,i^e+1)\!\!-\!\!f_i(U_t,i^e)\! ]\!≤\!- _ (c_A\!+\!c_B)E\! [\| G_t,i^e\|_F^2 ] +ηℓ2CACBGmax3+3L2ηℓ2(CA4+CB4)Gmax2 + _ ^2C_AC_BG_ ^3+ 3L2 _ ^2(C_A^4+C_B^4)G_ ^2 +3L2ηℓ4CA2CB2Gmax4, + 3L2 _ ^4C_A^2C_B^2G_ ^4, (31) Summing (31) over e=0,…,E−1e=0,…,E-1 gives [fi(Ut,iE)−fi(Ut,i)] \! [f_i(U_t,i^E)-f_i(U_t,i) ] ≤−ηℓ(cA+cB)∑e=0E−1[‖G¯t,ie‖F2]+Erloc, ≤- _ (c_A+c_B) _e=0^E-1E\! [\| G_t,i^e\|_F^2 ]+Er_loc, (32) where rloc=ηℓ2CACBGmax3+3L2ηℓ2(CA4+CB4)Gmax2+3L2ηℓ4CA2CB2Gmax4r_loc= _ ^2C_AC_BG_ ^3+ 3L2 _ ^2(C_A^4+C_B^4)G_ ^2+ 3L2 _ ^4C_A^2C_B^2G_ ^4. For the accumulated local gradients ∑e=0E−1[‖G¯t,ie‖F2] _e=0^E-1E\! [\| G_t,i^e\|_F^2 ] in (Appendix D Proof of Theorem 2), we have ∑e=0E−1[‖G¯t,ie‖F2]=[‖G¯t,i0‖F2+∑e=1E−1‖G¯t,ie‖F2] _e=0^E-1E\! [\| G_t,i^e\|_F^2 ]=E [ \| G_t,i^0 \|_F^2+ _e=1^E-1 \| G_t,i^e \|_F^2 ] ≥[‖G¯t,i0‖F2]≥κE[‖∇fi(Ut,i)‖F2], [ \| G_t,i^0 \|_F^2 ]≥κEE [ \|∇ f_i (U_t,i ) \|_F^2 ], (33) where κ∈[0,1E]κ∈[0, 1E]. Substituting (33) into the preceding bound yields [fi(Ut,iE)−fi(Ut,i)]≤ \! [f_i(U_t,i^E)-f_i(U_t,i) ]≤ Erloc−ηℓ(cA+cB)κE[‖∇fi(Ut,i)‖F2],i∈t. Er_loc\!-\! _ (c_A\!+\!c_B)κ EE\! [\|∇ f_i(U_t,i)\|_F^2 ],\ i _t. (34) We next bound the changes in the shared factor during local training and server aggregation. Telescoping the corresponding local factor updates gives ‖At,iE−At‖F \|A_t,i^E-A_t\|_F ≤ηℓECBGmax,for Share-A/Local-B, ≤ _ EC_BG_ ,\ for Share-A/Local-B, ‖Bt,iE−Bt‖F \|B_t,i^E-B_t\|_F ≤ηℓECAGmax,for Share-B/Local-A. ≤ _ EC_AG_ ,\ for Share-B/Local-A. (35) where both inequalities come from the triangle inequality, Assumptions 2 and 3. Consequently, either choice of the shared factor Q satisfies ‖Qt,iE−Qt‖F≤ηℓEC⋆Gmax,i∈t. \|Q_t,i^E-Q_t\|_F≤ _ EC_ G_ ,\ i _t. (36) Based on the server update in (3), we obtain ‖Qt+1−Qt‖F \|Q_t+1\!-\!Q_t\|_F =‖ηgSt∑k∈t(Qt,kE−Qt)‖F≤ηgηℓEC⋆Gmax. \!=\! \| _gS_t\! _k _t\!(Q_t,k^E\!-\!Q_t)\! \|_F\!\!≤\!\! _g _ EC_ G_ . (37) where the inequality follows from the triangle inequality and the upper bound of ‖Qt,kE−Qt‖F\|Q_t,k^E-Q_t\|_F in (36). After server aggregation, let Ut+1,iU_t+1,i denote the product formed from Qt+1Q_t+1 and client i’s retained local factor. For i∈ti _t, Ut+1,iU_t+1,i and Ut,iEU_t,i^E have the same local factor and differ only in Qt+1Q_t+1 and Qt,iEQ_t,i^E. For j∉tj _t, Ut+1,jU_t+1,j and Ut,jU_t,j have the same local factor and differ only in Qt+1Q_t+1 and QtQ_t. Based on Frobenius-norm submultiplicativity, we obtain ‖Ut+1,i−Ut,iE‖F≤Dsel:=(1+ηg)ηℓEC⋆2Gmax,i∈t, \|U_t+1,i\!-\!U_t,i^E\|_F\!≤\!D_sel:=\!(1\!+\! _g) _ EC_ ^2G_ ,\ i _t, (38) ‖Ut+1,j−Ut,j‖F≤Duns:=ηgηℓEC⋆2Gmax,j∉t. \|U_t+1,j\!-\!U_t,j\|_F\!≤\!D_uns:=\! _g _ EC_ ^2G_ ,\ j _t. (39) where the selected-client bound uses ‖Qt+1−Qt,iE‖F≤‖Qt+1−Qt‖F+‖Qt,iE−Qt‖F\|Q_t+1-Q_t,i^E\|_F≤\|Q_t+1-Q_t\|_F+\|Q_t,i^E-Q_t\|_F, whereas the unselected-client bound contains only ‖Qt+1−Qt‖F\|Q_t+1-Q_t\|_F. For a selected client i∈ti ^t, applying Assumption 1 between Ut,iEU_t,i^E and Ut+1,iU_t+1,i gives [fi(Ut+1,i)−fi(Ut,iE)]≤[⟨∇fi(Ut,iE),Ut+1,i−Ut,iE⟩F] \! [f_i(U_t+1,i)-f_i(U_t,i^E) ]\!≤\!E\! [ ∇ f_i(U_t,i^E),U_t+1,i-U_t,i^E _F ] +L2[‖Ut+1,i−Ut,iE‖F2]≤ηℓ2Gmax2+(12ηℓ+L2)Dsel2, \!+\! L2E\! [\|U_t+1,i-U_t,i^E\|_F^2\! ]\!≤\! _ 2G_ ^2+ ( 12 _ + L2 )D_sel^2, (40) where the second inequality follows from ⟨X,Y⟩F≤ηℓ2‖X‖F2+12ηℓ‖Y‖F2 X,Y _F≤ _ 2\|X\|_F^2+ 12 _ \|Y\|_F^2, Assumption 2, and (38). Combining (40) with (34) yields [fi(Ut+1,i)−fi(Ut,i)] \! [f_i(U_t+1,i)-f_i(U_t,i) ] ≤−ηℓ(cA+cB)κE[‖∇fi(Ut,i)‖F2]+ℛsel, ≤- _ (c_A+c_B)κ EE\! [\|∇ f_i(U_t,i)\|_F^2 ]+R_sel, (41) where ℛsel=Eηℓ2CACBGmax3+3LE2ηℓ2(CA4+CB4)Gmax2+3LE2ηℓ4CA2CB2Gmax4+ηℓ2Gmax2+(12ηℓ+L2)(1+ηg)2ηℓ2E2C⋆4Gmax2R_sel=E _ ^2C_AC_BG_ ^3+ 3LE2 _ ^2(C_A^4+C_B^4)G_ ^2+ 3LE2 _ ^4C_A^2C_B^2G_ ^4\!+\! _ 2G_ ^2\!+\! ( 12 _ \!+\! L2 )(1\!+\! _g)^2 _ ^2E^2C_ ^4G_ ^2. For an unselected client j∉tj ^t, although it performs no local update, its shared factor changes after aggregation: [fj(Ut+1,j)−fj(Ut,j)]≤[⟨∇fj(Ut,j),Ut+1,j−Ut,j⟩F] \! [f_j(U_t+1,j)-f_j(U_t,j) ] \! [ ∇ f_j(U_t,j),U_t+1,j-U_t,j _F ] +L2[‖Ut+1,j−Ut,j‖F2]≤ρηℓ2[‖∇fj(Ut,j)‖F2]+ℛuns, +\!\! L2E\! [\|U_t+1,j\!-\!U_t,j\|_F^2 ]\!\!≤\!\! ρ _ 2E\! [\!\|∇ f_j(U_t,j)\|_F^2\! ]\!\!+\!R_uns, (42) where ℛuns:=(12ρηℓ+L2)ηg2ηℓ2E2C⋆4Gmax2R_uns:= ( 12ρ _ + L2 ) _g^2 _ ^2E^2C_ ^4G_ ^2. ρ>0ρ>0 is the parameter in ⟨X,Y⟩F≤ρηℓ2‖X‖F2+12ρηℓ‖Y‖F2 X,Y _F≤ ρ _ 2\|X\|_F^2+ 12ρ _ \|Y\|_F^2, and the last inequality uses (39). Averaging (41) and (42) over all clients i∈i yields [Ft+1−Ft]≤StNℛsel+(1−StN)ℛuns [F_t+1-F_t]≤ S_tNR_sel+ (1- S_tN )R_uns −ηℓ(cA+cB)κE[tsel]+ρηℓ2[tuns], - _ (c_A+c_B)κ EE\! [G_t^sel ]+ ρ _ 2E\! [G_t^uns ], (43) where tsel:=1N∑i∈t‖∇fi(Ut,i)‖F2G_t^sel:= 1N _i _t\|∇ f_i(U_t,i)\|_F^2, and tuns:=1N∑i∉t‖∇fi(Ut,i)‖F2G_t^uns:= 1N _i _t\|∇ f_i(U_t,i)\|_F^2. For t:=tsel+tuns=1N∑i=1N‖∇fi(Ut,i)‖F2G_t:=G_t^sel+G_t^uns= 1N _i=1^N\|∇ f_i(U_t,i)\|_F^2, when t=0G_t=0, the terms −ηℓ(cA+cB)κE[tsel]+ρηℓ2[tuns]- _ (c_A+c_B)κ EE\! [G_t^sel ]+ ρ _ 2E\! [G_t^uns ] in (43) vanish. Otherwise, we have −ηℓ(cA+cB)κE[tsel]+ρηℓ2[tuns] - _ (c_A+c_B)κ EE\! [G_t^sel ]+ ρ _ 2E\! [G_t^uns ] (44) =−ηℓ[((cA+cB)κEλ−ρ2(1−λ))t]≤−ηℓc[t], =- _ E\! [ ((c_A\!+\!c_B)κ Eλ\!-\! ρ2(1\!-\!λ) )\!G_t ]\!≤\!- _ c\,E\! [G_t ], where c:=(cA+cB)κEλ−ρ2(1−λ)>0c:=(c_A+c_B)κ Eλ- ρ2(1-λ)>0 with ρ<2(cA+cB)κEλ(1−λ)ρ< 2(c_A+c_B)κ Eλ(1-λ), and λ should satisfy the gradient-mass coverage condition, as given by 0<λ≤tselt,∀t. 0<λ≤ G_t^selG_t,∀ t. (45) Here, (45) requires the selected clients to account for at least a fixed fraction λ of the total squared gradient mass in every round with t>0G_t>0. Equivalently, Gtsel≥λtG_t^sel≥ _t and Gtuns≤(1−λ)tG_t^uns≤(1-λ)G_t. This condition is imposed on the realized gradient mass rather than on the number or sampling distribution of the selected clients. Hence, it allows arbitrary client participation, provided that the selected subset does not capture an arbitrarily small fraction of the current gradient mass. Under full participation, one can set λ=1λ=1. Under partial participation, any round-independent lower bound λ>0λ>0 satisfying (45) is sufficient. Together with c>0c>0, this condition ensures that the descent contributed by the selected clients dominates the possible objective increase of the unselected clients caused by the shared-factor update. Substituting (44) into (43) and rearranging gives ηℓc[t]≤[Ft−Ft+1]+StNℛsel+(1−StN)ℛuns. _ c\,E\! [G_t ]\!≤\!E[F_t\!-\!F_t+1]\!+\! S_tNR_sel\!+\! (1- S_tN )R_uns. (46) Summing (46) over t=0,…,T−1t=0,…,T-1 and dividing by ηlcT _lcT, we obtain 1NT∑t=0T−1∑i=1N[‖∇fi(Ut,i)‖F2]≤F0−F⋆ηℓcT 1NT _t=0^T-1 _i=1^NE\! [\|∇ f_i(U_t,i)\|_F^2 ]≤ F_0-F_ _ cT +1ηℓcT∑t=0T−1StNℛsel+1ηℓcT∑t=0T−1(1−StN)ℛuns, +\! 1 _ cT _t=0^T-1 S_tNR_sel\!+\! 1 _ cT _t=0^T-1 (1\!-\! S_tN )R_uns, (47) which is exactly (14) and completes the proof. Figure 3: Alignment Between RSS Decisions and Training-Loss Preference Appendix E Full Experiment Results Hyperparameters Tables V and VI show the learning rates used for LoRA-based methods and rsLoRA-based methods, respectively. For the VeRA-based methods, we chose the AdamW optimizer and used separate learning rates for the classification head and the adapted layers as used in VeRA. The learning rates used for VeRA-based methods are shown in Table VII. Method MNLI-m MNLI-m SST-2 QNLI QQP RTE LoRA 1E-2 1E-2 2E-2 1E-2 1E-2 1E-2 FFA-LoRA 5E-2 5E-2 5E-2 2E-2 5E-2 2E-2 FedDPA-LoRA 1E-2 1E-2 1E-2 5E-2 5E-2 1E-2 FedSA-LoRA 2E-2 2E-2 1E-2 5E-3 2E-2 1E-2 FedAS-LoRA 2E-2 2E-2 1E-2 5E-3 2E-2 1E-2 TABLE V: Learning rates used for LoRA-based methods on the GLUE benchmark. Method MNLI-m MNLI-m SST-2 QNLI QQP RTE rsLoRA 5E-3 5E-3 1E-2 2E-3 5E-3 2E-3 FFA-rsLoRA 2E-2 2E-2 2E-2 1E-2 2E-2 1E-2 FedDPA-rsLoRA 5E-3 5E-3 1E-2 1E-3 5E-3 1E-2 FedSA-rsLoRA 5E-3 5E-3 5E-3 1E-3 2E-3 2E-3 FedAS-rsLoRA 5E-3 5E-3 5E-3 1E-3 2E-3 2E-3 TABLE VI: Learning rates used for rsLoRA-based methods on the GLUE benchmark. Method Position MNLI- m MNLI- m SST-2 QNLI QQP RTE VeRA VeRA 1E-2 1E-2 2E-2 2E-3 2E-3 1E-2 Head 6E-3 6E-3 2E-3 3E-4 3E-4 2E-4 FFA-VeRA VeRA 2E-2 2E-2 1E-2 1E-2 1E-2 1E-2 Head 2E-3 2E-3 6E-3 2E-4 6E-3 2E-4 FedDPA-VeRA VeRA 1E-2 1E-2 1E-2 2E-3 2E-2 1E-2 Head 6E-3 6E-3 6E-3 3E-4 2E-3 2E-4 FedSA-VeRA VeRA 2E-3 2E-3 1E-2 1E-2 2E-3 1E-2 Head 3E-5 3E-5 3E-4 3E-4 3E-4 1E-4 FedAS-VeRA VeRA 2E-3 2E-3 1E-2 1E-2 2E-3 1E-2 Head 3E-5 3E-5 3E-4 3E-4 3E-4 1E-4 TABLE VII: Learning rates used for VeRA-based methods on the GLUE benchmark. Evaluation of RSS-Based Sharing-Side Selection Alignment with Test-Accuracy Preference. Table VIII provides the full comparison between the sharing-side decisions made by RSS and the empirical test accuracies of Share-A/Local-B and Share-B/Local-A. The experiments cover six evaluation sets, multiple client partitions, and LoRA ranks r∈2,4,8,16r∈\2,4,8,16\. For each setting, RSS is computed before federated training, while the empirical winner is determined by comparing the test accuracies obtained at the best validation rounds of the two sharing strategies. Across the 38 evaluated settings, the RSS decision agrees with the empirically better sharing strategy in all 33 cases with a strict accuracy difference. The remaining five cases are marked as empirical ties in Table VIII. The preferred sharing side also changes with the data partition and LoRA rank. For example, QNLI and MNLI favor Share-A/Local-B under the evaluated IID settings, whereas Share-B/Local-A is preferred in most of their Dirichlet settings. For MNLI under Dirichlet α=1α=1, the preferred strategy further changes from Share-B/Local-A at r=4r=4 to Share-A/Local-B at r=8r=8. Similarly, SST-2 favors Share-A/Local-B under IID partitions but mainly favors Share-B/Local-A under the non-IID partitions. These results support the use of a partition- and rank-aware selection rule instead of a fixed sharing policy. RSS should nevertheless be interpreted as a sharing-side selector rather than an estimator of the final accuracy gap between the two strategies. Alignment with the training objective. Table VIII compares the RSS decision with the final test-accuracy preference of the two fixed sharing strategies. We further examine whether the decision direction of RSS agrees with their relative behavior under the actual training objective. For visual illustration, we randomly sample several evaluated settings and compare their RSS decision margins with the corresponding training losses of Share-A/Local-B and Share-B/Local-A. For rank r, we define the relative RSS decision margin as mRSS(r)=τ(r)−Δ(r)τ(r)+Δ(r). m_RSS(r)= τ(r)- (r)τ(r)+ (r). (48) Since RSS selects Share-A/Local-B when Δ(r)≤τ(r) (r)≤τ(r), a positive mRSS(r)m_RSS(r) favors Share-A/Local-B, whereas a negative value favors Share-B/Local-A. We also define the relative training-loss advantage as mloss=LB−LALB+LA, m_loss= L_B-L_AL_B+L_A, (49) where LAL_A and LBL_B denote the training losses obtained by Share-A/Local-B and Share-B/Local-A, respectively. Accordingly, mloss>0m_loss>0 means that Share-A/Local-B attains a lower training loss, while mloss<0m_loss<0 means that Share-B/Local-A attains a lower training loss. As shown in Fig. 3, all sampled cases lie in either the upper-right or lower-left quadrant. Thus, the sharing side selected by RSS also attains the lower training loss in these cases. This result provides additional evidence that the RSS decision is consistent with the optimization behavior of the two sharing strategies under the actual task objective. This directional agreement provides empirical support for the residual-based selection principle in Theorem 1. The theorem shows that the preferred sharing side is determined by which shared-subspace constraint yields the smaller aggregate projection residual. RSS translates this principle into a training-free decision rule by assessing whether the common rank-r input-side subspace required by Share-A/Local-B is sufficient for the client representations. When this subspace is sufficient, RSS favors Share-A/Local-B; otherwise, it favors Share-B/Local-A. The consistent training-loss preference therefore indicates that RSS identifies the sharing constraint that better fits the client data under the actual task objective. This result supports RSS as a practical proxy for the residual-based sharing-side preference characterized in Theorem VIII. Dataset Split r RSS metric Accuracy (%) Alignment Δ(r) (r) τ(r)τ(r) Decision Share-A Share-B Winner QNLI IID 4 2.46413×10−52.46413\!×\!10^-5 3.91172×10−53.91172\!×\!10^-5 Share-A 92.26 90.18 Share-A Match 8 4.28222×10−54.28222\!×\!10^-5 7.07299×10−57.07299\!×\!10^-5 Share-A 91.12 90.66 Share-A Match Dirichlet α=0.5α=0.5 2 3.01960×10−53.01960\!×\!10^-5 3.46345×10−53.46345\!×\!10^-5 Share-A 89.46 88.84 Share-A Match 4 5.89316×10−55.89316\!×\!10^-5 3.80830×10−53.80830\!×\!10^-5 Share-B 92.01 92.80 Share-B Match 8 7.37366×10−57.37366\!×\!10^-5 6.79052×10−56.79052\!×\!10^-5 Share-B 91.13 92.94 Share-B Match 16 1.33945×10−41.33945\!×\!10^-4 1.31365×10−41.31365\!×\!10^-4 Share-B 89.12 89.56 Share-B Match Dirichlet α=1α=1 4 5.90747×10−55.90747\!×\!10^-5 3.92434×10−53.92434\!×\!10^-5 Share-B 93.28 94.69 Share-B Match 8 8.31570×10−58.31570\!×\!10^-5 6.96169×10−56.96169\!×\!10^-5 Share-B 90.89 91.58 Share-B Match QQP Dirichlet α=0.5α=0.5 4 2.11905×10−52.11905\!×\!10^-5 1.28376×10−51.28376\!×\!10^-5 Share-B 85.68 86.70 Share-B Match 8 4.14507×10−54.14507\!×\!10^-5 1.67622×10−51.67622\!×\!10^-5 Share-B 86.87 87.95 Share-B Match SST-2 IID 4 3.89196×10−43.89196\!×\!10^-4 5.07871×10−45.07871\!×\!10^-4 Share-A 96.65 96.14 Share-A Match 8 4.35904×10−44.35904\!×\!10^-4 6.02930×10−46.02930\!×\!10^-4 Share-A 96.10 95.87 Share-A Match Dirichlet α=0.5α=0.5 2 1.45398×10−31.45398\!×\!10^-3 5.83336×10−45.83336\!×\!10^-4 Share-B 96.56 96.79 Share-B Match 4 1.84450×10−31.84450\!×\!10^-3 5.25775×10−45.25775\!×\!10^-4 Share-B 96.33 96.33 Tie Actual tie 8 2.20251×10−32.20251\!×\!10^-3 6.38468×10−46.38468\!×\!10^-4 Share-B 95.72 97.17 Share-B Match 16 3.63222×10−33.63222\!×\!10^-3 1.01451×10−31.01451\!×\!10^-3 Share-B 95.73 95.72 Tie Actual tie Dirichlet α=1α=1 4 1.18446×10−31.18446\!×\!10^-3 5.79445×10−45.79445\!×\!10^-4 Share-B 97.02 97.02 Tie Actual tie 8 1.43308×10−31.43308\!×\!10^-3 6.62178×10−46.62178\!×\!10^-4 Share-B 96.56 96.79 Share-B Match MNLI-m IID 4 5.98420×10−65.98420\!×\!10^-6 1.02906×10−51.02906\!×\!10^-5 Share-A 87.06 86.11 Share-A Match 8 2.91890×10−52.91890\!×\!10^-5 4.13822×10−54.13822\!×\!10^-5 Share-A 89.43 89.06 Share-A Match Dirichlet α=0.5α=0.5 2 1.03136×10−41.03136\!×\!10^-4 1.11881×10−51.11881\!×\!10^-5 Share-B 89.60 89.63 Share-B Match 4 4.62360×10−54.62360\!×\!10^-5 9.96230×10−69.96230\!×\!10^-6 Share-B 90.04 90.28 Share-B Match 8 9.33286×10−59.33286\!×\!10^-5 3.98546×10−53.98546\!×\!10^-5 Share-B 89.75 89.95 Share-B Match 16 8.07826×10−58.07826\!×\!10^-5 3.45278×10−53.45278\!×\!10^-5 Share-B 89.04 89.73 Share-B Match Dirichlet α=1α=1 4 1.37597×10−51.37597\!×\!10^-5 1.02455×10−51.02455\!×\!10^-5 Share-B 89.50 89.62 Share-B Match 8 3.26219×10−53.26219\!×\!10^-5 3.80408×10−53.80408\!×\!10^-5 Share-A 89.02 88.37 Share-A Match MNLI-m IID 4 5.98420×10−65.98420\!×\!10^-6 1.02906×10−51.02906\!×\!10^-5 Share-A 86.68 85.23 Share-A Match 8 2.91890×10−52.91890\!×\!10^-5 4.13822×10−54.13822\!×\!10^-5 Share-A 88.07 87.91 Share-A Match Dirichlet α=0.5α=0.5 4 4.62360×10−54.62360\!×\!10^-5 9.96230×10−69.96230\!×\!10^-6 Share-B 88.47 89.97 Share-B Match 8 9.33286×10−59.33286\!×\!10^-5 3.98546×10−53.98546\!×\!10^-5 Share-B 87.82 88.86 Share-B Match Dirichlet α=1α=1 4 1.37597×10−51.37597\!×\!10^-5 1.02455×10−51.02455\!×\!10^-5 Share-B 87.14 88.79 Share-B Match 8 3.26219×10−53.26219\!×\!10^-5 3.80408×10−53.80408\!×\!10^-5 Share-A 90.44 89.97 Share-A Match RTE IID 4 2.07749×10−32.07749\!×\!10^-3 3.52261×10−33.52261\!×\!10^-3 Share-A 88.14 85.82 Share-A Match 8 3.99091×10−33.99091\!×\!10^-3 5.98761×10−35.98761\!×\!10^-3 Share-A 87.50 87.50 Tie Actual tie Dirichlet α=0.5α=0.5 4 1.81187×10−31.81187\!×\!10^-3 3.83366×10−33.83366\!×\!10^-3 Share-A 88.71 88.71 Tie Actual tie 8 3.63808×10−33.63808\!×\!10^-3 5.98369×10−35.98369\!×\!10^-3 Share-A 87.77 86.34 Share-A Match Dirichlet α=1α=1 4 1.82618×10−31.82618\!×\!10^-3 3.55810×10−33.55810\!×\!10^-3 Share-A 87.19 86.47 Share-A Match 8 3.45208×10−33.45208\!×\!10^-3 5.75546×10−35.75546\!×\!10^-3 Share-A 86.26 85.35 Share-A Match TABLE VIII: RSS metric values, threshold decisions, and Share-A/Local-B versus Share-B/Local-A accuracies under the evaluated three-client settings. Scalability Analysis under Uniform and Non-Uniform Client Sampling To further evaluate scalability, we increase the number of clients to N=50N=50 under uniform and non-uniform sampling. As shown in Table IX, FedAS-LoRA achieves the highest accuracy on QNLI, SST-2, and MNLI-m under both sampling settings. Compared with FedSA-LoRA, its average accuracy is higher by 0.85 and 0.76 percentage points under uniform and non-uniform sampling, respectively. These results show that selecting the sharing side remains beneficial with a larger client population and uneven participation probabilities. Method Uniform Non-uniform LoRA 87.27 ± 0.48 83.24 ± 0.72 FFA-LoRA 86.18 ± 0.45 84.90 ± 0.85 FedDPA-LoRA 87.50 ± 0.41 84.29 ± 0.64 FedSA-LoRA 88.12 ± 0.55 85.68 ± 0.44 FedAS-LoRA (Ours) 89.37 ± 0.37 86.80 ± 0.56 LoRA 93.42 ± 0.67 92.80 ± 0.77 FFA-LoRA 93.61 ± 0.71 93.51 ± 0.78 FedDPA-LoRA 94.87 ± 0.52 94.70 ± 0.89 FedSA-LoRA 95.01 ± 0.58 96.48 ± 0.77 FedAS-LoRA (Ours) 95.64 ± 0.73 96.89 ± 0.79 LoRA 83.91 ± 0.53 81.25 ± 0.60 FFA-LoRA 84.23 ± 0.56 83.93 ± 0.99 FedDPA-LoRA 85.07 ± 0.80 83.19 ± 0.95 FedSA-LoRA 86.89 ± 0.83 84.28 ± 0.99 FedAS-LoRA (Ours) 87.55 ± 0.91 85.04 ± 0.84 TABLE IX: Performance on the QNLI, SST-2, and MNLI-m tasks under uniform and non-uniform client sampling with N=50N=50. From top to bottom, the three blocks correspond to QNLI, SST-2, and MNLI-m, respectively. Results under Structured Input Skew We further evaluate the compared methods under two structured input-skew partitions while controlling the client label distributions. For SST-2, we construct a label-balanced input-length-skew partition. Within each sentiment label, the samples are ordered according to their input lengths and divided into three non-overlapping groups of approximately equal size. Each client receives the corresponding length group from both sentiment labels. This construction keeps the positive and negative label proportions similar across clients, while making the client input-length distributions different. For MNLI, we construct a genre-based partition using the genre annotations provided by MultiNLI Williams et al. [2018]. The three clients contain premise–hypothesis pairs from the telephone, government, and fiction genres, respectively. We balance the entailment, neutral, and contradiction labels across the clients so that the main source of heterogeneity is the textual genre rather than the label distribution. We report performance on the MNLI matched evaluation set, denoted as MNLI-m. As shown in Table X, FedAS-LoRA achieves the highest accuracy in all four evaluated settings. On SST-2, it obtains accuracies of 96.47%96.47\% and 96.56%96.56\% at r=4r=4 and r=8r=8, respectively, exceeding LoRA by 0.60 and 1.38 percentage points. On MNLI-m, FedAS-LoRA reaches 87.43%87.43\% at r=4r=4 and 86.53%86.53\% at r=8r=8, improving over LoRA by 0.80 and 0.12 percentage points. It also exceeds FFA-LoRA by 1.99 and 3.16 percentage points, and FedSA-LoRA by 0.36 and 0.34 percentage points at r=4r=4 and r=8r=8, respectively. These results show that selecting the sharing side remains beneficial when clients mainly differ in input characteristics rather than label proportions. Rank Method SST-2 MNLI-m r=4r=4 LoRA 95.87 ± 0.59 86.63 ± 0.21 FFA-LoRA 95.56 ± 0.39 85.44 ± 0.42 FedSA-LoRA 92.98 ± 0.45 87.07 ± 0.11 FedAS-LoRA (Ours) 96.47 ± 0.19 87.43 ± 0.20 r=8r=8 LoRA 95.18 ± 0.48 86.41 ± 0.29 FFA-LoRA 93.35 ± 0.52 83.37 ± 0.10 FedSA-LoRA 94.04 ± 0.44 86.19 ± 0.11 FedAS-LoRA (Ours) 96.56 ± 0.04 86.53 ± 0.34 TABLE X: Test accuracy under structured input-skew partitions with different LoRA ranks r. SST-2 uses a label-balanced input-length-skew partition, while MNLI-m uses a label-balanced genre partition with telephone, government, and fiction clients. Results on GSM8K with Llama 3 8B The results on the GSM8K dataset are shown in Table XI, demonstrating that the proposed FedAS-LoRA outperforms other methods in complex natural language generation tasks. From the given example, it can be seen that both LoRA and FFA-LoRA have reasoning errors, but FedAS-LoRA can reason accurately, demonstrating the superiority of the proposed method. LoRA FFA-LoRA FedAS-LoRA Accuracy 55.14 ± 0.25 54.51 ± 0.32 56.28 ± 0.18 Question: Every day, Wendi feeds each of her chickens three cups of mixed chicken feed, containing seeds, mealworms and vegetables to help keep them healthy. She gives the chickens their feed in three separate meals. In the morning, she gives her flock of chickens 15 cups of feed. In the afternoon, she gives her chickens another 25 cups of feed. How many cups of feed does she need to give her chickens in the final meal of the day if the size of Wendi’s flock is 20 chickens? Output: Wendi gives her chickens 15 cups of feed in the morning and 25 cups of feed in the afternoon, for a total of 15+25=4015+25=40 cups of feed. She has 20 chickens, so each chicken needs 40/20=240/20=2 cups of feed in the final meal of the day. The answer is 2. [✗] Wendi feeds her chickens 15 cups of feed in the morning and 25 cups of feed in the afternoon. So she has already given them 15+25=4015+25=40 cups of feed. She has 20 chickens. So she needs to give them 40/20=240/20=2 cups of feed each. The answer is 2. [✗] Wendi has 15 cups of feed in the morning and 25 cups in the afternoon. So she has 15+25=4015+25=40 cups of feed. She has 20 chickens and she needs to give each chicken 3 cups of feed. So she needs 20×3=6020× 3=60 cups of feed. She has 40 cups of feed and she needs 60 cups of feed. So she needs 60−40=2060-40=20 more cups of feed. The answer is 20. [✓] TABLE XI: Performance of different methods on the GSM8K dataset and example answers generated by each method. Method Trainable Parm. Per-round Communicated Parm. Per-round Computation Cost Communication Round (MNLI-m / SST-2) MNLI-m SST-2 MNLI-m SST-2 LoRA 1.839M / 1.838M 0.786M 14.12s 6.24s 186 210 FFA-LoRA 1.446M / 1.445M 0.393M 13.94s 6.04s 477 467 FedDPA-LoRA 2.626M / 2.625M 0.786M 19.46s 11.56s 454 398 FedSA-LoRA 1.839M / 1.838M 0.393M 14.14s 6.25s 495 453 FedAS-LoRA 1.839M / 1.838M 0.393M 14.18s 6.26s 472 384 TABLE XII: Time and space costs for each method on the MNLI-m and SST-2 tasks. Communication Round denotes the communication-round index at which the best validation performance is obtained. Communication Cost Table XII compares the trainable parameter count, per-round communicated model size, per-round computation time, and the communication-round index of the best validation checkpoint. Share-A/Local-B and Share-B/Local-A train both LoRA factors and therefore retain the same number of trainable parameters as LoRA, i.e., 1.839M on MNLI-m and 1.838M on SST-2. However, each strategy communicates only the selected shared factor. Their per-round communicated model size is therefore 0.393M parameters, which is 50% lower than the 0.786M parameters communicated by LoRA and FedDPA-LoRA. The reduction in per-round communication does not introduce substantial additional local computation. On MNLI-m, the per-round computation times of Share-A/Local-B and Share-B/Local-A are 14.14s and 14.18s, respectively, compared with 14.12s for LoRA. On SST-2, the corresponding times are 6.25s and 6.26s, compared with 6.24s for LoRA. FFA-LoRA has a slightly lower per-round computation time because it freezes one factor, whereas the two sharing strategies keep both factors trainable. FedDPA-LoRA incurs a higher parameter and computation cost because it maintains additional global and personalized LoRA modules. These results show that factor-wise sharing reduces the communicated model size per round while preserving the training capacity of both LoRA factors. Relative Heterogeneity Amplification of Local LoRA Changes To further examine how client data heterogeneity affects the two LoRA factors, we compare the client-specific changes of A and B after local training on MNLI. We consider three non-IID partitioning schemes: Dirichlet distributions with α=1α=1 and α=0.5α=0.5, and the label-balanced MNLI-genre partition. For each scheme, we independently generate five three-client partitions using different partition seeds. Each non-IID partition is paired with an IID partition generated using the same seed. The model initialization and all training-related random seeds are fixed across the partition instances. Therefore, the variation across the five instances is caused by the client data partitions rather than by model initialization or stochastic training. For this analysis, the clients perform local training independently without server aggregation. We measure the change in each LoRA factor relative to its common initialization as ΔQi Q_i. We analyze these changes rather than the terminal factor values to exclude the initial factor values from the interpretation. This is particularly important for factor A, which is randomly initialized with nonzero values under the standard LoRA initialization, whereas factor B is initialized to zero. Using ΔQi Q_i therefore places both factors on the same reference and focuses the analysis on the changes learned from each client’s local data. For a given partition and factor Q, we quantify the cross-client disagreement using the mean pairwise Frobenius distance DQ=2N(N−1)∑1≤i<j≤N‖ΔQi−ΔQj‖F.D_Q= 2N(N-1) _1≤ i<j≤ N \| Q_i- Q_j \|_F. (50) For each non-IID partition instance, the relative heterogeneity amplification is defined with respect to its seed-matched IID reference as RHA=DAnon-IID/DAIIDDBnon-IID/DBIID.RHA= D_A^non -IID/D_A^IIDD_B^non -IID/D_B^IID. (51) An RHA value greater than one means that changing from the paired IID partition to the non-IID partition produces a larger proportional increase in the cross-client disagreement of ΔA A than in that of ΔB B. Conversely, an RHA value below one indicates a larger relative increase in the disagreement of ΔB B. Since RHA compares the non-IID-to-IID amplification of each factor, RHA>1RHA>1 does not necessarily imply that DA>DBD_A>D_B in absolute value. As shown in Fig. 4(a), the geometric-mean RHA is above one under all three non-IID partitioning schemes. Relative to their seed-matched IID references, the evaluated non-IID partitions therefore amplify the cross-client disagreement of ΔA A more strongly than that of ΔB B on average. Among the three settings, the MNLI-genre partition yields the largest overall RHA, and the five individual partition instances all remain above the RHA=1RHA=1 reference line. This result indicates a relatively stable amplification of ΔA A disagreement when the clients are separated by textual genre. The two Dirichlet settings show greater variation across partition instances. Although their geometric-mean RHA values are above one, each setting includes an individual instance with an RHA below one. Hence, the Dirichlet concentration parameter alone does not determine whether client heterogeneity increases the relative disagreement of ΔA A or ΔB B more strongly. Even under the same α, the result depends on the realized client partition. A fixed sharing policy based only on the nominal heterogeneity level is therefore insufficient, which supports the partition-aware design of RSS. Figs. 4(b) and 4(c) further show that the relative amplification varies across Transformer layers and projection types. For the query projection, the mean log RHA is positive in most layers under the three partitioning schemes. The positive values are particularly evident in several middle and later layers under the MNLI-genre partition, indicating stronger relative disagreement in ΔA A at these layers. The value projection exhibits a less uniform pattern. While many early and middle layers have positive mean log RHA, several later layers under the Dirichlet partitions have negative values, indicating stronger relative disagreement in ΔB B. These mixed layer-wise patterns show that an average factor-level trend does not fully characterize the behavior of all adapted layers and projections. This further supports evaluating shared-subspace sufficiency from the representations of the actual client partition, as done by RSS. Overall, the relative disagreement between the two LoRA factors depends on the realized client partition and varies across Transformer layers and projection types. Therefore, neither the Dirichlet concentration parameter nor an average factor-level trend is sufficient to prescribe a fixed sharing side across federated settings. RSS addresses this issue by assessing, for the current client partition and target LoRA rank r, whether a shared rank-r input subspace is sufficient for the local data distributions. These results support selecting the sharing side according to the observed client partition rather than applying a fixed factor-sharing policy. (a) Overall relative heterogeneity amplification. (b) Layer-wise RHA for the query projection. (c) Layer-wise RHA for the value projection. Figure 4: Relative heterogeneity amplification of LoRA factors across MNLI client partitions. The clients perform local training independently without server aggregation. For each of the three non-IID partitioning schemes, five three-client partitions are independently generated using different partition seeds. Each non-IID partition is paired with an IID reference partition generated using the same seed, while the model initialization and training-related random seeds are fixed across all partition instances. (a) Bars show the geometric mean across the five partition instances, white circles denote individual partition instances, and error bars represent one sample standard deviation in log-RHA space. The dashed line marks RHA=1RHA=1. (b,c) Layer-wise mean log RHA for the query and value projections, respectively. Both heatmaps use the same color scale; positive values indicate stronger relative disagreement in ΔA A, whereas negative values indicate stronger relative disagreement in ΔB B.