Paper deep dive
Wireless Federated Multi-Task LLM Fine-Tuning via Sparse-and-Orthogonal LoRA
Nuocheng Yang, Sihua Wang, Ouwen Huan, Mingzhe Chen, Tony Q. S. Quek, Changchuan Yin
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 95%
Last extracted: 7/20/2026, 2:39:17 PM
Summary
The paper proposes a decentralized federated learning (DFL) framework for fine-tuning large language models (LLMs) using a novel Sparse-and-Orthogonal Low-Rank Adaptation (LoRA) method. It addresses three key issues in multi-task DFL: catastrophic knowledge forgetting, inefficient communication, and multi-task knowledge interference. The solution includes a sparse-and-orthogonal LoRA to ensure update orthogonality, a cluster-based device connection topology to optimize communication, and an implicit mixture of experts (MoE) mechanism to isolate task-specific knowledge during inference. Simulations show a 73% reduction in communication resources and a 5% performance improvement over traditional LoRA.
Entities (8)
Relation Signals (6)
Decentralized Federated Learning → enables → Large Language Model
confidence 98% · Decentralized federated learning (DFL) based on low-rank adaptation (LoRA) enables mobile devices with multi-task datasets to collaboratively fine-tune a large language model
Sparse-and-Orthogonal LoRA → addresses → Catastrophic Knowledge Forgetting
confidence 95% · Sparse-and-Orthogonal LoRA that ensures orthogonality between model updates to eliminate direction conflicts during fine-tuning
Sparse-and-Orthogonal LoRA → ispartof → Decentralized Federated Learning
confidence 95% · we develop a collaborative LLM fine-tuning framework in DFL by integrating a sparse-and-orthogonal LoRA method
Implicit Mixture of Experts → mitigates → Multi-task Knowledge Interference
confidence 95% · propose an implicit mixture of experts (MoE) mechanism to avoid the coexistence of incompatible knowledge during inference
Sparse-and-Orthogonal LoRA → reduces → Communication Resource Consumption
confidence 92% · Simulation results demonstrate that the proposed approach effectively reduces communication resource consumption by up to 73%
Cluster-based Topology → optimizes → Communication Efficiency
confidence 90% · cluster-based topology design that can accelerate DFL convergence while reducing communication overhead
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Decentralized federated learning (DFL) based on low-rank adaptation (LoRA) enables mobile devices with multi-task datasets to collaboratively fine-tune a large language model (LLM) by exchanging locally updated parameters with a subset of neighboring devices via wireless connections for knowledge this http URL, directly aggregating parameters fine-tuned on heterogeneous datasets induces three primary issues across the DFL life-cycle: (i) \textit{catastrophic knowledge forgetting during fine-tuning process}, arising from conflicting update directions caused by data heterogeneity; (ii) \textit{inefficient communication and convergence during model aggregation process}, due to bandwidth-intensive redundant model transmissions; and (iii) \textit{multi-task knowledge interference during inference process}, resulting from incompatible knowledge representations coexistence during inference. To address these issues in a fully decentralized scenario, we first propose a sparse-and-orthogonal LoRA that ensures orthogonality between model updates to eliminate direction conflicts during this http URL, we analyze how device connection topology affects multi-task performance, prompting a cluster-based topology design during this http URL, we propose an implicit mixture of experts (MoE) mechanism to avoid the coexistence of incompatible knowledge during inference. Simulation results demonstrate that the proposed approach effectively reduces communication resource consumption by up to $73\%$ and enhances average performance by $5\%$ compared with the traditional LoRA method.
Tags
Links
- Source: https://arxiv.org/abs/2602.20492v1
- Canonical: https://arxiv.org/abs/2602.20492v1
Trouble viewing inline? Open PDF directly →
Full Text
76,171 characters extracted from source content.
Expand or collapse full text
Wireless Federated Multi-Task LLM Fine-Tuning via Sparse-and-Orthogonal LoRA Nuocheng Yang, Student Member, IEEE, Sihua Wang, Member, IEEE, Ouwen Huan, Student Member, IEEE, Mingzhe Chen, Senior Member, IEEE, Tony Q. S. Quek, Fellow, IEEE, and Changchuan Yin, Senior Member, IEEE N. Yang, S. Wang, O. Huan, and C. Yin are with the Beijing Laboratory of Advanced Information Network, and the Beijing Key Laboratory of Network System Architecture and Convergence, Beijing University of Posts and Telecommunications, Beijing 100876, China (emails: yangnuocheng, sihuawang, ouwenh, ccyin@bupt.edu.cn).M. Chen is with the Department of Electrical and Computer Engineering and Institute for Data Science and Computing, University of Miami, Coral Gables, FL, 33146 USA (email: mingzhe.chen@miami.edu).Tony Q. S. Quek with the Information Systems Technology and Design Pillar, Singapore University of Technology and Design, 487372, Singapore (email: tonyquek@sutd.edu.sg). Abstract Decentralized federated learning (DFL) based on low-rank adaptation (LoRA) enables mobile devices with multi-task datasets to collaboratively fine-tune a large language model (LLM) by exchanging locally updated parameters with a subset of neighboring devices via wireless connections for knowledge integration. However, directly aggregating parameters fine-tuned on heterogeneous datasets induces three primary issues across the DFL life-cycle: (i) catastrophic knowledge forgetting during fine-tuning process, arising from conflicting update directions caused by data heterogeneity; (i) inefficient communication and convergence during model aggregation process, due to bandwidth-intensive redundant model transmissions; and (i) multi-task knowledge interference during inference process, resulting from incompatible knowledge representations coexistence during inference. To address these issues in a fully decentralized scenario, we first propose a sparse-and-orthogonal LoRA that ensures orthogonality between model updates to eliminate direction conflicts during fine-tuning. Then, we analyze how device connection topology affects multi-task performance, prompting a cluster-based topology design during aggregation. Finally, we propose an implicit mixture of experts (MoE) mechanism to avoid the coexistence of incompatible knowledge during inference. Simulation results demonstrate that the proposed approach effectively reduces communication resource consumption by up to 73%73\% and enhances average performance by 5%5\% compared with the traditional LoRA method. Index Terms: Decentralized federated learning, large language model (LLM), catastrophic knowledge forgetting, mixture of experts. I Introduction Large language models (LLMs) have demonstrated their comprehension and reasoning capabilities across general tasks, surpassing traditional artificial intelligence (AI) models tailored to a single task [1, 2, 3, 4]. Despite their capabilities in general tasks, adapting to specific downstream applications requires continuous fine-tuning based on the low-rank adaptation (LoRA) method [5] and multi-task dataset distributed across devices [6]. To fine-tune LLMs on multi-task datasets, traditional centralized methods usually require a server for raw data or model updates collection from massive devices through wireless connections, which introduces a transmission bottleneck and privacy concerns. To address these issues, a decentralized federated learning (DFL) framework was proposed, which leverages devices to collaboratively fine-tune a global model [7]. In this paradigm, devices exchange locally fine-tuned model parameters via device-to-device connections for aggregation, thereby eliminating the dependency on centralized coordination. However, directly aggregating model updates fine-tuned on heterogeneous downstream tasks’ datasets from all available neighbors in DFL may lead to unstable and inefficient convergence, which is primarily caused by three issues across the DFL life-cycle: catastrophic knowledge forgetting, inefficient communication and convergence, and multi-task knowledge interference [8, 9, 10]. Firstly, catastrophic knowledge forgetting stems from the directional conflict between local and aggregated neighbor’s updates caused by data heterogeneity, which leads to the forgetting of prior knowledge during the fine-tuning process [8]. Secondly, inefficient communication and convergence caused by the redundant inter-device model exchanges, which force the aggregation of conflicting updates that impede convergence, leading to catastrophic forgetting and wasted bandwidth during aggregation process [9]. Thirdly, multi-task knowledge interference induced by merging multiple task-specific updates into a single aggregated model, which blurs task boundaries and leads to output distortion during the inference process [10]. To mitigate the catastrophic knowledge forgetting, the existing works have proposed the orthogonality subspace updates method for eliminating direction conflict between newly aggregated model updates and the prior model updates [11, 12, 13, 14]. This method ensures that the model updates fine-tuned on heterogeneous downstream tasks are projected onto orthogonal subspaces, thereby confining interference to distinct dimensions and effectively mitigating forgetting. However, in these studies, the subspaces of model update evolve continuously during the fine-tuning process, necessitating a centralized server for model update collection and enforce orthogonality, which is incompatible with the DFL setting. To address inefficient communication and convergence, the relationship between multi-task performance and device connection schemes must be analyzed. Nevertheless, existing works [15, 16, 17] overlooked the impact of model parameter orthogonality and failed to account for the unique constraints of DFL, where devices must autonomously determine a subset of neighbours for local model-update transmission. To reduce the multi-task knowledge interference, mixture of experts (MoE) methods are employed that regard model parameters fine-tuned on heterogeneous datasets as experts independently and introduce an additional MoE router for expert selection, and hence achieving knowledge isolation during the inference process [18, 19, 20]. Even so, training the MoE router incurs computational overhead that scales with the number of devices, burdening the resource-constrained devices in DFL. Furthermore, the parameter isolation inherent in MoE design completely hinders the integration of knowledge among devices. To fill these gaps, in this paper, we develop a collaborative LLM fine-tuning framework in DFL by integrating a sparse-and-orthogonal LoRA method, a task-aware implicit MoE mechanism, and a cluster-based device connection topology design. Specifically, our key contributions are as follows: • Sparse-and-Orthogonal LoRA for Reducing Catastrophic Knowledge Forgetting during the Fine-tuning Process: We decompose parameter updates into a static low-rank projection matrix, sampled from a Gauss-sampled, and a low-rank, sparse-activated expansion matrix. Consequently, the model updates fine-tuned on heterogeneous datasets are projected onto mutually orthogonal subspaces; this property is maintained by the fixed projection matrices, which inherit their orthogonality from independent Gaussian sampling. Furthermore, the sparse activated expansion matrix is introduced to further mitigate parameter collision while reducing the number of parameters that need to be updated and transmitted. • Cluster-based Device Connection Topology Design for Communication-Efficient and DFL Acceleration during the Model Aggregation Process: Based on the proposed sparse-and-orthogonal LoRA, we further propose a novel device connection topology design for efficient model updates transmission in multi-task DFL. Specifically, we first analyze how the device connection topology affects the cumulative error of the aggregated LoRA representation, which affects the fine-tuning performance. Then, we develop a cluster-based device connection topology design that can accelerate DFL convergence while reducing communication overhead. • Implicit MoE Design for Minimizing Multi-task Knowledge Interference during the Inference Process: We propose an implicit MoE mechanism that embeds task-specific information into the static projection matrix, coupled with a top-k expert activation strategy. This method selectively activates a subset of parameters within the aggregated expansion matrix (regarded as experts) that exhibit the highest responsiveness to the input tokens. Therefore, the projection matrix embedded with task-specific information can act as an implicit MoE router that activates to the related experts, while filtering out interference from irrelevant experts without an additional training process. An extensive experiment shows that the proposed method can effectively reduce communication overhead by up to 73%73\% while improve overall LLM performance by up to 5%5\% compared with the traditional LoRA method. I Related Works I-A Decentralized Federated Learning based on LoRA LoRA is one of the widely used parameter-efficient fine-tuning techniques that enables resource-constrained devices to adapt pre-trained LLMs with task-specific knowledge locally [5]. In particular, LoRA integrates trainable adapters into each pre-trained LLM layer. Each adapter is characterized by a pair of low-rank matrices, named the projection matrix A and expansion matrix B, respectively. During fine-tuning, only these low-rank matrices are updated while the pre-trained LLM remain frozen. Since low-rank matrices contain significantly fewer parameters than the pre-trained LLM, this approach drastically reduces the computation overhead of fine-tuning. Furthermore, multi-task fine-tuning of LLM traditionally relies on a centralized server to collect raw data or updated models from all participating devices, which incurs a prohibitive transmission bottleneck and raises significant privacy concerns [21, 22, 23]. To fill these gaps without centralization constraints, DFL, which enables devices to exchange updated local matrices for integration, has been proposed. To further accommodate the heterogeneous computational and communication resources of individual devices in DFL, strategies such as adaptive rank selection [24], adaptive trainable layers selection [25], and adaptive adapters quantization strategy [26] have been explored. However, directly aggregating model updates on heterogeneous datasets poses two key bottlenecks in DFL. First, due to the datasets’ heterogeneity, the adapters exhibit divergent update directions; these updates tend to offset the datasets’ heterogeneity, resulting in catastrophic knowledge forgetting. Second, the coexisting adapters trained based on heterogeneous datasets in the aggregated adapter may cause multitask knowledge interference due to the competing objectives of different downstream tasks. I-B Interference-free with Orthogonality Constraints To address the catastrophic knowledge forgetting problem caused by interference between adapters with divergent update directions, several studies have proposed a model orthogonality-based approach [11, 12, 13, 14]. The authors in [11] first attributed catastrophic knowledge forgetting to conflicts between model updates. Then, they proposed an orthogonal gradient descent (OGD) method to prevent conflicts in model update directions by introducing a direction orthogonal constraint into the local objective function. Note that, to ensure the mutual orthogonality between model update directions, all devices’ model updates must be collected and constrained simultaneously. Inspired by [11], the authors in [12] first introduced OGD into multi-task LLM fine-tuning, and the authors in [13] expanded it to the federated learning scenario, where the centralized server collects model updates from devices and ensures orthogonality. Beyond maintaining orthogonality via explicit constraints, the authors in [14] demonstrated that parameter orthogonality can also be achieved by ensuring non-overlapping updates across multi-tasks. However, these works [11, 12, 13, 14] can only be employed with a centralized server to coordinate the disruption of the orthogonality caused by local fine-tuning and non-parameter collisions between updates. On the other hand, the absence of a central server in DFL, coupled with constrained communication resources, prevents devices from accessing all participants’ updates to ensure orthogonality [7]. I-C Multi-task Knowledge Isolation based on MoE To address multi-task knowledge interference caused by multiple knowledge coexisting in the aggregated adapter, several works [18, 19, 20] have focused on MoE based approaches. These methods treat each adapter fine-tuned on heterogeneous datasets as an expert and introduce a router to assign inputs to the most appropriate subset of experts. The authors in [18] initialized FL with MoE-based adapters, where each device trains a lightweight gating network to selectively activate a personalized subset of experts to enhance the multi-task fine-tuning performance. Building upon a similar intuition, the authors in [19] employed an attention-based MoE router to manage input dispatching within a DFL framework. However, in these works [18, 19], both the number of experts and the router’s training overhead scale linearly with the increasing number of devices, leading to significant scalability challenges in large-scale networks. To eliminate the drawbacks caused by an additional MoE router, the authors in [20] employed singular value decomposition (SVD) to decompose adapters into experts, which are dynamically engaged according to their singular values. These works [18, 19, 20] also face challenges in aggregating experts into the converged MoE models, particularly in balancing expert utilization and designing effective mixing strategies. Additionally, due to limited communication capability, devices can only exchange updates with a few neighbors, necessitating an efficient connection topology to optimize expert aggregation and accelerate MoE convergence. I-D Communication-Efficient Topology for DFL Convergence Acceleration Due to the limited communication resources of each device in the DFL, they must select a subset of neighbors for exchanging the locally updated adapters, which also impacts communication overhead and DFL convergence speed. An inefficient device connection scheme not only incurs substantial communication overhead but also prevents the aggregated model from converging, due to catastrophic knowledge forgetting and multi-task knowledge interference problems. To fill these gaps, the authors in [15] analyzed how device connection topology affects the model transmission consumption and the convergence speed in DFL with single-task and proposed a decentralized graph neural network based topology design approach. The authors in [16] proved that the device connection topology also impacts the knowledge aggregation order in single-task learning, and they optimize the device connection topology by considering graph connectivity and model alignment. The authors in [17] proposed a device topology design method based on model distillation and feature mapping method to simultaneously enhance DFL aggregation performance and ensure security. However, these works [15, 16, 17] still suffer from catastrophic knowledge forgetting and multi-task knowledge interference. Therefore, it is desirable to design a new collaborative LLM fine-tuning method in the DFL scenario by jointly considering the catastrophic knowledge forgetting, multi-task knowledge interference, and the substantial communication resource consumption under DFL resource constraints. I System Setup and Problem Formulation Consider a distributed wireless network consisting of a set ℳM of M devices that collaboratively train an LLM with L layers that can adapt to multiple downstream tasks. We assume that each device i∈ℳi has a local dataset iD_i for a unique type of downstream task KiK_i, containing NiN_i data samples with N=∑i=1MNiN= _i=1^MN_i being the total number of data samples across devices. Each data sample n consists of an input feature vector i,n∈ℝNI×1 x_i,n ^N_ I× 1 and a corresponding label vector i,n∈ℝNO×1 y_i,n ^N_ O× 1. In particular, the local objective function of device i is given by F(0,i,t,i)=1Ni∑n=1Nif(ϕ(0,i,t,i,n),i,n),F ( W_0, w_i,t,D_i )\!=\! 1N_iΣ _n=1^N_i\!f\! (φ ( W_0, w_i,t, x_i,n ), y_i,n ), (1) where 0=0,1,⋯,0,L W_0= \ w_0,1,·s, w_0,L \ denotes the fixed pre-trained LLM parameters set, where 0,l∈ℝdl×kl w_0,l ^d_l× k_l being the parameters of l-th pre-trained LLM layer. Similarly, i,t=i,1,t,⋯,i,L,t w_i,t= \ w_i,1,t,·s, w_i,L,t \ is the trainable adapter parameters set of device i at iteration t , where i,l,t∈ℝdl×kl w_i,l,t ^d_l× k_l being the parameters of l-th fine-tuned LLM layer. ϕ(0,i,t,i,n)φ ( W_0, w_i,t, x_i,n ) denotes the output of fine-tuned LLM, and f(ϕ(0,i,t,i,n),i,n)f (φ ( W_0, w_i,t, x_i,n ), y_i,n ) is the loss function that measures the difference between the output ϕ(0,i,t,i,n)φ ( W_0, w_i,t, x_i,n ) and label i,n y_i,n. I-A DFL Local Fine-tuning Process Figure 1: Illustration of the considered multi-task DFL framework. To reduce the computation overhead caused by full parameter updating, LoRA is employed to decompose adapter i,l,t∈ℝdl×kl w_i,l,t ^d_l× k_l into two low-rank matrices i,l,t∈ℝdl×rl B_i,l,t ^d_l× r_l and i,l,t∈ℝrl×kl A_i,l,t ^r_l× k_l in each device with which has fewer parameters, which is given by i,l,t=i,l,ti,l,t, w_i,l,t= B_i,l,t A_i,l,t, (2) where i,l,t B_i,l,t is called expansion matrix and i,l,t A_i,l,t is called projection matrix, respectively. Expansion matrix and projection matrix share the same rank rl≪min(dl,kl)r_l (d_l,k_l ). To further reduce the computational overhead for local LoRA parameter updating, a sparsity-activation-based method is employed, which activates only a subset of parameters in i,l,t B_i,l,t during local fine-tuning. The sparsification mask for i,l,t B_i,l,t is i,l,tB∈0,1dl×rl M^B_i,l,t∈\0,1\^d_l× r_l where [i,l,tB]x,y=1 [ M^B_i,l,t ]_x,y=1 implies the x,y\x,y\-th parameter in i,l,t B_i,l,t is non-zero and trainable during updating, and [i,l,tB]x,y=0 [ M^B_i,l,t ]_x,y=0, otherwise. Thus, the adapter of device i in (2) can be rewritten as i,l,t=(i,l,t⊙i,l,tB)i,l,t, w_i,l,t= ( B_i,l,t M^B_i,l,t ) A_i,l,t, (3) where ⊙ is the element-wise multiplication function. Then, we define the sparsity rate of i,l,t B_i,l,t, which represents the average activation probability on each parameter position of i,l,t B_i,l,t during fine-tuning as si,l,t=ji,l,tBjjdl×rl, s_i,l,t= || M^B_i,l,t||d_l× r_l, (4) where ji,l,tBj=∑x=1dl∑y=1rl[i,l,tB]x,y|| M^B_i,l,t||=Σ _x=1^d_lΣ _y=1^r_l [ M^B_i,l,t ]_x,y is the number of the activated parameter that device i will transmit to its neighbors. Generally, in each iteration t, each device i performs a local model update based on the local dataset iD_i as follows i,l,t′ A _i,l,t =i,l,tΓη∇i,l,tF(0,i,t,i), = A_i,l,t-η _ A_i,l,tF ( W_0, w_i,t,D_i ), (5) i,l,t′ B _i,l,t =i,l,tΓη∇i,l,t⊙i,l,tBF(0,i,t,i), = B_i,l,t-η _ B_i,l,t M^B_i,l,tF ( W_0, w_i,t,D_i ), (6) where η is the learning rate, ∇i,l,tF(0,i,t,i) _ A_i,l,tF ( W_0, w_i,t,D_i ) and ∇i,l,t⊙i,l,tBF(0,i,t,i) _ B_i,l,t M^B_i,l,tF ( W_0, w_i,t,D_i ) are the cumulative gradient. The computation overhead of device i in the local fine-tuning process is given by [27] ci,t=εQi,t(i,t)NiB, c_i,t= Q_i,t ( s_i,t )N_iB, (7) where ε donate the computing correlation coefficient related to the device’s computing capability, Qi,t(i,t)=∑l=1Ldlrlsi,l,t+rlklQ_i,t ( s_i,t )= _l=1^Ld_lr_ls_i,l,t+r_lk_l being the number of parameters that need to be updated. i,t=[i,1,t,…,i,L,t] s_i,t= [ s_i,1,t,…, s_i,L,t ] is the sparsity rate matrix. , and B≤BmaxB≤ B_max is the batch size of local fine-tuning and BmaxB_max is the maximum batch size. I-B DFL Model Transmission and Aggregation Process After local fine-tuning, each device i will transmit the updated LoRA parameters with its neighbors for model aggregation. Note that due to the limited transmission resources, only a subset of neighboring devices can be selected for model transmission. We adopt an orthogonal frequency division multiple access (OFDMA) transmission scheme for model transmission. Let W be the bandwidth that the device can use to transmit model parameter updates i,t w_i,t and pi,j,tp_i,j,t be the transmit power. The data transmission delay of device i transmitting updated model parameters to device j is li,j,t(Qi,t(i,t),i,t,t,pi,j,t)=Qi,t(i,t)W‖i,t‖log(1+pi,j,thi,j,t(t)σN2), l_i,j,t\! (Q_i,t ( s_i,t ),\! u_i,t,\! _t,\!p_i,j,t )\!=\! Q_i,t ( s_i,t ) W|| u_i,t|| (1+ p_i,j,th_i,j,t ( _t )σ^2_ N ), (8) where i,t=[ui,1,t,⋯,ui,M,t] u_i,t=[u_i,1,t,·s,u_i,M,t] is a model transmission vector of device i with ui,j,t=1u_i,j,t=1 implying that device i will exchange its local model with device j at iteration t, and ui,j,t=0u_i,j,t=0, otherwise. ji,tj=∑j∈ℳui,j,t|| u_i,t||= _j u_i,j,t is the number of devices that will exchange the updated parameters to device i. The subset of neighbors selected by device i with which it transmit its updated parameter is ℳi,t=jjui,j,t=1M_i,t= \j|u_i,j,t=1 \. hi,j,t=ρi,j,tdi,j,t−2h_i,j,t= _i,j,td_i,j,t^-2 is the channel gain between device i and j with ρi,j,t _i,j,t being the Rayleigh fading parameters, and di,j,td_i,j,t is the distance between device i and j. The location of each device i at iteration t is captured by a vector i,t=[εi,t,1,εi,t,2] _i,t=[ _i,t,1, _i,t,2], with t=[1,t,…,M,t] _t= [ _1,t,..., _M,t ] being the location matrix across devices. σN2σ^2_ N represents the variance of additive white Gaussian noise. Given the neighbor’s update model parameter, in the traditional DFL framework, devices will aggregate the received model into a single model [28], which is given by: i,l,t+1=∑j∈ℳαjuj,i,tjji,tjj,l,t′, A_i,l,t+1= _j _j u_j,i,t|| u_i,t|| A _j,l,t, (9) i,l,t+1=∑j∈ℳαjuj,i,tjji,tjj,l,t′, B_i,l,t+1= _j _j u_j,i,t|| u_i,t|| B _j,l,t, (10) where αj∈[0,1] _j∈ [0,1 ] is the aggregation weight based on training data volume and device connection density, which satisfy ∑j∈ℳαj=1 _j _j=1. I-C Problem Formulation Our goal is to minimize the DFL training loss across various downstream tasks while accounting for transmission delay, communication and computational resource constraints simultaneously. The optimization problem is formulated as min,,1M∑i=1M∑j=1MF(0,i,t,j), _ U, P, S 1MΣ _i=1^MΣ _j=1^MF ( W_0, w_i,t,D_j ),\\ (11) s.t. s.t.\,\, li,j,t(Qi,t(i,t),i,t,t,pi,j,t)⩽Γ,∀i,j∈ℳ,∀t∈, l_i,j,t (Q_i,t ( s_i,t ), u_i,t, _t,p_i,j,t ) ,∀ i,j ,∀t , (11a) ∑j∈ℳi,tpi,j,t⩽pmax,∀i∈ℳ,∀t∈, _j _i,tp_i,j,t p_max,∀ i ,∀t , (11b) ci,t⩽ci,max,∀i∈ℳ,∀t∈, c_i,t c_i,max,∀ i ,∀t , (11c) where =[1,⋯,T]⊤ U=[ u_1,·s, u_T] is the model transmission matrix, =[1,⋯,T]⊤ P=[ p_1,·s, p_T] is the transmit power matrix, and =[1,⋯,T]⊤ S=[ s_1,·s, s_T] is the sparsity rate matrix where t=[si,t,⋯,sM,t] s_t= [s_i,t,·s,s_M,t ] is the sparsity rate of each layer at iteration t. pmaxp_max is the transmit power constraint. Γ is the maximum model transmission delay. pmaxp_max and ci,maxc_i,max are the maximum transmission power and maximum computing resources, respectively. (11a) is a constraint on the model transmission delay per iteration, (11b) is the transmit power constraint, and (11c) is the computing resources constraint. Second, the design of U and sparsity masks is challenging, and their interplay with DFL performance needs to be investigated. IV Proposed Methods Figure 2: Comparison between different LoRA methods during the aggregation process. IV-A Issues of Existing Works Based on the modeling above, we identify three primary challenges for collaborative LLM fine-tuning within the DFL life-cycle: • Problem 1: Absence of a decentralized mechanism against catastrophic knowledge forgetting during fine-tuning process. The existing works introduce orthogonal constraints into the local objective function (1) to enforce the model updates of different tasks separately in orthogonal subspaces, which is given by Forth(0,i,t,i)= F_orth ( W_0, w_i,t,D_i )= 1Ni∑n=1Nif(ϕ(0,i,t,i,n),i,n) 1N_iΣ _n=1^N_i\!f\! (φ ( W_0, w_i,t, x_i,n ), y_i,n ) (12) +∑j=1Mλiji,l,ti,l,t⊤j2. +Σ _j=1^M _i| A_i,l,t A_i,l,t |^2. The second term in (12) is the orthogonal loss function, λi _i is the penalty factor. Note that each device needs to collect i,l,t,∀i∈ℳ A_i,l,t,∀ i for minimizing Forth(0,i,t,i)F_orth ( W_0, w_i,t,D_i ) contradicts the decentralized setting. • Problem 2: Inadequacy of theoretical analysis on the relationship between multi-task performance and device connection scheme design during the aggregation process. The existing DFL works focus on single-task fine-tuning scenario, omitting the analysis of its relationship with multi-task performance. • Problem 3: Lack of an computationally inexpensive method against multi-task knowledge interference reduction method during the inference process. For example, consider i,l,t w_i,l,t and j,l,t w_j,l,t are aggregated into a single model based on the traditional method shown in (9) and (10), which is given by l,t′ W _l,t =0,l+(αii,l,t′+αjj,l,t′)(αii,l,t′+αjj,l,t′). \!=\! W_0,l\!+\! ( _i B _i,l,t\!+\! _j B _j,l,t )\!\! ( _i A _i,l,t+ _j A _j,l,t ). (13) Let i,l,input H_i,l,input be the l-th layer input for task i, derived from dataset iD_i, the accordingly inference output of the merged model l,t′ W _l,t is given by l,t′i,l,input=0,li,l,input+αi2i,l,t′i,l,t′i,l,input W _l,t H_i,l,input= W_0,l H_i,l,input+ _i^2 B _i,l,t A _i,l,t H_i,l,input (14) +(αiαj∑k,l∈i,j,k≠lk,l,t′l,l,t′+αj2j,l,t′j,l,t′)i,l,input, \!\!+\!\! ( _i _j\!\!\!\!Σ _k,l∈ \i,j \,k≠ l\!\!\!\! B _k,l,t A _l,l,t+ _j^2 B _j,l,t A _j,l,t ) H_i,l,input, where the second term is the desired output from the fine-tuned adapter i,l,t w_i,l,t. The third term of (14) can be viewed as a ”interference” introduced by the knowledge from other adapters, which needs to be minimized for enhancing LLM performance. To achieve this, the existing MoE-based approaches employ a MoE router to determine gating weights for experts (e.g., adapters) during inference process based on their knowledge. The output of the MoE-based model can be given by l,MoE′i,l,input=(0,l+∑k=1Mβkk,l,t′k,l,t′)i,l,input, W _l,MoE H_i,l,input\!= (\! W_0,l\!+\!Σ _k=1^M _k B _k,l,t A _k,l,t ) H_i,l,input, (15) where βk _k is the gating weight. The MoE-based approach introduces a router, incurring additional training overhead that scales with the number of devices and limits its applicability in large-scale deployments. IV-B Static-and-Orthogonal Projection Matrix and Layer-Wise Sparsity Expansion Matrix Design to Solve Problem 1 Inspired by [11], keeping the projection matrix i,l,t A_i,l,t orthogonality between different tasks (i.e., ∑ji,l,ti,l,t⊤j2≈0,∀i,j∈ℳ,∀l∈[1,L]Σ|| A_i,l,t A_i,l,t ||^2≈ 0,∀ i,j ,∀ l∈ [1,L ]) can effectively avoid knowledge forgetting during parameter aggregation process. However, the orthogonality between i,l,t A_i,l,t will be compromised during each local model update, and hence the updated projection matrix from all other devices needs to be collected in real-time to ensure orthogonality, which compromises the communication resources and leads to unstable fine-tuning. To satisfy the orthogonality between the projection matrix i,l,t A_i,l,t in the DFL scenario without relying on collecting the latest updated parameters from all devices, we regard i,l,t A_i,l,t as a static matrix with entries drawn from a zero-mean, unit-variance Gaussian distribution independently, inspired by LoRI [29]. In this way, i,l,t,∀i∈ℳ A_i,l,t,∀ i are mutually orthogonal during the fine-tuning process, which is given by [Theorem 3.4, [30]] ∑ji,l,t⊤j,l,tj2≈r×r,[i,l,t]x,y∼(0,1),∀i,j∈ℳ. Σ| A_i,l,t A_j,l,t|^2≈ 0_r× r, [ A_i,l,t ]_x,y (0,1 ),∀ i,j . (16) Since i,l,t,∀i∈ℳ A_i,l,t,∀ i are frozen during fine-tuning and aggregation, we assume they are stored separately, which is different from the standard aggregation step (9) in traditional DFL. In the proposed framework, each device i transmit its local projection matrix i,l,t A_i,l,t with its neighbors only at the first iteration of parameter exchange, and separately save them into the projection matrix set i,l,tji∈ℳ,l∈[1,L] \ A_i,l,t|i ,l∈ [1,L ] \. Further, we abbreviated static i,l,t A_i,l,t as i,l A_i,l in the following section. The number of parameters that need to be updated can also be rewritten as Qi,t(i,t)=∑l=1Ldlrlsi,l,t. Q_i,t ( s_i,t )= _l=1^Ld_lr_ls_i,l,t. (17) Here the transmission overhead of i,l A_i,l is ignored since it only requires to be transmitted once between devices. Note that keeping i,l A_i,l static does not compromise the fine-tuning accuracy compared to traditional DFL where i,l A_i,l is continuously updated, as i,l,ti,l B_i,l,t A_i,l preserves the same subspace dimensionality. Furthermore, i,l A_i,l only needs to be transmitted once across different pairs of devices, which can also reduce the transmission overhead compared to the traditional approaches where i,l A_i,l is transmitted at each iteration. However, parameter interference during the aggregation of expansion matrix i,l,t B_i,l,t persists even with the implementation of the orthogonal projection matrix. To further avoid parameter interference between the expansion matrix i,l,t B_i,l,t during aggregation as well as decreasing communication overhead, we propose a sparse-activation mechanism as shown in (3). This mechanism selectively activates only a subset of parameters in i,l,t B_i,l,t during local fine-tuning, independently based on their communication and computing resources through sparsification mask for i,l,tB∈0,1rl×kl M^B_i,l,t∈\0,1\^r_l× k_l. Figure 3: Comparison of update conflicts in conventional LoRA and the proposed sparse-and-orthogonal LoRA. Specifically, to preserve essential model performance, we activate the parameters with the highest si,l,t%s_i,l,t\% value within i,l,t B_i,l,t during the initial fine-tuning mini-batch, which can be given by: [i,l,tB]x,y=([i,l,t]x,y≥τsi,l,t), [ M_i,l,t^B ]_x,y= 1 ( [ B_i,l,t ]_x,y≥ _s_i,l,t ), (18) where x=1 1_ \x \=1 if condition x is true, x=0 1_ \x \=0, otherwise. τsi,l,t _s_i,l,t is the threshold corresponding to the top si,l,t%s_i,l,t\% value in i,l,t B_i,l,t. Since the projection matrix i,l A_i,l are randomly generated on each device and i,l,t B_i,l,t are activated based on their fine-tuned expansion matrix, we assume that at each parameter position, i,l,t B_i,l,t has a fully independent activation probability. Then, we can define the parameter collision rate during aggregating set of expansion matrix i,l,t⊙i,l,tBji∈ℳi,t\ B_i,l,t M_i,l,t^B|i _i,t\ which measure the parameter interference during model aggregation as Sℳi,t,l=1 S_M_i,t,l=1 Γ∏j∈ℳi,t(1Γsj,l,t)Γ∑j∈ℳi,tsj,l,t∏z∈ℳi,t,z≠j(1Γsz,l,t), \!-\!\!\!Π _j _i,t\!\!\! (1\!-\!s_j,l,t )\!-\!\!Σ _j _i,t\!\!s_j,l,t\!\!\!\!Π _z _i,t,z≠ j\!\!\! (1-s_z,l,t ), (19) where Sℳi,t,l∈[0,1]S_M_i,t,l∈ [0,1 ]. We can see that the parameter collision rate Sℳi,t,lS_M_i,t,l depends on both the devices in the aggregation set ℳi,tM_i,t and their activation probability si,l,ts_i,l,t, respectively. To minimize the parameter interference rate Sℳi,t,lS_M_i,t,l of devices in ℳi,tM_i,t, we need to design the device set and their activation probabilities si,l,ts_i,l,t, respectively. Since the computing and communication resources of each device are limited, the maximum number of parameters that need to be updated and transmitted is restricted as shown in (11c). At the same time, heterogeneous datasets exhibit diversity across multiple dimensions, leading to varied representational requirements across different layers [31]. This motivates the design of a task-specific sparsity rate allocation method across layers, which can better match the representational demands of each task and reduce overall parameter collisions across devices in ℳi,tM_i,t. To measure the heterogeneous task’s representational demands across layers for sparsity rates allocation, we need to measure these discrepancies. Let i,l,input0 H^0_i,l,input be the average input of pre-trained model 0 W_0 at l-th layer for a small batch of data from iD_i, which is stable and independent of the fine-tuning process. Then, we introduce the covariance i,l=i,l,input0(i,l,input0)⊤ C_i,l= H^0_i,l,input ( H^0_i,l,input ) and derive a SVD as follows ^i,l=SVD(0,li,l)i,l−1=∑i=1Ri,lσi,li,li,l⊤, W_i,l=SVD ( W_0,l C_i,l ) C_i,l^-1= _i=1^R_i,l _i,lu_i,lv_i,l , (20) where σi,l _i,l is the singular value, i,lu_i,l and i,lv_i,l represent corresponding vectors, and Ri,l=Rank(^i,l)R_i,l=Rank ( W_i,l ) is the rank dimension of ^i,l W_i,l. Since the singular value can measure the complexity of the representation required by task in the l-th layer, we can compute the spectral entropy of the singular values σi,l _i,l for task i in the l-th layer as ℋ(i,l)=Γ∑j=1Rpj,llog(pj,l), ( σ_i,l )=- _j=1^Rp_j,l (p_j,l ), (21) where i,l=[σi,1,⋯,σi,Ri,l] σ_i,l= [ _i,1,·s, _i,R_i,l ] and pj,l=σj,l∑z=1Ri,lσz,lp_j,l= _j,l _z=1^R_i,l _z,l is the normalized probability. Under this fixed total number of trainable parameter budget Qi,budget=ci,maxBεNiQ_i,budget= c_i,maxB N_i which is restricted by the communication (11a) and computation resources (11c), the sparsity rate of task i in layer l can be allocated by si,l,t=max[Qi,budget×[Softmax(ℋ())]l,dl×rl]dl×rl, s_i,l,t= max[Q_i,budget× [Softmax (H ( σ ) ) ]_l,d_l× r_l]d_l× r_l, (22) where [Softmax(ℋ())]l [Softmax (H ( σ ) ) ]_l is the normalized distribution for l-th layer based on Softmax function. By introducing a fixed total dimension budget and layer-wise importance, we employ a task-aware sparsity rate design method that can accommodate heterogeneous requirements under resource constraints. IV-C Cluster-based Device Connection Topology Design based on DFL Convergence Analysis to Solve Problem 2 To simplify the device connection U optimization in (11), we must analyze the impact of U on the DFL model convergence under the proposed orthogonal i,l A_i,l, sparse activation i,l,t B_i,l,t, and refinement design using task-aware vector first. Since the each projection matrix i,l A_i,l is independently sampled by Gaussian distribution, and each expansion matrix i,l,t B_i,l,t is updated based on independent dataset and sparse mask, each adapter i,l,t w_i,l,t and the masks i,l,t M_i,l,t are independent with each other, we can make the following assumptions: • Assumption1~1: For device i,j∈ℳi,j , we assume Ej∇i,l,tF(0,i,t,i)i,lj≤GE|| _ B_i,l,tF ( W_0, w_i,t,D_i ) A_i,l||≤ G. • Assumption2~2: For each device i∈ℳi , we assume Ej1|ℳi,t|∑j∈ℳi,tj,l,ti,lj≤PE|| 1|M_i,t|Σ _j _i,t B_j,l,t A_i,l||≤ P. These assumptions are natural, where Assumption 11 stems from the fact that the upper bound of the local gradient exists, and Assumption 22 assumes an upper bound of the product between Gauss-sampled i,l A_i,l and independent updated j,l,t B_j,l,t exists. Then, we analyze the impact of U on the upper bound of the gap between the local fine-tuned adapter and the aggregated global adapter, which can be given by Theorem 1. Given the model transmission matrix U, an upper bound of the gap between each local model (i,l,t⊙i,l,t)i,l,refined ( B_i,l,t M_i,l,t ) A_i,l,refined of device i and the average model (1|ℳi,t|∑j∈ℳi,tj,l,t⊙j,l,t)i,l,refined ( 1|M_i,t| _j _i,t B_j,l,t M_j,l,t ) A_i,l,refined of all neighboring devices in ℳi,t=jjui,j,t=1,∀j∈ℳM_i,t= \j|u_i,j,t=1,∀ j \ can be given by ∑i∈ℳi,tEj(i,l,t⊙i,l,t)i,l,refinedΓ Σ _i _i,t\!\!\!\!E| ( B_i,l,t M_i,l,t ) A_i,l,refined- (23) (1jℳi,tj∑j∈ℳi,tj,l,t⊙j,l,t)i,l,refinedj2 ( 1|M_i,t| _j _i,t B_j,l,t M_j,l,t ) A_i,l,refined|^2 ≤2rk∑i=1M∑j=1MSℳi,t,l(G+P). ≤ 2rkΣ _i=1^MΣ _j=1^MS_M_i,t,l (G+P ). Proof. See Appendix A. ∎ From Theorem 1, we can see that the gap between each local adapter of device i and the aggregated adapters of all devices in ℳi,tM_i,t is affected by Sℳi,t,lS_M_i,t,l, which depends on the device connection topology U. From Theorem 1, we can also observe that this gap decreases as the parameter collision rate Sℳi,t,lS_M_i,t,l decreases, which implies that the device connection topology design that can satisfy the parameter collision-free condition between i,l,t B_i,l,t can effectively enhance the DFL performance. This is due to the fact that each device updates i,l,t⊙i,l,t B_i,l,t M_i,l,t with activation overlap simultaneously, which will cause a multi-party parameter interference, leading to inconsistencies in the optimization directions, thus reducing the DFL performance. Furthermore, since the parameter collision rate Sℳi,t,lS_M_i,t,l grows with the number of devices in ℳi,tM_i,t, it is essential to perform aggregation separately. To enhance DFL performance by reducing Sℳi,t,lS_M_i,t,l, we propose a cluster-based method where devices are grouped into clusters according to their resources and si,l,ts_i,l,t, and the aggregation of i,l,t B_i,l,t is performed separately within each cluster (named inner-cluster aggregation), thereby guaranteeing Sℳi,t,lS_M_i,t,l does not exceed the predefined threshold SmaxS_ . After iterations of inner-cluster aggregation, inter-cluster parameter exchange is performed, where parameters exchanged without further aggregation and thus fundamentally expanding the scope of knowledge while avoiding parameter collisions between clusters. In particular, we employ an AGNES-based cluster algorithm considering devices’ computing, communication resources, and parameter collision rate constraints, where each device is initially considered as a single-element cluster (leaf). At each step of the cluster algorithm, neighbor clusters exchange their communication resources and sparsity rate with each other and find clusters with the lowest parameter collision rate, as well as satisfy the transmission power constraint in (11b), and combine into a new, bigger cluster. This procedure is iterated until all devices are members of a single, large cluster, or until the predefined parameter collision rate SmaxS_ is reached. The parameter aggregation and exchange in the proposed cluster-based algorithm can be summarized as follows: 1. Devices aggregate into different clusters ℳi,t⊆M_i,t based on cluster algroithm. 2. Each device i perform local fine-tuning and exchange parameter i,l A_i,l and i,l,t B_i,l,t with other devices available in cluster for aggregation. 3. Each device in cluster ℳi,tM_i,t store the received i,l A_i,l and aggregate the received i,l,t B_i,l,t based on (10). 4. After iterations of updating, clusters exchange the stored i,l A_i,l and i,l,t B_i,l,t with each other for knowledge sharing. Steps 2)-4) are performed until the DFL model convergence. Hence, each device keeps a group of adapters generated by different clusters =ℳi,tC=\M_i,t\, the adapter used for the specific input i,l H_i,l can be given by i,l,ti,l,input0= w_i,l,t H^0_i,l,input= ∑ℳi,t⊆∑j∈ℳi,t(1jℳi,tj∑j∈ℳi,tj,l,t⊙j,t) Σ _M_i,t Σ _j _i,t ( 1|M_i,t|Σ _j _i,t B_j,l,t M_j,t ) (24) j,l,refinedi,l,input0. A_j,l,refined H^0_i,l,input. After obtaining the device connection scheme U, the optimal transmit power can be given by [Lemma 1, [15]] as follows: Lemma 1. The optimal transmit power pi,j,tp_i,j,t of device i for transmitting its FL model to device j is pi,j,t∗=ui,j,tσN2hi,j,t(t)(2Qi,t(i,t)‖i‖WΓ1). splitp_i,j,t = u_i,j,t _N^2h_i,j,t ( _t ) (2 Q_i,t ( s_i,t )|| u_i||W -1 ). split (25) IV-D Task-Aware Coding and Implicit MoE Design to Solve Problem 3 To address Problem 3 without introducing additional computational overhead from the MoE router, we propose an implicit MoE mechanism that combines task-aware coding with a static, orthogonal-based projection matrix design. To minimize the interference term in (14), we refined static and orthogonal-based projection matrix i,l A_i,l in (16) with i,l,input0 H^0_i,l,input which is given by i,l,refined=i,li,l,input0ji,l,input0j, A_i,l,refined= A_i,l H^0_i,l,input|| H^0_i,l,input||, (26) where i,l,input0‖i,l,input0‖ H^0_i,l,input|| H^0_i,l,input|| is the normalized latent vector. By incorporating normalized i,l,input0 H^0_i,l,input of task i into the projection matrix i,l A_i,l, the interference term in (14) can be given by (i,l,t⊙j,l,tB)i,l,refinedl,input ( B_i,l,t M^B_j,l,t ) A_i,l,refined H_l,input (27) = = (i,l,t⊙j,l,tB)i,li,l,input0l,inputjji,l,input0j. ( B_i,l,t M^B_j,l,t ) A_i,l H^0_i,l,input H_l,input|| H^0_i,l,input||. The interference term (LABEL:eq:interference_term) scaled down by i,l,input0l,input‖i,l,input0‖ H^0_i,l,input H_l,input|| H^0_i,l,input|| when l,input H_l,input has completely different representation with i,l,input0 H^0_i,l,input. On the other hand, when i,l,input0 H^0_i,l,input and l,input H_l,input have a highly similar representation, it implies the adapter fine-tuned based on task i and l,input H_l,input equipped with similar knowledge, which can also be scaled up simultaneously. Hence, the proposed method that refines the projection matrix i,l A_i,l with task-aware input vector i,l,input0 H^0_i,l,input based on the unified pre-trained model 0 W_0 can re-scale the interference from interference from other unrelated tasks, but also promotes the knowledge integration. It can also be proved that the proposed orthogonality, sparsity activation, and refinement method does not change the orthogonality between i,l,refined A_i,l,refined as follows: Theorem 2. (Subspace Orthogonality under Sparsity Activation and Task-Aware Refinement) Let i,l,t=(i,l,t⊙i,l,tB)i,l,refined w_i,l,t= ( B_i,l,t M_i,l,t^B ) A_i,l,refined and j,l,t=(j,l,t⊙j,l,tB)j,l,refined w_j,l,t= ( B_j,l,t M_j,l,t^B ) A_j,l,refined. The expectation of their product reveals: ⟨i,l,t,j,l,t⊤⟩F=0. < w_i,l,t, w_j,l,t >_F=0. (28) Proof. See Appendix B. ∎ From Theorem 2, we can observe that both the proposed sparse activation (for minimizing interference during fine-tuning) and the refinement method (for minimizing interference during inference) maintain orthogonality between adapters, thereby ensuring that inter-adapter interference is minimized. Next, to activate related expert after obtaining the relevant coefficient through the matching between specific inputs l,input H_l,input and the refined projection matrix i,l,refined A_i,l,refined, we must filter out irrelevant activation in i,l,refinedl,input A_i,l,refined H_l,input. Thus, we further propose a sparse expert-activated mechanism based on a task-aware vector during inference, where the task type of the input is unknown. Given an input l,input H_l,input of layer l, the output of the proposed top-k activation method is given by =(i,l,t⊙i,l,tB)topk(i,l,refinedl,input), y= ( B_i,l,t M^B_i,l,t )topk ( A_i,l,refined H_l,input ), (29) where topk()=⊙ktopk ( x )= x M_k and [Mk]x,y=1 [M_k ]_x,y=1 if []x,y [ x ]_x,y is among the k-th largest value in x, and [Mk]x,y=0 [M_k ]_x,y=0, otherwise. By selecting the top-k expert, the task-irrelevant parts of i,l,refinedl,input A_i,l,refined H_l,input are effectively suppressed. It can be seen as a implicit MoE strategy is achieved without an additional explicit MoE router, but rather through implicit task-aware coding. V Simulation Results For our simulations, we consider a DFL framework with a circular area and uniformly distributed M≤15M≤ 15 devices across the region, each possessing heterogeneous computing and communication capabilities. Each device is equipped with the same pre-trained Qwen 2.5-1.5B-Instruct LLM or Qwen 2.5-7B-Instruct LLM, and a fine-tuning dataset that focuses on four capabilities: (i) natural language understanding, which includes BoolQ, Piqa, and SocialIqa, (i) mathematical reasoning and science, which includes GSM8K, Arc-easy, Arc-challenge, (i) code generation, which includes HumanEval and MBPP, and (iv) comprehensive ability, which includes DollyTails, Hellaswag, and ScienceQA. For comparison, we utilize five baselines : • A decentralized fine-tuning method based on parameter orthogonalization and a sparsification method in LoRI [29], where devices generate a random device connection under communication and computing resources constraints (labelled ”LoRI” in plots). • A decentralized fine-tuning based on traditional LoRA [5] (without parameter orthogonalization and sparsification) and a random device connection under communication resources constraints (labelled ”LoRA” in plots). • A hard-routing MoE mechanism, where a perfect MoE router is introduced that can assigned task to the related experts that are fine-tuned based on the specific task dataset accordingly (labelled ”Hard-routing MoE” in plots). • A sparse-and-orthogonal with implicit MoE mechanism, which has the same set as the proposed method and devices are allocated into the same cluster (labelled ”Proposed (Single Cluster)” in plots). • A full parameter fine-tuning (FPFT) method (labelled ”FPFT” in the plots). Figure 4: Average Parameter Collision Rate vs. the Number of Devices. Figure 5: Spectral Entropy and Sparsity Rate vs. Layers. Figure 6: Test accuracy vs. the Number of Devices. TABLE I: Comparison of different fine-tuning methods using Qwen2.5 models across various benchmarks with rl=32r_l=32 in DFL setting. Bold indicates the best-performing method, and underline indicates the second-best. Method # Params (%) # Trans. (%) BoolQ Piqa SocI.qa GSM8K ARC-E. ARC-C. Hella.S. HumanE. Avg. Qwen2.5-1.5B-Instruct FPFT 3.2G (100%) 3.2G (100%) 79.81 80.72 79.16 77.83 84.49 64.2 87.35 62.05 76.95 LoRA 65.5M (2.01%) 65.5M (2.01%) 76.94 78.59 79.20 71.90 62.30 52.00 68.30 47.90 68.03 LoRI 32M (1.05%) 32M (1.05%) 79.29 81.48 78.64 74.12 82.78 62.11 84.4 58.58 75.17 Hard-routing MoE 3.2G (100%) 3.2G (100%) 83.23 84.87 83.04 82.44 87.56 69.12 90.91 66.29 80.93 Proposed (Single) 16M (0.54%) 8M (0.27%) 80.21 81.69 78.38 78.21 82.95 64.25 84.74 61.09 76.44 Proposed Method 16M (0.54%) 8M (0.27%) 81.45 83.36 79.49 80.04 84.41 65.97 86.09 62.67 78.00 Qwen2.5-7B-Instruct FPFT 14.2G (100%) 14.2G (100%) 84.25 85.65 83.96 92.57 93.31 73.29 91.8 87.57 86.55 LoRA 362M (2.5%) 362M (2.5%) 81.32 83.88 80.99 88.56 89.38 70.25 88.18 82.95 83.19 LoRI 168M (1.16%) 168M (1.16%) 82.72 83.22 83.4 90.33 90.82 70.56 89.12 84.67 84.36 Hard-routing MoE 14.2G (100%) 14.2G (100%) 87.67 90.46 87.11 96.84 96.95 78.21 95.58 92.13 90.62 Proposed (Single) 87M (0.60%) 42M (0.30%) 83.33 85.74 83.74 91.14 90.69 75.17 89.78 88.08 85.96 Proposed Method 87M (0.60%) 42M (0.30%) 84.76 86.86 85.52 92.49 92.6 76.23 91.32 89.35 87.39 TABLE I: Comparison of different fine-tuning methods using Qwen2.5 models across various benchmarks with rl=32r_l=32 in single task. Bold indicates the best-performing method, and underline indicates the second-best. Method # Params (%) BoolQ Piqa SocI.qa GSM8K ARC-E. ARC-C. Hella.S. HumanE. Avg. Qwen2.5-1.5B-Instruct FPFT 3.2G (100%) 83.23 84.87 83.04 82.44 87.56 69.12 90.91 66.29 80.93 LoRA 65.5M (2.01%) 80.09 83.65 79.33 75.78 84.41 63.38 85.15 60.5 76.53 LoRI 32M (1.05%) 81.72 84.19 80.82 77.07 85.14 64.93 86.47 61.17 77.69 Proposed Method 16M (0.54%) 82.88 84.54 81.25 81.36 86.30 67.02 87.63 63.94 79.37 Qwen2.5-7B-Instruct FPFT 14.2G (100%) 87.67 90.46 87.11 96.84 96.95 78.21 95.58 92.13 90.62 LoRA 362M (2.5%) 84.54 87.73 84.40 92.32 92.52 74.18 91.76 86.22 86.71 LoRI 168M (1.16%) 85.15 87.03 85.55 93.89 93.60 74.48 92.26 87.34 87.41 Proposed Method 87M (0.06%) 86.18 88.71 86.79 94.08 93.73 77.97 93.23 90.71 88.93 In Fig. 4, we show how the average test accuracy changes as the number of devices varies. From Fig. 4 we can also observe that the hard-routing MoE and FPFT have the highest and lowest average performance, respectively, and their performance does not change with the number of devices. This is due to the fact that the hard-routing MoE mechanism employs a perfect MoE-router that can assign the tasks to the most related expert, which can effectively prevent catastrophic knowledge forgetting and multi-task knowledge interference. On the other hand, FPFT suffers from the heavily catastrophic knowledge forgetting and multi-task knowledge interference since parameters sub-space direction and parameter overlap with each other. From Fig. 4, we can observe that the average test accuracy of the considered methods decreases as the number of devices increases, except for the hard-routing MoE and FPFT method. This is due to the fact that catastrophic knowledge forgetting and multi-task knowledge interference occur when more devices participate in the parameter aggregation process, which leads to an average accuracy drop. From Fig. 4, we can see that the proposed method achieves a 1.3%1.3\% performance increase compared with the proposed method (single cluster). This is because the one cluster aggregation may causes unaccaptable parameter collision rate, which also leads to catastrophic knowledge forgetting and multi-task knowledge interference and decreases collaborative fine-tuning performance. Similarly, the proposed method exceeds the LoRA and LoRI methods due to its orthogonal projection matrix design and layer-wise sparsity expansion matrix, respectively. In Fig. 5, we show how the spectral entropy and sparsity rate change across layers in Qwen2.5-7B-Instruct. Specifically, for simplicity, we only present results for the ScienceQA and Arc-challenge tasks. As shown in Fig. 5, the spectral entropy of the ScienceQA and Arc-challenge tasks exhibits a complementary and alternating pattern across layers, rather than following a synchronized trend. Specifically, in the initial layers, the ScienceQA task exhibits lower spectral entropy compared to the higher values of the Arc-challenge task; this relationship is reversed in the final layers, where the two tasks mirror each other’s earlier behaviour. This implies the difference between tasks’ requirements across layers, and leaves room for adaptive sparsity rate allocation accordingly. From Fig. 5, we can also observe that the proposed method reallocates the sparsity rate of these tasks across layers based on their spectral entropy distribution. On the other hand, he considered baseline LoRA and LoRI employ a uniform and non-sparse strategy across layers, respectively, which caused a mismatch between sthe parsity rate and the task requirements of different layers. Similar to the observation from Fig.4, which indicates that our proposed method reallocates the sparsity rate across layers, can achieve a lower parameter collision rate and higher performance. In Fig. 6, we show how the average parameter collision rate changes as the number of devices varies. From Fig. 6, we can observe that the proposed method achieves the lowest average parameter collision rate compared with the LoRI and LoRA method, which is strictly constrained by the predefined threshold SmaxS_ , whereas the other baselines exhibit higher parameter collision rates. This is due to the fact that the proposed algorithm can reduce the parameter collision rate through cluster-based DFL device connection topology design, which considers each device’s sparsity rate and the number of devices in the cluster. In contrast, the average parameter collision rate of LoRI increases as the number of devices participating in the DFL increases, while LoRA, without parameter sparsity, consistently experiences a 100%100\% parameter collision rate. Due to the increased parameter collision rate between devices, the corresponding performance is shown in Fig. 4. As the average parameter collision rate of LoRI increases as the number of devices participating in the DFL increases, while LoRA, lacking parameter sparsity, consistently experiences a 100%100\% parameter collision rate. Taking both Fig. 4 and Fig. 6 into consideration, we can see that the proposed cluster method can significantly improve the performance of the static-and-orthogonal projection matrix and layer-wise sparsity expansion matrix design by comparing the performance between the proposed method and the proposed (single cluster) scheme. We can also observe from both Fig. 4 and Fig. 6 that the cluster method and subspace orthogonal projection matrix can effectively reduce parameter interference and enable convergence of fine-tuning. In Tab. I and Tab. I, we show the detailed multi-task DFL average performance and single-task fine-tuning performance without DFL, respectively. In particular, we show the detailed performance of each task that contains 1010 devices fine-tuning on different strategies in the DFL scenario in Tab. I. From Tab. I, we can observe that the proposed method can reduce the number of parameters that need to be updated compared to the traditional LoRA method and FPFT method by up to 73%73\% and 99.46%99.46\%, respectively. We can also observe that the proposed method can reduce the number of parameters that need to be transmitted compared to the traditional LoRA method and FPFT method by up to 86.56%86.56\% and 99.7%99.7\%, respectively. This is due to the fact that in the proposed method, i,l A_i,l is kept static during fine-tuning, which does not need to be updated and transmitted, which can reduce both the computing and communication overhead. From Tab. I, we can observe that despite a lower computational budget, the proposed method outperforms traditional LoRA and LoRI baseline variants with the same rank across all datasets. The outperformance of the LoRA is due to the static projection matrix, which will not compromise accuracy but also achieve a more stable convergence than LoRA, which has twice the number of parameters to be fine-tuned. It can also prove that compared to the broader parameter space of the pre-trained model, the fixed parameter sub-spaces projected by fixed i,l A_i,l are affordable for the specific task. Besides, the fixed i,l A_i,l with a lower degree of freedom can achieve a stable convergence and higher performance compared to the traditional methods. The outperformance of the proposed method compared to the LoRI method can be attributed to the layer-wise sparsity expansion design. In particular, the layer-wise sparsity expansion design can achieve an efficient sparsity assignment across layers, thus makes an task-orient sparsity scheme that can achieve a better performance. VI Conclusion In this paper, we developed a novel DFL framework that enables devices to collaboratively fine-tune an LLM via exchanging their local updates based on heterogeneous datasets with their neighboring device without reliance on a centralized server. We formulated an optimization problem whose goal is to minimize fine-tuning loss across datasets while meeting the delay, transmit power, and computing power resource constraints in each DFL round. Then, we identified three key problems around the life-cycle of DFL: (i) catastrophic forgetting during fine-tuning, (i) communication inefficiencies during aggregation, and (i) multi-task interference during inference. To solve these problems, we first introduced a static-and-orthogonal along with a layer-wise sparsity LoRA method to ensure update orthogonality and reduce parameter overlap while reducing the communication overhead. Then, we designed a cluster-based device connection topology based on the analysis of the impact of device connection and multi-task performance. Finally, we proposed a task-aware coding and implicit MoE mechanism that can activate the specified expert and filter the unrelated parameters during inference. Simulation results demonstrated that the proposed algorithm can achieve robust DFL compared to traditional LoRA and LoRI methods, while ensuring lower computational and communication resource consumption. For future work, we will explore more generalized scenarios where local fine-tuning is unavailable. In such cases, lightweight techniques, specifically textual inversion and collaborative inference, will be central to our problem formulation. VII Appendix VII-A Proof of Theorem 1 Under assumptions, we have ∑i∈ℳi,tEji,li,l,t⊙i,tΓ1jℳi,tji,l∑j∈ℳi,tj,l,t⊙j,tj2 Σ _i _i,tE| A_i,l B_i,l,t M_i,t\!\!-\!\! 1|M_i,t| A_i,l\!\!Σ _j _i,t B_j,l,t\!\! \!\! M_j,t|^2 (30) ≤ ≤ ∑i∈ℳi,t2Eji,li,l,t⊙i,tΓ1jℳi,tji,l∑j∈ℳi,tj,l,t⊙i,tj \!\!Σ _i _i,t\!\!2E| A_i,l B_i,l,t M_i,t\!-\! 1|M_i,t| A_i,l\!\!Σ _j _i,t B_j,l,t\!\! \!\! M_i,t| +∑i∈ℳi,t2Ej1jℳi,tji,l∑j∈ℳi,tj,l,t⊙i,t +Σ _i _i,t2E| 1|M_i,t| A_i,lΣ _j _i,t B_j,l,t M_i,t Γ1jℳi,tji,l∑j∈ℳi,tj,l,t⊙j,tj. - 1|M_i,t| A_i,lΣ _j _i,t B_j,l,t M_j,t|. For the first term in (LABEL:proof_1), we have ∑i∈ℳi,tEji,li,l,t⊙i,tΓ1jℳi,tji,l∑j∈ℳi,tj,l,t⊙i,tj Σ _i _i,tE| A_i,l B_i,l,t M_i,t\!\!-\!\! 1|M_i,t| A_i,l\!\!\!\!Σ _j _i,t\!\! B_j,l,t M_i,t| (31) = = ∑i∈ℳi,tEjji,l1jℳi,tj∑j∈ℳi,t(i,l,tΓj,l,t)⊙(j,t⊙i,t Σ _i _i,tE| A_i,l 1|M_i,t|Σ _j _i,t ( B_i,l,t- B_j,l,t ) ( M_j,t M_i,t +j,tΓi,tj⊙i,t)j + | M_j,t- M_i,t | M_i,t)| = = ∑i∈ℳi,tEji,l1jℳi,tj∑j∈ℳi,t(i,l,tΓj,l,t)⊙i,j,tcj. Σ _i _i,tE| A_i,l 1|M_i,t|Σ _j _i,t ( B_i,l,t- B_j,l,t ) M_i,j,t^c|. Due to the definition of parameter collision rate, (31) can be further given by rk∑i∈ℳi,t∑j∈ℳi,tSℳi,t,lEji,l(i,l,tΓ1jℳi,tj∑j∈ℳi,tj,l,t)j rkΣ _i _i,t\!\!\!Σ _j _i,tS_M_i,t,lE| A_i,l ( B_i,l,t\!-\! 1|M_i,t|Σ _j _i,t\!\!\! B_j,l,t )| (32) = = rk∑i∈ℳi,t∑j∈ℳi,tSℳi,t,lEjΓηi,l∇i,l,tFij rkΣ _i _i,t\!\!\!Σ _j _i,tS_M_i,t,lE|-η A_i,l _ B_i,l,tF_i| ≤ ≤ rk∑i∈ℳi,t∑j∈ℳi,tSℳi,t,lG. rkΣ _i _i,t\!\!\!Σ _j _i,tS_M_i,t,lG. The second equation is due to the fact that i,l A_i,l is independent to 1|ℳi,t|∑j∈ℳi,t(i,l,tΓj,l,t)jj,tΓi,tj⊙i,t 1|M_i,t|Σ _j _i,t ( B_i,l,t- B_j,l,t ) | M_j,t- M_i,t | M_i,t. Similarly, we can rewrite the second term in (LABEL:proof_1) as ∑i∈ℳi,tEj1jℳi,tji,l∑j∈ℳi,tj,l,t⊙j,t Σ _i _i,t\!\!\!E|\! 1|M_i,t| A_i,l\!\!\!Σ _j _i,t\!\!\! B_j,l,t M_j,t (33) Γ1jℳi,tji,l∑j∈ℳi,tj,l,t⊙i,tj≤rk∑i∈ℳi,tSℳi,tP. -\! 1|M_i,t| A_i,l\!\!\!\!Σ _j _i,t\!\!\! B_j,l,t M_i,t|≤ rkΣ _i _i,tS_M_i,tP. Thus, we have ∑i∈ℳi,tEji,li,l,t⊙i,tΓ1jℳi,tji,l∑i∈ℳi,tj,l,t⊙j,tj2 Σ _i _i,tE| A_i,l B_i,l,t M_i,t\!-\! 1|M_i,t| A_i,l\!\!Σ _i _i,t\!\! B_j,l,t M_j,t|^2 (34) ≤2rk∑i∈ℳi,t∑j∈ℳi,tSℳi,t,l(G+P). ≤ 2rkΣ _i _i,tΣ _j _i,tS_M_i,t,l (G+P ). This ends the proof. VII-B Proof of Theorem 2 By the definition of the Frobenius inner product: ⟨i,l,t,j,l,t⊤⟩F=Tr(i,l,t⊤,j,l,t) < w_i,l,t, w_j,l,t >_F=Tr ( w_i,l,t , w_j,l,t ) (35) = = Tr((j,l,t⊙j,l,tB)j,l,refinedi,l,refined⊤(i,l,t⊙i,l,tB)⊤) ( ( B_j,l,t M_j,l,t^B ) A_j,l,refined A_i,l,refined ( B_i,l,t M_i,l,t^B ) ) = = Tr((j,l,t⊙j,l,tB)[j,l0]⊤jj,l0jj,l⊤i,li,l0ji,l0j(i,l,t⊙i,l,tB)⊤) \! (\!\! ( B_j,l,t\! \! M_j,l,t^B )\!\! [ H^0_j,l ] || H^0_j,l|| A_j,l A_i,l\!\! H^0_i,l|| H^0_i,l||\! ( B_i,l,t\! \! M_i,l,t^B ) \!\!\! ) = = Tr((j,l,t⊙j,l,tB)[j,l0]⊤jj,l0j(i,lj,l⊤)i,l0ji,l0j(i,l,t⊙i,l,tB)⊤), \! (\!\!\! ( B_j,l,t\! \! M_j,l,t^B )\!\! [ H^0_j,l ] || H^0_j,l||\!\! (\!\!\! A_i,l A_j,l \! )\!\! H^0_i,l|| H^0_i,l||\!\! ( B_i,l,t\!\! \!\! M_i,l,t^B ) \!\!\! ), where i,l0 H^0_i,l is the abbreviation of i,l,input0 H^0_i,l,input. Since elements in i,l A_i,l are random generated from Gauss distribution, i,lj,l⊤=0 A_i,l A_j,l =0. Then, we can prove that ⟨i,l,t,j,l,t⊤⟩F=Tr(i,l,t⊤,j,l,t)=0 < w_i,l,t, w_j,l,t >_F=Tr ( w_i,l,t , w_j,l,t )=0. This completes the proof. References [1] K. B. Kan, H. Mun, G. Cao, and Y. Lee, “Mobile-LLaMA: Instruction Fine-Tuning Open-Source LLM for Network Analysis in 5G Networks,” IEEE Network, vol. 38, no. 5, p. 76–83, July 2024. [2] Y. Su, Y. Du, Y. Deng, and M. Dohler, “Towards Communication Efficient Multi-Agent Cooperations: Reinforcement Learning and LLM,” IEEE Transactions on Vehicular Technology, p. 1–15, Oct. 2025, early access. [3] Y. Huang, H. Du, X. Zhang, D. Niyato, J. Kang, Z. Xiong, S. Wang, and T. Huang, “Large Language Models for Networking: Applications, Enabling Techniques, and Challenges,” IEEE Network, vol. 39, no. 1, p. 235–242, July 2024. [4] N. Xiong, Y. Zhou, H. Zeng, Z. Chen, F. Huang, S. Bi, L. Zhang, and Z. Zhao, “Token-Level LLM Collaboration via FusionRoute,” ArXiv, vol. 2601.05106, Jan. 2026. [5] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen, “LoRA: Low-Rank Adaptation of Large Language Models,” vol. 2106.09685, Oct. 2021. [6] J. Hu, D. Wang, Z. Wang, X. Pang, H. Xu, J. Ren, and K. Ren, “Federated Large Language Model: Solutions, Challenges and Future Directions,” IEEE Wireless Communications, vol. 32, no. 4, p. 82–89, Aug. 2025. [7] M. Chen, H. V. Poor, W. Saad, and S. Cui, “Wireless communications for collaborative federated learning,” IEEE Communications Magazine, vol. 58, no. 12, p. 48–54, Dec. 2020. [8] E. Holton, L. Braun, J. Thompson, J. Grohn, and Christopher C. Summerfield, “Humans and neural networks show similar patterns of transfer and interference during continual learning,” Nature Human Behaviour, vol. 10, no. 1, p. 111–125, Oct. 2025. [9] C. Zhang, X. Lyu, Z. Liang, C. Ren, Y. Hou, and Q. Cui, “Diameter-Constrained Topology Orchestration for Communication-Convergence Tradeoffs in Decentralized Federated Learning,” IEEE Transactions on Cognitive Communications and Networking, vol. 12, p. 5393–5407, Jan. 2026. [10] J. Feng, Z. Pu, T. Hu, D. Li, X. Ai, and H. Wang, “OMoE: Diversifying Mixture of Low-Rank Adaptation by Orthogonal Finetuning,” ArXiv, vol. 2501.10062, Jul. 2025. [11] M. Farajtabar, N. Azizan, A. Mott, and A. Li, “Orthogonal Gradient Descent for Continual Learning,” ArXiv, vol. 1910.07104, Oct. 2019. [12] X. Wang, T. Chen, Q. Ge, H. Xia, R. Bao, R. Zheng, Q. Zhang, T. Gui, and X. Huang, “Orthogonal subspace learning for language model continual learning,” in Findings of the Association for Computational Linguistics: EMNLP, Singapore, Dec. 2023. [13] Y.-S. Liang and W.-J. Li, “Adaptive Plasticity Improvement for Continual Learning,” in Conference on Computer Vision and Pattern Recognition (CVPR), Vancouver, BC, Canada, Aug. 2023. [14] S. Yang, K. Ning, Y. Liu, J. Yao, Y. Tian, Y. Song, and L. Yuan, “Is Parameter Collision Hindering Continual Learning in LLMs?,” ArXiv, vol. 2410.10179, Dec. 2024. [15] N. Yang, S. Wang, Y. Liu, C. G. Brinton, C. Yin, and M. Chen, “Graph neural networks for the optimization of collaborative federated learning energy efficiency,” IEEE Transactions on Mobile Computing, vol. 24, no. 11, p. 12049–12060, June 2025. [16] X. Wang, X. Li, Z. Zhou, C. Li, and Y. Liu, “ADF-LoRA: Alternating Low-Rank Aggregation for Decentralized Federated Fine-Tuning,” ArXiv, vol. 2511.18291, Nov. 2025. [17] S. Inderjeet, V.-G. Eleonore, O. Andikan, and S. Motoyoshi, “Learning to Collaborate: An Orchestrated-Decentralized Framework for Peer-to-Peer LLM Federation,” ArXiv, vol. 2601.17133, Jan. 2026. [18] G. Hu, Y. Teng, P. Wu, and N. Wang, “FFT-MoE: Efficient Federated Fine-Tuning for Foundation Models via Large-scale Sparse MoE under Heterogeneous Edge,” ArXiv, vol. 2508.18663, Aug. 2025. [19] L. Xie, T. Luan, W. Cai, G. Yan, Z. Chen, N. Xi, Y. Fang, Q. Shen, Z. Wu, and J. Yuan, “dFLMoE: Decentralized Federated Learning via Mixture of Experts for Medical Data Analysis,” in Conference on Computer Vision and Pattern Recognition (CVPR), Nashville, TN, USA, Aug. 2025. [20] S. Yuan, Y. Zheng, T. Wang, B. Liu, and H. Xu, “MoORE: SVD-based Model MoE-ization for Conflictand Oblivion-Resistant Multi-Task Adaptation,” ArXiv, vol. 2506.14436, Oct. 2025. [21] N. Yan, Y. Su, Y. Deng, and R. Schober, “Federated Fine-Tuning of LLMs: Framework Comparison and Research Directions,” IEEE Communications Magazine, vol. 63, no. 10, p. 52–58, Sep. 2025. [22] Z. Hu, L. Zhang, S. Dai, S. Gong, and Q. Shi, “FedQLoRA: Federated Quantization-Aware LoRA for Large Language Models,” in International Conference on Learning Representations (ICLR), Singapore, Apr. 2025. [23] X. Yi, C. Hu, B. Cai, H. Huang, Y. Chen, and K. Wang, “FedALoRA: Adaptive Local LoRA Aggregation for Personalized Federated Learning in LLM,” IEEE Internet of Things Journal, June 2025. [24] B. Kim and W. Choi, “Communication-Efficient Wireless Federated Fine-Tuning for Large-Scale AI Models,” ArXiv, vol. 2505.00333, 2017. [25] Z. Zhang, P. Liu, J. Xu, and R. Hu, “Fed-HeLLo: Efficient Federated Foundation Model Fine-Tuning With Heterogeneous LoRA Allocation,” Transactions on Neural Networks and Learning Systems, vol. 36, no. 10, p. 17556–17569, July 2025. [26] J. Liu, R. Li, H. Xu, Q. Ma, J. Yan, and L. Huang, “FedQuad: Adaptive Layer-wise LoRA Deployment and Activation Quantization for Federated Fine-Tuning,” IEEE Transactions on Mobile Computing, p. 1–15, Nov. 2025, early access. [27] M. Chen, Z. Yang, W. Saad, C. Yin, H. V. Poor, and S. Cui, “A Joint Learning and Communications Framework for Federated Learning Over Wireless Networks,” IEEE Transactions on Wireless Communications, vol. 20, no. 1, p. 269–283, Oct. 2021. [28] H. B. McMahan, M. Eider, R. Daniel, H. Seth, and A. A. Blaise, “Communication-Efficient Learning of Deep Networks from Decentralized Data,” vol. abs/1602.05629, Feb. 2016. [29] J. Zhang, J. You, A. Panda, and T. Goldstein, “LoRI: Reducing Cross-Task Interference in Multi-Task Low-Rank Adaptation,” in Conference on Language Modeling (COLM), Montreal, Canada, Oct. 2025. [30] H. Zou, Y. Zang, W. Xu, Y. Zhu, and X. Ji, “FlyLoRA: Boosting Task Decoupling and Parameter Efficiency via Implicit Rank-Wise Mixture-of-Experts,” ArXiv, vol. 2510.08396, Oct. 2025. [31] J. Liang, W. Huang, X. Guo, G. Wan, B. Du, and M. Ye, “ThanoRA: Task Heterogeneity-Aware Multi-Task Low-Rank Adaptation,” ArXiv, vol. 2505.18640, Sep. 2025.