Paper deep dive
Task-Centric Personalized Federated Fine-Tuning of Language Models
Gabriel U. Talasso, Meghdad Kurmanji, Allan M. de Souza, Nicholas D. Lane, Leandro A. Villas
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 96%
Last extracted: 4/2/2026, 3:22:26 AM
Summary
FedRouter is a task-centric personalized federated learning framework designed to address generalization failures and intra-client task interference in large language model fine-tuning. By employing local and global clustering mechanisms, it creates specialized adapters for specific tasks rather than individual clients, and utilizes an adaptive evaluation router to route test samples to the most relevant adapter, significantly improving performance under distribution shifts.
Entities (5)
Relation Signals (3)
FedRouter â implements â Federated Learning
confidence 98% · we propose FedRouter, a clustering-based pFL that builds specialized models for each task
FedRouter â uses â LoRA
confidence 95% · FedRouter uses adapters to personalize models... such as the Low-Rank Adapters (LoRA)
FedRouter â utilizes â Flower
confidence 95% · We implement FedRouter using the Flower framework
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Federated Learning (FL) has emerged as a promising technique for training language models on distributed and private datasets of diverse tasks. However, aggregating models trained on heterogeneous tasks often degrades the overall performance of individual clients. To address this issue, Personalized FL (pFL) aims to create models tailored for each client's data distribution. Although these approaches improve local performance, they usually lack robustness in two aspects: (i) generalization: when clients must make predictions on unseen tasks, or face changes in their data distributions, and (ii) intra-client tasks interference: when a single client's data contains multiple distributions that may interfere with each other during local training. To tackle these two challenges, we propose FedRouter, a clustering-based pFL that builds specialized models for each task rather than for each client. FedRouter uses adapters to personalize models by employing two clustering mechanisms to associate adapters with specific tasks. A local clustering that associate adapters with task data samples and a global one that associates similar adapters from different clients to construct task-centric personalized models. Additionally, we propose an evaluation router mechanism that routes test samples to the best adapter based on the created clusters. Experiments comparing our method with existing approaches across a multitask dataset, FedRouter demonstrate strong resilience in these challenging scenarios performing up to 6.1% relatively better under tasks interference and up to 136% relative improvement under generalization evaluation.
Tags
Links
- Source: https://arxiv.org/abs/2604.00050v1
- Canonical: https://arxiv.org/abs/2604.00050v1
Trouble viewing inline? Open PDF directly â
Full Text
36,262 characters extracted from source content.
Expand or collapse full text
Task-Centric Personalized Federated Fine-Tuning of Language Models Gabriel U. Talasso 1â Meghdad Kurmanji 2 Allan M. de Souza 1 Nicholas D. Lane 2,3 Leandro A. Villas 1 1 Universidade Estadual de Campinas 2 University of Cambridge 3 Flower Labs Abstract Federated Learning (FL) has emerged as a promising technique for training lan- guage models on distributed and private datasets of diverse tasks. However, ag- gregating models trained on heterogeneous tasks often degrades the overall per- formance of individual clients. To address this issue, Personalized FL (pFL) aims to create models tailored for each clientâs data distribution. Although these ap- proaches improve local performance, they usually lack robustness in two aspects: (i) generalization: when clients must make predictions on unseen tasks, or face changes in their data distributions, and (i) intra-client tasks interference: when a single clientâs data contains multiple distributions that may interfere with each other during local training. To tackle these two challenges, we propose FedRouter, a clustering-based pFL that builds specialized models for each task rather than for each client. FedRouter uses adapters to personalize models by employing two clustering mechanisms to associate adapters with specific tasks. A local clustering that associate adapters with task data samples and a global one that associates simi- lar adapters from different clients to construct task-centric personalized models. Additionally, we propose an evaluation router mechanism that routes test samples to the best adapter based on the created clusters. Experiments comparing our method with existing approaches across a multitask dataset, FedRouter demon- strate strong resilience in these challenging scenarios performing up toâŒ6.1% relatively better under tasks interference and up toâŒ136% relative improvement under generalization evaluation. 1 Introduction Foundation models have gained significant attention in recent years, particularly due to their remark- able ability to be applied across diverse domains. In particular, Large Language Models (LLMs) have been successfully employed in a wide range of applications such as mobile devices, healthcare, and law Zhuang et al. [2023]. To achieve strong performance in these domains, LLMs are fine-tuned, in a post-training process, for specific tasks to enhance their specialization and effectiveness Hu et al. [2022], Tian et al. [2024]. However, this adaptation process requires access to large amounts of high-quality, domain-specific data to properly align the modelâs behavior with the target task Ye et al. [2024]. In this context, Federated Learning (FL) emerges as a promising paradigm for adapting foundation models, as it enables access to distributed, high-quality datasets located across multiple clients while preserving data privacy by sharing only model parameters rather than raw data Zhuang et al. [2023], Ye et al. [2024], Iacob et al. [2024], Sani et al. [2025]. â Correspondence: g235078@dac.unicamp.br Preprint. Under review. arXiv:2604.00050v1 [cs.LG] 30 Mar 2026 Client 2 Client 3 Client 1 Client 1 Client 2 Client 3 Base Model Embed- dings Client's Data Base Model Embed- dings Client's Data Base Model Embed- dings Client's Data Client's Local Centroids Client's Cluster Adapters Global Aggregated Task Adapters Global Centroids 1 2 3 4 5 6 Figure 1: FedRouter Workflow Overview. Each client first computes embeddings from its local data and applies clustering to partition the dataset into task-specific subsets. The client then sends the resulting centroids and adapters to the server, which performs global clustering to associate similar tasks across clients and aggregate their corresponding adapters collaboratively. Finally, the server sends the updated adapters back to the clients, which then associate each received model with the appropriate local task-specific dataset for next round of training. Since fine-tuning large-scale models is expensive, Parameter-Efficient Fine-Tuning (PEFT) Xu et al. [2023] methods, such as the Low-Rank Adapters (LoRA) Hu et al. [2022], have been explored to significantly reduce resource demands and have also been adapted to FL scenarios Ye et al. [2024], Iacob et al. [2024], Zhang et al. [2024], Kuang et al. [2024]. A second major challenge arises from the data heterogeneity across clients in FL, where datasets may differ in domains, distributions, or even underlying tasks. This variability often leads to degraded performance when aggregating locally trained models into a unique global one. To mitigate this issue, state-of-the-art approaches focus on personalized FL (pFL), aiming to produce tailored models for each client Sun et al. [2024], Guo et al. [2024], Long et al. [2024]. These methods seek to balance collaborative knowledge sharing with client-specific adaptation, thereby improving overall performance in heterogeneous scenarios Smith et al. [2017]. Personalized FL Challenges.Client-specific models introduce several challenges, including gener- alization and intra-client tasks interference, that we aim to address in this paper. First is generalization. On the one hand, pFL methods tailor models for each client; changes in local data distributions, such as the rise of new tasks at test time, can lead to significant performance degradation, as the models were not optimized for these scenarios Long et al. [2024]. On the other hand, tasks interference arises from divergent tasks within the same clientâs dataset. In such cases, training a single adapter per client is suboptimal, as it must fit to potentially conflicting objectives, similar to multi-task learning techniques Crawshaw [2020], Tian et al. [2024]. This negative interference Crawshaw [2020] often degrades adaptation, consequently reducing overall performance across heterogeneous client datasets. Contributions.We propose FedRouter, a clustering-based pFL method for federated adaptation of LLMs across multiple tasks. Our approach shifts the personalization perspective from a client-centric paradigm, where individual models are tailored for each client, to a task-centric one, where specialized models are created for each task. This design mitigates issues caused by both generalization failures and tasks interference. During training, FedRouter leverages two complementary clustering methods: (i) a local clustering process that partitions each clientâs data into distinct tasks and trains a specialized adapter for each of them to avoid divergences issues; and (i) a global clustering process that groups similar tasks across clients. During inference, we introduce an adaptive router evaluation mechanism with local and global modes. In the local mode, new data samples are routed to the most relevant local task clusters, ensuring personalized inference. In the global mode, samples can be matched to any task cluster present across the federation, enabling generalized inference even under test-time distribution shifts or the presence of previously unseen tasks. Our main contributions are summarized as follows: 2 âąWe first identify and analyze two major challenges within pFL settings:(i)the generalization problem under test-time distribution shifts such as the introduction of new tasks in client test datasets, and(i)the negative impact of intra-client tasks interference, when clients possess multiple, heterogeneous datasets leading to conflicting optimization objectives. âą We propose FedRouter, a task-centric pFL method that leverages both local and global clustering mechanisms to collaboratively train specialized models for each task, effec- tively mitigating generalization issues and intra-client tasks interference. And made the implementation publicly available 2 . âąWe propose a two-mode adaptive inference pipeline, where new samples are dynamically routed either to local adapters for personalization or to the full adapter pool for global generalization, enabling unified personalized and generalized evaluation at test time. âąWe conduct an extensive empirical evaluation of FedRouter under the identified challenging scenarios, demonstrating its superior performance compared to traditional client-centric personalization approaches, performing until 3.5% (âŒ6.1% relative) better under tasks interference and until 33.6% (âŒ136% relative) improvement under generalization evaluation. 2 Related Works Federated Fine-Tuning of Language Models. With the evolution of current language models, there is an increasing need for large amounts of high-quality data for training. In this context, several studies use FL as an approach to access more training data while preserving user privacy McMahan et al. [2017], Iacob et al. [2024]. In particular, works such as FedIT Zhang et al. [2024], OpenFedLLM Ye et al. [2024] and FederatedScope-LLM Kuang et al. [2024] apply FL for post-training of these models, especially by leveraging Parameter-Efficient Fine-Tuning (PEFT) Xu et al. [2023], Hu et al. [2022] techniques to improve training efficiency and reduce communication costs. These studies highlight the limitations related to fine-tuning models across multiple domains and tasks, claiming the need for new personalization approaches tailored for these scenarios Tan et al. [2022]. Personalized Federated Language Models. Based on the limitations of fine-tuning language models using FL in heterogeneous tasks and domains, some solutions address this problem by creating personalized models for each client or group of clients. The first class of works focuses on creating models for clusters of similar clients by measuring their similarity and aggregating models separately, thus avoiding the negative interference caused by different data distributions Sattler et al. [2020], Ghosh et al. [2020], Talasso et al. [2024, 2025]. Other approaches personalize the model by dividing the total training parameters into two groups: one to learn the general knowledge shared across the federation and another to learn the specific knowledge of each client. For example, FedDPA Long et al. [2024] proposes a globally shared adapter trained individually on each client, along with a local adapter trained with consideration of the global one to adapt the model to local task distributions. On the other hand, FFA-LoRA Sun et al. [2024] and FedSA Guo et al. [2024] train part of an adapter locally (the B matrix of LoRA in both cases) while either freezing (in FFA-LoRA) or sharing (in FedSA) the A matrix, which represents the general knowledge. Although these solutions represent advances in training for multiple tasks in a federated manner, several challenges remain for such approaches. The first concerns generalization: although in some cases clients share a common component (as in FedDPA and FedSA), changes in data distributions and the addition of new tasks lead to a significant drop in the performance of models that are specific to each client. Another challenge involves the existence of multiple domains or tasks locally, which hinders model optimization even before sharing and aggregation step. These open challenges are the focus of our proposal in this work. 3 FedRouter We propose FedRouter, a task-centric personalization approach that leverages clustering to create specialized models in FL. FedRouter improves both generalization and tasks interference challenges in the presence of heterogeneous data and unseen tasks and domains. Our proposed approach is composed of three main components (i) Local clustering of raw data (i) Global clustering of 2 Code base available after acceptance. 3 task centroids and (i) Evaluation Router Mechanism. Additionally, each of these components is composed of steps that are described in Figure 1 for clustering mechanisms and Figure 2 for routing details. 3.1 Local Clustering Mechanism Data can be produced by various applications and sources on the clientâs side. Consequently, it is not reasonable to assume that all clients have their data separated by task in the same way. Furthermore, a single task can be subdivided into several others or grouped with similar tasks to improve model performance. As a result, clientsâ data may be mixed, and the definition of tasks and subtasks can become ambiguous. To address this problem, FedRouterfirstly aims to separate different tasks within each clientâs local dataset in order to train specialized adapters for each of them. To this end, the first step 1 involves computing the embeddings of the training data using the pre-trained base model. This step is performed only once as a prerequisite to starting the federation and results in a set of embeddings (E i ) for clienti, with dimensionsD(dataset length) byE(model- dependent embedding size). Next, in the second step2, clients perform local clustering on the setE i using the clustering algorithmK(E i |n l ), wheren l denotes the number of local clusters to be created. In our case, we setKto K-Means, without loss of generality, since other methods capable of generating a centroid representing each cluster could also be used. Additionally, the number of clusters may vary across clients depending on the degree of local task heterogeneity. More details about these hyperparameters are provided in the results section. Finally, once the embeddings have been computed and local clusters created, the third step 3 involves training a specialized adapter for each cluster and sharing the corresponding parameters A i and the set of centroidsC i with the server. To avoid excessive communication and computation costs associated with training and transmitting multiple models, we propose a round-robin-based approach, in which only one cluster is trained per round and its adapter is shared. The coordination of which adapter will be trained is made by the server. Thus, in later rounds, each client shares only one centroid and one adapter. 3.2 Global Clustering Mechanism As the clients may have similar tasks of other clients, we need a way to associate the same cluster of tasks with different clients. For this, the second main component of FedRouter is the server-side clustering mechanism, which aims to associate similar tasks across different clients while aggregating the corresponding models for each task. In the fourth step 4 , during the first communication round, all clients share their local centroids, which serve as proxies for each clientâs task data. The server then performs another clustering over all shared centroids, denoted asK(C|n g ), whereC =C 1 âȘC 2 âȘ ...âȘC N andn g represents the number of global clusters, corresponding to the total number of tasks in the federation. Using the resulting global centroidsGfor each task, the server aggregates the associated adapters through averaging in step5. As in the local phase, we employ K-Means as the clustering algorithm, but other clustering methods could be used without loss of generality. Likewise, while we use averaging for aggregation, other aggregation strategies could also be applied. To avoid broadcasting all adapters to clients in every round, the server selects only the next centroid and its corresponding adapter to be sent in a round-robin-based coordination, i.e, each client will train only one adapter of one task per round, avoiding increasing computer and communication requirements. In a later round, the client may receive other adapters and tasks to be trained. This process is managed by the server, which sends the correct centroids to each client train. Finally, in step6, each client associates the received global centroid with its corresponding local centroid by computing the Euclidean distance between them, and then retrains the associated adapter locally using the data from that local cluster. This cycle is repeated forTrounds or until a defined stopping criterion. 4 3.3 Evaluation Router Mechanism As illustrated in Figure 2, we introduce a novel inference mechanism, which aims to promote generalization in scenarios with test-time distribution shifts Long et al. [2024], where clients must evaluate on data from distributions unseen during local training. Such situations occur when new tasks are introduced in the test datasets or when client data distributions change over time. To address this, FedRouter supports two possible evaluation modes, illustrated in Figure 2, that can be chosen depending on the existence of new tasks on test-time dataset, allowing the generalization. In the first step 1 of both modes, the embeddings of the new test samples are computed only using the pre-trained base model. In the second step2, each embedding is associated with its nearest adapter by finding the minimum Euclidean distance to the centroids, ensuring that each sample is evaluated using the most appropriate task-specific adapter. To improve efficiency when testing multiple data points, all samples are first assigned to their corresponding nearest adapter. Then, each adapter is set once, and the corresponding batches are evaluated together, thereby avoiding unnecessary adapter switching. Finally, in the third step3the nearest chosen adapter is set to model inference process. Option 1: Pesonalized Evaluation Option 2: Generalized Evaluation Base Model Embedding New Data Local Centroids Chosen Adapter 1 Global Centroids 2 3 1 2 3 Figure 2: FedRouter Evaluation Modes. During inference, each client computes the embedding of a new data sample and associates it with the nearest centroid based on the minimum Euclidean distance. The association can be performed using either the local centroids, to obtain a personalized evaluation, or the global centroids, to enable a generalized evaluation across the federation. The difference between the two evaluation modes (local and global) lies only in the centroids used to route the new data. In the local mode, only the clientâs locally computed centroids are available for association, whereas in the global mode, all global centroids are accessible. The local mode enhances routing accuracy in scenarios without test-time distribution shifts, as fewer options lead to more precise associations. Conversely, the global mode improves performance when distribution shifts occur, since it allows the use of adapters associated with previously unseen tasks. 4 Results This section presents an extensive evaluation of our proposed method, FedRouter, under scenarios where most existing pFL approaches struggle to perform effectively. We begin by describing the experimental setup, including the models, datasets, scenarios, and baseline methods for comparison (Section 4.1). Next, we report and discuss the main results, first to answer if FedRouter improves task interference issues (Section 4.2) and then if it improve generalization issues (Section 4.2). Finally, we conduct ablation studies to analyze the scalability and impact of each component and hyperparameter in our framework (Section 4.4). 4.1 Evaluation Setup Datasets. To evaluate data heterogeneity in multitask language model fine-tuning, we select a subset of four tasks from FLAN Wei et al. [2021]: QQP for paraphrase detection, WebNLG for structure-to- text generation, Samsum for dialogue summarization, and GigaWord for general text summarization. Following Long et al. [2024], we use ROUGE-1 as the primary evaluation metric. Baselines. To assess the effectiveness of FedRouter, we compare it against several representative baselines: FedIT Zhang et al. [2024] (the instruction-tuning variant of FedAvg), Local (a non- federated, independent fine-tuning baseline), FedCluster (a clustering-based method inspired by Sattler et al. [2020], Ghosh et al. [2020] and adapted for LLM fine-tuning), FedDPA Long et al. 5 Table 1: Performance comparison (mean±std) across different data scenarios. The last column reports the average performance across scenarios. Best results per column are highlighted in bold. MethodSingleDualAllAverage FedIT0.546± 0.0120.550± 0.0090.560± 0.0100.552 Local0.553± 0.0050.525± 0.0140.534± 0.0050.537 FedCluster0.561± 0.0080.551± 0.0120.553± 0.0120.555 FedSA0.554± 0.0080.530± 0.0100.531± 0.0100.538 FedDPA0.556± 0.0090.551± 0.0130.549± 0.0100.552 FedRouter0.561± 0.0040.558± 0.0190.566± 0.0340.562 FedRouter* 0.562± 0.013 0.563± 0.012 0.575± 0.012 0.567 [2024], and FedSA Guo et al. [2024], which represent state-of-the-art personalized FL approaches. We also compare with a variant, called FedRouter*, to computer budge flexibility to the training. Evaluation Scenarios. We design three evaluation settings to capture different levels of intra-client tasks interference. In the Single scenario, each client has only one task, representing the standard assumption in most prior works, without significant interference. In the Dual scenario, each client has two distinct tasks simultaneously. Finally, in the All scenario, each client has all tasks, resulting in the highest degree of tasks interference. Model Setup. We used Llama 3.2 with 1B parameters in the experiments. All models are instruction- tuned with a maximum sequence length of 1024 tokens. Experiments are performed on an NVIDIA A100 80GB GPU, using a batch size of 16 and a learning rate of5Ă 10 â4 for 10 steps each round. All LoRA adapters are configured with a rank of 8 and an α value of 16. Federated Setup. We implement FedRouter using the Flower framework Beutel et al. [2020] and the OpenFedLLM base code Ye et al. [2024] and made the implementation public available 3 . All experiments were conducted with 8 clients participating in the federation for 25 rounds, where each pair of clients receives a similar dataset under each evaluation scenario. To emulate a realistic data-scarcity condition Long et al. [2024], McMahan et al. [2017], the training data for each client is limited to 600 samples, while the test set contains 300 samples per client. 4.2 Task Interference Results The results, summarized by the average of 5 runs in Table 1, show that FedRouter achieves perfor- mance comparable to other methods in the Single scenario, where no tasks interference is present, demonstrating its ability to provide effective personalization similar to state-of-the-art approaches, especially when compared with no-personalization as FedIT. However, as client divergence increases in the Dual and All scenarios, where clients hold data from multiple tasks, the performance of other methods degrades significantly faster. In contrast, FedRouter consistently outperforms competing methods, achieving the highest scores in most cases and on average across all scenarios. These results highlight the advantages of shifting from a client-centric to a task-centric personalization paradigm, where training specialized models per task, rather than per client, effectively mitigates tasks interference and enhances robustness under heterogeneous data conditions. We also report in Table 1 the FedRouter*, a variant that makes the computation and communication budget of our approach more flexible. In contrast to standard FedRouter, where each client trains and communicates the same number of adapters per round, thereby normalizing the training effort at the client level as described in Section 3.2, FedRouter* updates all adapters available at each client in every round. This normalization criterion moves from the client level to the adapter level budget, ensuring that each adapter receives (in the All Scenario, and proportional for the others) an equivalent amount of training across rounds. The results show that in this scenario, where there is no limitation on standardizing client resources, FedRouter achieves even greater performance, standing out more than other solutions. 3 Code base available after acceptance. 6 4.3 Generalization Results To evaluate test-time distribution shift scenarios, where clients are required to perform inference on unseen tasks, we present in Table 3 the final performance of all compared methods trained on the Single scenario. The results reveal a substantial performance drop in most personalization-based methods, as these approaches train client-specific models that fail to generalize to unseen tasks. Two exceptions are observed: FedIT Zhang et al. [2024], which shows limited degradation due to aggregating updates from clients across all tasks without explicit personalization, and FedDPA Long et al. [2024], which maintains a stable performance through its global adapter and inference mechanism designed to mitigate test-time distribution shifts. Finally, FedRouter demonstrates the most robust generalization among all evaluated methods. By training task-specific adapters and leveraging global evaluation to reuse adapters from unseen tasks, FedRouter achieves superior performance under distribution shifts, reinforcing the advantages of its task-centric personalization strategy in FL. Figure 3:Performance comparison (mean±std) in the single-task training scenario, evaluated on all tasks at test time to assess generalization capability and robustness under test-time distribu- tion shift. MethodTest-Time Gen. FedIT0.570± 0.013 Local0.255± 0.006 FedCluster0.252± 0.008 FedSA0.247± 0.008 FedDPA0.461± 0.009 FedRouter0.583± 0.005 10.07.55.02.50.02.55.07.5 t-SNE Component 1 10.0 7.5 5.0 2.5 0.0 2.5 5.0 7.5 10.0 t-SNE Component 2 t-SNE of Client Local Data Embeddings Client 0 Client 1 Client 2 Client 3 Client 4 Client 5 Client 6 Client 7 Figure 4: t-SNE visualization of client test data embeddings in single scenario. Additionally, we evaluated the quality of the local clustering method to better understand the remaining sources of error that may explain the small performance decay observed in some cases. Figure 4 illustrates the t-SNE of clientsâ local test data embeddings, in the single scenario, showing clearly separated tasks across different clients while showing that similar clients, with the same tasks, remain close together. This supports the fact that our method trains a specific adapter for each task by clustering the local data. Furthermore, the clustering accuracy evaluated on each clientâs test data reached 100%, 100%, and 95.4% for the Single, Dual, and All domain scenarios, respectively. These results indicate that scenarios involving a larger number of tasks are more challenging, often leading to reduced clustering accuracy due to overlapping tasks across clients. 4.4 Ablation Studies To fully explore FedRouterâs behavior under considering different conditions, we performed ablation experiments varying the method and scenario to ensure its effectiveness and stability. Below, we first present experiments related to the scalability of federated scenarios, followed by experiments regarding the correct selection of the number of clusters. 4.5 Scaling Figure 5 shows the performance of FedRouter when scaling the model size. In this experiment, three model sizes of LLamaâs 3 model family (Grattafiori et al. [2024]), 1 billion, 3 billion, and 8 billion parameters models in the âsingleâ evaluation scenario were used, using the presented method, and limiting the batch size to 8. The results showed that FedRouter scales the performance without a bottleneck when scaling the number of parameters, showing that our proposed method can also be used in both small and larger model scenarios and applications. Additionally, in Figure 6 we show the results of scaling the number of clients on the federation and consequently on each cluster, also in the single scenario. The results showed that, due to the 7 1B3B8B Model Size 0.40 0.45 0.50 0.55 0.60 0.65 0.70 Mean ROUGE-1 Score at Round 25 0.518 0.572 0.650 Figure 5: Scaling model size of Llama models using FedRouter in single scenario. 4 (1/c)8 (2/c)16 (4/c) Number of Clients - Total (per cluster) 0.400 0.425 0.450 0.475 0.500 0.525 0.550 0.575 0.600 Mean ROUGE-1 Score at Round 25 0.549 0.563 0.570 Figure 6: Scaling the number of clients, total and per cluster of FedRouter in single scenario. availability of more data to train the specialized adapters per task, the performance improves with more clients, which is beneficial when scaling FedRouter to be used in scenarios with more users. 4.6 Number of Clusters As the number of clusters is a hyperparameter of FedRouter in both the local clustering process and the global clustering process, we also analyze methods to choose the correct number of clusters. Figure 7 shows the results of the Silhouette Score method to find the correct number of global clusters based on the centroids reported by the clients on each of the three proposed scenarios. The results show that in all cases it is possible to clearly find the correct number of clusters, in this case 4, due the good separability of the embeddings tasks as shown previously. It results in improved performance of FedRouter, as it can effectively cluster similar tasks from different clients while avoiding the aggregation of different tasks into the same adapter. 234567 0.4 0.6 0.8 Silhouette Score Single Domain 234567 Number of Global Clusters (k) 0.5 0.6 0.7 0.8 0.9 Dual Domain 234567 0.6 0.7 0.8 All Domain Figure 7: Silhouette Score method to choose the number of global clusters in different scenarios for FedRouter. Additionally, Figure 8 presents the results of applying the Silhouette Score method to find the number of local clusters in the Dual scenario, where the correct number is 2 tasks per client and the all scenario where the correct number is 4, in both cases the method was capable of clearly finding the best number of clusters. We do not consider the single scenario because each client has only one cluster locally, and Silhouette Score is not applied. Confidence bars represent the standard deviation due the measure on different clientsâ local datasets. 5 Conclusion Personalization in FL fine-tuning is promising in heterogeneous and multi-task scenarios, but existing methods often ignore negative interference among tasks within clients and performance degradation under test-time distribution shifts. In this work, we proposed FedRouter, a task-centric personalization framework that combines both local and global clustering mechanisms to train specialized adapters. By structuring adapters around task clusters and enabling routing at inference time, FedRouter mitigates negative transfer and improves test-time generalization. Future work involves exploring scenarios with even more tasks on the federation and the possible cross-task collaboration methods. 8 234567 Number of Local Clusters (k) 0.05 0.10 0.15 0.20 0.25 0.30 0.35 Mean Silhouette Score Dual Domains 234567 Number of Local Clusters (k) 0.100 0.125 0.150 0.175 0.200 0.225 0.250 All Domains Figure 8: Silhouette Score method to choose the number of local clusters in different scenarios for FedRouter. References Weiming Zhuang, Chen Chen, and Lingjuan Lyu. When foundation model meets federated learning: Motivations, challenges, and future directions. arXiv preprint arXiv:2306.15546, 2023. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022. Chunlin Tian, Zhan Shi, Zhijiang Guo, Li Li, and Cheng-Zhong Xu. Hydralora: An asymmetric lora architecture for efficient fine-tuning. Advances in Neural Information Processing Systems, 37: 9565â9584, 2024. Rui Ye, Wenhao Wang, Jingyi Chai, Dihan Li, Zexi Li, Yinda Xu, Yaxin Du, Yanfeng Wang, and Siheng Chen. Openfedllm: Training large language models on decentralized private data via federated learning. In Proceedings of the 30th ACM SIGKDD conference on knowledge discovery and data mining, pages 6137â6147, 2024. Alex Iacob, Lorenzo Sani, Bill Marino, Preslav Aleksandrov, William F Shen, and Nicholas Donald Lane. Worldwide federated training of language models. arXiv preprint arXiv:2405.14446, 2024. Lorenzo Sani, Alex Iacob, Zeyu Cao, Royson Lee, Bill Marino, Yan Gao, Wanru Zhao, Dongqi Cai, Zexi Li, Xinchi Qiu, and Nicholas D. Lane. Photon: Federated LLM pre-training. In Eighth Conference on Machine Learning and Systems, 2025. URLhttps://openreview.net/forum? id=AQgYcfg5EI. Lingling Xu, Haoran Xie, Si-Zhao Joe Qin, Xiaohui Tao, and Fu Lee Wang. Parameter-efficient fine-tuning methods for pretrained language models: A critical review and assessment. arXiv preprint arXiv:2312.12148, 2023. Jianyi Zhang, Saeed Vahidian, Martin Kuo, Chunyuan Li, Ruiyi Zhang, Tong Yu, Guoyin Wang, and Yiran Chen. Towards building the federatedgpt: Federated instruction tuning. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6915â6919. IEEE, 2024. Weirui Kuang, Bingchen Qian, Zitao Li, Daoyuan Chen, Dawei Gao, Xuchen Pan, Yuexiang Xie, Yaliang Li, Bolin Ding, and Jingren Zhou. Federatedscope-llm: A comprehensive package for fine-tuning large language models in federated learning. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 5260â5271, 2024. Youbang Sun, Zitao Li, Yaliang Li, and Bolin Ding. Improving lora in privacy-preserving federated learning. arXiv preprint arXiv:2403.12313, 2024. Pengxin Guo, Shuang Zeng, Yanran Wang, Huijie Fan, Feifei Wang, and Liangqiong Qu. Selective aggregation for low-rank adaptation in federated learning. arXiv preprint arXiv:2410.01463, 2024. Guodong Long, Tao Shen, Jing Jiang, Michael Blumenstein, et al. Dual-personalizing adapter for federated foundation models. Advances in Neural Information Processing Systems, 37:39409â 39433, 2024. 9 Virginia Smith, Chao-Kai Chiang, Maziar Sanjabi, and Ameet S Talwalkar. Federated multi-task learning. Advances in neural information processing systems, 30, 2017. Michael Crawshaw. Multi-task learning with deep neural networks: A survey. arXiv preprint arXiv:2009.09796, 2020. Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-Efficient Learning of Deep Networks from Decentralized Data. In Aarti Singh and Jerry Zhu, editors, Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, volume 54 of Proceedings of Machine Learning Research, pages 1273â1282. PMLR, 20â22 Apr 2017. URL https://proceedings.mlr.press/v54/mcmahan17a.html. Alysa Ziying Tan, Han Yu, Lizhen Cui, and Qiang Yang. Towards personalized federated learning. IEEE transactions on neural networks and learning systems, 34(12):9587â9603, 2022. Felix Sattler, Klaus-Robert MĂŒller, and Wojciech Samek. Clustered federated learning: Model- agnostic distributed multitask optimization under privacy constraints. IEEE transactions on neural networks and learning systems, 32(8):3710â3722, 2020. Avishek Ghosh, Jichan Chung, Dong Yin, and Kannan Ramchandran. An efficient framework for clustered federated learning. Advances in neural information processing systems, 33:19586â19597, 2020. Gabriel U. Talasso, Allan M. de Souza, Luiz F. Bittencourt, Eduardo Cerqueira, Antonio A. F. Loureiro, and Leandro A. Villas. Fedsccs: Hierarchical clustering with multiple models for federated learning. In ICC 2024 - IEEE International Conference on Communications, pages 3280â3285, 2024. doi: 10.1109/ICC51166.2024.10622346. Gabriel U Talasso, Allan M de Souza, Luis FG Gonzalez, Eduardo Cerqueira, Antonio AF Loureiro, and Leandro A Villas. Leveraging federated learning for multilingual and private language models via model clustering. In 2025 3rd International Conference on Federated Learning Technologies and Applications (FLTA), pages 25â32. IEEE, 2025. Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652, 2021. Daniel J Beutel, Taner Topal, Akhil Mathur, Xinchi Qiu, Javier Fernandez-Marques, Yan Gao, Lorenzo Sani, Kwing Hei Li, Titouan Parcollet, Pedro Porto Buarque De GusmĂŁo, et al. Flower: A friendly federated learning research framework. arXiv preprint arXiv:2007.14390, 2020. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. 10