Paper deep dive
MoE Router-Guided Clustering for Heterogeneous Federated Instruction Tuning
Ankita Sharma, Bahar Farahani, Sanaz Rahimi Moosavi, Amir Rrahmani, Farshad Firouzi, Krishnendu Chakrabarty
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%
Last extracted: 8/18/2026, 6:11:34 AM
Summary
The paper introduces ClientMorpher, a routing-aware personalized federated instruction fine-tuning framework for Mixture-of-Experts (MoE) Large Language Models. It addresses the challenge of negative transfer in heterogeneous federated settings by leveraging MoE routing signatures to cluster clients or experts before aggregation. Two strategies are proposed: ClientMorpher-C, which clusters clients based on expert activation profiles, and ClientMorpher-E, which clusters experts based on cross-client usage signatures to derive client groups. Experiments on the Databricks Dolly-15K dataset demonstrate that these methods improve personalized performance over conventional federated averaging and local training while maintaining communication efficiency.
Entities (9)
Relation Signals (9)
ClientMorpher → evaluatedon → Databricks Dolly-15k
confidence 97% · We evaluate ClientMorpher for federated instruction fine-tuning on the Databricks Dolly-15K dataset
ClientMorpher-C → clusters → Clients
confidence 95% · ClientMorpher-C, which directly clusters clients using expert activation profiles
ClientMorpher-E → clusters → experts
confidence 95% · ClientMorpher-E, which first clusters experts based on their cross-client usage signatures
MoE → hascomponent → Router
confidence 95% · the router of a pretrained MoE model provides a compact signature of each client’s data distribution
ClientMorpher → uses → routing signatures
confidence 95% · ClientMorpher, a routing-aware, personalized federated instruction fine-tuning framework that leverages routing signatures
K-Means → usedby → ClientMorpher-E
confidence 92% · Apply K-Means to expert signatures
K-Means → usedby → ClientMorpher-C
confidence 92% · Apply K-Means to client profiles
ClientMorpher → improvesperformanceover → FedAvg
confidence 90% · routing-aware collaboration consistently improves personalized performance compared to conventional federated averaging
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Federated instruction fine-tuning enables Large Language Models (LLMs) to adapt to decentralized, privacy-sensitive data without requiring data sharing. Recent Mixture-of-Experts (MoE) LLMs are particularly attractive for federated learning because their sparse activation reduces computation and communication while scaling model capacity. However, existing federated MoE methods primarily focus on parameter aggregation and personalization, overlooking the routing behavior of MoE models as a source of information for client collaboration. Under heterogeneous instruction distributions, indiscriminate aggregation can lead to negative transfer, highlighting the need to identify which clients should collaborate during federated optimization. We propose ClientMorpher, a routing-aware, personalized federated instruction fine-tuning framework that leverages routing signatures from pretrained MoE models to organize client collaboration prior to aggregation. We investigate two complementary clustering strategies: ClientMorpher-C, which directly clusters clients using expert activation profiles, and ClientMorpher-E, which first clusters experts based on their cross-client usage signatures and then derives client collaboration groups. We evaluate ClientMorpher for federated instruction fine-tuning on the Databricks Dolly-15K dataset, using pathological and Dirichlet-based heterogeneous client distributions across multiple instruction-following tasks. Experimental results show that routing-aware collaboration consistently improves personalized performance compared to conventional federated averaging and local training, while maintaining the same communication cost. Furthermore, our study shows that client-centric and expert-centric clustering provides an effective and scalable approach for personalized federated instruction fine-tuning of sparse MoE LLMs.
Tags
Links
- Source: https://arxiv.org/abs/2608.15311v1
- Canonical: https://arxiv.org/abs/2608.15311v1
Trouble viewing inline? Open PDF directly →
Full Text
34,923 characters extracted from source content.
Expand or collapse full text
MoE Router-Guided Clustering for Heterogeneous Federated Instruction Tuning Ankita Sharma Bahar Farahani Affiliation: Member, IEEE Sanaz Rahimi Moosavi Affiliation: California State University, Dominguez Hills Amir Rahmani Affiliation: University of California, Irvine Farshad Firouzi Affiliation: Johns Hopkins University Krishnendu Chakrabarty [1ex] Arizona State University Abstract Federated instruction fine-tuning enables Large Language Models (LLMs) to adapt to decentralized, privacy-sensitive data without requiring data sharing. Recent Mixture-of-Experts (MoE) LLMs are particularly attractive for federated learning because their sparse activation reduces computation and communication while scaling model capacity. However, existing federated MoE methods primarily focus on parameter aggregation and personalization, overlooking the routing behavior of MoE models as a source of information for client collaboration. Under heterogeneous instruction distributions, indiscriminate aggregation can lead to negative transfer, highlighting the need to identify which clients should collaborate during federated optimization. We propose ClientMorpher, a routing-aware, personalized federated instruction fine-tuning framework that leverages routing signatures from pretrained MoE models to organize client collaboration prior to aggregation. We investigate two complementary clustering strategies: ClientMorpher-C, which directly clusters clients using expert activation profiles, and ClientMorpher-E, which first clusters experts based on their cross-client usage signatures and then derives client collaboration groups. Together, these strategies exploit routing behavior as a structural signal for personalization while maintaining the communication efficiency of sparse MoE fine-tuning. We evaluate ClientMorpher for federated instruction fine-tuning on the Databricks Dolly-15K dataset, using pathological and Dirichlet-based heterogeneous client distributions across multiple instruction-following tasks. Experimental results demonstrate that routing-aware collaboration consistently improves personalized performance compared to conventional federated averaging and local training, while maintaining the same communication cost. Furthermore, our study shows that client-centric and expert-centric clustering capture complementary forms of task similarity, providing an effective and scalable approach for personalized federated instruction fine-tuning of sparse MoE LLMs. Index Terms: Large Language Model, Mixture of Experts, Federated Instruction Tuning, Personalization. I Introduction Large Language Models (LLMs) have significantly advanced natural language processing by achieving state-of-the-art performance across a wide range of instruction-following tasks, including text classification, summarization, question answering, and information extraction. As these models become increasingly integrated into real-world applications, fine-tuning them on organization-specific data has become essential for adapting to specialized domains and user requirements [5]. However, such data are often distributed across multiple organizations, institutions, or edge devices and cannot be centralized because of privacy regulations, proprietary constraints, or communication limitations. Federated Learning (FL) [16, 11, 10] addresses this challenge by enabling multiple clients to collaboratively fine-tune a shared model while keeping local data decentralized. More recently, Mixture-of-Experts (MoE) architectures [19] have emerged as an attractive foundation for large-scale federated deployment because they activate only a sparse subset of experts for each input token, allowing models to scale to billions of parameters while maintaining efficient computation and communication [4]. Beyond computational efficiency, the routing mechanism of MoE models determines which experts process each token, implicitly capturing characteristics of the underlying instruction distribution. This routing behavior provides an additional source of information that is unique to sparse MoE models [21, 1] and has not yet been fully exploited in federated instruction fine-tuning. Fig. 1: Motivation for ClientMorpher. (a) Local training provides personalization but no collaboration. (b) FedAvg aggregates all MoE experts across clients, ignoring expert usage patterns. (c) ClientMorpher clusters clients using sparse expert activation profiles and performs cluster-wise aggregation of shared parameters. Fig. 2: The end-to-end workflow of the ClientMorpher Framework. Despite these advantages, federated instruction fine-tuning presents significant challenges under heterogeneous client distributions. In practical deployments, clients rarely possess identically distributed data; instead, they often specialize in different domains or instruction types, such as summarization, classification, question answering, or information extraction, as depicted in Fig. 1. Conventional Federated Averaging (FedAvg) aggregates updates from all participating clients into a single global model, implicitly assuming that every client should contribute equally to a shared representation. Under highly heterogeneous settings, however, indiscriminate aggregation frequently introduces negative transfer, reducing both personalization and overall model quality [12, 14]. Recent personalized federated learning methods [20] attempt to alleviate this issue by adapting subsets of parameters or maintaining client-specific models, while federated MoE frameworks such as FedMoE [17], FedMoE-DA [25], and OpenFedLLM [22] further exploit sparse architectures to reduce communication overhead and improve scalability. Nevertheless, these methods largely treat collaboration as a predefined optimization process and overlook MoE model’s routing behavior. In particular, existing approaches do not investigate whether routing patterns can reveal latent similarities among clients before federated optimization begins, nor whether such information can guide more effective collaboration among heterogeneous participants supporting clustering [6]. Since the router dynamically selects experts for each instruction, clients with similar instruction distributions are expected to exhibit similar routing behavior. These routing signatures, therefore, offer an observable representation of client similarity before federated optimization begins. Based on this insight, we propose ClientMorpher, a routing-aware personalized federated learning framework that leverages routing signatures to form collaboration groups prior to model aggregation. We investigate two complementary perspectives: ClientMorpher-C, which groups clients directly according to their expert activation profiles, and ClientMorpher-E, which first discovers expert communities from expert usage signatures and then derives client collaboration groups. In summary, this paper makes the following contributions: • We propose ClientMorpher, a routing-aware personalized federated instruction fine-tuning framework as illustrated in Fig. 2 that exploits the routing behavior of sparse MoE LLMs to guide client collaboration. • We develop two complementary clustering strategies: ClientMorpher-C, which clusters clients using expert activation profiles, and ClientMorpher-E, which clusters experts using cross-client usage signatures before assigning clients to expert-induced collaboration groups. • We conduct an empirical study on heterogeneous splits of the Dolly-15k dataset [2], comparing local training, MoE-FedAvg, ClientMorpher-C, and ClientMorpher-E across pathological and Dirichlet partitions with varying instruction-task skew. The remainder of this paper is organized as follows. Section I reviews related work. Section I presents the proposed framework. Section IV describes the experimental setup, evaluation, and results. Finally, Section V concludes the paper. I Related Work I-A Clustering Clients for Personalization Client clustering is an effective personalization strategy for federated learning in the presence of statistical heterogeneity. Instead of optimizing a single global model, clustered federated learning groups clients with similar data distributions or optimization behavior and trains a shared model for each cluster, improving convergence while reducing negative transfer. Representative approaches include IFCA [6], CFL [18], FedSEM [26], FedGroup [3], and StoCFL [24], which construct client clusters using gradients, model parameters, feature representations, or optimization dynamics. Although these methods have demonstrated strong performance with conventional neural networks, they are not designed for Mixture-of-Experts (MoE) models, in which sparse expert routing naturally captures functional specialization. However, existing clustering methods fail to leverage routing behavior as a lightweight, semantically meaningful signal to identify similar clients. I-B Federated Learning with Mixture-of-Experts Models Mixture-of-Experts (MoE) architectures have recently been introduced into federated learning to increase model capacity while maintaining communication and computational efficiency through sparse expert activation. Existing methods, including PFL-MoE [7], FedMoE [17], Fed-MoE, pFedMoE [23], PM-MoE [9], and FLEx [15], primarily focus on personalized adaptation, expert selection, modular aggregation, or communication-efficient training. While these approaches leverage the MoE routing mechanism during optimization, they largely treat routing as a means of personalization rather than as a source of structural information about client relationships. In contrast, our work utilizes expert-routing statistics to construct expert signatures, discover client similarity through expert-overlap clustering, and perform cluster-personalized federated optimization without exchanging expert parameters. I Methodology We present ClientMorpher, a routing-aware personalization framework for federated fine-tuning of MoE-LLMs. The core idea is that the router of a pretrained MoE model provides a compact signature of each client’s data distribution. Clients who activate similar experts are likely to require similar adaptation behavior, whereas clients who route to different experts should not necessarily share a single globally averaged model. ClientMorpher, therefore, replaces global aggregation with cluster-specific federated optimization as illustrated in Fig. 3. We study two variants of the same framework. ClientMorpher-C clusters clients directly using their expert-activation profiles. ClientMorpher-E first clusters experts according to their cross-client usage signatures and then assigns clients to expert clusters. Both variants share the same profiling step and the same cluster-personalized federated training step; they differ only in how client clusters are constructed, as explained in Algorithm 1. I-A Federated MoE Setup Consider N clients indexed by i∈1,…,Ni∈\1,…,N\, where client i owns a local dataset iD_i. The server initializes a pretrained MoE language model with E routed experts. For an input token, the MoE router produces logits over the experts and selects the top-k experts. The pretrained MoE backbone and expert parameters are kept frozen, and only a small set of adaptation parameters θa _a is optimized. The objective is to learn personalized adaptation parameters without requiring each client to train in isolation or forcing all clients into a single global FedAvg model. ClientMorpher solves this by learning a cluster assignment ai∈1,…,Ka_i∈\1,…,K\ for each client and maintaining one adaptation state θa(k) _a^(k) per cluster. Fig. 3: The depiction of router-guided client clustering for MoE-LLMs. Algorithm 1 The ClientMorpher Workflow. 1: Client datasets ii=1N\D_i\_i=1^N; pretrained MoE model with E experts; top-k router selections; profile budget B; clusters K; rounds R; local steps L; variant v∈C,Ev∈\C,E\ 2: Cluster-specific adaptation states θa(k,R)k=1K\ _a^(k,R)\_k=1^K 3: Freeze pretrained MoE backbone and expert parameters 4: Initialize trainable adaptation parameters θa _a ⊳ Shared routing profiling 5: for each client i∈1,…,Ni∈\1,…,N\ do 6: Run up to B batches from iD_i through the frozen MoE router 7: Count expert selections ci,e=∑t∈i[e∈TopKi,t]c_i,e= _t _i1[e _i,t] 8: Normalize ui,e=ci,e/∑e′=1Eci,e′u_i,e=c_i,e/ _e =1^Ec_i,e 9: Store client profile i=[ui,1,…,ui,E]u_i=[u_i,1,…,u_i,E] 10: end for 11: Build client-expert matrix ∈[0,1]N×EU∈[0,1]^N× E from rows iu_i ⊳ Variant-specific clustering 12: if v=Cv=C then 13: Apply K-Means to client profiles ii=1N\u_i\_i=1^N 14: Assign clients directly: ai=argminℓ∈1,…,K‖i−ℓ‖22a_i= _ ∈\1,…,K\\|u_i- μ_ \|_2^2 15: else 16: for each expert e∈1,…,Ee∈\1,…,E\ do 17: Construct and normalize expert signature e=:,e/∥:,e∥2s_e=U_:,e/\|U_:,e\|_2 18: end for 19: Apply K-Means to expert signatures ee=1E\s_e\_e=1^E 20: Obtain expert labels ze=argminℓ∈1,…,K‖e−ℓ‖22z_e= _ ∈\1,…,K\\|s_e- ν_ \|_2^2 21: Build ∈0,1E×KM∈\0,1\^E× K where Me,ℓ=1M_e, =1 iff ze=ℓz_e= 22: Assign clients by expert-cluster mass: ai=argmax∑e=1Eℓ∈1,…,Kui,eMe,ℓa_i= _ ∈\1,…,K\ _e=1^Eu_i,eM_e, 23: end if 24: Form client clusters k=i:ai=kC_k=\i:a_i=k\ ⊳ Shared cluster-PFL 25: for each cluster k∈1,…,Kk∈\1,…,K\ do 26: Initialize cluster state θa(k,0)←θa _a^(k,0)← _a 27: end for 28: for round r=0,…,R−1r=0,…,R-1 do 29: for each non-empty cluster kC_k do 30: for each client i∈ki _k do 31: Send θa(k,r) _a^(k,r) to client i 32: θa,i(r+1)←LocalTrain(θa(k,r),i,L) _a,i^(r+1) ( _a^(k,r),D_i,L) 33: end for 34: Aggregate within cluster: θa(k,r+1)←∑i∈kni∑j∈knjθa,i(r+1) _a^(k,r+1)← _i _k n_i _j _kn_j _a,i^(r+1) 35: end for 36: end for 37: return θa(k,R)k=1K\ _a^(k,R)\_k=1^K I-B Phase 0: Shared Client Expert-Usage Profiling Both ClientMorpher-C and ClientMorpher-E begin by profiling how each client’s local data activates the frozen MoE router. For client i, we run up to B batches from iD_i through the model without updating parameters. Let iT_i denote the profiled token positions across all captured MoE layers. For token position t∈it _i, let TopKi,t⊆1,…,ETopK_i,t \1,…,E\ be the set of top-k experts selected by the router. We count how often each expert is selected: ci,e=∑t∈i[e∈TopKi,t],c_i,e= _t _i1 [e _i,t ], (1) where ci,ec_i,e is the number of routing assignments from client i to expert e. The normalized expert-usage profile for client i is ui,e=ci,e∑e′=1Eci,e′,i=[ui,1,…,ui,E]∈[0,1]E.u_i,e= c_i,e _e =1^Ec_i,e , _i=[u_i,1,…,u_i,E]∈[0,1]^E. (2) The vectors are stacked into the client-expert usage matrix =[1⊤2⊤N⊤]∈[0,1]N×E.U= bmatrixu_1 \\ u_2 \\ \\ u_N bmatrix∈[0,1]^N× E. (3) The i-th row of U represents the routing profile of client i. This matrix is the common input to both ClientMorpher variants. I-C ClientMorpher-C: Direct Client-Profile Clustering ClientMorpher-C directly clusters clients in the expert-usage space. Each client is represented by its row vector iu_i from the usage matrix U. We apply K-Means to the set of client profiles ii=1N\u_i\_i=1^N: ai=argmink∈1,…,K‖i−k‖22,a_i= _k∈\1,…,K\ \|u_i- μ_k \|_2^2, (4) where k μ_k is the centroid of cluster k. This produces client clusters k=i:ai=kC_k=\i:a_i=k\. This variant treats the entire expert-activation profile as a client signature. If two clients route to experts with similar frequencies, they are assigned to the same aggregation group. I-D ClientMorpher-E: Expert-Overlap Clustering ClientMorpher-E uses the same usage matrix U, but it clusters experts before assigning clients. The routing signature of expert e is the cross-client usage vector given by the e-th column of U: e=:,e∈ℝN.s_e=U_:,e ^N. Each expert signature is normalized before clustering. We then compute pairwise expert similarity using cosine similarity. The experts are clustered into K expert groups. We use K-Means on the expert signatures: ze=argmink∈1,…,K‖e−k‖22,z_e= _k∈\1,…,K\ \|s_e- ν_k \|_2^2, (5) where zez_e is the expert-cluster assignment and k ν_k is an expert-cluster centroid. We define an expert-membership matrix ∈0,1E×KM∈\0,1\^E× K: Me,k=1,ze=k,0,otherwise.M_e,k= cases1,&z_e=k,\\ 0,&otherwise. cases (6) Each client is then assigned to the expert cluster that receives the largest mass from its local routing profile. The score of client i for the expert cluster k is qi,k=∑e=1Eui,eMe,k.q_i,k= _e=1^Eu_i,eM_e,k. (7) The final client assignment is ai=argmaxk∈1,…,Kqi,k.a_i= _k∈\1,…,K\q_i,k. The resulting client clusters are again k=i:ai=kC_k=\i:a_i=k\. This variant is denoted ClientMorpher-E in the experimental section. Unlike ClientMorpher-C, which clusters clients directly, ClientMorpher-E first discovers groups of experts that are shared across clients and then derives client groups from those expert clusters. I-E Shared Phase 2: Cluster-Personalized Federated Learning After either ClientMorpher-C or ClientMorpher-E produces client clusters, the training phase is identical. The server maintains one adaptation state θa(k) _a^(k) for each non-empty cluster kC_k. At communication round r, the server sends θa(k,r) _a^(k,r) to clients in cluster kC_k. Each client performs local optimization on its dataset iD_i for a fixed number of local steps: θa,i(r+1)=LocalTrain(θa(ai,r),i). _a,i^(r+1)=LocalTrain ( _a^(a_i,r),D_i ). (8) The server then aggregates updates only within the same cluster using FedAvg: θa(k,r+1)=∑i∈kni∑j∈knjθa,i(r+1), _a^(k,r+1)= _i _k n_i _j _kn_j _a,i^(r+1), (9) where ni=|i|n_i=|D_i| is the number of local training samples for client i. If a cluster contains a single client, its cluster state is simply that client’s locally trained state. TABLE I: Comparison of the two clustering strategies. C EC K-Means input Client profiles Expert signatures Input shape N×EN× E E×NE× N Output Client labels aia_i Expert labels zez_e Assignment Direct (aia_i) argmax∑ekui,eMe,k _k _eu_i,eM_e,k Aggregation Cluster-wise FedAvg Cluster-wise FedAvg IV Experimental Design and Evaluation Results TABLE I: 4-client pathological split. All federated methods exchange identical LoRA parameters (0.0878% of model/round). Method CLF CQA IE SUM Avg. Comm.% Local 0.6036 0.4096 0.5087 0.4075 0.4824 0.00 MoE-FedAvg (LoRA) 0.6299 0.4232 0.5085 0.4090 0.4927 0.0878 ClientMorpher-C 0.5937 0.3785 0.5243 0.3960 0.4731 0.0878 ClientMorpher-E 0.6411 0.4254 0.5099 0.4181 0.4986 0.0878 TABLE I: Comparison of Local Training, MoE-FedAvg, Client Clustering, and Expert Clustering under different Dirichlet concentration parameters (α). Client IDs remain fixed across experiments, while task assignments may differ under different α values. ROUGE-L is reported for each client (higher is better). α=0.1α=0.1 α=1.0α=1.0 Client Task Local MoE-FedAvg Client Cluster Expert Cluster Task Client Cluster Expert Cluster C0 QA 0.4679 0.5175 0.4732 0.4702 CLS 0.4833 0.4748 C1 IE 0.4660 0.5215 0.5085 0.5129 Summ 0.5174 0.4953 C2 Summ 0.3179 0.3552 0.3849 0.3798 Summ 0.4253 0.4269 C3 CLS 0.5383 0.5529 0.5472 0.5327 CLS 0.5648 0.5835 C4 QA 0.4013 0.4294 0.4767 0.4875 IE 0.4781 0.4766 C5 IE 0.4274 0.5102 0.5323 0.5301 IE 0.4972 0.5176 C6 IE 0.4294 0.5183 0.4940 0.4988 CLS 0.5549 0.5517 C7 IE 0.4642 0.5627 0.5502 0.5318 Summ 0.4088 0.4220 Average – 0.4391 0.4960 0.4959 0.4929 – 0.4912 0.4936 Classification – 0.5383 0.5529 0.5472 0.5327 – 0.5343 0.5367 Information Extraction – 0.4467 0.5282 0.5213 0.5184 – 0.4876 0.4971 Closed QA – 0.4346 0.4734 0.4750 0.4788 – – – Summarization – 0.3179 0.3552 0.3849 0.3798 – 0.4505 0.4481 IV-A Federated Setting We consider a federated learning setting with N clients, each holding a local dataset DiD_i drawn from a heterogeneous data distribution. Each client is associated with one or more NLP tasks, including text classification, closed-domain question answering, information extraction, and summarization. To evaluate the proposed method under varying degrees of statistical heterogeneity, we consider two non-IID data partitioning strategies. First, the Pathological partition, in which each client receives data exclusively from a single task. Second, the Dirichlet partition, where the task distribution of each client is sampled from a Dirichlet distribution with concentration parameter α. We evaluate α=0.1α=0.1, which represents highly heterogeneous client distributions, and α=1.0α=1.0, which represents a substantially more balanced non-IID setting. IV-B Dataset and Client Partitioning The Databricks Dolly-15k instruction-following dataset is used as the source dataset. Dolly-15k consists of instruction-response examples organized into multiple task categories. To simulate a strongly heterogeneous federated learning environment, we construct a pathological non-IID split in which each client receives examples from only one task category based on data distribution. The federated setting contains four tasks: • CLS: classification • QA: closed-domain question answering • IE: information extraction • Summ: summarization This partitioning creates distinct client distributions and allows us to evaluate whether a federated method can avoid negative transfer between task-specific clients. Each example is formatted as an Alpaca-style instruction-following prompt [13]. During training, the response is included in the sequence, but the loss is computed only over response tokens by masking the prompt tokens. During evaluation, only the prompt is given to the model, and the generated response is compared with the reference answer. IV-C Compared Methods We compare the following adaptation strategies: Local Training, MoE-FedAvg, Client Clustering, and Expert Clustering. IV-C1 Local Training In this baseline, each client fine-tunes its own LoRA [8] parameters using only its local data. No model parameters are exchanged or aggregated during training, resulting in zero communication overhead and no cross-client collaboration. Each client learns a fully personalized model tailored to its local data distribution. IV-C2 MoE FedAvg MoE-FedAvg implements the standard FedAvg algorithm in the proposed MoE architecture. In each communication round, every client performs local fine-tuning on its private data, after which the server aggregates the trainable non-expert parameters using sample-weighted averaging. The resulting global adaptation state is then broadcast back to all clients for the next training round. The server computes a sample-weighted average over all client updates. IV-C3 ClientMorpher-C The server first computes an expert-activation profile for each client using the frozen MoE router. Clients are then clustered directly using these profiles, and federated averaging is performed only within each cluster. IV-C4 ClientMorpher-E This variant first summarizes how experts are used across clients. Experts are clustered by their cross-client usage signatures. Each client is then assigned to the expert cluster where most of its routing mass goes. Federated averaging is again performed within the induced client groups. IV-D Baseline Comparison Table I compares Local Training, MoE-FedAvg, ClientMorpher-C, and ClientMorpher-E on the four-client pathological split. All federated methods share the same trainable LoRA parameters, which correspond to only 0.0878% of the full model per communication round. Consequently, all federated methods operate under an identical communication budget, allowing improvements to be attributed solely to more effective client collaboration rather than additional communication. Among the compared methods, ClientMorpher-E achieves the highest average ROUGE-L score (0.4986), outperforming both Local Training (0.4824) and MoE-FedAvg (0.4927). It also achieves the best performance in Classification, Closed Question Answering, and Summarization, while ClientMorpher-C achieves the highest score in Information Extraction. These results suggest that organizing collaboration according to routing signatures enables more effective knowledge sharing than aggregating updates globally, particularly when clients exhibit highly heterogeneous instruction distributions. IV-E Task-Wise Results Across Heterogeneity Table I compares ClientMorpher-C and ClientMorpher-E under two heterogeneous data distributions generated using Dirichlet parameters α=0.1α=0.1 and α=1.0α=1.0. Across both settings, the MoE backbone, LoRA configuration, clustering parameters, and training schedule remain identical. Under the highly heterogeneous setting (α=0.1α=0.1), both ClientMorpher variants achieve performance comparable to MoE-FedAvg while substantially improving over Local Training. ClientMorpher-C achieves the best performance on Summarization and remains competitive on Closed Question Answering, whereas ClientMorpher-E performs better on Classification and maintains competitive performance on Information Extraction. As the client distribution becomes more balanced (α=1.0α=1.0), the performance gap between the two variants narrows, with ClientMorpher-E achieving the highest overall average ROUGE-L (0.4936) and stronger performance on Classification and Information Extraction, while ClientMorpher-C remains slightly better on Summarization. These observations suggest that the two routing-aware clustering strategies capture complementary aspects of client similarity. Directly clustering clients using activation profiles is particularly effective when routing patterns are highly separated under extreme heterogeneity. In contrast, clustering experts based on shared usage signatures produces more stable collaboration groups as client distributions become less skewed. Finally, because no client is dominated by the Closed Question Answering task under the sampled α=1.0α=1.0 partition, task-specific results for that category are not reported. V Conclusion This paper introduces ClientMorpher, a routing-aware, personalized, federated instruction fine-tuning framework for sparse Mixture-of-Experts (MoE) LLMs. We addressed the challenge of collaborative learning under heterogeneous client instruction distributions by leveraging routing signatures as a signal for client data distribution. ClientMorpher forms personalized groups based on routing behavior, enabling clients with similar instruction characteristics to exchange knowledge more effectively. To investigate different perspectives on routing-aware collaboration, we proposed two variants: ClientMorpher-C, which clusters clients directly using expert activation profiles, and ClientMorpher-E, which uses expert clusters via cross-client expert usage signatures before building client groups. Extensive experiments on federated instruction fine-tuning using the Databricks Dolly-15K dataset under both pathological and Dirichlet-based heterogeneous data distributions demonstrate that routing-aware collaboration consistently improves personalized performance over traditional Federated Averaging and local training while maintaining the same communication cost. Our analysis further shows that client-centric and expert-centric clustering capture complementary aspects of task similarity. Direct client clustering is particularly effective under highly skewed client distributions, whereas expert-centric clustering provides more robust collaboration as client heterogeneity decreases. References [1] M. S. R. Avinash (2026) Task-conditioned routing signatures in sparse mixture-of-experts transformers. arXiv preprint arXiv:2603.11114. Cited by: §I. [2] M. Conover, M. Hayes, A. Mathur, J. Xie, J. Wan, S. Shah, A. Ghodsi, P. Wendell, M. Zaharia, and R. Xin (2023)Free dolly: introducing the world’s first truly open instruction-tuned llm(Website) External Links: Link Cited by: 3rd item. [3] M. Duan, D. Liu, X. Ji, R. Liu, L. Liang, X. Chen, and Y. Tan (2021) Fedgroup: efficient federated learning via decomposed similarity-based clustering. In 2021 IEEE Intl Conf on parallel & distributed processing with applications, big data & cloud computing, sustainable computing & communications, social computing & networking (ISPA/BDCloud/SocialCom/SustainCom), p. 228–237. Cited by: §I-A. [4] W. Fedus, B. Zoph, and N. Shazeer (2022) Switch transformers: scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research 23 (120), p. 1–39. Cited by: §I. [5] F. Firouzi, A. Ray, B. Farahani, M. Daneshmand, J. Song, S. Wu, and K. Chakrabarty (2026) Generative iot (giot): advancing iot with generative ai and large language models. Digital Communications and Networks. Cited by: §I. [6] A. Ghosh, J. Chung, D. Yin, and K. Ramchandran (2020) An efficient framework for clustered federated learning. Advances in neural information processing systems 33, p. 19586–19597. Cited by: §I, §I-A. [7] B. Guo, Y. Mei, D. Xiao, W. Wu, Y. Yin, and H. Chang (2020) PFL-moe: personalized federated learning based on mixture of experts. arXiv preprint arXiv:2012.15589. Cited by: §I-B. [8] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, W. Chen, et al. (2022) Lora: low-rank adaptation of large language models.. Iclr 1 (2), p. 3. Cited by: §IV-C1. [9] J. Jiang, Y. Chen, X. Liu, H. Jiang, and C. Fan (2025) Heterogeneous federated learning with scalable server mixture-of-experts. In Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence, p. 5480–5488. Cited by: §I-B. [10] S. Jiang, F. Firouzi, and K. Chakrabarty (2023) Low-overhead clustered federated learning for personalized stress monitoring. IEEE Internet of Things Journal 11 (3), p. 4335–4347. Cited by: §I. [11] S. Jiang, Y. Li, F. Firouzi, and K. Chakrabarty (2024) Federated clustered multi-domain learning for health monitoring. Scientific reports 14 (1), p. 903. Cited by: §I. [12] P. Kairouz and H. B. McMahan (2021) Advances and open problems in federated learning. Foundations and trends in machine learning 14 (1-2), p. 1–210. Cited by: §I. [13] C. A. Köhler, D. Ulianych, S. Grün, S. Decker, and M. Denker (2023) Facilitating the sharing of electrophysiology data analysis results through in-depth provenance capture. External Links: 2311.09672 Cited by: §IV-B. [14] T. Li, A. K. Sahu, M. Zaheer, M. Sanjabi, A. Talwalkar, and V. Smith (2020) Federated optimization in heterogeneous networks. Proceedings of Machine learning and systems 2, p. 429–450. Cited by: §I. [15] F. Liu, B. Pan, Z. Wang, X. Yao, X. Tang, J. Wang, and Y. Shi (2025) FLEx: personalized federated learning for mixture-of-experts llms via expert grafting. arXiv preprint arXiv:2506.00965. Cited by: §I-B. [16] B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas (2017) Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, p. 1273–1282. Cited by: §I. [17] H. Mei, D. Cai, A. Zhou, S. Wang, and M. Xu (2024) Fedmoe: personalized federated learning via heterogeneous mixture of experts. arXiv preprint arXiv:2408.11304. Cited by: §I, §I-B. [18] F. Sattler, K. Müller, and W. Samek (2019) Clustered federated learning: model-agnostic distributed multitask optimization under privacy constraints. IEEE Transactions on Neural Networks and Learning Systems 32, p. 3710–3722. External Links: Link Cited by: §I-A. [19] N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. Le, G. Hinton, and J. Dean (2017) Outrageously large neural networks: the sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701.06538. Cited by: §I. [20] A. Z. Tan, H. Yu, L. Cui, and Q. Yang (2022) Towards personalized federated learning. IEEE transactions on neural networks and learning systems 34 (12), p. 9587–9603. Cited by: §I. [21] X. Wang, S. Hayou, and E. Nalisnick (2026) The myth of expert specialization in moes: why routing reflects geometry, not necessarily domain expertise. arXiv preprint arXiv:2604.09780. Cited by: §I. [22] R. Ye, W. Wang, J. Chai, D. Li, Z. Li, Y. Xu, Y. Du, Y. Wang, and S. Chen (2024) Openfedllm: training large language models on decentralized private data via federated learning. In Proceedings of the 30th ACM SIGKDD conference on knowledge discovery and data mining, p. 6137–6147. Cited by: §I. [23] L. Yi, H. Yu, C. Ren, H. Zhang, G. Wang, X. Liu, and X. Li (2024) PFedMoE: data-level personalization with mixture of experts for model-heterogeneous personalized federated learning. ArXiv abs/2402.01350. External Links: Link Cited by: §I-B. [24] D. Zeng, X. Hu, S. Liu, Y. Yu, Q. Wang, and Z. Xu (2025) StoCFL: a stochastically clustered federated learning framework for non-iid data with dynamic client participation. Neural Networks 187, p. 107278. Cited by: §I-A. [25] Z. Zhan, W. Zhao, Y. Li, W. Liu, X. Zhang, C. W. Tan, C. Wu, D. Guo, and X. Chen (2024) FedMoE-da: federated mixture of experts via domain aware fine-grained aggregation. arXiv preprint arXiv:2411.02115. Cited by: §I. [26] X. Zhou, Y. Li, and J. Zhao (2025) FedSem: a resource allocation scheme for federated learning assisted semantic communication. arXiv preprint arXiv:2503.06058. Cited by: §I-A.