Paper deep dive

FedTreeLoRA: Reconciling Statistical and Functional Heterogeneity in Federated LoRA Fine-Tuning

Jieming Bian, Lei Wang, Letian Zhang, Jie Xu

Year: 2026Venue: arXiv preprintArea: cs.LGType: PreprintEmbeddings: 135

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 88%

Last extracted: 7/20/2026, 6:43:51 AM

Summary

The paper introduces FedTreeLoRA, a framework for Federated Learning with Low-Rank Adaptation (LoRA) that addresses both statistical heterogeneity (horizontal) and functional heterogeneity (vertical) in Large Language Models. It proposes a tree-structured aggregation mechanism that dynamically constructs an aggregation hierarchy, allowing clients to share broad consensus on shallow layers while specializing on deeper layers, thereby outperforming existing flat-model approaches.

Entities (10)

FedTreeLoRA · method · 98%Federated Learning · concept · 95%Low-Rank Adaptation · technique · 95%Functional Heterogeneity · concept · 90%Statistical Heterogeneity · concept · 90%Agglomerative Hierarchical Clustering · algorithm · 85%Flat-Model Assumption · concept · 85%GLUE · benchmark · 80%MNLI · dataset · 80%RoBERTa-large · model · 80%

Relation Signals (6)

FedTreeLoRA → addresses → Statistical Heterogeneity

confidence 90% · FedTreeLoRA, a framework employing tree-structured aggregation for fine-grained, layer-wise alignment... reconciling generalization and personalization.

FedTreeLoRA → addresses → Functional Heterogeneity

confidence 90% · FedTreeLoRA, a framework employing tree-structured aggregation for fine-grained, layer-wise alignment... reconciling generalization and personalization.

Flat-Model Assumption → ignores → Functional Heterogeneity

confidence 85% · existing personalized methods predominantly operated under a restrictive Flat-Model Assumption... ignoring the functional heterogeneity across LLM layers.

FedTreeLoRA → outperforms → state-of-the-art methods

confidence 85% · Experiments on NLU and NLG benchmarks demonstrate that FedTreeLoRA significantly outperforms state-of-the-art methods

Agglomerative Hierarchical Clustering → usedby → FedTreeLoRA

confidence 85% · We construct a binary merge tree T via Agglomerative Hierarchical Clustering (AHC) on D_global

RoBERTa-large → usedin → FedTreeLoRA

confidence 80% · We fine-tune a RoBERTa-Large (Liu et al., 2019) model on the GLUE benchmark... to quantify how vertical heterogeneity... interacts

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Federated Learning (FL) with Low-Rank Adaptation (LoRA) has become a standard for privacy-preserving LLM fine-tuning. However, existing personalized methods predominantly operated under a restrictive Flat-Model Assumption: they addressed client-side \textit{statistical heterogeneity} but treated the model as a monolithic block, ignoring the \textit{functional heterogeneity} across LLM layers. We argue that these two statistical (horizontal) and functional (vertical) dimensions, are \textit{orthogonal in source yet coupled in interaction}, implying that the optimal depth of parameter sharing is functionally dependent on client similarity. To address this, we propose \textbf{FedTreeLoRA}, a framework employing tree-structured aggregation for fine-grained, layer-wise alignment. By dynamically constructing an aggregation hierarchy, FedTreeLoRA allows clients to share broad consensus on shallow `trunks' while progressively specializing on deep `branches'. Experiments on NLU and NLG benchmarks demonstrate that FedTreeLoRA significantly outperforms state-of-the-art methods by effectively reconciling generalization and personalization.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

133,635 characters extracted from source content.

Expand or collapse full text

FedTreeLoRA: Reconciling Statistical and Functional Heterogeneity in Federated LoRA Fine-Tuning Jieming Bian Lei Wang Letian Zhang Jie Xu Abstract Federated Learning (FL) with Low-Rank Adaptation (LoRA) has become a standard for privacy-preserving LLM fine-tuning. However, existing personalized methods predominantly operated under a restrictive Flat-Model Assumption: they addressed client-side statistical heterogeneity but treated the model as a monolithic block, ignoring the functional heterogeneity across LLM layers. We argue that these two statistical (horizontal) and functional (vertical) dimensions, are orthogonal in source yet coupled in interaction, implying that the optimal depth of parameter sharing is functionally dependent on client similarity. To address this, we propose FedTreeLoRA, a framework employing tree-structured aggregation for fine-grained, layer-wise alignment. By dynamically constructing an aggregation hierarchy, FedTreeLoRA allows clients to share broad consensus on shallow ‘trunks’ while progressively specializing on deep ‘branches’. Experiments on NLU and NLG benchmarks demonstrate that FedTreeLoRA significantly outperforms state-of-the-art methods by effectively reconciling generalization and personalization. Machine Learning, ICML 1 Introduction Large Language Models (LLMs) have demonstrated significant potential across a wide range of domains, ranging from natural language understanding and code generation to complex mathematical reasoning (Devlin et al., 2019; Touvron et al., 2023a; Achiam et al., 2023; Touvron et al., 2023b; Team et al., 2023). To leverage these capabilities for specialized downstream applications, fine-tuning is often essential. However, the substantial scale of modern LLMs renders full fine-tuning computationally expensive and resource-intensive. Furthermore, in many real-world scenarios, high-quality domain data is dispersed across decentralized institutions or edge devices, where data sharing is restricted by privacy regulations and competitive concerns. Federated Learning (FL) (McMahan et al., 2017) has emerged as an effective paradigm to address these challenges, enabling collaborative training across distributed data sources without exchanging raw data. To make FL feasible for LLMs, Parameter-Efficient Fine-Tuning (PEFT) methods (Han et al., 2024), particularly Low-Rank Adaptation (LoRA) (Hu et al., 2021), are widely adopted to mitigate communication and computation overhead by updating only low-rank matrices while keeping pre-trained weights frozen. Despite these advancements, applying LoRA in FL settings presents a critical challenge: heterogeneity. Existing works (Bian et al., 2025a) have predominantly focused on horizontal heterogeneity, which refers to the non-IID (non-independent and identically distributed) nature of client data. Early approaches (Zhang et al., 2024; Bian et al., 2025b; Wang et al., 2024b) aggregated all clients into a single global model, which often yields suboptimal performance on diverse data distributions. More recent personalized methods have attempted to mitigate this issue through distinct aggregation strategies. One line of research, represented by methods (Qi et al., 2024; Yang et al., 2024; Hao et al., 2025; Bian et al., 2026), employs a dual-module approach, maintaining separate global and local LoRA modules to balance generic knowledge sharing with personalized adaptation. Another line of research, exemplified by FedLEASE (Wang et al., 2025), addresses data heterogeneity by adaptively clustering clients based on representation similarity to train shared experts within groups. While these strategies improve upon simple global aggregation, they still rely on a restrictive Flat-Model Assumption: regardless of using dual modules or client clustering, they treat LoRA as a single monolithic unit, assuming that sharing decisions must remain uniform across layers. We argue that this assumption overlooks a second, equally important dimension: the Intra-Model Functional Heterogeneity (which we term vertical heterogeneity in the context of model depth). This is an architectural property, distinct from traditional vertical federated learning (data feature partitioning). Empirical studies indicate that LLM layers exhibit distinct representational properties: shallow layers typically extract general linguistic features, whereas deep layers encode specific semantic information (Gao et al., 2024; Qian et al., 2025). Consequently, the optimal degree of parameter sharing is not uniform but varies with model depth. By ignoring this vertical dimension, existing aggregation strategies face a fundamental dilemma: they either aggressively aggregate deep layers, risking negative transfer due to data conflicts, or conservatively isolate shallow layers, thereby under-utilizing shared general knowledge. In this paper, we propose that these two dimensions of heterogeneity, the statistical horizontal and the functional vertical, are orthogonal in source yet coupled in interaction. They are orthogonal because horizontal heterogeneity stems from external data distributions, while vertical heterogeneity arises solely from the internal LLM architecture. However, they are coupled because the effective aggregation strategy depends on their intersection: the depth at which clients should cease sharing parameters is functionally dependent on their data similarity. To empirically verify this hypothesis, we first conduct extensive motivational studies analyzing the interaction between client data distributions and layer-wise representations. Our analysis confirms that clients with heterogeneous data distributions can benefit from sharing shallow ‘trunks’ of the model while diverging at deeper layers to preserve personalization. This implies that the optimal aggregation structure is not a set of disjoint clusters, but a hierarchical tree. Based on these insights and empirical verifications, we introduce FedTreeLoRA, a novel framework that addresses this dual-heterogeneity through tree-structured aggregation. Unlike prior methods, including coarse-grained client grouping approaches such as FedLEASE (Wang et al., 2025), FedTreeLoRA implements a fine-grained, layer-wise alignment strategy. It utilizes a data-driven hierarchical approach to dynamically construct an aggregation tree, where the root represents shared shallow layers among all clients, and branches represent progressive specialization towards deeper layers for specific client subgroups. This structure naturally determines the optimal sharing boundary for each client group, ensuring that general capabilities are consolidated while specific data patterns are adapted locally. Our main contributions are summarized as follows: • We identify the problem of dual-heterogeneity in federated LLM fine-tuning, revealing that ignoring the coupling between client-side data distribution and layer-wise representation leads to suboptimal aggregation. • We propose FedTreeLoRA, a tree-structured framework that adaptively determines the optimal parameter-sharing depth across layers, effectively reconciling generalization and personalization. • We conduct extensive experiments on diverse benchmarks, demonstrating that FedTreeLoRA significantly outperforms state-of-the-art federated LoRA methods. 2 Related Works Personalized Federated Learning. Federated Learning (McMahan et al., 2017; Huang et al., 2026; Wang et al., 2024a; Bian et al., 2024; Zhang et al., 2025a; Liu et al., 2024, 2025; Peng et al., 2024) enables collaborative training while preserving data privacy, yet it faces significant challenges from statistical data heterogeneity (Li et al., 2020). To address this, Personalized FL (PFL) has been extensively studied, employing techniques such as regularization (Karimireddy et al., 2020), meta-learning (Fallah et al., 2020), and clustering-based methods (Ghosh et al., 2020; Sattler et al., 2020) that group clients with similar distributions. While prior work has explored layer-wise aggregation strategies, selectively sharing shallow layers while personalizing deep ones, these have been almost exclusively limited to small models like CNNs (e.g., FedPer (Arivazhagan et al., 2019), LG-FedAvg (Liang et al., 2020)). However, extending these insights to LLMs is non-trivial. Unlike CNNs, where hierarchy is defined by spatial feature progression, Transformer-based LLMs consist of architecturally identical layers where hierarchy emerges from semantic specialization (from syntax to reasoning) (Jawahar et al., 2019; Gao et al., 2024). To the best of our knowledge, no prior work has systematically examined how LoRA parameters should be aggregated differentially across Transformer depth based on client similarity, a gap this work aims to fill. Federated Fine-tuning with LoRA. Given the immense computational cost of full LLM fine-tuning, integrating Parameter-Efficient Fine-Tuning methods (Houlsby et al., 2019), particularly Low-Rank Adaptation (Hu et al., 2021), into FL has become a dominant paradigm. Early approaches, such as FedIT (Zhang et al., 2024) and SLoRA (Babakniya et al., 2023), focused on training a single shared global LoRA module. However, these global methods often struggle with domain shifts in real-world heterogeneous scenarios. Consequently, recent research has pivoted towards personalized federated LoRA. FedSA (Guo et al., 2025), proposes splitting the LoRA module (e.g., aggregating matrix A while keeping B local). Another direction, exemplified by FedDPA (Yang et al., 2024) and FedALT (Bian et al., 2026), utilizes a dual-branch architecture containing both global and local LoRA modules. More recently, clustering-based methods like FedLEASE (Wang et al., 2025) group clients to train shared domain experts. Despite their architectural differences, these personalized methods share a common limitation: they rely on a Flat-Model Assumption. They treat the LoRA module as a monolithic unit, overlooking the intrinsic vertical heterogeneity of LLMs. Our proposed FedTreeLoRA challenges this assumption by introducing a tree-structured aggregation mechanism that explicitly aligns aggregation depth with client similarity. 3 Preliminaries 3.1 Low-Rank Adaptation (LoRA) for LLMs LoRA (Hu et al., 2021) is a parameter-efficient fine-tuning method predicated on the hypothesis that the change in weights during model adaptation has a low intrinsic dimension. For a pre-trained Transformer model consisting of L layers, let W0∈ℝdout×dinW_0 ^d_out× d_in denote a specific frozen weight matrix (e.g., a query or value projection) within a given layer. LoRA bypasses the update of W0W_0 by injecting two trainable low-rank decomposition matrices, B∈ℝdout×rB ^d_out× r and A∈ℝr×dinA ^r× d_in, where the rank r≪min⁡(din,dout)r (d_in,d_out). The forward pass for an input x∈ℝdinx ^d_in is modified as: h=W0x+ΔWx=W0x+BAx,h=W_0x+ Wx=W_0x+BAx, (1) where A is typically initialized with a random Gaussian distribution and B is initialized to zero, ensuring that the training begins with the original model behavior (i.e., ΔW=0 W=0). We denote the collection of all LoRA adapters across the model layers as . 3.2 Problem Formulation: Federated Fine-tuning We consider a federated learning system comprising N clients, indexed by k∈1,…,Nk∈\1,…,N\, each possessing a private dataset kD_k drawn from a heterogeneous distribution kP_k. In the context of personalized federated fine-tuning, each client k aims to obtain a specific set of effective LoRA parameters, denoted as k _k, to adapt the shared frozen backbone 0W_0. The overarching goal is to solve the objective: minkk=1N∑k=1Npkξ∼k[ℓ(k;0,ξ)], _\ _k\_k=1^N _k=1^Np_kE_ξ _k[ ( _k;W_0,ξ)], (2) where pkp_k is the relative weight of client k. Crucially, in our framework, k _k is not necessarily trained in isolation. Instead, it is a composite mapping derived from the federated aggregation mechanism. Our specific goal is to design a structural dependence such that k _k leverages shared knowledge from similar clients (via the aggregation tree) while retaining task-specific adaptability. 4 Motivational Studies While the hierarchical nature of LLMs is well-established in centralized settings (Jawahar et al., 2019), its implications for federated adaptation remain under-investigated. We posit that treating all LoRA layers uniformly, as done in current Flat-Model approaches, leads to inefficient parameter sharing. In this section, we conduct two motivational studies to quantify how vertical heterogeneity (layer depth) interacts with horizontal heterogeneity (client data distribution) to determine the optimal aggregation strategy. Due to space constraints, we provide the full experimental setup and extended results in Appendix B. Figure 1: Vertical Heterogeneity. Aggregating only shallow layers significantly outperforms aggregating deep layers. Observation 1: Vertical (Intra-Model Functional) heterogeneity dictates the stability of federated aggregation. We fine-tune a RoBERTa-Large (Liu et al., 2019) model on the GLUE benchmark (Wang et al., 2018) with ten clients partitioned under a non-IID distribution (Dirichlet α=0.5α=0.5). We vary the aggregation scope: aggregating only shallow layers, aggregating only deep layers, full model aggregation, and purely local training. As illustrated in Figure 1, aggregating shallow layers yields significantly higher accuracy than aggregating deep layers. Crucially, aggregating deep layers is detrimental, resulting in performance even worse than local training. Furthermore, full model aggregation under heterogeneous conditions degrades performance relative to the shallow-only strategy. These results indicate that the hierarchical nature of LLMs directly translates into aggregation sensitivity: deep layers are highly vulnerable to negative transfer when clients diverge, whereas shallow layers provide a robust common ground for collaboration. Thus, vertical heterogeneity is not merely an architectural feature but a critical determinant of effective parameter sharing and aggregation stability. Observation 2: The optimal sharing boundary is coupled with horizontal heterogeneity. We further investigate the interaction between vertical depth and statistical divergence through a controlled study. Specifically, we evaluate two clients on the MNLI dataset using RoBERTa-Large across three distinct regimes: Homogeneous, Moderate, and Heterogeneous. We vary the aggregation scope to encompass the first 8, 16, or all 24 layers. As illustrated in Figure 2, we observe a distinct shift in the optimal aggregation depth based on the severity of data heterogeneity. In the Homogeneous setting, performance improves monotonically as aggregation extends to deeper layers, benefiting from maximum knowledge sharing. However, as heterogeneity increases, the performance peak shifts sharply toward shallower layers. In the Heterogeneous setting, aggregating beyond the shallow layers becomes detrimental, yielding results inferior to those of partial aggregation. These findings demonstrate that the “safe” depth for parameter sharing is not an architectural constant but is dynamically coupled with the statistical similarity between clients. This coupling implies that a valid aggregation strategy must determine a hierarchical sharing boundary rather than simply grouping clients into disjoint, flat clusters as in prior work (Wang et al., 2025). This motivates the design of FedTreeLoRA, which constructs a tree-structured mechanism to ensure that general capabilities are shared at the root while task-specific reasoning diverges at the branches. Figure 2: The Coupling Effect of Dual Heterogeneity. As client distributions diverge (from Homogeneous to Heterogeneous), the optimal sharing boundary shifts from deep to shallow layers. 5 Methodology: FedTreeLoRA We introduce FedTreeLoRA to reconcile statistical and functional heterogeneity. Unlike static flat-model approaches, FedTreeLoRA leverages a global dependency tree to enforce structural consistency while enabling adaptive, layer-wise granularity control across the Transformer architecture. 5.1 Global Topological Structure Modeling The foundation of our approach is the construction of a global hierarchy that captures the overall statistical relationships among clients. The process begins with a warmup phase where each client k∈1,…,Nk∈\1,…,N\ independently performs local fine-tuning on the pre-trained backbone using its private dataset kD_k for EwarmE_warm epochs. This yields an initial set of LoRA parameters. Consistent with prior findings that the B matrices capture task-specific semantic variations (Tian et al., 2024), we utilize the accumulated updates in B to quantify client similarity, and we further provide a layer-wise heterogeneity analysis in Appendix G to empirically justify this choice. Specifically, we compute the Global Distance Matrix Dglobal∈ℝN×ND^global ^N× N via the average distance between client updates: Di,jglobal=1L∑l=1Ldist(Bl,i,Bl,j),D^global_i,j= 1L _l=1^Ldist(B_l,i,B_l,j), (3) where L denotes the total number of layers and dist(⋅,⋅)dist(·,·) is a distance operator in the parameter space. We default to the Frobenius distance due to its robustness in high-dimensional spaces (Aggarwal et al., 2001). Alternatives like cosine distance are evaluated in Section C.4. We construct a binary merge tree T via Agglomerative Hierarchical Clustering (AHC) on DglobalD^global, encoding a spectrum of partitions from P1P_1 (universal sharing) to PNP_N (full personalization). Crucially, constructing T from global rather than layer-wise information prevents topological incoherence, where contradictory client groupings across adjacent layers disrupt semantic continuity (e.g., clusters 1,2\1,2\ and 3,4\3,4\ at layer l are reshuffled into 1,3\1,3\ and 2,4\2,4\ at layer l+1l+1). By establishing this global skeleton, we ensure every layer-specific clustering is a valid cut of a unified topology, guaranteeing structural consistency: clients separated at shallow layers remain specialized at deeper layers, thereby preserving the logical hierarchy of expert specialization. Figure 3: Overview of FedTreeLoRA. (1) Global Topological Structure Modeling: A hierarchy tree is built via AHC on client LoRA B matrices during warmup to capture cross-client relationships. (2) Adaptive Layer-wise Alignment: For each layer l, the optimal cluster count cl∗c_l^* is dynamically selected under a monotonicity constraint. (3) Cluster-External Expert Mechanism: Each client synthesizes parameters by mixing a Cluster Expert with an External Expert via a learnable coefficient λl,k _l,k. 5.2 Adaptive Layer-wise Depth Alignment While the global tree T provides structural candidates, the optimal aggregation resolution varies by depth, shifting from broad consensus in shallow layers to fine-grained specialization in deep ones. To identify the optimal cluster count cl∗c_l^* for layer l, we first quantify local functional heterogeneity via the Layer-wise Distance Matrix D(l)D^(l): Di,j(l)=dist(Bl,i,Bl,j).D^(l)_i,j=dist(B_l,i,B_l,j). (4) Unlike the global metric, D(l)D^(l) captures layer-specific data distributions, serving as the basis for evaluating the fitness of any candidate partition PcP_c at this specific depth. To enforce the hierarchical prior that specialization increases with depth, we impose a monotonicity constraint: cl≥cl−1c_l≥ c_l-1. We set c0∗=1c_0^*=1 to represent the shared global root. For each Transformer layer l∈1,…,Ll∈\1,…,L\, given the optimal cl−1∗c_l-1^* from the previous layer, we define the feasible search space Ωl _l, constrained by the window size K: Ωl=c∈ℤ∣cl−1∗≤c<min⁡(N,cl−1∗+K). _l=\c c_l-1^*≤ c< (N,c_l-1^*+K)\. (5) This formulation ensures that even for the first layer (l=1l=1), the algorithm dynamically searches for the optimal granularity starting from c=1c=1 (global sharing) up to K, allowing the model to maintain the current granularity or refine it by splitting existing clusters. We select the optimal cluster count from Ωl _l using the Silhouette Coefficient (Rousseeuw, 1987), denoted as Sil(Pc,D(l))Sil(P_c,D^(l)), which measures cluster validity based on the layer-specific distance. To handle the singleton case (c=1c=1) where the Silhouette score is undefined, we introduce a heterogeneity threshold τ as a baseline. We define the scoring function ϕ(c;D(l))φ(c;D^(l)) as τ if c=1c=1, and Sil(Pc,D(l))Sil(P_c,D^(l)) otherwise. This threshold τ effectively controls the resistance to splitting; the algorithm transitions from a global shared model to specialized clusters only when the distinguishability of client distributions exceeds τ. The optimal cluster count for layer l is thus determined by: cl∗=argmaxc∈Ωlϕ(c;D(l)).c_l^*= *argmax_c∈ _lφ(c;D^(l)). (6) This sequential optimization ensures that the aggregation structure evolves dynamically from the root to the leaves, strictly governed by the functional properties of each layer. 5.3 Cluster-External Expert Mechanism The layer-wise partitions derived in Sec. 5.2 provide a topological blueprint of client relationships, specifically identifying peer groups with high functional similarity. To operationalize this topology for federated fine-tuning, we must define how parameters are synthesized based on these structural priors. While our tree-structured alignment is compatible with various interaction mechanisms (as discussed later), we propose a parameter-efficient Cluster-External aggregation strategy as the primary instantiation. This design allows clients to leverage high-fidelity consensus from their specific peer cluster while retaining a pathway to broad global knowledge, effectively preventing information isolation without incurring the high computational cost of complex routing. For a given layer l, the optimal cluster count cl∗c_l^* induces the partition Pcl∗=1,…,cl∗P_c_l^*=\C_1,…,C_c_l^*\. For any client k, let k(l)∈Pcl∗S_k^(l)∈ P_c_l^* denote the cluster containing k (peer group), and let ℛk(l)R_k^(l) denote the remaining clients (external group). Expert Construction. Instead of aggregating generic parameter sets, we explicitly construct two specific LoRA experts for client k at layer l: the Cluster Expert and the External Expert. Let Φ∈A,B ∈\A,B\ denote a LoRA parameter matrix. We compute the aggregated experts as: Φ¯l,kclus _l,k^clus =1|k(l)|∑j∈k(l)Φl,j, = 1|S_k^(l)| _j _k^(l) _l,j, (7) Φ¯l,kext _l,k^ext =1|ℛk(l)|∑j∈ℛk(l)Φl,j. = 1|R_k^(l)| _j _k^(l) _l,j. Note that if k(l)S_k^(l) contains all clients (e.g., at the root layer), the External Expert is zeroed to avoid redundancy. Forward Process and Updating. To balance peer-group specialization and global knowledge sharing, we introduce a learnable mixing coefficient λl,k∈[0,1] _l,k∈[0,1] for each layer. The forward pass for input x at layer l combines the frozen pre-trained weight W0,lW_0,l with the weighted contributions of the Cluster and External Experts: hl(x)=W0,lx h_l(x)=W_0,lx +λl,k(B¯l,kclusA¯l,kclusx) + _l,k ( B_l,k^clus A_l,k^clusx ) (8) +(1−λl,k)(B¯l,kextA¯l,kextx). +(1- _l,k) ( B_l,k^ext A_l,k^extx ). During local training, client k updates only the Cluster Expert parameters (A¯l,kclus,B¯l,kclus)( A_l,k^clus, B_l,k^clus) and the coefficient λl,k _l,k, while keeping the External Expert frozen. This ensures that the client refines the consensus of its peer group while retaining static access to broader global features. Crucially, the proposed tree structure defines the topology of sharing, specifically identifying optimal peer groups for constructing layer-wise Cluster Experts. This topological contribution is orthogonal to the specific combination strategy used to integrate these experts during inference. While Eq. (8) adopts a parameter-efficient scalar mixing approach with a consolidated External Expert to minimize communication and computational overhead, our framework is inherently compatible with diverse interaction mechanisms. Specifically, the framework can be instantiated with: (1) a learnable Mixture-of-Experts (MoE) router (Jordan and Jacobs, 1994) replacing the scalar coefficient λ for input-dependent dynamic selection; (2) a decomposed set of distinct external experts corresponding to specific peer clusters, rather than a single aggregated External Expert; or (3) an isolationist strategy that utilizes solely the Cluster Expert, discarding external information entirely. In our experiments, we explore these variations to analyze the trade-offs between performance and efficiency, demonstrating that the benefits of our layer-wise topological alignment persist regardless of the specific combination operator employed. 5.4 Convergence Analysis Our analysis focuses on the proposed adaptive update rule, where the effective weight update is driven by the trainable Cluster Expert while conditioned on the frozen External Expert. To facilitate the analysis, we adopt standard assumptions commonly used in federated optimization: local objective functions are σ-smooth and stochastic gradients are unbiased with bounded variance G2G^2 (Assumptions A.1 and A.2 in Appendix). In addition, specific to low-rank adaptation, we follow the formulation in (Guo et al., 2025) and assume that the LoRA matrices are bounded by constants MAM_A and MBM_B, and satisfy a gradient-alignment condition with coefficients μA,μB>0 _A, _B>0, ensuring that optimization within the low-rank subspace yields meaningful descent directions for the full parameter space (Assumption A.3). Theorem 5.1. Let Assumptions A.1–A.3 hold. Let E be the number of local SGD steps and choose stepsize η>0η>0. Define the composite constant Γ collecting all O(η2)O(η^2) terms from local updates and tree-structured aggregation as: Γ=LMA2MB2G2+CσLG2(MA4+MB4+MA4MB4), =LM_A^2M_B^2G^2+C\,σ LG^2 (M_A^4+M_B^4+M_A^4M_B^4 ), for some constant C>0C>0. The average squared gradient norm of the iterates generated by FedTreeLoRA satisfies 1NT∑k=1N∑t=1T[‖∇ℒk(k(t))‖F2]≤2μA+μBΔ⋅ΓT, 1NT _k=1^N _t=1^TE [\| _k(W_k^(t))\|_F^2 ]\;≤\; 2 _A+ _B · T, where Δ denotes the initial optimality gap. Theorem 5.1 shows an (1/T)O(1/ T) convergence rate under standard smooth non-convex assumptions, matching FedAvg (Yu et al., 2019) and FedSA (Guo et al., 2025). Detailed assumptions and proofs are provided in Appendix A. Table 1: Performance comparison on GLUE benchmarks (RoBERTa-Large-355M) under non-IID settings (α=0.5α=0.5). Methods Rank % Param MNLI QNLI SST2 QQP Average FedIT (Zhang et al., 2024) 4 0.1107%0.1107\% 83.18±0.7483.18± 0.74 87.03±0.4387.03± 0.43 93.65±0.6393.65± 0.63 84.93±0.5984.93± 0.59 87.2087.20 - FFA-LoRA (Sun et al., 2024) 4 0.0553%0.0553\% 83.02±0.4283.02± 0.42 87.47±0.1387.47± 0.13 93.73±0.1893.73± 0.18 83.48±0.3683.48± 0.36 86.9386.93 −0.27-0.27 8 0.1107%0.1107\% 83.13±0.5683.13± 0.56 88.90±0.3088.90± 0.30 93.95±0.2093.95± 0.20 83.62±0.4883.62± 0.48 87.4087.40 +0.20+0.20 FedSA (Guo et al., 2025) 4 0.1107%0.1107\% 83.63±0.6983.63± 0.69 91.32±0.1091.32± 0.10 95.87±0.3095.87± 0.30 89.33±0.4389.33± 0.43 90.0490.04 +2.84+2.84 FedDPA (Yang et al., 2024) 4 0.1107%0.1107\% 83.97±0.7583.97± 0.75 91.31±0.5191.31± 0.51 95.72±0.1295.72± 0.12 89.74±0.5889.74± 0.58 90.1990.19 +2.99+2.99 FedALT (Bian et al., 2026) 4 0.1383%0.1383\% 84.03±0.5984.03± 0.59 90.77±0.4690.77± 0.46 96.16±0.2996.16± 0.29 89.27±0.5089.27± 0.50 90.0690.06 +2.86+2.86 FedLEASE (Wang et al., 2025) 4 0.1521%0.1521\% 86.21±0.3686.21± 0.36 92.56±0.7792.56± 0.77 95.63±0.3495.63± 0.34 90.36±0.4690.36± 0.46 91.1991.19 +3.99+3.99 FedTreeLoRA (Ours) 4 0.1107%0.1107\% 88.15±0.2588.15± 0.25 93.37±0.6293.37± 0.62 96.56±0.0796.56± 0.07 91.35±0.1791.35± 0.17 92.3692.36 +5.16+5.16 Table 2: Performance comparison on FLAN benchmarks (LLaMA-2-7B). We report ROUGE-1 scores. Methods Rank % Param Text Edit Struct2Text Sentiment Reasoning Average FedIT (Zhang et al., 2024) 8 0.0622%0.0622\% 59.84±1.1759.84± 1.17 51.71±1.1451.71± 1.14 44.53±1.3044.53± 1.30 74.42±0.7474.42± 0.74 57.6257.62 - FFA-LoRA (Sun et al., 2024) 8 0.0311%0.0311\% 58.64±0.9658.64± 0.96 51.83±0.4851.83± 0.48 44.26±0.3144.26± 0.31 73.62±0.4373.62± 0.43 57.0957.09 −0.53-0.53 16 0.0622%0.0622\% 59.15±0.1659.15± 0.16 52.85±0.5452.85± 0.54 44.96±2.2244.96± 2.22 73.77±0.2873.77± 0.28 57.6857.68 +0.06+0.06 FedSA (Guo et al., 2025) 8 0.0622%0.0622\% 63.80±1.8863.80± 1.88 54.48±0.5354.48± 0.53 46.98±0.5546.98± 0.55 74.37±0.9474.37± 0.94 59.9159.91 +2.29+2.29 FedDPA (Yang et al., 2024) 8 0.0622%0.0622\% 64.33±0.9264.33± 0.92 54.18±1.1254.18± 1.12 48.13±1.0248.13± 1.02 75.55±1.4375.55± 1.43 60.5560.55 +2.93+2.93 FedALT (Bian et al., 2026) 8 0.0699%0.0699\% 67.61±1.8067.61± 1.80 54.06±1.5154.06± 1.51 48.57±1.2648.57± 1.26 76.84±1.4876.84± 1.48 61.7761.77 +4.15+4.15 FedLEASE (Wang et al., 2025) 8 0.0895%0.0895\% 66.31±1.4366.31± 1.43 54.80±1.0754.80± 1.07 49.32±0.8849.32± 0.88 76.40±1.0776.40± 1.07 61.7161.71 +4.09+4.09 FedTreeLoRA (Ours) 8 0.0622%0.0622\% 68.63±0.8768.63± 0.87 55.59±0.8655.59± 0.86 51.27±0.7151.27± 0.71 77.27±0.7377.27± 0.73 63.1963.19 +5.57+5.57 6 Experiments Baseline Methods. We compare against representative baselines categorized into two groups: General Federated Fine-Tuning: (1) FedIT (Zhang et al., 2024): Applies FedAvg directly to LoRA parameters; (2) FFA-LoRA (Sun et al., 2024): Freezes A matrices and fine-tunes only B for communication efficiency. Personalized Federated Fine-Tuning: (3) FedSA (Guo et al., 2025): Aggregates global A matrices while keeping B local; (4) FedDPA (Yang et al., 2024): Decouples knowledge via dual global and local modules; (5) FedALT (Bian et al., 2026): Mixes continuously trained local modules with frozen Rest-of-World parameters; (6) FedLEASE (Wang et al., 2025): A strong flat-clustering baseline that trains shared domain experts, serving as a direct contrast to our hierarchical approach. Detailed descriptions of these baseline methods and their specific implementation settings are provided in Appendix H. 6.1 Natural Language Understanding NLU Setup. We employ RoBERTa-Large (355M) (Liu et al., 2019), consisting of 24 Transformer layers, as the backbone model. Evaluation is conducted on four datasets from the GLUE benchmark (Wang et al., 2018): QQP, QNLI, MNLI, and SST2. We simulate a federated system with N=20N=20 clients, each possessing 1,000 training samples. To emulate statistical heterogeneity, data is partitioned using a Dirichlet distribution with α=0.5α=0.5. Training is performed with a batch size of 128 for E=2E=2 local epochs over T=30T=30 communication rounds. LoRA adapters with rank r=4r=4 are applied to the query and value projections, while the classification head remains frozen. Learning rates are tuned via grid search over η∈1E−4,3E−4,5E−4,1E−3,3E−3,5E−3η∈\1E-4,3E-4,5E-4,1E-3,3E-3,5E-3\ for each method. We report Accuracy as the evaluation metric. All reported results are averaged over three independent runs. Performance Comparison. Table 1 presents the comparative results on the NLU benchmarks under the non-IID setting. Several key observations arise from this evaluation. First, personalized federated fine-tuning methods consistently outperform general approaches (e.g., FedIT), validating the necessity of personalization in heterogeneous environments. Second, regarding efficiency-performance trade-offs, FFA-LoRA yields suboptimal results; even when doubling the rank to r=8r=8 to equalize the trainable parameter count with other baselines, it fails to recover the performance drop caused by freezing the A matrices. Third, while FedLEASE outperforms prior baselines by leveraging clustering and adaptive expert selection, this comes at the cost of significantly increased training workload due to the additional parameters required for its adaptive top-M routers. In sharp contrast, FedTreeLoRA achieves state-of-the-art performance across all four tasks with negligible parameter overhead (limited to layer-wise scalar mixing coefficients λl,k _l,k). This result is theoretically significant: it demonstrates that the performance bottleneck in prior personalized methods stems from focusing solely on horizontal (statistical) heterogeneity. By explicitly addressing vertical (functional) heterogeneity via our tree-structured alignment, FedTreeLoRA unlocks superior generalization and personalization efficiency without the computational burden of complex routing networks. Visualization of Adaptive Structure. To verify that FedTreeLoRA dynamically adapts to diverse data distributions, we visualize the layer-wise evolution of the cluster count (cl∗c_l^*) across the evaluated datasets in Figure 4. The distinct trajectories observed for each dataset confirm that our framework does not enforce a rigid template; instead, it automatically learns a tailored topological structure driven by the specific heterogeneity profile of each condition. We further provide detailed structural visualizations under extreme task heterogeneity in Appendix C. Figure 4: Layer-wise cluster counts (cl∗c_l^*) across different datasets. FedTreeLoRA adaptively identifies the optimal aggregation granularity specific to each data distribution. Ablation Studies. We conduct comprehensive ablations on GLUE to isolate the contribution of each component in FedTreeLoRA. For brevity, we report average accuracy here, while detailed task-wise results are provided in Appendix D. (1) Impact of Interaction Mechanisms. We compare the scalar-mixed Cluster-External aggregation with three alternatives enabled by our topological framework: Isolationist (Cluster Expert only), Decomposed Experts (distinct experts for every peer cluster), and MoE Router (replacing λl,k _l,k with a learnable MLP router (Jordan and Jacobs, 1994)). As shown in Table 3, the Decomposed Experts variant achieves the highest average accuracy (92.57%92.57\%), but incurs substantial communication overhead due to exchanging multiple expert modules. Interestingly, the Isolationist variant still attains 91.40%91.40\%, surpassing the strongest baseline (FedLEASE, 91.19%91.19\%, Table 1). This highlights that our tree-structured alignment (Sec. 5.2) is the dominant performance contributor: precise peer matching already mitigates most heterogeneity without complex routing or dual branches. Finally, Scalar-Mixed achieves the best practicality–performance trade-off, recovering most of the remaining gap with negligible parameter overhead (≈0.020%≈ 0.020\%) and minimal communication cost, making it highly suitable for resource-constrained federated environments. Table 3: Ablation on interaction mechanisms. “Params ↑ ” denotes the relative increase in trainable parameters vs. standard LoRA. Interaction Variant Params ↑ Comm. Avg. Acc. Isolationist (Cluster-Only) 0%0\% Low 91.4091.40 Decomposed Experts ≈0.028%≈ 0.028\% High 92.5792.57 MoE Router 25%25\% Low 92.0292.02 Scalar-Mixed (Ours) ≈0.020%≈ 0.020\% Low 92.36¯ 92.36 (2) Effect of Layer-wise Adaptivity. A core premise of our work is that optimal aggregation granularity varies with model depth to address vertical heterogeneity. To validate this, we compare our adaptive depth search (cl∗c_l^*) against Fixed-Depth strategies, where we force a uniform number of clusters (k=1,4,8k=1,4,8) across all layers. As shown in Table 4, applying a uniform clustering depth yields suboptimal results. A global model (k=1k=1), such as FedIT, underfits deep-layer heterogeneity, while enforcing high granularity (k=8k=8 everywhere) harms shallow-layer representation learning due to data fragmentation. Our adaptive approach outperforms the best fixed setting, confirming that decoupling the aggregation depth across layers is essential for reconciling functional heterogeneity. Table 4: Comparison between Fixed-Depth aggregation (Flat-Model Assumption) and Layer-wise Adaptive aggregation. Aggregation Depth Constraint Avg. Acc. Fixed (k=1k=1) Global Only 87.2087.20 Fixed (k=4k=4) Coarse Clusters 91.4591.45 Fixed (k=8k=8) Fine-grained 90.7490.74 Layer-wise Adaptive Dynamic (cl∗c_l^*) 92.3692.36 (3) Importance of Global Structural Consistency. Finally, we examine the necessity of constructing a global dependency tree (Sec. 5.1) versus performing Independent Layer-wise Clustering. In the latter, clients are clustered at each layer based solely on that layer’s local distance matrix D(l)D^(l), without enforcing the topological constraints of the global tree. Table 5 shows that independent clustering degrades performance. We attribute this to topological incoherence, where client groupings fluctuate chaotically between adjacent layers, disrupting the semantic continuity of the forward pass. By anchoring decisions to a global skeleton, FedTreeLoRA ensures smooth transitions in expert specialization, which is critical for stable fine-tuning. Table 5: Impact of the Global Tree Skeleton on performance. Structural Prior Consistency Avg. Acc. Independent Clustering Low 89.4789.47 Global Tree Skeleton High 92.3692.36 Sensitivity Analysis. We further conduct comprehensive sensitivity analyses to evaluate the robustness of FedTreeLoRA under varying system configurations, including the number of local epochs, LoRA rank, client population size, and degrees of data heterogeneity. We also analyze the impact of our framework-specific hyperparameters: the heterogeneity threshold τ and search range K. The detailed results and discussions are provided in Appendix C. 6.2 Natural Language Generation To validate the generalizability of FedTreeLoRA beyond classification tasks, we evaluate its performance on Natural Language Generation (NLG) benchmarks. NLG Setup. We employ LLaMA-2-7B (Touvron et al., 2023a), quantized to 8-bit precision, as the base model. To construct a realistic heterogeneous federated setting, we utilize four diverse datasets from the FLAN collection (Chung et al., 2024): Text Editing, Struct to Text, Sentiment Analysis, and Commonsense Reasoning. Unlike NLU tasks where heterogeneity is often modeled via label skew, NLG tasks exhibit inherent functional diversity. Therefore, we adopt the task-heterogeneous setup proposed in (Wang et al., 2025): a total of N=8N=8 clients are partitioned such that each dataset is assigned to two specific clients. Each client possesses 600 training samples and 200 test samples. Training is conducted using the AdamW optimizer with a batch size of 8 for E=2E=2 local epochs over T=10T=10 communication rounds. We apply LoRA adapters with rank r=8r=8 to the query and value projections. Learning rates are tuned via grid search over η∈1E−4,3E−4,1E−3,3E−3,1E−2η∈\1E-4,3E-4,1E-3,3E-3,1E-2\. Following standard evaluation protocols (Yang et al., 2024), we report ROUGE-1 scores as the primary metric. Performance Comparison. Table 2 summarizes the results on the NLG benchmarks. Consistent with our NLU findings, FedTreeLoRA achieves the best performance across all four generation tasks, with a parameter budget comparable to FedIT. It outperforms the strongest baselines by a clear margin despite using fewer trainable parameters, with particularly strong gains on tasks requiring structured planning and semantic reasoning (e.g., Text Editing). This confirms that our hierarchical aggregation strategy effectively preserves the delicate generative capabilities required for diverse tasks while mitigating the interference typically caused by aggregating conflicting domains. Additional results on a model from an alternative LLM family are reported in Appendix F. 7 Conclusion In this paper, we challenge the conventional ‘Flat-Model Assumption’ in federated fine-tuning and propose FedTreeLoRA, a framework that reconciles statistical and functional heterogeneity via tree-structured aggregation. By aligning the aggregation granularity with the layer-wise hierarchy of LLMs, sharing shallow linguistic features while personalizing deep semantic reasoning, FedTreeLoRA consistently outperforms state-of-the-art methods across diverse NLU and NLG benchmarks. We achieve these gains with negligible parameter overhead, empirically validating that topological precision is more effective than indiscriminate capacity expansion. Our work offers a new perspective on efficiently scaling personalized federated fine-tuning. Acknowledgments The work of Jieming Bian, Lei Wang and Jie Xu is partially supported by NSF under grants 2433886, 2505381 and 2515982. The work of Letian Zhang is partially supported by NSF under grant 2348279 and also supported by MTSU Stark Land project. Impact Statement This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here. References J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al. (2023) Gpt-4 technical report. arXiv preprint arXiv:2303.08774. Cited by: §1. C. C. Aggarwal, A. Hinneburg, and D. A. Keim (2001) On the surprising behavior of distance metrics in high dimensional space. In International conference on database theory, p. 420–434. Cited by: §5.1. M. G. Arivazhagan, V. Aggarwal, A. K. Singh, and S. Choudhary (2019) Federated learning with personalization layers. arXiv preprint arXiv:1912.00818. Cited by: §2. S. Babakniya, A. R. Elkordy, Y. H. Ezzeldin, Q. Liu, K. Song, M. El-Khamy, and S. Avestimehr (2023) Slora: federated parameter efficient fine-tuning of language models. arXiv preprint arXiv:2308.06522. Cited by: §2. J. Bian, Y. Peng, L. Wang, Y. Huang, and J. Xu (2025a) A survey on parameter-efficient fine-tuning for foundation models in federated learning. arXiv preprint arXiv:2504.21099. Cited by: §1. J. Bian, L. Wang, K. Yang, C. Shen, and J. Xu (2024) Accelerating hybrid federated learning convergence under partial participation. IEEE Transactions on Signal Processing 72 (), p. 3258–3271. External Links: Document Cited by: §2. J. Bian, L. Wang, L. Zhang, and J. Xu (2025b) LoRA-fair: federated lora fine-tuning with aggregation and initialization refinement. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), p. 3737–3746. Cited by: §1. J. Bian, L. Wang, L. Zhang, and J. Xu (2026) FedALT: federated fine-tuning through adaptive local training with rest-of-world lora. Proceedings of the AAAI Conference on Artificial Intelligence 40 (24), p. 19728–19736. External Links: Link, Document Cited by: Table 10, Table 11, Table 12, Table 13, Table 6, Table 7, Table 8, Table 9, Table 21, Appendix H, §1, §2, Table 1, Table 2, §6. H. W. Chung, L. Hou, S. Longpre, B. Zoph, Y. Tay, W. Fedus, Y. Li, X. Wang, M. Dehghani, S. Brahma, et al. (2024) Scaling instruction-finetuned language models. Journal of Machine Learning Research 25 (70), p. 1–53. Cited by: §6.2. J. Devlin, M. Chang, K. Lee, and K. Toutanova (2019) BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), J. Burstein, C. Doran, and T. Solorio (Eds.), Minneapolis, Minnesota, p. 4171–4186. External Links: Link, Document Cited by: §1. A. Fallah, A. Mokhtari, and A. Ozdaglar (2020) Personalized federated learning: a meta-learning approach. arXiv preprint arXiv:2002.07948. Cited by: §2. C. Gao, K. Chen, J. Rao, B. Sun, R. Liu, D. Peng, Y. Zhang, X. Guo, J. Yang, and V. Subrahmanian (2024) Higher layers need more lora experts. arXiv preprint arXiv:2402.08562. Cited by: §1, §2. A. Ghosh, J. Chung, D. Yin, and K. Ramchandran (2020) An efficient framework for clustered federated learning. Advances in neural information processing systems 33, p. 19586–19597. Cited by: §2. P. Guo, S. Zeng, Y. Wang, H. Fan, F. Wang, and L. Qu (2025) Selective aggregation for low-rank adaptation in federated learning. In International Conference on Learning Representations, Cited by: §A.1, §A.1, §A.2, Table 10, Table 11, Table 12, Table 13, Table 6, Table 7, Table 8, Table 9, Table 21, Appendix G, Appendix H, §2, §5.4, §5.4, Table 1, Table 2, §6. Z. Han, C. Gao, J. Liu, J. Zhang, and S. Q. Zhang (2024) Parameter-efficient fine-tuning for large models: a comprehensive survey. arXiv preprint arXiv:2403.14608. Cited by: §1. J. Hao, Y. Wu, A. Payani, M. Lee, and M. Liu (2025) Personalized federated fine-tuning for heterogeneous data: an automatic rank learning approach via two-level lora. arXiv preprint arXiv:2503.03920. Cited by: §1. N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. De Laroussilhe, A. Gesmundo, M. Attariyan, and S. Gelly (2019) Parameter-efficient transfer learning for nlp. In International conference on machine learning, p. 2790–2799. Cited by: §2. E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen (2021) Lora: low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685. Cited by: §1, §2, §3.1. W. Huang, Y. Liao, W. Lei, G. Wan, X. Rong, C. Wen, H. Li, M. Ye, Q. Wu, and B. Du (2026) Federated clinical concept and disease semantic learning for congenital heart disease diagnosis. npj Digital Medicine. Cited by: §2. G. Jawahar, B. Sagot, and D. Seddah (2019) What does BERT learn about the structure of language?. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, A. Korhonen, D. Traum, and L. Màrquez (Eds.), Florence, Italy, p. 3651–3657. External Links: Link, Document Cited by: §2, §4. M. I. Jordan and R. A. Jacobs (1994) Hierarchical mixtures of experts and the em algorithm. Neural computation 6 (2), p. 181–214. Cited by: §5.3, §6.1. S. P. Karimireddy, S. Kale, M. Mohri, S. Reddi, S. Stich, and A. T. Suresh (2020) Scaffold: stochastic controlled averaging for federated learning. In International conference on machine learning, p. 5132–5143. Cited by: §2. T. Le Scao, A. Fan, C. Akiki, E. Pavlick, S. Ilić, D. Hesslow, R. Castagné, A. S. Luccioni, F. Yvon, M. Gallé, et al. (2023) Bloom: a 176b-parameter open-access multilingual language model. Cited by: Appendix F. T. Li, A. K. Sahu, A. Talwalkar, and V. Smith (2020) Federated learning: challenges, methods, and future directions. IEEE signal processing magazine 37 (3), p. 50–60. Cited by: §2. P. P. Liang, T. Liu, L. Ziyin, N. B. Allen, R. P. Auerbach, D. Brent, R. Salakhutdinov, and L. Morency (2020) Think locally, act globally: federated learning with local and global representations. arXiv preprint arXiv:2001.01523. Cited by: §2. J. Liu, Y. Liu, F. Shang, H. Liu, J. Liu, and W. Feng (2025) Improving generalization in federated learning with highly heterogeneous data via momentum-based stochastic controlled weight averaging. In Forty-second International Conference on Machine Learning, Cited by: §2. J. Liu, F. Shang, Y. Liu, H. Liu, Y. Li, and Y. Gong (2024) Fedbcgd: communication-efficient accelerated block coordinate gradient descent for federated learning. In Proceedings of the 32nd ACM International Conference on Multimedia, p. 2955–2963. Cited by: §2. Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov (2019) Roberta: a robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. Cited by: §4, §6.1. B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas (2017) Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, p. 1273–1282. Cited by: §1, §2. Y. Peng, J. Bian, and J. Xu (2024) Fedmm: federated multi-modal learning with modality heterogeneity in computational pathology. In ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Vol. , p. 1696–1700. External Links: Document Cited by: §2. J. Qi, Z. Luan, S. Huang, C. Fung, H. Yang, and D. Qian (2024) FDLoRA: personalized federated learning of large language model via dual lora tuning. arXiv preprint arXiv:2406.07925. Cited by: §1. Y. Qian, Y. Xu, Z. Zhang, P. Zhao, and Z. Zhou (2025) TreeLoRA: efficient continual learning via layer-wise loras guided by a hierarchical gradient-similarity tree. arXiv preprint arXiv:2506.10355. Cited by: Appendix I, Appendix I, §1. P. J. Rousseeuw (1987) Silhouettes: a graphical aid to the interpretation and validation of cluster analysis. Journal of computational and applied mathematics 20, p. 53–65. Cited by: §5.2. F. Sattler, K. Müller, and W. Samek (2020) Clustered federated learning: model-agnostic distributed multitask optimization under privacy constraints. IEEE transactions on neural networks and learning systems 32 (8), p. 3710–3722. Cited by: §2. Y. Sun, Z. Li, Y. Li, and B. Ding (2024) Improving lora in privacy-preserving federated learning. arXiv preprint arXiv:2403.12313. Cited by: Table 10, Table 10, Table 11, Table 11, Table 12, Table 12, Table 13, Table 6, Table 6, Table 7, Table 8, Table 9, Table 9, Table 21, Table 21, Appendix H, Table 1, Table 2, §6. G. Team, R. Anil, S. Borgeaud, J. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, K. Millican, et al. (2023) Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805. Cited by: §1. C. Tian, Z. Shi, Z. Guo, L. Li, and C. Xu (2024) HydraLoRA: an asymmetric lora architecture for efficient fine-tuning. arXiv preprint arXiv:2404.19245. Cited by: §5.1. H. Touvron, T. Lavril, G. Izacard, X. Martinet, M. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al. (2023a) Llama: open and efficient foundation language models. arXiv preprint arXiv:2302.13971. Cited by: §1, §6.2. H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, et al. (2023b) Llama 2: open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288. Cited by: §1. A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman (2018) GLUE: a multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461. Cited by: §4, §6.1. L. Wang, J. Bian, L. Zhang, C. Chen, and J. Xu (2024a) Taming cross-domain representation variance in federated prototype learning with heterogeneous data domains. arXiv preprint arXiv:2403.09048. Cited by: §2. L. Wang, J. Bian, L. Zhang, and J. Xu (2025) Adaptive lora experts allocation and selection for federated fine-tuning. In Advances in Neural Information Processing Systems, D. Belgrave, C. Zhang, H. Lin, R. Pascanu, P. Koniusz, M. Ghassemi, and N. Chen (Eds.), Vol. 38, p. 76018–76045. Cited by: §C.3, Table 10, Table 11, Table 12, Table 13, Table 6, Table 7, Table 8, Table 9, Table 21, Appendix G, Appendix H, §1, §1, §2, §4, Table 1, Table 2, §6.2, §6. Z. Wang, Z. Shen, Y. He, G. Sun, H. Wang, L. Lyu, and A. Li (2024b) Flora: federated fine-tuning large language models with heterogeneous low-rank adaptations. arXiv preprint arXiv:2409.05976. Cited by: §1. Y. Yang, G. Long, T. Shen, J. Jiang, M. Blumenstein, et al. (2024) Dual-personalizing adapter for federated foundation models. Advances in Neural Information Processing Systems 37, p. 39409–39433. Cited by: Table 10, Table 11, Table 12, Table 13, Table 6, Table 7, Table 8, Table 9, Table 21, Appendix H, §1, §2, Table 1, Table 2, §6.2, §6. H. Yu, S. Yang, and S. Zhu (2019) Parallel restarted sgd with faster convergence and less communication: demystifying why model averaging works for deep learning. In Proceedings of the AAAI conference on artificial intelligence, Vol. 33, p. 5693–5700. Cited by: §5.4. J. Zhang, S. Vahidian, M. Kuo, C. Li, R. Zhang, T. Yu, G. Wang, and Y. Chen (2024) Towards building the federatedgpt: federated instruction tuning. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), p. 6915–6919. Cited by: Table 10, Table 11, Table 12, Table 13, Table 6, Table 7, Table 8, Table 9, Table 21, Appendix H, §1, §2, Table 1, Table 2, §6. L. Zhang, B. Chen, J. Bian, L. Wang, and J. Xu (2025a) FedEL: federated elastic learning for heterogeneous devices. In Advances in Neural Information Processing Systems, D. Belgrave, C. Zhang, H. Lin, R. Pascanu, P. Koniusz, M. Ghassemi, and N. Chen (Eds.), Vol. 38, p. 35665–35696. Cited by: §2. Z. Zhang, P. Liu, J. Xu, and R. Hu (2025b) Fed-hello: efficient federated foundation model fine-tuning with heterogeneous lora allocation. IEEE Transactions on Neural Networks and Learning Systems. Cited by: Appendix I. Contents of Appendix This appendix contains the following sections: • Appendix A: Theoretical Convergence Analysis (Appendix A) – Assumptions (Section A.1) – Proof of Theorem 5.1 (Section A.2) • Appendix B: Detailed Motivational Studies (Appendix B) – Experimental Setup (Section B.1) – Extended Observation 1: Impact of Functional Heterogeneity (Section B.2) – Extended Observation 2: Coupling of Dual Heterogeneity (Section B.3) • Appendix C: Sensitivity and Robustness Analysis (Appendix C) – Impact of Framework Hyperparameters (τ and K) (Section C.1) * Impact of Threshold τ * Impact of Search Range K – Robustness to Training Configurations (Section C.2) * Effect of Local Epochs * Effect of LoRA Rank * Impact of Client Numbers – Robustness to Data Distribution Shifts (Section C.3) * Label Distribution Skew * Task Heterogeneity – Impact of Different Distance Metrics (Section C.4) – Effect of Warm-up Duration and Fixed Tree Topology (Section C.5) – Robustness to Dynamic Client Participation (Section C.6) • Appendix D: Detailed Results of Ablation Studies (Appendix D) – Detailed Analysis of Interaction Mechanisms (Section D.1) – Detailed Impact of Layer-wise Adaptivity (Section D.2) – Effect of the Global Tree Skeleton (Section D.3) • Appendix E: Computational Overhead and Wall-Clock Training Time (Appendix E) • Appendix F: Extended NLG Results on an Alternative LLM Family (Appendix F) • Appendix G: Justification for Similarity Measurement via LoRA B Matrices (Appendix G) • Appendix H: Compared Methods (Appendix H) • Appendix I: Clarification with Existing Methods (Appendix I) • Appendix J: Reproducibility and Code Availability (Appendix J) • Appendix K: FedTreeLoRA Algorithm (Appendix K) Appendix A Convergence Analysis: Assumptions and Proof In this appendix we formalize the assumptions used in Section 5.4 and provide a complete proof of Theorem 5.1. We recall that the effective weights of client k at communication round t are denoted by k(t)=Wl,k(t)l=1L,W_k^(t)= \W_l,k^(t) \_l=1^L, and for each layer l the forward pass (cf. Eq. (8)) is hl(x)=W0,lx+λl,k(t)B¯l,kclus,(t)A¯l,kclus,(t)x+(1−λl,k(t))B¯l,kext,(t)A¯l,kext,(t)x.h_l(x)=W_0,lx+ _l,k^(t) B_l,k^clus,(t) A_l,k^clus,(t)x+ (1- _l,k^(t) ) B_l,k^ext,(t) A_l,k^ext,(t)x. (9) During local training in round t, each client k updates only its Cluster Expert (A¯l,kclus,(t),B¯l,kclus,(t)) ( A_l,k^clus,(t), B_l,k^clus,(t) ) and mixing coefficient λl,k(t) _l,k^(t), while the External Expert (A¯l,kext,(t),B¯l,kext,(t)) ( A_l,k^ext,(t), B_l,k^ext,(t) ) is frozen; the experts themselves are obtained by averaging client-level LoRA factors according to the tree-structured aggregation in Eq. (7). A.1 Assumptions We adopt standard smoothness and stochastic gradient assumptions from federated optimization, together with a LoRA-specific boundedness and alignment condition that is in line with the formulation of Guo et al. (2025). Assumption A.1 (σ-smoothness). Each local objective ℒkL_k is σ-smooth with respect to the adapted weights. That is, for any two collections of weights ,′W,W , ℒk(′)≤ℒk()+⟨′−,∇ℒk()⟩F+σ2‖′−‖F2.L_k(W )\;≤\;L_k(W)+ -W,\, _k(W) _F+ σ2 \|W -W \|_F^2. Assumption A.2 (Unbiased stochastic gradients with bounded variance). Let ξk,t,e _k,t,e be sampled uniformly from client k’s local dataset at local step e in communication round t, and let k(t,e)W_k^(t,e) denote the local model at that step (with k(t,0)=k(t)W_k^(t,0)=W_k^(t)). Then ξk,t,e[∇ℒk(k(t,e);ξk,t,e)]=∇ℒk(k(t,e)),ξk,t,e[‖∇ℒk(k(t,e);ξk,t,e)‖F2]≤G2,E_ _k,t,e [ _k(W_k^(t,e); _k,t,e) ]= _k(W_k^(t,e)), _ _k,t,e [ \| _k(W_k^(t,e); _k,t,e) \|_F^2 ]\;≤\;G^2, for some constant G>0G>0. Assumption A.3 (LoRA parameter bounds and alignment). There exist constants MA,MB>0M_A,M_B>0 and μA,μB>0 _A, _B>0 such that, for all clients k, layers l, and rounds t, the underlying client-level LoRA factors Al,k(t)A_l,k^(t) and Bl,k(t)B_l,k^(t) satisfy ‖Al,k(t)‖F≤MA,‖Bl,k(t)‖F≤MB. \|A_l,k^(t) \|_F≤ M_A, \|B_l,k^(t) \|_F≤ M_B. Let Gl,k(t):=∇Wl,kℒk(k(t))G_l,k^(t):= _W_l,kL_k(W_k^(t)) denote the layer-wise gradient at the adapted weights k(t)W_k^(t). We assume the following alignment inequalities hold: ∑l=1L⟨(Al,k(t))⊤Al,k(t),(Gl,k(t))⊤Gl,k(t)⟩F≥μA‖∇ℒk(k(t))‖F2, _l=1^L (A_l,k^(t) ) A_l,k^(t),\, (G_l,k^(t) ) G_l,k^(t) _F\;≥\; _A\, \| _k(W_k^(t)) \|_F^2, (10) ∑l=1L⟨Bl,k(t)(Bl,k(t))⊤,Gl,k(t)(Gl,k(t))⊤⟩F≥μB‖∇ℒk(k(t))‖F2. _l=1^L B_l,k^(t) (B_l,k^(t) ) ,\,G_l,k^(t) (G_l,k^(t) ) _F\;≥\; _B\, \| _k(W_k^(t)) \|_F^2. (11) In FedTreeLoRA, both the Cluster Expert and the External Expert are constructed as (weighted) averages of the client-level factors Al,j(t),Bl,j(t)j\A_l,j^(t),B_l,j^(t)\_j over appropriate subsets defined by the tree structure (cf. Eq. (7)). Therefore, they inherit the same Frobenius-norm bounds and alignment properties, possibly with smaller constants, which we absorb into MA,MB,μA,μBM_A,M_B, _A, _B for notational simplicity. Furthermore, the mixing coefficients satisfy λl,k(t)∈[0,1] _l,k^(t)∈[0,1] for all (l,k,t)(l,k,t). Assumptions A.1 and A.2 are standard in non-convex federated optimization, while Assumption A.3 follows the low-rank alignment formulation of Guo et al. (2025), adapted to the expert-based aggregation of FedTreeLoRA. A.2 Proof of Theorem 5.1 We now prove the convergence result stated in Theorem 5.1. The proof follows the standard “one-round descent + telescoping” structure, but tailored to the low-rank updates and expert mixing in FedTreeLoRA. Notation within one round. Fix a communication round t. We index local steps by e∈0,1,…,Ee∈\0,1,…,E\ and write k(t,0):=k(t),k(t,E):=k(t),W_k^(t,0):=W_k^(t), _k^(t,E):=U_k^(t), for the model at the beginning and after the E local updates, respectively. For client k and layer l, we denote by Al,k(t,e),Bl,k(t,e),λl,k(t,e)A_l,k^(t,e), B_l,k^(t,e), _l,k^(t,e) the trainable low-rank factors and mixing coefficient at step e. These correspond to the Cluster Expert and its mixing weight in Eq. (9); the External Expert remains fixed throughout the round and is thus not indexed by e. Let Gl,k(t,e):=∇Wl,kℒk(k(t,e);ξk,t,e),G_l,k^(t,e):= _W_l,kL_k(W_k^(t,e); _k,t,e), and denote the full gradient at the beginning of the round by ∇ℒk(k(t)):=Gl,k(t,0)l=1L. _k(W_k^(t)):= \G_l,k^(t,0) \_l=1^L. Stage 1: descent during local updates. Using Eq. (9) and the chain rule, the stochastic gradients with respect to the low-rank factors and mixing coefficient at step e can be written as ∇Bl,kℒk(k(t,e);ξk,t,e) _B_l,kL_k(W_k^(t,e); _k,t,e) =λl,k(t,e)Gl,k(t,e)(Al,k(t,e))⊤, = _l,k^(t,e)\,G_l,k^(t,e) (A_l,k^(t,e) ) , (12) ∇Al,kℒk(k(t,e);ξk,t,e) _A_l,kL_k(W_k^(t,e); _k,t,e) =λl,k(t,e)(Bl,k(t,e))⊤Gl,k(t,e), = _l,k^(t,e)\, (B_l,k^(t,e) ) G_l,k^(t,e), (13) ∇λl,kℒk(k(t,e);ξk,t,e) _ _l,kL_k(W_k^(t,e); _k,t,e) =⟨Bl,k(t,e)Al,k(t,e)−B¯l,kext,(t)A¯l,kext,(t),Gl,k(t,e)⟩F. = B_l,k^(t,e)A_l,k^(t,e)- B_l,k^ext,(t) A_l,k^ext,(t),\,G_l,k^(t,e) _F. (14) The SGD updates with stepsize η>0η>0 are Bl,k(t,e+1) B_l,k^(t,e+1) =Bl,k(t,e)−η∇Bl,kℒk(k(t,e);ξk,t,e), =B_l,k^(t,e)-η\, _B_l,kL_k(W_k^(t,e); _k,t,e), (15) Al,k(t,e+1) A_l,k^(t,e+1) =Al,k(t,e)−η∇Al,kℒk(k(t,e);ξk,t,e), =A_l,k^(t,e)-η\, _A_l,kL_k(W_k^(t,e); _k,t,e), (16) λl,k(t,e+1) _l,k^(t,e+1) =λl,k(t,e)−η∇λl,kℒk(k(t,e);ξk,t,e). = _l,k^(t,e)-η\, _ _l,kL_k(W_k^(t,e); _k,t,e). (17) Summing (15) over e and applying the triangle inequality, together with Assumptions A.2 and A.3 (boundedness of Al,k(t,e)A_l,k^(t,e) and Bl,k(t,e)B_l,k^(t,e)), we obtain the incremental bounds ‖Bl,k(t,E)−Bl,k(t,0)‖F \|B_l,k^(t,E)-B_l,k^(t,0) \|_F ≤η∑e=0E−1‖Gl,k(t,e)(Al,k(t,e))⊤‖F≤ηEMAG, ≤η _e=0^E-1 \|G_l,k^(t,e) (A_l,k^(t,e) ) \|_F\;≤\;η EM_AG, (18) ‖Al,k(t,E)−Al,k(t,0)‖F \|A_l,k^(t,E)-A_l,k^(t,0) \|_F ≤η∑e=0E−1‖(Bl,k(t,e))⊤Gl,k(t,e)‖F≤ηEMBG. ≤η _e=0^E-1 \| (B_l,k^(t,e) ) G_l,k^(t,e) \|_F\;≤\;η EM_BG. (19) Similarly, using (14) and the product bound ‖Bl,k(t,e)Al,k(t,e)−B¯l,kext,(t)A¯l,kext,(t)‖F≤2MAMB \|B_l,k^(t,e)A_l,k^(t,e)- B_l,k^ext,(t) A_l,k^ext,(t) \|_F≤ 2M_AM_B (from Assumption A.3 and the construction of the experts), we obtain |λl,k(t,E)−λl,k(t,0)|≤ηE⋅2MAMBG. | _l,k^(t,E)- _l,k^(t,0) |\;≤\;η E· 2M_AM_BG. (20) We now relate the change in the model parameters to the gradient of ℒkL_k. By σ-smoothness (Assumption A.1), we have ℒk(k(t))≤ℒk(k(t))+⟨k(t)−k(t),∇ℒk(k(t))⟩F+σ2‖k(t)−k(t)‖F2.L_k(U_k^(t)) _k(W_k^(t))+ _k^(t)-W_k^(t),\, _k(W_k^(t)) _F+ σ2 \|U_k^(t)-W_k^(t) \|_F^2. (21) The perturbation k(t)−k(t)U_k^(t)-W_k^(t) is induced by the cumulative changes in Al,k(t,e),Bl,k(t,e),λl,k(t,e)l,e\A_l,k^(t,e),B_l,k^(t,e), _l,k^(t,e)\_l,e. Using (18)–(20), the submultiplicativity of the Frobenius norm, and the fact that |λl,k(t,e)|≤1| _l,k^(t,e)|≤ 1, a standard calculation shows that [‖k(t)−k(t)‖F2]≤ 24Lη2E2G2(MA4+MB4+MA4MB4).E [ \|U_k^(t)-W_k^(t) \|_F^2 ]\;≤\;24Lη^2E^2G^2 (M_A^4+M_B^4+M_A^4M_B^4 ). (22) To bound the inner-product term in (21), we compare the actual trajectory with a “reference” gradient step in the low-rank subspace at the beginning of the round. Define Zl,kA,(t):=Gl,k(t,0)(Al,k(t,0))⊤,Zl,kB,(t):=(Bl,k(t,0))⊤Gl,k(t,0).Z_l,k^A,(t):=G_l,k^(t,0) (A_l,k^(t,0) ) , Z_l,k^B,(t):= (B_l,k^(t,0) ) G_l,k^(t,0). Using the Frobenius trace identity ‖XY⊤‖F2=⟨Y⊤Y,X⊤X⟩F\|XY \|_F^2= Y Y,X X _F together with the alignment conditions (10)–(11), we obtain ∑l=1L‖Zl,kA,(t)‖F2 _l=1^L\|Z_l,k^A,(t)\|_F^2 =∑l=1L⟨(Al,k(t,0))⊤Al,k(t,0),(Gl,k(t,0))⊤Gl,k(t,0)⟩F≥μA‖∇ℒk(k(t))‖F2, = _l=1^L (A_l,k^(t,0) ) A_l,k^(t,0),\, (G_l,k^(t,0) ) G_l,k^(t,0) _F\;≥\; _A \| _k(W_k^(t)) \|_F^2, (23) ∑l=1L‖Zl,kB,(t)‖F2 _l=1^L\|Z_l,k^B,(t)\|_F^2 =∑l=1L⟨Bl,k(t,0)(Bl,k(t,0))⊤,Gl,k(t,0)(Gl,k(t,0))⊤⟩F≥μB‖∇ℒk(k(t))‖F2. = _l=1^L B_l,k^(t,0) (B_l,k^(t,0) ) ,\,G_l,k^(t,0) (G_l,k^(t,0) ) _F\;≥\; _B \| _k(W_k^(t)) \|_F^2. (24) Following the same variance-decomposition argument as in Guo et al. (2025), one can show that the cumulative effect of the SGD updates (15)–(17) yields [⟨k(t)−k(t),∇ℒk(k(t))⟩F]≤−η(μA+μB)E[‖∇ℒk(k(t))‖F2]+η2E2LMA2MB2G2.E [ _k^(t)-W_k^(t),\, _k(W_k^(t)) _F ]\;≤\;-η( _A+ _B)EE [ \| _k(W_k^(t)) \|_F^2 ]+η^2E^2LM_A^2M_B^2G^2. (25) The negative term comes from the aligned reference step quantified by (23)–(24), while the O(η2)O(η^2) term collects the higher-order deviations caused by (i) evaluating gradients at intermediate points k(t,e)W_k^(t,e) and (i) the updates of the mixing coefficients λl,k(t,e) _l,k^(t,e). Substituting (22) and (25) into (21), and grouping the O(η2)O(η^2) terms, we obtain the Stage-1 descent inequality [ℒk(Uk(t))]≤[ℒk(Wk(t))] [L_k(U_k^(t))] [L_k(W_k^(t))] −η(μA+μB)E[‖∇ℒk(Wk(t))‖F2] -η( _A+ _B)EE[|| _k(W_k^(t))||_F^2] (26) +η2E2LMA2MB2G2+12σLη2E2G2(MA4+MB4+MA4MB4). +η^2E^2LM_A^2M_B^2G^2+2σ Lη^2E^2G^2(M_A^4+M_B^4+M_A^4M_B^4). Stage 2: tree-structured aggregation. We now make precise the effect of the tree-structured aggregation: (i) the mismatch between a client’s locally updated Cluster Expert and the aggregated Cluster Expert shared by all peers in the same node; and (i) the change in the External Expert induced by the peers outside the node. Fix a round t, a client k, and a layer l. After the E local updates in Stage 1, the layer used by client k has the form Ul,k(t)=W0,l+λl,k(t,E)Bl,k(t,E)Al,k(t,E)+(1−λl,k(t,E))B¯l,kext,(t)A¯l,kext,(t),U_l,k^(t)=W_0,l+ _l,k^(t,E)\,B_l,k^(t,E)A_l,k^(t,E)+ (1- _l,k^(t,E) )\, B_l,k^ext,(t) A_l,k^ext,(t), (27) where Al,k(t,E),Bl,k(t,E),λl,k(t,E)A_l,k^(t,E),B_l,k^(t,E), _l,k^(t,E) are the locally updated Cluster Expert and mixing coefficient, and A¯l,kext,(t),B¯l,kext,(t) A_l,k^ext,(t), B_l,k^ext,(t) is the (frozen) External Expert from the previous round. Let k(l)S_k^(l) denote the set of clients that share the same tree node as k at layer l, and ℛk(l)R_k^(l) its complement (cf. Section 5.3). Tree-structured aggregation at the end of round t updates the experts as follows (cf. Eq. (7)): A¯l,kclus,(t+1) A_l,k^clus,(t+1) =1|k(l)|∑j∈k(l)Al,j(t,E), = 1|S_k^(l)| _j _k^(l)A_l,j^(t,E), B¯l,kclus,(t+1)=1|k(l)|∑j∈k(l)Bl,j(t,E), B_l,k^clus,(t+1)= 1|S_k^(l)| _j _k^(l)B_l,j^(t,E), (28) A¯l,kext,(t+1) A_l,k^ext,(t+1) =1|ℛk(l)|∑j∈ℛk(l)Al,j(t,E), = 1|R_k^(l)| _j _k^(l)A_l,j^(t,E), B¯l,kext,(t+1)=1|ℛk(l)|∑j∈ℛk(l)Bl,j(t,E). B_l,k^ext,(t+1)= 1|R_k^(l)| _j _k^(l)B_l,j^(t,E). The post-aggregation layer used by client k at layer l in the next round is Wl,k(t+1)=W0,l+λl,k(t,E)B¯l,kclus,(t+1)A¯l,kclus,(t+1)+(1−λl,k(t,E))B¯l,kext,(t+1)A¯l,kext,(t+1).W_l,k^(t+1)=W_0,l+ _l,k^(t,E)\, B_l,k^clus,(t+1) A_l,k^clus,(t+1)+ (1- _l,k^(t,E) )\, B_l,k^ext,(t+1) A_l,k^ext,(t+1). (29) Subtracting (27) from (29), we obtain the decomposition Wl,k(t+1)−Ul,k(t) W_l,k^(t+1)-U_l,k^(t) =λl,k(t,E)(B¯l,kclus,(t+1)A¯l,kclus,(t+1)−Bl,k(t,E)Al,k(t,E))⏟(i) Cluster Expert aggregation difference = _l,k^(t,E) ( B_l,k^clus,(t+1) A_l,k^clus,(t+1)-B_l,k^(t,E)A_l,k^(t,E) )_(i) Cluster Expert aggregation difference (30) +(1−λl,k(t,E))(B¯l,kext,(t+1)A¯l,kext,(t+1)−B¯l,kext,(t)A¯l,kext,(t))⏟(i) External Expert update difference. + (1- _l,k^(t,E) ) ( B_l,k^ext,(t+1) A_l,k^ext,(t+1)- B_l,k^ext,(t) A_l,k^ext,(t) )_(i) External Expert update difference. We now bound the two terms separately in Frobenius norm. (i) Cluster Expert aggregation difference. Fix l and k(l)S_k^(l). By construction of FedTreeLoRA, all clients j∈k(l)j _k^(l) share the same Cluster Expert at the beginning of round t: Al,j(t,0)=Al,k(t,0),Bl,j(t,0)=Bl,k(t,0)∀j∈k(l).A_l,j^(t,0)=A_l,k^(t,0), B_l,j^(t,0)=B_l,k^(t,0) ∀ j _k^(l). After E local updates, we can write Al,j(t,E)=Al,k(t,0)+ΔAl,j(t),Bl,j(t,E)=Bl,k(t,0)+ΔBl,j(t),A_l,j^(t,E)=A_l,k^(t,0)+ A_l,j^(t), B_l,j^(t,E)=B_l,k^(t,0)+ B_l,j^(t), with ‖ΔAl,j(t)‖F≤ηEMBG\| A_l,j^(t)\|_F≤η EM_BG and ‖ΔBl,j(t)‖F≤ηEMAG\| B_l,j^(t)\|_F≤η EM_AG for all j by (19)–(18). In particular, Al,k(t,E)=Al,k(t,0)+ΔAl,k(t)A_l,k^(t,E)=A_l,k^(t,0)+ A_l,k^(t), Bl,k(t,E)=Bl,k(t,0)+ΔBl,k(t)B_l,k^(t,E)=B_l,k^(t,0)+ B_l,k^(t). From (28), the aggregated Cluster Expert can be expressed as A¯l,kclus,(t+1)=Al,k(t,0)+1|k(l)|∑j∈k(l)ΔAl,j(t),B¯l,kclus,(t+1)=Bl,k(t,0)+1|k(l)|∑j∈k(l)ΔBl,j(t). A_l,k^clus,(t+1)=A_l,k^(t,0)+ 1|S_k^(l)| _j _k^(l) A_l,j^(t), B_l,k^clus,(t+1)=B_l,k^(t,0)+ 1|S_k^(l)| _j _k^(l) B_l,j^(t). Therefore, A¯l,kclus,(t+1)−Al,k(t,E) A_l,k^clus,(t+1)-A_l,k^(t,E) =1|k(l)|∑j∈k(l)ΔAl,j(t)−ΔAl,k(t), = 1|S_k^(l)| _j _k^(l) A_l,j^(t)- A_l,k^(t), (31) B¯l,kclus,(t+1)−Bl,k(t,E) B_l,k^clus,(t+1)-B_l,k^(t,E) =1|k(l)|∑j∈k(l)ΔBl,j(t)−ΔBl,k(t). = 1|S_k^(l)| _j _k^(l) B_l,j^(t)- B_l,k^(t). (32) Using the triangle inequality and the bounds on ΔAl,j(t) A_l,j^(t), ΔBl,j(t) B_l,j^(t), we obtain ‖A¯l,kclus,(t+1)−Al,k(t,E)‖F \| A_l,k^clus,(t+1)-A_l,k^(t,E) \|_F ≤1|k(l)|∑j∈k(l)‖ΔAl,j(t)‖F+‖ΔAl,k(t)‖F≤ 2ηEMBG, ≤ 1|S_k^(l)| _j _k^(l) \| A_l,j^(t) \|_F+ \| A_l,k^(t) \|_F\;≤\;2η EM_BG, (33) ‖B¯l,kclus,(t+1)−Bl,k(t,E)‖F \| B_l,k^clus,(t+1)-B_l,k^(t,E) \|_F ≤2ηEMAG. ≤ 2η EM_AG. (34) Moreover, since each Al,j(t,E),Bl,j(t,E)A_l,j^(t,E),B_l,j^(t,E) remains inside the Frobenius ball of radius MA,MBM_A,M_B (Assumption A.3), their averages also satisfy ‖A¯l,kclus,(t+1)‖F≤MA,‖B¯l,kclus,(t+1)‖F≤MB. \| A_l,k^clus,(t+1) \|_F≤ M_A, \| B_l,k^clus,(t+1) \|_F≤ M_B. We now bound the difference between the two Cluster-Expert products. Using B¯A¯−BA=B¯(A¯−A)+(B¯−B)A B A-BA= B( A-A)+( B-B)A and the submultiplicativity of the Frobenius norm, we have ‖B¯l,kclus,(t+1)A¯l,kclus,(t+1)−Bl,k(t,E)Al,k(t,E)‖F \| B_l,k^clus,(t+1) A_l,k^clus,(t+1)-B_l,k^(t,E)A_l,k^(t,E) \|_F ≤‖B¯l,kclus,(t+1)‖F‖A¯l,kclus,(t+1)−Al,k(t,E)‖F+‖B¯l,kclus,(t+1)−Bl,k(t,E)‖F‖Al,k(t,E)‖F ≤ \| B_l,k^clus,(t+1) \|_F \| A_l,k^clus,(t+1)-A_l,k^(t,E) \|_F+ \| B_l,k^clus,(t+1)-B_l,k^(t,E) \|_F \|A_l,k^(t,E) \|_F ≤MB⋅2ηEMBG+MA⋅2ηEMAG ≤ M_B· 2η EM_BG+M_A· 2η EM_AG =2ηEG(MA2+MB2). =2η EG (M_A^2+M_B^2 ). (35) Multiplying by |λl,k(t,E)|≤1| _l,k^(t,E)|≤ 1 and using (a+b)2≤2(a2+b2)(a+b)^2≤ 2(a^2+b^2), we deduce that the squared norm of the Cluster Expert aggregation difference in (30) is bounded by ‖λl,k(t,E)(B¯l,kclus,(t+1)A¯l,kclus,(t+1)−Bl,k(t,E)Al,k(t,E))‖F2≤ 8η2E2G2(MA4+MB4), \| _l,k^(t,E) ( B_l,k^clus,(t+1) A_l,k^clus,(t+1)-B_l,k^(t,E)A_l,k^(t,E) ) \|_F^2\;≤\;8η^2E^2G^2 (M_A^4+M_B^4 ), (36) up to an absolute numerical constant. (i) External Expert update difference. We now turn to the second term in (30). At the beginning of round t, all clients in ℛk(l)R_k^(l) share the same External Expert A¯l,kext,(t),B¯l,kext,(t) A_l,k^ext,(t), B_l,k^ext,(t), which is a fixed average of Cluster Experts from the previous round. During local training in round t, this External Expert is kept frozen, and only the Cluster Experts Al,j(t,e),Bl,j(t,e)\A_l,j^(t,e),B_l,j^(t,e)\ are updated. At the end of the round, the new External Expert is defined in (28) as an average of the updated Cluster Experts of clients in ℛk(l)R_k^(l): A¯l,kext,(t+1)=1|ℛk(l)|∑j∈ℛk(l)Al,j(t,E),B¯l,kext,(t+1)=1|ℛk(l)|∑j∈ℛk(l)Bl,j(t,E). A_l,k^ext,(t+1)= 1|R_k^(l)| _j _k^(l)A_l,j^(t,E), B_l,k^ext,(t+1)= 1|R_k^(l)| _j _k^(l)B_l,j^(t,E). Writing again Al,j(t,E)=Al,j(t,0)+ΔAl,j(t)A_l,j^(t,E)=A_l,j^(t,0)+ A_l,j^(t), Bl,j(t,E)=Bl,j(t,0)+ΔBl,j(t)B_l,j^(t,E)=B_l,j^(t,0)+ B_l,j^(t) and using the fact that A¯l,kext,(t) A_l,k^ext,(t) and B¯l,kext,(t) B_l,k^ext,(t) are averages of Al,j(t,0),Bl,j(t,0)j∈ℛk(l)\A_l,j^(t,0),B_l,j^(t,0)\_j _k^(l), we can write A¯l,kext,(t+1)−A¯l,kext,(t)=1|ℛk(l)|∑j∈ℛk(l)ΔAl,j(t),B¯l,kext,(t+1)−B¯l,kext,(t)=1|ℛk(l)|∑j∈ℛk(l)ΔBl,j(t). A_l,k^ext,(t+1)- A_l,k^ext,(t)= 1|R_k^(l)| _j _k^(l) A_l,j^(t), B_l,k^ext,(t+1)- B_l,k^ext,(t)= 1|R_k^(l)| _j _k^(l) B_l,j^(t). By (19)–(18), ‖A¯l,kext,(t+1)−A¯l,kext,(t)‖F≤ηEMBG,‖B¯l,kext,(t+1)−B¯l,kext,(t)‖F≤ηEMAG, \| A_l,k^ext,(t+1)- A_l,k^ext,(t) \|_F≤η EM_BG, \| B_l,k^ext,(t+1)- B_l,k^ext,(t) \|_F≤η EM_AG, (37) and both External Experts remain bounded: ‖A¯l,kext,(t)‖F,‖A¯l,kext,(t+1)‖F≤MA \| A_l,k^ext,(t) \|_F, \| A_l,k^ext,(t+1) \|_F≤ M_A, ‖B¯l,kext,(t)‖F,‖B¯l,kext,(t+1)‖F≤MB \| B_l,k^ext,(t) \|_F, \| B_l,k^ext,(t+1) \|_F≤ M_B. Analogously to (35), we obtain ‖B¯l,kext,(t+1)A¯l,kext,(t+1)−B¯l,kext,(t)A¯l,kext,(t)‖F \| B_l,k^ext,(t+1) A_l,k^ext,(t+1)- B_l,k^ext,(t) A_l,k^ext,(t) \|_F ≤MB⋅ηEMBG+MA⋅ηEMAG ≤ M_B·η EM_BG+M_A·η EM_AG =ηEG(MA2+MB2), =η EG (M_A^2+M_B^2 ), (38) and therefore ‖(1−λl,k(t,E))(B¯l,kext,(t+1)A¯l,kext,(t+1)−B¯l,kext,(t)A¯l,kext,(t))‖F2≤ 2η2E2G2(MA4+MB4), \| (1- _l,k^(t,E) ) ( B_l,k^ext,(t+1) A_l,k^ext,(t+1)- B_l,k^ext,(t) A_l,k^ext,(t) ) \|_F^2\;≤\;2η^2E^2G^2 (M_A^4+M_B^4 ), (39) again up to an absolute constant. Combining (i) and (i). By (30), the per-layer difference satisfies ‖Wl,k(t+1)−Ul,k(t)‖F2≤ 2‖(i)‖F2+2‖(i)‖F2, \|W_l,k^(t+1)-U_l,k^(t) \|_F^2\;≤\;2 \|(i) \|_F^2+2 \|(i) \|_F^2, so that, combining (36) and (39), ‖Wl,k(t+1)−Ul,k(t)‖F2≤C0η2E2G2(MA4+MB4), \|W_l,k^(t+1)-U_l,k^(t) \|_F^2\;≤\;C_0\,η^2E^2G^2 (M_A^4+M_B^4 ), (40) for some universal constant C0>0C_0>0. Summing over l=1,…,Ll=1,…,L and taking expectation, we obtain the aggregate bound [‖k(t+1)−k(t)‖F2]≤C0Lη2E2G2(MA4+MB4),E [ \|W_k^(t+1)-U_k^(t) \|_F^2 ]\;≤\;C_0Lη^2E^2G^2 (M_A^4+M_B^4 ), (41) Applying Assumption A.1 at k(t)U_k^(t), setting the learning rate η=O(1EL)η=O( 1EL) and using Cauchy–Schwarz to bound the corresponding inner-product term (41), we obtain [ℒk(k(t+1))]≤[ℒk(k(t))]+C~ση2E2LG2(MA4+MB4),E [L_k(W_k^(t+1)) ]\;≤\;E [L_k(U_k^(t)) ]+ C\,ση^2E^2LG^2 (M_A^4+M_B^4 ), (42) for some absolute constant C~>0 C>0. One-round recursion and telescoping. Combining the Stage 1 descent inequality (cf. Eq. (26)) with the Stage 2 aggregation bound (42), we obtain the following one-round recursion for each client k and communication round t: [ℒk(k(t+1))]≤[ℒk(k(t))]−η(μA+μB)E[‖∇ℒk(k(t))‖F2]+η2E2Γ,E [L_k(W_k^(t+1)) ]\;≤\;E [L_k(W_k^(t)) ]-η( _A+ _B)EE [\| _k(W_k^(t))\|_F^2 ]+η^2E^2 , (43) where Γ is given by Γ:=LMA2MB2G2+CσLG2(MA4+MB4+MA4MB4), :=LM_A^2M_B^2G^2+C\,σ LG^2 (M_A^4+M_B^4+M_A^4M_B^4 ), (44) for some absolute constant C>0C>0. Summing (43) over t=0,…,T−1t=0,…,T-1 and using telescoping yields ∑t=0T−1[‖∇ℒk(k(t))‖F2]≤ΔkηE(μA+μB)+ηEΓTμA+μB, _t=0^T-1E [ \| _k(W_k^(t)) \|_F^2 ]\;≤\; _kη E( _A+ _B)+ η E T _A+ _B, (45) where Δk:=ℒk(k(0))−ℒk(k⋆) _k:=L_k(W_k^(0))-L_k(W_k ) is the initial function gap for client k. Let Δ≥maxk⁡Δk ≥ _k _k, sum over all k, and divide both sides by NTNT to obtain 1NT∑k=1N∑t=0T−1[‖∇ℒk(k(t))‖F2]≤ΔηE(μA+μB)+ηEΓμA+μB. 1NT _k=1^N _t=0^T-1E [ \| _k(W_k^(t)) \|_F^2 ]\;≤\; η E( _A+ _B)+ η E _A+ _B. (46) Choosing ηE=Δ/(ΓT)η E= /( T) minimizes the right-hand side of (46) and yields 1NT∑k=1N∑t=1T[‖∇ℒk(k(t))‖F2]≤2μA+μBΔ⋅ΓT, 1NT _k=1^N _t=1^TE [ \| _k(W_k^(t)) \|_F^2 ]\;≤\; 2 _A+ _B · T, which coincides with the convergence guarantee stated in Theorem 5.1. Appendix B Detailed Motivational Studies In this section, we provide the comprehensive experimental setup and extended results for the motivational studies discussed in Section 4. These studies aim to further substantiate our observations regarding vertical heterogeneity and the coupling between model depth and data similarity across a broader range of tasks. (a) Obs 1: SST2 & QQP (b) Obs 2: QNLI (c) Obs 2: SST2 (d) Obs 2: QQP Figure 5: Extended Motivational Studies. (a) Substantiates Observation 1 (Vertical Heterogeneity) on SST2 and QQP datasets. (b)–(d) Substantiate Observation 2 (Coupling Effect) across different tasks, confirming that the optimal sharing boundary consistently shifts towards shallower layers as data heterogeneity increases. B.1 Experimental Setup For all motivational studies, we employ RoBERTa-Large as the backbone model. We utilize the GLUE benchmark datasets, including MNLI, QNLI, SST2 and QQP. Following standard federated settings, data is partitioned using a Dirichlet distribution Dir(α)Dir(α). To observe the impact of layer-wise aggregation, we divide the 24 Transformer layers into Shallow (layers 1–12) and Deep (layers 13–24). B.2 Extended Analysis of Observation 1: Impact of Functional Heterogeneity In the main paper, Figure 1 illustrates that aggregating shallow layers is more stable than aggregating deep layers for MNLI and QNLI with 10 clients (α=0.5α=0.5), where each client holds 1000 training samples. We extend this analysis to SST2 and QQP. As shown in Figure 5(a), we observe a consistent trend across all four datasets: deep-layer aggregation frequently leads to performance degradation (negative transfer) compared to local-only training. This confirms that the functional role of shallow layers as general feature extractors is universal across diverse NLU tasks. B.3 Extended Analysis of Observation 2: Coupling of Dual Heterogeneity Rationale for the Controlled 2-Client Setup. To rigorously isolate the coupling effect between functional depth and statistical heterogeneity, we prioritize a controlled 2-client environment over a large-scale population with stochastic Dirichlet-based partitioning (e.g., N=20,α=0.5N=20,α=0.5). The primary rationale is the avoidance of spurious similarities inherent in random sampling. In a multi-client Dirichlet setting, even under low global α, sub-groups of clients may accidentally share near-identical distributions. Such stochastic interference masks the precise layer-wise divergence: if two clients are randomly similar, separating them at shallow layers would fail to capture the true performance gap that occurs when they could have safely shared deeper parameters. By utilizing two clients with strictly defined label skew profiles, we eliminate these confounding variables and directly measure how the “safe” aggregation boundary moves in response to deterministic distribution shifts. Experimental Settings and Label Skew Profiles. We conduct evaluations on RoBERTa-Large (24 layers) across three settings. For binary classification tasks (QNLI, QQP, SST2), the label distributions for Client 1 and Client 2 are defined as: Homogeneous: [0.50,0.50][0.50,0.50] vs. [0.50,0.50][0.50,0.50]; Moderate: [0.60,0.40][0.60,0.40] vs. [0.40,0.60][0.40,0.60]; and Heterogeneous: [0.70,0.30][0.70,0.30] vs. [0.30,0.70][0.30,0.70]. For ternary classification task (MNLI) with distributions [p1,p2,p3][p_1,p_2,p_3], the settings are: Homogeneous with both clients sharing [0.40,0.40,0.20][0.40,0.40,0.20]; Moderate with Client 1: [0.50,0.30,0.20][0.50,0.30,0.20] and Client 2: [0.30,0.50,0.10][0.30,0.50,0.10]; and Heterogeneous with Client 1: [0.60,0.20,0.20][0.60,0.20,0.20] and Client 2: [0.20,0.60,0.20][0.20,0.60,0.20]. Consistent with the findings in Figure 2, these results in Figure 5 demonstrate a systematic phase transition: as label skew intensifies, the optimal aggregation depth shifts from 24 layers toward the first 8 layers. This dataset-agnostic behavior underscores that parameter sharing must be dynamically aligned with the measured similarity between clients, a core capability of the FedTreeLoRA framework. Appendix C Sensitivity and Robustness Analysis In this section, we provide a comprehensive evaluation of FedTreeLoRA’s robustness against variations in framework-specific hyperparameters, training configurations, and data distribution settings. Unless otherwise stated, the experimental setup follows the default configuration used in the main NLU experiments (N=20,α=0.5,r=4,E=2N=20,α=0.5,r=4,E=2). C.1 Impact of Framework Hyperparameters (τ and K) Our framework introduces two hyperparameters that jointly govern the evolution of the aggregation tree: the heterogeneity threshold τ, which regulates the resistance to splitting when deciding whether to transition from a globally shared cluster to finer partitions, and the search range K, which constrains how rapidly the model is allowed to refine aggregation granularity across Transformer depth. Figure 6: Sensitivity analysis of hyperparameters τ and K. FedTreeLoRA demonstrates a clear robustness plateau within τ∈[0,0.06]τ∈[0,0.06] and K∈3,4,5K∈\3,4,5\, while still outperforming all baselines even in suboptimal settings. Impact of Threshold τ. Recall that τ acts as the baseline score for the single-cluster case (c=1c=1), and thus determines how much statistical evidence must accumulate before a layer is allowed to branch. Empirically, the layer-wise Silhouette statistics in FedTreeLoRA naturally lie in the order of 10−210^-2 across tasks, implying that the “effective” decision scale of τ is inherently small. Motivated by this observation, we sweep τ over a range that is already wide relative to this intrinsic scale (τ∈[−0.02,0.10]τ∈[-0.02,0.10]), as shown in Figure 6(a). Within the range τ∈[0,0.06]τ∈[0,0.06], FedTreeLoRA exhibits a clear robustness plateau: performance remains highly stable across MNLI, QNLI, QQP, and SST-2, indicating that τ does not require fine-grained tuning. Outside this interval, the system behaves consistently with our design intuition. When τ<0τ<0, splitting is triggered too aggressively, leading to premature fragmentation and mild performance drops. When τ becomes overly conservative (e.g., τ=0.10τ=0.10), branching is postponed and the tree drifts toward a near-global solution, again slightly hurting accuracy. Overall, these results confirm that τ is a meaningful yet non-fragile control signal: as long as it is chosen within the natural Silhouette scale, FedTreeLoRA reliably induces appropriate hierarchical specialization. Impact of Search Range K. Figure 6(b) analyzes the effect of the monotonicity-constrained search window K, which controls how smoothly the layer-wise specialization structure is allowed to evolve with depth. When K is too small (K=2K=2), the framework has limited freedom to adjust aggregation structure across layers, resulting in slightly weaker performance, particularly on MNLI and QNLI. Expanding the window to a moderate range (K∈3,4,5K∈\3,4,5\) consistently yields the best accuracy and forms a stable robustness plateau, indicating that FedTreeLoRA benefits from controlled structural flexibility without requiring delicate tuning. Interestingly, for QQP and SST-2 the curves remain almost flat once K≥2K≥ 2. This behavior is fully consistent with the layer-wise structural patterns observed in Figure 4: for these datasets, the hierarchy stabilizes extremely early, typically transitioning only once from a global cluster (11) to two subgroups (22) before remaining stable. As a result, even a small search window already suffices to capture their limited specialization behavior, making them inherently less sensitive to K. When K becomes excessively large (K=20K=20), we observe mild degradation mainly on NLI tasks, supporting our hypothesis that overly aggressive structural jumps introduce topological incoherence and noisier specialization. Nevertheless, even under the weakest configurations (K=2K=2 or K=20K=20), FedTreeLoRA still outperforms all baseline methods in Table 1, while the moderate range K∈3,4,5K∈\3,4,5\ delivers the strongest and most stable improvements. C.2 Robustness to Training Configurations We investigate the adaptability of FedTreeLoRA under varying resource constraints and training intensities. Effect of Local Epochs. To further assess the robustness of our method under different local training budgets, we increase the number of local epochs to E=4E=4. As shown in Table 6, FedTreeLoRA consistently maintains its superiority. Our tree-structured aggregation effectively mitigates drift by ensuring that clients only aggregate with structurally similar peers, thereby preserving high performance even with increased local steps. Table 6: Performance with increased Local Epochs (E=4E=4). Methods MNLI QNLI SST2 QQP Average FedIT (Zhang et al., 2024) 82.47±0.3682.47± 0.36 86.36±0.3786.36± 0.37 93.01±0.9993.01± 0.99 83.83±0.6083.83± 0.60 86.4286.42 FFA-LoRA (r=4r=4) (Sun et al., 2024) 82.68±0.4982.68± 0.49 90.32±0.2590.32± 0.25 93.32±0.1093.32± 0.10 83.16±0.4283.16± 0.42 87.3787.37 FFA-LoRA (r=8r=8) (Sun et al., 2024) 82.75±0.5982.75± 0.59 90.73±0.6490.73± 0.64 93.82±0.1893.82± 0.18 83.66±0.1783.66± 0.17 87.7487.74 FedSA (Guo et al., 2025) 84.12±0.6484.12± 0.64 92.15±0.4592.15± 0.45 96.07±0.4196.07± 0.41 89.15±0.1089.15± 0.10 90.3790.37 FedDPA (Yang et al., 2024) 84.69±0.5484.69± 0.54 92.20±0.5392.20± 0.53 95.91±0.4095.91± 0.40 89.50±0.2589.50± 0.25 90.5890.58 FedALT (Bian et al., 2026) 86.28±0.2586.28± 0.25 91.70±0.3891.70± 0.38 96.51±0.2096.51± 0.20 88.97±0.5388.97± 0.53 90.8790.87 FedLEASE (Wang et al., 2025) 86.48±0.3486.48± 0.34 93.51±0.2793.51± 0.27 96.18±0.5896.18± 0.58 90.13±0.3990.13± 0.39 91.5891.58 FedTreeLoRA (Ours) 88.27±0.1888.27± 0.18 94.03±0.1494.03± 0.14 96.88±0.1596.88± 0.15 92.07±0.2892.07± 0.28 92.8192.81 Effect of LoRA Rank. To assess parameter efficiency and capacity, we evaluated performance under a low-rank regime (r=2r=2) and a high-rank regime (r=6r=6). The results are summarized in Table 7 and 8. FedTreeLoRA maintains its lead over baselines in both settings. In the strictly constrained r=2r=2 setting, the efficiency of our layer-wise expert allocation becomes critical, yielding gains over the rigid allocation strategies of methods like FedLEASE. Table 7: Performance comparison under different LoRA Ranks (r=2r=2). Methods MNLI QNLI SST2 QQP Average FedIT (Zhang et al., 2024) 81.08±0.4681.08± 0.46 86.15±0.5886.15± 0.58 92.42±0.5092.42± 0.50 83.78±0.4683.78± 0.46 85.8685.86 FFA-LoRA (Sun et al., 2024) 81.74±0.5681.74± 0.56 86.41±0.1986.41± 0.19 93.13±0.3793.13± 0.37 83.16±0.7383.16± 0.73 86.1186.11 FedSA (Guo et al., 2025) 82.55±0.9082.55± 0.90 90.79±0.2790.79± 0.27 95.24±0.5595.24± 0.55 89.33±0.4989.33± 0.49 89.4889.48 FedDPA (Yang et al., 2024) 82.74±0.6082.74± 0.60 91.07±0.3391.07± 0.33 95.16±0.1995.16± 0.19 89.69±0.6789.69± 0.67 89.6789.67 FedALT (Bian et al., 2026) 84.45±0.3084.45± 0.30 90.90±0.1890.90± 0.18 95.83±0.1795.83± 0.17 89.49±0.5489.49± 0.54 90.1790.17 FedLEASE (Wang et al., 2025) 85.49±0.2985.49± 0.29 92.77±0.2092.77± 0.20 95.02±0.2795.02± 0.27 90.35±0.3890.35± 0.38 90.9190.91 FedTreeLoRA (Ours) 86.60±0.4386.60± 0.43 93.48±0.1693.48± 0.16 96.30±0.3996.30± 0.39 91.46±0.0891.46± 0.08 91.9691.96 Table 8: Performance comparison under different LoRA Ranks (r=6r=6). Methods MNLI QNLI SST2 QQP Average FedIT (Zhang et al., 2024) 82.84±0.2482.84± 0.24 87.78±0.5287.78± 0.52 93.61±0.5293.61± 0.52 84.17±0.2984.17± 0.29 87.1087.10 FFA-LoRA (Sun et al., 2024) 82.78±0.9382.78± 0.93 88.43±0.5888.43± 0.58 93.87±0.6993.87± 0.69 83.49±0.7883.49± 0.78 87.1487.14 FedSA (Guo et al., 2025) 83.70±0.2483.70± 0.24 91.30±0.8591.30± 0.85 96.05±0.1496.05± 0.14 89.83±0.6789.83± 0.67 90.2290.22 FedDPA (Yang et al., 2024) 84.03±0.5384.03± 0.53 91.20±0.8291.20± 0.82 96.02±0.3496.02± 0.34 89.45±0.5389.45± 0.53 90.1890.18 FedALT (Bian et al., 2026) 81.45±0.4181.45± 0.41 91.93±0.1391.93± 0.13 96.12±0.4296.12± 0.42 89.50±0.1889.50± 0.18 89.7589.75 FedLEASE (Wang et al., 2025) 86.03±0.1586.03± 0.15 93.34±0.3693.34± 0.36 95.25±0.7995.25± 0.79 90.66±0.2090.66± 0.20 91.3291.32 FedTreeLoRA (Ours) 88.37±0.3288.37± 0.32 94.13±0.2294.13± 0.22 96.71±0.0896.71± 0.08 91.45±0.2091.45± 0.20 92.6792.67 Impact of Client Numbers. We evaluate scalability by varying the system size to N=10N=10 and N=40N=40 while maintaining the same total dataset size to simulate data fragmentation. As shown in Table 9 and 10, FedTreeLoRA demonstrates superior scalability. Flat clustering methods could struggle to form robust clusters due to noise. In contrast, our hierarchical structure allows sparse clients to share knowledge effectively at the root (shallow layers) while specializing only where necessary. Table 9: Performance comparison under different Client Numbers (N=10N=10). Methods MNLI QNLI SST2 QQP Average FedIT (Zhang et al., 2024) 82.33±0.5082.33± 0.50 89.20±0.3589.20± 0.35 93.14±0.2193.14± 0.21 84.91±0.5784.91± 0.57 87.4087.40 FFA-LoRA (r = 4) (Sun et al., 2024) 82.40±0.8982.40± 0.89 89.68±0.8589.68± 0.85 93.34±0.1693.34± 0.16 84.93±0.7484.93± 0.74 87.5987.59 FFA-LoRA (r = 8) (Sun et al., 2024) 83.33±0.3583.33± 0.35 90.10±0.2190.10± 0.21 93.60±0.4893.60± 0.48 85.43±0.7385.43± 0.73 88.1288.12 FedSA (Guo et al., 2025) 83.52±0.9283.52± 0.92 90.78±0.3590.78± 0.35 94.75±0.6394.75± 0.63 91.35±0.1791.35± 0.17 90.1090.10 FedDPA (Yang et al., 2024) 83.73±0.3083.73± 0.30 90.25±0.4790.25± 0.47 94.91±0.2394.91± 0.23 91.15±0.1391.15± 0.13 90.0190.01 FedALT (Bian et al., 2026) 83.50±0.7183.50± 0.71 90.68±0.6090.68± 0.60 94.72±0.4994.72± 0.49 91.14±0.1691.14± 0.16 90.0190.01 FedLEASE (Wang et al., 2025) 86.19±0.1286.19± 0.12 91.97±0.2091.97± 0.20 94.82±0.4194.82± 0.41 91.45±0.2691.45± 0.26 91.1191.11 FedTreeLoRA (Ours) 87.10±0.2187.10± 0.21 92.88±0.4292.88± 0.42 95.76±0.3095.76± 0.30 92.65±0.3692.65± 0.36 92.1092.10 Table 10: Performance comparison under different Client Numbers (N=40N=40). Methods MNLI QNLI SST2 QQP Average FedIT (Zhang et al., 2024) 79.28±0.5679.28± 0.56 85.39±0.7485.39± 0.74 91.99±0.7191.99± 0.71 80.05±0.9780.05± 0.97 84.1884.18 FFA-LoRA (r = 4) (Sun et al., 2024) 81.71±0.4681.71± 0.46 86.62±0.7986.62± 0.79 91.97±0.7491.97± 0.74 82.41±0.7282.41± 0.72 85.6885.68 FFA-LoRA (r = 8) (Sun et al., 2024) 82.35±0.4882.35± 0.48 86.98±0.6286.98± 0.62 92.74±0.8592.74± 0.85 82.71±0.6282.71± 0.62 86.2086.20 FedSA (Guo et al., 2025) 83.88±0.4183.88± 0.41 89.13±0.3189.13± 0.31 94.08±0.5494.08± 0.54 85.96±0.4785.96± 0.47 88.2688.26 FedDPA (Yang et al., 2024) 83.45±0.0983.45± 0.09 88.53±0.7788.53± 0.77 93.52±0.7893.52± 0.78 85.63±0.6985.63± 0.69 87.7887.78 FedALT (Bian et al., 2026) 82.53±0.4482.53± 0.44 89.11±0.3389.11± 0.33 91.78±0.4291.78± 0.42 85.68±0.0585.68± 0.05 87.2887.28 FedLEASE (Wang et al., 2025) 84.13±0.3584.13± 0.35 90.99±0.0890.99± 0.08 94.11±0.9494.11± 0.94 87.71±0.1587.71± 0.15 89.2489.24 FedTreeLoRA (Ours) 87.28±0.3387.28± 0.33 92.20±0.3092.20± 0.30 94.87±0.2694.87± 0.26 89.48±0.1089.48± 0.10 90.9690.96 C.3 Robustness to Data Distribution Shifts Finally, we examine the resilience of our method against different forms of statistical heterogeneity, varying from label skew to distinct task assignments. Label Distribution Skew. In the main experiments, we have already evaluated FedTreeLoRA under a highly heterogeneous setting (α=0.5α=0.5). To further examine robustness to label skew, we additionally consider two milder non-IID regimes with α∈0.7,1α∈\0.7,1\. Both settings introduce label imbalance across clients while being less extreme than the primary benchmark. Across these configurations, FedTreeLoRA consistently achieves the best performance among all compared methods. This indicates that the proposed tree-structured aggregation remains reliably effective even as the degree of heterogeneity varies, and does not rely on extremely skewed data to demonstrate benefits. As α increases and the client distributions become closer to IID, our adaptive layer-wise specialization mechanism naturally performs fewer splits, promoting more parameter sharing while still preserving the hierarchical structure when beneficial. These results further validate the robustness of FedTreeLoRA to varying levels of label skew, as summarized in Table 11 and Table 12. Table 11: Performance comparison under different Label Skew (α=1α=1). Methods MNLI QNLI SST2 QQP Average FedIT (Zhang et al., 2024) 83.13±0.2083.13± 0.20 88.04±0.5688.04± 0.56 93.67±0.6193.67± 0.61 83.88±0.3683.88± 0.36 87.1887.18 FFA-LoRA (r = 4) (Sun et al., 2024) 83.41±0.8383.41± 0.83 87.82±0.9987.82± 0.99 94.12±0.3394.12± 0.33 83.78±0.5383.78± 0.53 87.2887.28 FFA-LoRA (r = 8) (Sun et al., 2024) 83.69±0.1383.69± 0.13 88.94±0.2388.94± 0.23 94.27±0.6094.27± 0.60 84.13±0.2384.13± 0.23 87.7687.76 FedSA (Guo et al., 2025) 85.32±0.0885.32± 0.08 89.17±0.9789.17± 0.97 94.50±0.4094.50± 0.40 87.89±0.3987.89± 0.39 89.2289.22 FedDPA (Yang et al., 2024) 85.38±0.2185.38± 0.21 89.93±0.3389.93± 0.33 94.74±0.6394.74± 0.63 88.07±0.5288.07± 0.52 89.5389.53 FedALT (Bian et al., 2026) 85.85±0.8885.85± 0.88 90.10±0.4590.10± 0.45 94.47±0.2994.47± 0.29 87.95±0.1187.95± 0.11 89.5989.59 FedLEASE (Wang et al., 2025) 86.98±0.0686.98± 0.06 91.83±0.1491.83± 0.14 95.29±0.6195.29± 0.61 88.16±0.2088.16± 0.20 90.5790.57 FedTreeLoRA (Ours) 87.43±0.2687.43± 0.26 93.03±0.3293.03± 0.32 95.62±0.3195.62± 0.31 89.23±0.1089.23± 0.10 91.3391.33 Table 12: Performance comparison under different Label Skew (α=0.7α=0.7). Methods MNLI QNLI SST2 QQP Average FedIT (Zhang et al., 2024) 81.90±0.1581.90± 0.15 87.28±0.2987.28± 0.29 93.99±0.1093.99± 0.10 82.87±0.6082.87± 0.60 86.5186.51 FFA-LoRA (r = 4) (Sun et al., 2024) 83.11±0.3183.11± 0.31 89.18±0.7689.18± 0.76 94.30±0.9394.30± 0.93 83.23±0.7983.23± 0.79 87.4687.46 FFA-LoRA (r = 8) (Sun et al., 2024) 83.32±0.9383.32± 0.93 89.60±0.2789.60± 0.27 94.56±0.3694.56± 0.36 83.51±0.5683.51± 0.56 87.7587.75 FedSA (Guo et al., 2025) 84.73±0.3984.73± 0.39 89.84±0.2289.84± 0.22 94.68±0.3994.68± 0.39 88.98±0.2788.98± 0.27 89.5689.56 FedDPA (Yang et al., 2024) 84.19±0.3684.19± 0.36 89.69±0.3989.69± 0.39 94.79±0.5194.79± 0.51 89.13±0.3989.13± 0.39 89.4589.45 FedALT (Bian et al., 2026) 83.23±0.8783.23± 0.87 89.55±0.1189.55± 0.11 90.86±0.5190.86± 0.51 88.71±0.1988.71± 0.19 88.0988.09 FedLEASE (Wang et al., 2025) 85.80±0.3385.80± 0.33 92.70±0.1392.70± 0.13 95.65±0.0995.65± 0.09 90.23±0.1490.23± 0.14 91.1091.10 FedTreeLoRA (Ours) 87.90±0.1387.90± 0.13 93.12±0.1093.12± 0.10 96.40±0.3296.40± 0.32 91.31±0.0191.31± 0.01 92.1892.18 Task Heterogeneity. Following the protocol of FedLEASE (Wang et al., 2025), we evaluate a task-heterogeneous setting where client diversity arises from performing different NLP tasks rather than label skew. We simulate a federated system with N=16N=16 clients, partitioned into four groups of four label IID clients each, corresponding to four GLUE tasks: MNLI, QNLI, SST-2, and QQP. We use Cosine distance in this setting, as it is more suitable for measuring cross-task differences in low-rank update directions. Performance Analysis. As reported in Table 13, FedTreeLoRA achieves the highest average accuracy under task heterogeneity. While FedLEASE is specifically designed for this scenario, it enforces a flat task-wise separation. In contrast, FedTreeLoRA exploits the hierarchical nature of Transformer representations by allowing clients to share low-level linguistic features while gradually specializing toward task-specific decision boundaries. This hierarchical sharing enables stronger positive transfer in shallow layers while preventing negative interference in deeper layers. Structural Analysis. Figure 7 visualizes the learned layer-wise clustering structure. We observe a clear and interpretable hierarchy that aligns with task semantics. All 16 clients initially share a single cluster in shallow layers, reflecting common syntactic and lexical representations. At intermediate depths, the model separates MNLI and QQP clients into distinct branches, while QNLI and SST-2 remain grouped, indicating that their low-rank updates remain geometrically similar at this stage. In deeper layers, the hierarchy further refines into four task-specific clusters, each corresponding to one GLUE task. This progression from a global cluster to intermediate task groups and finally to task-specific leaves demonstrates that FedTreeLoRA does not impose a rigid task partition. Instead, it automatically discovers a hierarchical task topology that reflects both shared linguistic structure and task-dependent specialization. Such adaptive, depth-aware decomposition is precisely what enables FedTreeLoRA to outperform flat task separation methods in heterogeneous federated environments. Figure 7: Layer-wise client clustering under task heterogeneity. Clients assigned to the same GLUE task form stable branches at deeper layers, while all tasks share common representations in shallow layers, illustrating the hierarchical specialization discovered by FedTreeLoRA. Table 13: Performance under Task Heterogeneity (N=16N=16 clients, 4 per task). Methods MNLI QNLI SST2 QQP Average FedIT (Zhang et al., 2024) 71.46±1.7471.46± 1.74 83.71±1.9583.71± 1.95 93.25±0.3193.25± 0.31 78.54±3.0378.54± 3.03 81.7481.74 FFA-LoRA (Sun et al., 2024) 72.17±2.7472.17± 2.74 82.75±2.2682.75± 2.26 93.79±0.6893.79± 0.68 81.25±2.5681.25± 2.56 82.4982.49 FedSA (Guo et al., 2025) 74.29±1.0974.29± 1.09 86.33±0.7586.33± 0.75 93.87±0.1893.87± 0.18 82.12±0.7482.12± 0.74 84.1584.15 FedDPA (Yang et al., 2024) 75.00±1.7075.00± 1.70 86.29±0.1686.29± 0.16 93.58±0.1693.58± 0.16 81.92±2.0181.92± 2.01 84.2084.20 FedALT (Bian et al., 2026) 73.25±1.4073.25± 1.40 83.00±1.3583.00± 1.35 93.69±0.1993.69± 0.19 84.31±0.4484.31± 0.44 83.5683.56 FedLEASE (Wang et al., 2025) 79.91±1.8679.91± 1.86 87.54±1.3587.54± 1.35 94.17±0.3794.17± 0.37 83.46±0.7983.46± 0.79 86.2786.27 FedTreeLoRA (Ours) 82.94±0.9482.94± 0.94 89.31±0.1989.31± 0.19 94.19±0.3194.19± 0.31 84.75±1.2584.75± 1.25 87.8087.80 C.4 Impact of Different Distance Metrics To evaluate the sensitivity of FedTreeLoRA to the geometric definition of client similarity, we conduct a comparative study using two widely adopted distance functions for constructing the global topological skeleton: Frobenius distance (our default) and Cosine distance. The Frobenius distance captures the absolute magnitude of parameter deviations, while Cosine distance focuses on the directional orientation of the LoRA updates. As summarized in Table 14, the performance remains remarkably consistent across both metrics. On the GLUE benchmark, the average accuracy gap between the two implementations is a marginal 0.09%0.09\% (92.45%92.45\% for Cosine vs. 92.36%92.36\% for Frobenius). The consistency underscores the robustness of FedTreeLoRA, demonstrating that its performance gains are rooted in the structural hierarchy and layer-wise alignment rather than the specific distance definition. We default to Frobenius distance for label non-IID but task-homogeneous settings, where LoRA update differences are mainly reflected in their magnitudes. Table 14: Sensitivity analysis of distance metrics on GLUE benchmarks. Results demonstrate that FedTreeLoRA is metric-agnostic, maintaining high performance regardless of the specific similarity definition. Metric MNLI QNLI SST2 QQP Average Cosine Distance 88.29±0.3088.29± 0.30 93.47±0.2593.47± 0.25 96.41±0.0296.41± 0.02 91.61±0.0991.61± 0.09 92.45 Frobenius Distance 88.15±0.2588.15± 0.25 93.37±0.6293.37± 0.62 96.56±0.0796.56± 0.07 91.35±0.1791.35± 0.17 92.36 C.5 Effect of Warm-up Duration and Fixed Tree Topology In FedTreeLoRA, the tree topology is constructed once after the warm-up stage and then kept fixed during subsequent optimization. The rationale behind this design is that the warm-up phase already provides a sufficiently informative estimate of inter-client relationships, rather than merely serving as a rough initialization. As illustrated in Figure 7, the learned hierarchy exhibits strong semantic consistency: clients tend to remain unified in shallow layers while progressively separating into task-related subgroups at deeper layers. This observation suggests that the discovered topology captures a stable and interpretable collaborative structure. To further validate whether a short warm-up is sufficient for reliable topology discovery, we vary the number of warm-up epochs and report the downstream performance in Table 15. The results show that FedTreeLoRA remains highly stable across different warm-up durations. Increasing the warm-up stage from 5 to 10 epochs yields nearly identical average accuracy, indicating that the hierarchical relationships among clients emerge early during adaptation. Table 15: Sensitivity analysis of warm-up duration on GLUE benchmarks. Results show that FedTreeLoRA remains highly stable across different warm-up stages, indicating that reliable client topology can be identified early during adaptation. Warm-up Epochs MNLI QNLI SST2 QQP Average 5 82.94 89.31 94.19 84.75 87.80 10 82.66 89.83 94.62 84.13 87.81 Based on these observations, FedTreeLoRA decouples topology discovery from parameter optimization. Maintaining a fixed hierarchy throughout training preserves structural consistency across layers while avoiding repeated similarity computation and hierarchical clustering at every communication round, thereby significantly reducing computational overhead. C.6 Robustness to Dynamic Client Participation Although FedTreeLoRA primarily focuses on personalized federated fine-tuning under a fixed client population, an important practical scenario is dynamic client participation, where new clients may join the system during training. While this setting is more closely related to federated continual learning, FedTreeLoRA can be naturally extended to support incremental client integration. Specifically, when a new client joins, the client first performs a short local warm-up stage to obtain an initial LoRA representation. The resulting LoRA B matrices are then used to measure similarity against the existing hierarchy. Based on this similarity, the client is assigned to the closest branch of the learned tree and subsequently follows the same layer-wise aggregation strategy as existing clients. To evaluate this behavior, we conduct experiments on the MNLI benchmark where a new client joins the training at communication round 10 or 20. For each setting, we randomly sample a different client to join the system in order to avoid bias toward a specific client distribution. We report: (1) the performance of the original clients without client joining, (2) the performance of existing clients after the new client joins, (3) the performance of the new client after joining the hierarchy, and (4) a local-only baseline where the new client trains independently without collaboration. Table 16: Robustness to dynamic client participation on MNLI. Results show that integrating new clients causes negligible degradation to existing clients while significantly improving the performance of the newly joined client compared to local-only training. Join Round Original Clients Existing Clients New Client Local Only 10 87.90 87.43 84.50 82.00 20 87.90 87.50 86.50 79.00 The results demonstrate that the learned hierarchy provides a stable and reusable topology for incremental personalization. Existing clients experience only marginal performance changes after client insertion, while the newly joined client consistently benefits from collaborative adaptation compared to isolated local training. Appendix D Detailed Results of Ablation Studies Due to space limitations in the main paper, we reported only the average accuracy (Avg. Acc.) to summarize the findings of our ablation studies. To provide a more comprehensive and granular analysis of how each component of FedTreeLoRA contributes to the results, we here report the detailed performance breakdown for each of the four evaluation tasks: MNLI, QNLI, SST2, and QQP, corresponding to the summarized results presented in the main paper. D.1 Detailed Analysis of Interaction Mechanisms As discussed in the main paper, we compared our Cluster-External aggregation against three variants: Isolationist, Decomposed Experts, and the MoE Router. Table 17 provides the task-specific accuracy for each mechanism, providing insight into the trade-off between performance and efficiency. Table 17: Detailed performance of different Interaction Mechanisms across GLUE tasks. Accuracy is reported in (%). Interaction Variant MNLI QNLI SST2 QQP Average Isolationist (Cluster-Only) 86.60±0.2186.60± 0.21 92.57±0.4492.57± 0.44 95.96±0.3995.96± 0.39 90.48±0.3990.48± 0.39 91.4091.40 Decomposed Experts 88.46±0.2388.46± 0.23 93.91±0.3693.91± 0.36 96.54±0.0496.54± 0.04 91.35±0.1391.35± 0.13 92.5792.57 MoE Router 87.92±0.8287.92± 0.82 93.35±0.5893.35± 0.58 96.15±0.4696.15± 0.46 90.67±0.4590.67± 0.45 92.0292.02 VAScalar-Mixed 88.15±0.2588.15± 0.25 93.37±0.6293.37± 0.62 96.56±0.0796.56± 0.07 91.35±0.1791.35± 0.17 92.36¯ 92.36 D.2 Detailed Impact of Layer-wise Adaptivity Table 18 extends the results from the main paper regarding layer-wise adaptivity. It compares our adaptive depth search against fixed clustering granularities (k=1,4,8k=1,4,8) across each specific task to validate the necessity of layer-wise adaptivity. Table 18: Detailed performance of Fixed-Depth vs. Layer-wise Adaptive aggregation. Aggregation Depth MNLI QNLI SST2 QQP Average Fixed (k=1k=1, Global Only) 83.18±0.7483.18± 0.74 87.03±0.4387.03± 0.43 93.65±0.6393.65± 0.63 84.93±0.5984.93± 0.59 87.2087.20 Fixed (k=4k=4, Coarse) 86.75±0.6386.75± 0.63 92.95±0.2292.95± 0.22 96.13±0.1696.13± 0.16 89.98±0.5989.98± 0.59 91.4591.45 Fixed (k=8k=8, Fine-grained) 85.81±0.6885.81± 0.68 92.02±0.7192.02± 0.71 95.17±0.1395.17± 0.13 89.96±0.8989.96± 0.89 90.7490.74 Layer-wise Adaptive (cl∗c_l^*) 88.15±0.2588.15± 0.25 93.37±0.6293.37± 0.62 96.56±0.0796.56± 0.07 91.35±0.1791.35± 0.17 92.3692.36 D.3 Effect of the Global Tree Skeleton Finally, we report the detailed breakdown of the structural consistency analysis presented in the main paper. Table 19 demonstrates the importance of the global tree skeleton in maintaining topological coherence across model layers for each individual dataset. Table 19: Detailed impact of Global Structural Consistency on individual datasets. Structural Prior MNLI QNLI SST2 QQP Average Independent Clustering 83.84±0.3283.84± 0.32 89.74±1.9589.74± 1.95 95.06±0.2195.06± 0.21 89.23±1.5689.23± 1.56 89.4789.47 Global Tree Skeleton 88.15±0.2588.15± 0.25 93.37±0.6293.37± 0.62 96.56±0.0796.56± 0.07 91.35±0.1791.35± 0.17 92.3692.36 Appendix E Computational Overhead and Wall-Clock Training Time We analyze the computational efficiency of FedTreeLoRA compared to baseline methods. Fairness of Training Budget. To ensure a fair comparison, we maintain an identical total training budget across all evaluated methods. For NLU tasks, the total communication budget is fixed at T=30T=30 rounds. For methods requiring a warmup phase to capture client relationships (e.g. FedLEASE and the proposed FedTreeLoRA), we allocate Ewarm=10E_warm=10 epochs (i.e. 5 rounds) for local warmup, followed by 2525 rounds of federated fine-tuning. Similarly, for NLG tasks, the total budget is T=10T=10 rounds, with Ewarm=2E_warm=2 epochs (i.e. 1 round) dedicated to the warmup phase. This configuration ensures that all methods undergo the same cumulative number of local updates and communication iterations, isolating the performance gains to the structural aggregation mechanism rather than an increased training budget. Efficiency of Topology Generation. Our tree topology generation step is performed only once during the initialization phase (immediately following warmup) and is not repeated during the iterative communication rounds. Consequently, its impact on the total training time is negligible. Furthermore, utilizing only the LoRA B matrices for similarity computation provides an efficient and lightweight proxy for task alignment. This is because the dimensionality of B is significantly smaller than that of full model weights or accumulated BABA products. As shown in Table 20, we measured the topology generation time to be approximately 4.21 seconds on an Intel Xeon Platinum 8570 CPU. This is orders of magnitude shorter than the total training duration, which is approximately 70.82 seconds when utilizing NVIDIA B200 GPUs. Notably, FedTreeLoRA achieves superior performance with a total training time comparable to standard FedAvg-based methods like FedIT (65.1565.15s), and remains lower than complex clustering-based baselines such as FedLEASE (77.9977.99s) which require heavy routing computations and additional parameter overhead. Table 20: Comparison of Computational Overhead. All times are measured in seconds (s). Time (s) FedIT FFA-LoRA FedSA FedDPA FedALT FedLEASE FedTreeLoRA Local Training (Per Epoch) 1.0545 1.0369 1.0582 1.0668 1.2091 1.1883 1.0716 Global Aggregation 0.0628 0.0594 0.0693 0.0723 0.0782 0.0908 0.0770 Topology/Clustering Time - - - - - 3.97 4.21 Total Training Time 65.154 63.996 65.571 66.177 74.892 77.992 70.816 Appendix F Extended NLG Results on an Alternative LLM Family To evaluate the architectural robustness of FedTreeLoRA, we further extend our Natural Language Generation (NLG) experiments by instantiating the framework on a large language model from an alternative LLM family, specifically BLOOM-7B (Le Scao et al., 2023). Experimental Configuration. Following the protocol established in Section 6.2, we utilize N=8N=8 clients partitioned into four distinct NLG task groups from the FLAN collection: Text Editing, Struct to Text, Sentiment Analysis, and Commonsense Reasoning. Each task is assigned to two clients to simulate task-level heterogeneity. We maintain identical hyperparameters to the main LLaMA-2 experiments: a batch size of 8, E=2E=2 local epochs, and T=10T=10 communication rounds using the AdamW optimizer. LoRA adapters with rank r=8r=8 are applied to the query and value projections. Consistent with standard evaluation protocols, we report the average ROUGE-1 scores. Results and Analysis. As shown in Table 21, FedTreeLoRA consistently outperforms both general federated fine-tuning baselines and flat-clustering methods on the BLOOM model. The results indicate that the “Flat-Model Assumption”, which treats LLM layers as a monolithic block for aggregation, is equally restrictive for BLOOM. By dynamically constructing an aggregation tree that aligns with the model’s functional depth, our method mitigates negative transfer between conflicting domains while maximizing the utility of shared linguistic knowledge. Table 21: Performance comparison on NLG benchmarks using BLOOM. We report ROUGE-1 scores averaged over three independent runs. Methods Text Edit Struct2Text Sentiment Reasoning Average FedIT (Zhang et al., 2024) 77.21±1.3977.21± 1.39 51.80±0.8951.80± 0.89 49.73±0.3649.73± 0.36 72.37±0.5872.37± 0.58 62.7862.78 FFA-LoRA (r = 8) (Sun et al., 2024) 77.05±1.3577.05± 1.35 51.10±0.6651.10± 0.66 48.72±1.4948.72± 1.49 72.32±0.4672.32± 0.46 62.3062.30 FFA-LoRA (r = 16) (Sun et al., 2024) 78.77±1.8178.77± 1.81 51.20±0.6051.20± 0.60 49.05±0.8449.05± 0.84 72.12±0.8672.12± 0.86 62.7962.79 FedSA (Guo et al., 2025) 84.93±1.3484.93± 1.34 52.93±0.2352.93± 0.23 49.38±0.8449.38± 0.84 72.64±1.1872.64± 1.18 64.9464.94 FedDPA (Yang et al., 2024) 88.56±0.8688.56± 0.86 53.78±0.3653.78± 0.36 51.31±0.5151.31± 0.51 72.16±1.7872.16± 1.78 66.4566.45 FedALT (Bian et al., 2026) 88.32±0.1988.32± 0.19 53.58±0.1953.58± 0.19 50.97±1.1350.97± 1.13 73.22±1.4573.22± 1.45 66.5266.52 FedLEASE (Wang et al., 2025) 87.07±0.4587.07± 0.45 52.22±0.3952.22± 0.39 52.12±0.1552.12± 0.15 73.75±0.9073.75± 0.90 66.7966.79 FedTreeLoRA (Ours) 88.84±0.3488.84± 0.34 55.20±1.0155.20± 1.01 52.85±0.4552.85± 0.45 74.23±0.6874.23± 0.68 67.7867.78 Appendix G Justification for Similarity Measurement via LoRA B Matrices In FedTreeLoRA, we utilize the distance between clients’ LoRA B matrices to construct the topological tree. This design choice is grounded in the asymmetric roles of adapter matrices observed in recent federated fine-tuning literature (Guo et al., 2025; Wang et al., 2025), which suggest that A matrices tend to capture general features while B matrices encode task-specific semantics. To verify and refine this hypothesis in a layer-wise and heterogeneity-aware manner, we conduct a controlled analysis on RoBERTa-Large for the QNLI task with two clients under an IID setting (identical label distributions) and a highly non-IID setting ([0.8,0.2][0.8,0.2] vs. [0.2,0.8][0.2,0.8]), reporting the average pairwise Frobenius distance of LoRA parameters across all 24 Transformer layers. As shown in Figure 8, the upper row (Proj A @ Query and Value) exhibits nearly identical distance trajectories under IID and non-IID across all layers, indicating that A learns a shared low-rank basis that is largely insensitive to data heterogeneity. In contrast, the lower row (Proj B @ Query and Value) consistently separates IID and non-IID settings, and this separation is strongly depth-dependent: in shallow layers the distances are close, reflecting shared general representations, whereas in deeper layers the non-IID distance grows substantially while the IID distance remains small, producing a widening gap that directly reflects the emergence of client-specific semantics. This layer-wise divergence in B explains why shallow layers can be safely aggregated while deeper layers require personalized or clustered aggregation, and it establishes LoRA B matrices as a sensitive and reliable signal for measuring client similarity. Since computing distances on B∈ℝdout×rB ^d_out× r is orders of magnitude cheaper than on the full adapter product BA∈ℝdout×dinBA ^d_out× d_in, using B to construct the topological tree is both empirically justified and computationally efficient. Figure 8: Layer-wise average pairwise Frobenius distance between two clients on QNLI under IID (identical label distributions) and Non-IID ([0.8,0.2][0.8,0.2] vs. [0.2,0.8][0.2,0.8]) settings. The upper-left and upper-right panels (Proj A at Query and Value) show nearly overlapping IID and Non-IID curves across all layers, indicating that A captures distribution-invariant shared representations. The lower-left and lower-right panels (Proj B at Query and Value) exhibit a progressively widening gap between IID and Non-IID as depth increases, revealing that client-specific semantics emerge primarily in deeper layers through the LoRA B matrices. This validates using B-matrix distances to quantify heterogeneity and to guide hierarchical, layer-wise aggregation in FedTreeLoRA. Appendix H Compared Methods We provide a detailed overview of the baseline methods used in our evaluation, highlighting their operational mechanisms and distinguishing them from our proposed FedTreeLoRA. FedIT [ICASSP 2024] (Zhang et al., 2024). FedIT represents the canonical application of Federated Learning to Instruction Tuning. It directly integrates LoRA with the standard FedAvg framework, aggregating all client updates globally. Under heterogeneous settings, this leads to significant “negative transfer” or cross-client interference, where the global aggregate dilutes task-specific features, often resulting in performance inferior to local-only training. FFA-LoRA [ICLR 2024] (Sun et al., 2024). FFA-LoRA focuses on communication efficiency by freezing the randomly initialized non-zero A matrices and fine-tuning only the zero-initialized B matrices. While this strategy reduces communication overhead, it severely constrains the expressivity of the adaptation. By locking the input projection A, the model loses the flexibility to align diverse client inputs into a shared latent space, resulting in suboptimal performance. FedSA [ICLR 2025] (Guo et al., 2025). FedSA proposes a selective aggregation strategy to address heterogeneity. It decomposes the LoRA update into global and local components: only the A matrices are aggregated server-side to learn shared representations, while B matrices remain strictly local to preserve personalization. While FedSA attempts to decouple personalization from generalization, it relies on a rigid, manually defined split. It overlooks that client-specific B matrices often contain valuable transferable knowledge for distinct sub-communities of clients, which FedTreeLoRA captures via its hierarchical aggregation. FedDPA [NeurIPS 2024] (Yang et al., 2024). FedDPA targets task heterogeneity by employing a dual-adapter architecture. It maintains both a global LoRA module (aggregated via FedAvg) and a local LoRA module (kept private). It utilizes a multi-phase training strategy to balance general knowledge acquisition and local refinement. FedDPA fundamentally relies on a “Global vs. Local” binary separation. It lacks the granularity to model relationships, where groups of clients share specific tasks. FedTreeLoRA replaces this binary dichotomy with a continuous tree structure, allowing parameter sharing to adapt dynamically based on cluster similarity rather than just global or local isolation. FedALT [AAAI 2026] (Bian et al., 2026). FedALT introduces a “Rest-of-World” (RoW) paradigm. Instead of a global model, each client maintains an Individual LoRA (trained locally) and a RoW LoRA (a frozen aggregate of all other clients’ modules). An input-dependent adaptive mixer (resembling a gate) dynamically weights the contribution of the individual and RoW modules during inference. FedALT mitigates interference by isolating the global signal into a frozen module. However, it operates on a coarse-grained “Self vs. Others” logic. It aggregates all other clients into a single RoW representation, potentially blending conflicting signals from unrelated tasks. In contrast, FedTreeLoRA selectively aggregates only relevant peers defined by the topological tree, ensuring that shared signals remain semantically coherent. FedLEASE [NeurIPS 2025] (Wang et al., 2025). FedLEASE addresses heterogeneity via flat clustering. It computes client similarity based on initial LoRA updates and groups clients into disjoint clusters, training distinct domain experts for each group. It further employs an adaptive top-M routing mechanism to allow clients to select multiple experts during inference. FedLEASE represents a strong clustering baseline. However, it relies on the Flat-Model Assumption: it generates a single partition of clients and applies expert selection uniformly. Crucially, it ignores vertical heterogeneity, the fact that shallow layers require broad consensus while deep layers need specialization. FedTreeLoRA advances beyond FedLEASE by constructing a hierarchical tree that enables layer-wise depth alignment, allowing the aggregation boundary to shift dynamically from the root (global) to the leaves (personalized) across the model depth. Appendix I Clarification with Existing Methods Distinction from TreeLoRA (Qian et al., 2025). We note that although our method shares the term ‘TreeLoRA’ with the recent work of (Qian et al., 2025), the two approaches differ fundamentally in problem setting, tree semantics, and optimization role. TreeLoRA (Qian et al., 2025) is proposed for continual learning, where the tree dynamically organizes tasks arriving sequentially based on gradient-direction similarity, and is incrementally expanded via a bandit-based search to mitigate catastrophic forgetting. In contrast, our proposed FedTreeLoRA operates in a federated fine-tuning setting, where the tree models statistical relationships among clients rather than tasks. The tree in FedTreeLoRA is constructed once after a warmup phase using the geometric similarity of LoRA adaptation parameters, serving as a fixed global topological skeleton shared across all Transformer layers. Moreover, the role of the tree is different: in TreeLoRA, the tree directly controls sparse gradient updates by selecting task-specific adapters, whereas in FedTreeLoRA, the tree only defines the topology of parameter sharing, constraining the feasible layer-wise partitions while leaving the parameter interaction mechanism to a Cluster-External expert aggregation with a learnable mixing coefficient. Finally, FedTreeLoRA introduces a layer-wise monotonic depth alignment to ensure structural coherence across model layers, a design that is specific to federated adaptation of deep architectures and has no counterpart in continual learning. Consequently, despite the terminological overlap, FedTreeLoRA represents a distinct federated learning framework focused on structural alignment rather than a variant of tree-based continual learning. Additional Distinction from Layer-aware Federated LoRA. We further note that recent federated LoRA methods have explored layer-aware adaptation under heterogeneous client resources (Zhang et al., 2025b). However, these approaches primarily focus on resource-efficient layer allocation across clients, rather than modeling the hierarchical relationship between client similarity and aggregation depth. In contrast, FedTreeLoRA studies how parameter sharing should evolve progressively across Transformer layers based on statistical similarity, leading to a tree-structured aggregation topology that jointly reconciles horizontal and vertical heterogeneity. Appendix J Reproducibility and Code Availability To ensure full reproducibility of our experimental results, we publicly release the complete implementation of FedTreeLoRA at: https://github.com/jmbian/FedTreeLoRA. The repository includes all components required to reproduce the results reported in both the main paper and appendix, including data preprocessing, model architectures, training scripts, evaluation pipelines, global tree construction, layer-wise adaptive clustering, and the cluster-external expert mechanism. We further provide detailed instructions for environment setup, dataset preparation, and command-line execution for all NLU and NLG benchmarks evaluated in this work. In addition, the codebase is designed to support extensibility and ablation studies. Researchers can easily modify the similarity metric (e.g., Frobenius or cosine distance), warm-up duration, search range K, and heterogeneity threshold τ, enabling further investigation into the trade-offs between personalization, communication efficiency, and federated adaptation performance. Appendix K FedTreeLoRA Algorithm We provide the completed pseudocode for FedTreeLoRA in Algorithm 1. Algorithm 1 FedTreeLoRA: Tree-Structured Aggregation 0: Clients 1,…,N\1,…,N\ with datasets k\D_k\; layers l=1,…,Ll=1,…,L; warmup epochs EwarmE_warm; rounds T; local epochs E; search range K; threshold τ. A¯l,kclus,B¯l,kclus,λl,k\ A_l,k^clus, B_l,k^clus, _l,k\. 1: — Phase 1: Warm-up and Topology Generation — 2: Client Side (Parallel): 3: for all k∈1,…,Nk∈\1,…,N\ do 4: Initialize Al,k,Bl,kl=1L\A_l,k,B_l,k\_l=1^L; 5: Warmup-train on kD_k for EwarmE_warm epochs; 6: Upload updated Bl,kB_l,k to Server. 7: end for 8: Server Side: 9: Compute global distance Di,jglobal=1L∑l=1Ldist(Bl,i,Bl,j)D^global_i,j= 1L _l=1^Ldist(B_l,i,B_l,j) via Eq. (3); 10: Build merge tree T via AHC; 11: Initialize c0∗←1c_0^*← 1; 12: for l=1l=1 to L do 13: Compute Di,j(l)=dist(Bl,i,Bl,j)D^(l)_i,j=dist(B_l,i,B_l,j); 14: Define search space Ωl=c∈ℤ∣cl−1∗≤c≤min⁡(N,cl−1∗+K−1) _l=\c c_l-1^*≤ c≤ (N,c_l-1^*+K-1)\; 15: Determine optimal clusters cl∗←argmaxc∈Ωlϕ(c;D(l))c_l^*← *argmax_c∈ _lφ(c;D^(l)); 16: Extract partition Pcl∗P_c_l^* and assign peer groups for each client. 17: end for 18: — Phase 2: Federated Fine-Tuning — 19: for t=1t=1 to T do 20: Server Side (Aggregation): 21: for all k∈1,…,Nk∈\1,…,N\ do 22: for l=1l=1 to L do 23: Identify peer group k(l)S_k^(l) and external group ℛk(l)R_k^(l) from Pcl∗P_c_l^*; 24: for all Φ∈A,B ∈\A,B\ do 25: Φ¯l,kclus←1|k(l)|∑j∈k(l)Φl,j(t) _l,k^clus← 1|S_k^(l)| _j _k^(l) _l,j^(t); 26: if |ℛk(l)|>0|R_k^(l)|>0 then 27: Φ¯l,kext←1|ℛk(l)|∑j∈ℛk(l)Φl,j(t) _l,k^ext← 1|R_k^(l)| _j _k^(l) _l,j^(t); 28: else 29: Φ¯l,kext← _l,k^ext 0; 30: end if 31: end for 32: end for 33: Send aggregated experts A¯kclus/ext,B¯kclus/ext\ A_k^clus/ext, B_k^clus/ext\ to Client k. 34: end for 35: Client Side (Parallel Update): 36: for all k∈1,…,Nk∈\1,…,N\ do 37: Receive experts; 38: Update (A¯l,kclus,B¯l,kclus,λl,k)( A_l,k^clus, B_l,k^clus, _l,k) on kD_k for E epochs (via Eq. 8); 39: Upload updated parameters to Server. 40: end for 41: end for

Instant research discovery