Paper deep dive
Prototype-guided Bilateral Alignment Multimodal Federated Learning
Tianchi Liao Tianchi_Liao, Lele Fu, Sheng Huang, Qing Hu, Hong-Ning Dai, Chuan Chen
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Multimodal federated learning (MFL) has emerged as a pivotal paradigm for leveraging distributed data to enhance model performance. However, existing methods predominantly rely on idealized assumptions of model homogeneity and balanced modality distributions, rendering them ill-suited for practical scenarios characterized by heterogeneous client architectures and severe modality imbalance. To address these challenges, we propose a \textbf{M}ultimodal \textbf{Fed}erated learning Prototype-guided Bilateral Alignment (MFedPBA) framework. MFedPBA facilitates robust knowledge synergy through a dual alignment mechanism: (i) at the feature level, it aligns heterogeneous feature spaces via a projection encoder optimized by contrastive learning and the Gromov-Wasserstein distance; (ii) at the decision level, it employs an entropy-weighted aggregation of naturally aligned logit prototypes. This novel design achieves robust MFL by jointly tackling heterogeneous feature spaces and collectively aggregating decisions. Extensive experiments demonstrate that our method significantly outperforms state-of-the-art baselines under conditions of model heterogeneity and modality imbalance.
Tags
Links
- Source: https://arxiv.org/abs/2609.38925v1
- Canonical: https://arxiv.org/abs/2609.38925v1
Trouble viewing inline? Open PDF directly →
Full Text
112,507 characters extracted from source content.
Expand or collapse full text
Prototype-guided Bilateral Alignment Multimodal Federated Learning Tianchi Liao Affiliation: Department of Computer Science, Hong Kong Baptist University, Hong Kong SAR, China Affiliation: School of Computer Science and Engineering, Sun Yat-Sen University, Guangzhou, China Lele Fu Affiliation: School of Computer Science and Engineering, Sun Yat-Sen University, Guangzhou, China Sheng Huang Affiliation: School of Computer Science and Engineering, Sun Yat-Sen University, Guangzhou, China Qing Hu Affiliation: School of Computer Science and Engineering, Sun Yat-Sen University, Guangzhou, China Hong-Ning Dai Affiliation: Department of Computer Science, Hong Kong Baptist University, Hong Kong SAR, China Chuan Chen Affiliation: School of Computer Science and Engineering, Sun Yat-Sen University, Guangzhou, China Correspondence to: chenchuan@mail.sysu.edu.cn Abstract Multimodal federated learning (MFL) has emerged as a pivotal paradigm for leveraging distributed data to enhance model performance. However, existing methods predominantly rely on idealized assumptions of model homogeneity and balanced modality distributions, rendering them ill-suited for practical scenarios characterized by heterogeneous client architectures and severe modality imbalance. To address these challenges, we propose a Multimodal Federated learning Prototype-guided Bilateral Alignment (MFedPBA) framework. MFedPBA facilitates robust knowledge synergy through a dual alignment mechanism: (i) at the feature level, it aligns heterogeneous feature spaces via a projection encoder optimized by contrastive learning and the Gromov-Wasserstein distance; (i) at the decision level, it employs an entropy-weighted aggregation of naturally aligned logit prototypes. This novel design achieves robust MFL by jointly tackling heterogeneous feature spaces and collectively aggregating decisions. Extensive experiments demonstrate that our method significantly outperforms state-of-the-art baselines under conditions of model heterogeneity and modality imbalance. Keywords: Machine Learning, ICML 1 Introduction With the rapid proliferation of multi-sensor Internet of Things devices and the growing demand for data privacy, personalized federated learning (FL) has emerged as an important paradigm for training customized local models on decentralized clients (Collins et al., 2021; Meng et al., 2024; Kou et al., 2026; Ma et al., ). Recently, the integration of heterogeneous sensory data has further advanced multimodal federated learning (MFL), which aims to exploit complementary information across modalities such as vision, audio, and text to improve model robustness and personalization (Qi et al., 2023; Che et al., 2023; Huang et al., 2024; Huang et al., 2026). However, the majority of existing multimodal federated learning frameworks are predicated on the idealized assumption of model homogeneity, mandating that all participating clients deploy an identical neural network architecture (Hu et al., 2024b; Liao et al., 2024; Cao et al., 2026). In practice, as illustrated in Figure 1(a), this assumption rarely holds true in real-world deployments. Given that practical clients typically exhibit profound heterogeneity in both hardware resource constraints (e.g., computational and storage capacities) and multimodal data distributions (such as missing modalities or data skewness), a single global model struggles to simultaneously accommodate the diverse conditions of all nodes. Consequently, designing and deploying personalized, heterogeneous model architectures tailored to different clients has emerged as an inevitable trend (Chen and Zhang, 2024; Rahman and Nguyen, 2024). To address model heterogeneity, prior work has explored prototype-based federated learning, where class embeddings are exchanged as communication units (Tan et al., 2022a; Huang et al., 2023; Li et al., 2025). Nevertheless, such approaches encounter severe bottlenecks in multimodal settings. Since heterogeneous encoders map multimodal data into disparate embedding spaces, the direct aggregation of prototypes on the server results in significant representation misalignment (Fu et al., 2025; Zhang et al., 2025c). As illustrated in Figure 1(b), when clients employ different encoders, prototypes corresponding to the same class may reside in incompatible feature spaces. Consequently, forced aggregation causes the global prototype to be close to class decision boundaries. Instead of yielding performance gains, such forced aggregation can lead to significant knowledge contamination. This issue often causes the global model to degrade to the point where its performance falls below that of a baseline trained purely on local data (Li et al., 2024; Chen et al., 2025; Xiao et al., 2026; Hu et al., 2024a). This raises a critical research challenge: How can we overcome feature space incompatibility to facilitate effective knowledge transfer when both client neural architectures and input modality configurations are inconsistent? Figure 1: (a) Illustration of the multimodal federated learning setting with complex architectural and data inconsistencies. (b) Visualization of feature space incompatibility across heterogeneous encoders, where direct prototype aggregation induces drift towards ambiguous regions, resulting in performance inferior to local training. Furthermore, existing MFL research typically operates under the assumption of balanced modality distributions. While some methods consider scenarios with missing modalities, they are often restricted to bimodal settings or assume that all clients share an identical set of modalities (Che et al., 2024; Wang et al., 2024; Pan et al., 2025; Yu et al., 2025). Such assumptions fail to capture the prevalent imbalance in modality availability across decentralized clients. In real-world applications, systems commonly involve multiple modalities, and certain clients may entirely lack data for specific modalities. This imbalance substantially increases the difficulty of collaborative learning by introducing a higher degree of information heterogeneity (Qi et al., 2025; He et al., 2025). This leads to another key challenge: How can we design an alignment mechanism that ensures precise collaboration in complex scenarios characterized by unbalanced data distributions and modality quantities exceeding the bimodal limit? To address these challenges, we propose a multimodal federated learning prototype-guided bilateral alignment framework (MFedPBA). The framework implements a bilateral alignment mechanism on the server that integrates knowledge from heterogeneous clients through modality-specific feature prototypes and logit prototypes. ❶ At the feature level, the server constructs an autoencoding alignment framework. It leverages contrastive learning to enhance intra-modal semantic consistency, while employing the Gromov–Wasserstein (GW) distance to align the geometric structures between heterogeneous feature spaces and the global space, enabling effective aggregation within a unified representation space. ❷ At the decision level, unlike feature prototypes that may reside in incompatible spaces, logits naturally reside in a shared semantic space aligned across heterogeneous models, as each dimension directly corresponds to a specific class. Leveraging this property, we employ an entropy-based weighting strategy during aggregation. This mechanism quantifies the reliability of local predictions to construct highly discriminative global logit prototypes. By synergizing feature-level and decision-level insights, our framework facilitates robust and efficient cross-client knowledge transfer while strictly preserving the architectural heterogeneity of local models. The primary contributions of this work are summarized as follows: • New perspective: We address a realistic MFL setting characterized by model heterogeneity and modality imbalance, transcending conventional homogeneous assumptions to enable robust knowledge transfer. • Effective algorithm: We propose MFedPBA, a bilateral alignment framework that achieves dual-level synergy: feature alignment via GW distance and contrastive learning, and decision alignment via entropy-based logit aggregation. We further provide theoretical convergence guaranties in non-convex settings. • Superior performance: Extensive experiments on multiple multimodal benchmarks demonstrate that MFedPBA consistently outperforms state-of-the-art baselines under diverse modality distribution settings. 2 Related Work 2.1 Multimodal Federated Learning Multimodal federated learning (Feng et al., 2023; Ouyang et al., 2023; Pan et al., 2024) has emerged as a pivotal paradigm for addressing privacy concerns in mobile sensing systems, finding broad application in real-world scenarios (Zhao et al., 2022; Zheng et al., 2023; Thrasher et al., 2025). Existing MFL literature predominantly addresses two settings: (1) unimodal clients, where each client possesses only a single modality (Zhang et al., 2025b; Deng et al., 2025), and (2) missing modalities, where clients hold incomplete subsets of views (Che et al., 2024; Liu et al., 2025). While these approaches aim to aggregate effective global models, real-world deployments often necessitate distinct feature extractors tailored to specific modalities (Chen et al., 2024). To handle such heterogeneity, recent work like FedMBridge (Chen and Zhang, 2024) proposes a topology-aware hypernetwork. However, this method incurs prohibitive communication overhead, which scales poorly with increasing model complexity and modality counts. Consequently, existing methods fail to adequately address the compounding challenges of inter-node modal imbalance and model heterogeneity. However, such methods incur prohibitive communication overhead and suffer from severe scalability bottlenecks as model complexity and the number of modalities increase. In contrast, our work proposes a communication-efficient heterogeneous federated learning framework via multimodal joint alignment. 2.2 Federated Prototype Learning Federated prototype learning has been extensively investigated across diverse tasks, wherein a prototype is defined as the mean feature vector of instances belonging to a specific class (Huang et al., 2022; Qi et al., 2025; Liao et al., 2025). Within the federated learning literature, prototypes serve as a vital mechanism to abstract knowledge while strictly preserving data privacy (Tan et al., 2022b; Zhao et al., 2024). Due to their robust representational capacity, prototypes are widely adopted to enforce local regularization and enhance communication efficiency (Huang et al., 2023; Yi et al., 2023). To mitigate the feature drift issue in FL, (Fu et al., 2025) proposed a federated domain-independent prototype learning approach, which achieves representation and parameter space alignment under feature shifts. Furthermore, FedPall (Zhang et al., 2025c) leverages prototype-based adversarial learning to unify the feature space and employs collaborative learning to reinforce class-specific information within the features. Conventionally, local prototypes are simply aggregated into a global prototype via weighted averaging on the server, yielding sub-optimal global knowledge that negatively impacts client performance. To address this limitation, FedTGP (Zhang et al., 2024) employs adaptive margin-enhanced contrastive learning to optimize trainable global prototypes at the server level. However, the feature misalignment arising from multimodal heterogeneity is significantly more severe than standard domain shifts. Consequently, naive aggregation fails to bridge the divergent feature spaces generated by heterogeneous local models. To address this challenge, we propose a learnable prototype framework on the server. By distilling knowledge from client-side prototypes, our approach effectively aligns the heterogeneous feature spaces. Figure 2: The framework of the MFedPBA. 3 Preliminary 3.1 General multimodal FL Framework Consider a heterogeneous MFL problem. The system consists of a central server and K clients. There are a total of M modality types (e.g. image, video, text, and audio, etc.) and C classes globally. The k-th client only possesses data from a subset of modalities ℳk⊆1,…,MM_k \1,…,M\, and has a combinatorial input space ℳk=(m|∀m∈ℳk)X_M_k=(X_m|∀ m _k), where mX_m is the subspace associated with the modality type m. The data distribution, quantity, and modality configuration of different clients are inconsistent. For client k and modality m∈ℳkm _k, its local data is denoted as k=(k,i,yk,i)i=1NkD_k=\( x_k,i,y_k,i)\_i=1^N_k, where yk,i∈1,…,Cy_k,i∈\1,…,C\. Each sample’s input consists of the modalities k,i=(k,im)m∈ℳk x_k,i=( x_k,i^m)_m _k present as in ℳkM_k, where k,im x_k,i^m denotes the modality m in k,i x_k,i. Following previous conventions (Tan et al., 2022a), we split each client k’s model θk _k into a feature extractor fkf_k parameterized by φk _k and a classifier hkh_k parameterized by wkw_k. Each client is equipped with |ℳk||M_k| modality specific feature extractors, denoted as fkm:ℳk→ℝdDf_k^m:X_M_k ^d_D. The client possesses a single classifier, denoted as hk:ℝdD→ℝdCh_k:R^d_D ^d_C. For inference or training, the client concatenates the feature vectors output by all modality specific feature extractors, and feeds this into the classifier hkh_k to obtain the final prediction. The global objective of MFL is formulated as: minθ1K∑k=1K1Nk∑i=1Nkℒk(θk)+Ω(θ1,θ2,⋯,θK), _θ 1K _k=1^K 1N_k _i=1^N_kL_k( _k)+ ( _1, _2,·s, _K), (1) which aims to jointly optimize the local objectives of all clients minθkℒk(θk)=(,y)∼kl(y,fk(,θk)) _ _kL_k( _k)=E_( x,y) _kl (y,f_k( x; _k) ), where l(⋅,⋅)l(·,·) is the loss function, and Eq. (1) utilizes a central server to encourage privacy-preserving knowledge sharing schemes Ω(⋅) (·) among clients in order to improve the local model performance of each client. 3.2 Prototype-based Federated Learning In contrast to conventional FL, which relies on aggregating model parameters, prototype-based FL achieves knowledge sharing by exchanging class prototypes between clients and the server, thereby offering a viable solution for scenarios involving model heterogeneity. Feature-based prototypes. During the local training phase, each client k computes its local prototype for each class c of each modality, using the following method: Ek,cm=1|k,cm|∑(,y)∈k,cmfkm(),E_k,c^m= 1|D_k,c^m| _( x,y) _k,c^mf_k^m( x), (2) where k,cmD_k,c^m represents a subset of kmD_k^m for the m-th modality of the local dataset, containing all data samples belonging to class c. Previous studies commonly upload local feature prototypes to the server and aggregate them via weighted averaging. However, in MFL, such direct aggregation is challenging due to heterogeneous feature extractors that project data into incompatible embedding spaces. Logit-based prototypes. In contrast to feature prototypes, which suffer from inherent space incompatibility across heterogeneous models, logit prototypes reside in a naturally aligned shared semantic space. Each dimension of a logit prototype directly corresponds to a specific class, thereby providing intrinsic semantic alignment across heterogeneous models. This property effectively avoids the collaborative degradation caused by inconsistent embedding spaces and enables more robust knowledge aggregation in heterogeneous federated learning scenarios. Therefore, the prototype based on logits can be defined as the average of the logit vectors of all nodes under category c: Ik,c=1|k,cm|∑(,y)∈k,cmhk(fkm()).I_k,c= 1|D_k,c^m| _( x,y) _k,c^mh_k(f_k^m( x)). (3) Considering that logits naturally occupy a common semantic space irrespective of the underlying model architectures, the direct alignment of output-level logit prototypes emerges as a more rational approach compared to utilizing feature-level prototypes. 3.3 Gromov Wasserstein Distance The Gromov-Wasserstein (GW) distance provides a rigorous framework for quantifying the discrepancy between two metric measure spaces (Mémoli, 2011). The GW distance is particularly well-suited for heterogeneous settings where a direct correspondence between feature dimensions is unavailable, which facilitates alignment by comparing the intrinsic geometric or relational structures of the data (Saha et al., 2025). Specifically, the GW framework evaluates the internal pairwise distance distributions within each space and seeks an optimal coupling (transport plan) that minimizes the relational distortion between these topologies. Formally, let (,CX,)(X,C_X,p) and (,CY,)(Y,C_Y,q) be two metric measure spaces, where CXC_X, CYC_Y denote the distance metrics of data X and Y, respectively (Peyré et al., 2016). The ,p,q represent the probability measures associated with each domain. The GW distance is defined as: GWp(CX,CY,,)=(min∑i,j,k,lT∈Π(,)L(CX(i,k),CY(j,l))pTijTkl)1/p, aligned &GW_p(C_X,C_Y,p,q)=\\ & ( _T∈ (p,q) _i,j,k,lL(C_X(i,k),\ C_Y(j,l))^p\,T_ijT_kl )^1/p, aligned (4) where Π(,)=T∈ℝ+n×m|Tm=,T⊤n= (p,q)= \T _+^n× m\, |\,T1_m=p,\ T 1_n=q \ is a joint distribution of all couplings from p to q, and p is the order of distance (commonly p=2). 4 Methodology The proposed MFedPBA aligns prototypes by collecting two types at the server. The MFedPBA framework is depicted in Figure 2, with the following details. 4.1 Global Prototype Modeling Each client first performs local training and then collects its local prototypes, denoted as Ek,cmk∈K\E_k,c^m\_k∈ K and Ik,ck∈K\I_k,c\_k∈ K, according to Eq. (2) and (3), and uploads them to the server. Upon receiving the bilateral prototypes from all clients, the server constructs global prototypes at both the logit level and the feature level, respectively. 4.1.1. Logit Prototype Aggregation Since logits are the final outputs of classifiers, they naturally reside in a semantically aligned shared representation space across heterogeneous models. Therefore, no additional processing is applied to the logit prototypes, and they are directly aggregated. Considering that data distribution discrepancies across clients lead to varying levels of predictive uncertainty in their logit prototypes, an entropy weighted aggregation strategy is adopted. Clients with lower predictive uncertainty, corresponding to lower entropy, are assigned higher weights, resulting in a more reliable global logit prototype. Accordingly, defining σ(⋅)σ(·) as the softmax, the entropy of a logit prototype is defined as: H(Ik,c)=−∑j=1dCσ(Ik,c)jlogσ(Ik,c)j.H(I_k,c)=- _j=1^d_Cσ(I_k,c)_j σ(I_k,c)_j. (5) The server weights the values based on the inverse of the entropy, obtaining a global logit prototype for class c: G,c=∑k=1Kqk,cIk,c,qk,c=(H(Ik,c)+ϵ)−1∑k′K(H(Ik′,c)+ϵ)−1,I_G,c= _k=1^Kq_k,cI_k,c, q_k,c= (H(I_k,c)+ε )^-1 _k ^K (H(I_k ,c)+ε )^-1, (6) where ϵε is a very small positive constant that avoids numerical collapse. 4.1.2. Feature Prototype Learning To effectively capture modality-specific discriminative characteristics and facilitate cross-modal collaboration, the server introduces an auto-encoding framework composed of modality-specific encoders Φm(⋅):ℝdD→ℝdD/2 _m(·):R^d_D ^d_D/2 and a shared decoder ψ(⋅):ℝdD/2→ℝdDψ(·):R^d_D/2 ^d_D. This projects feature prototypes of varying modalities and dimensionalities into a unified and comparable semantic space, while enforcing multiple constraints to achieve representation alignment across both clients and modalities. Therefore, we have: Pk,cm=ψ(Φm(Ek,cm)).P_k,c^m=ψ( _m(E_k,c^m)). (7) The modality-specific encoders project features of the same modality from different clients into a unified representation space, while the shared decoder enforces a common semantic structure during reconstruction. This promotes cross-modal consistency and constraining latent representations from different modalities within a shared and information-rich semantic space. Consequently, global modality feature prototypes are derived through the weighted aggregation of these intra-modality aligned prototypes: G,cm=1∑k(m∈ℳk)∑k,m∈ℳkPk,cm,P_G,c^m= 1 _kI(m _k) _k,m _kP_k,c^m, (8) here I denotes the indicator function. To make the learned global modal prototypes more discriminative, the server updates the network parameters by minimizing the following joint loss function: Prototype Reconstruction Loss: This loss ensures that the projection and reconstruction process preserves the essential information of the input prototypes: ℒrec=1|K|∑k,m,c‖Pk,cm−Ek,cm‖2.L_rec= 1|K| _k,m,c\|P_k,c^m-E_k,c^m\|^2. (9) Inter-modal Contrastive Loss: This loss enhances semantic consistency across modalities within the same class by pulling together projected features of different modalities for class c, while pushing apart features from different classes ℒcon=−∑c∑m≠m′logexp(sim(G,cm,G,cm′)/τ)∑c′exp(sim(G,cm,G,c′m′)/τ),L_con=- _c _m≠ m (sim\! (P_G,c^m,P_G,c^m )/τ ) _c (sim\! (P_G,c^m,P_G,c ^m )/τ ), (10) where sim(u,v)=u⊤v/(‖u‖‖v‖)sim(u,v)=u v/(\|u\|\|v\|), and τ is the temperature coefficient. Intra-modal Distribution Alignment Loss: This loss aligns the structural consistency of the same class across different modalities using the GW distance. This loss does not require strict dimensional alignment but preserves the proportional relationships of inter-class distances between the feature space and the logit space, thereby maintaining topological consistency. We compute the GW distance between the modality-specific feature prototypes G,cmP_G,c^m with the global logit prototypes G,cI_G,c space. Within each metric space (Mémoli, 2011), we characterize the intrinsic geometric topology using the pairwise squared Euclidean distance, defined as Ci,j=‖xi−xj‖22C_i,j=\|x_i-x_j\|_2^2. Consequently, we derive the cost matrix CG∈ℝdC×dCC_I_G ^d_C× d_C and CGm∈ℝdC×dCC_P_G^m ^d_C× d_C to represent the inter-class relational structures within the feature and logit spaces, respectively. Therefore, this distance loss can be expressed as: ℒalign=∑m(GW22(CG,CGm,,)−εH(Tm)) _align= _m (GW_2^2(C_I_G,C_P_G^m,p,q)- H(T^m) ) (11) =∑mmin∑i,j,k,lTm∈Π(,)(CG(i,k)−CGm(j,l))2TijmTklm = _m _T^m∈ (p,q) _i,j,k,l (C_I_G(i,k)-C_P_G^m(j,l) )^2\,T_ij^mT_kl^m +∑mε∑ijTijmlogTijm. + _m _ijT_ij^m T_ij^m. To solve this problem, we introduce entropy regularization H(T)H(T) to make the problem differentiable, where ε weights this regularization, and then use the Sinkhorn algorithm (Cuturi, 2013; Séjourné et al., 2021) to solve the optimal transmission problem of this regularization. Therefore, the server optimizes the global loss for S rounds. The total loss is formulated as: ℒserver=ℒrec+ℒcon+ℒalign.L_server=L_rec+L_con+L_align. (12) The refined client feature prototypes incorporate richer global semantics and modality-specific information. Therefore, the server constructs a global feature prototype for each modality m by averaging the learned prototypes. Finally, the server sends the global prototypes G,cmm=1M\P_G,c^m\_m=1^M and G,cI_G,c to each client, transferring complementary cross-modal knowledge to compensate for the information deficiency in clients with missing modalities. 4.2 Client Local Update Upon receiving the prototype set from the server, the objective of local training is to effectively distill knowledge from both the global feature prototypes and the global logit prototypes at the representation and decision levels, thereby maximally injecting global information into local representation learning. To this end, we propose a prototype-based supervised contrastive loss composed of two terms. Feature alignment loss is introduced to mitigate feature space drift caused by model heterogeneity and biased local data distributions. Since the local feature extractor fkm(⋅)f_k^m(·) tends to overfit limited modal client data, we directly anchor local feature representations to the global feature prototype space, encouraging all clients to optimize toward a shared and consensus feature distribution. Specifically, we employ the mean squared error as the alignment metric, defined as ℒfea=1|ℳk|∑m∈ℳk1C∑c=1C‖G,cm−Ek,cm‖22.L_fea= 1|M_k| _m _k 1C _c=1^C\|P_G,c^m-E_k,c^m\|_2^2. (13) Logit alignment loss performs knowledge distillation at the decision level. The global logit prototypes G,cI_G,c encode rich information about inter-class relationships and global decision structures. By enforcing the local classifier outputs to match the global logit prototypes, robust and transferable decision knowledge is distilled into local models. Accordingly, we measure the distribution discrepancy using the Kullback–Leibler (KL) divergence: ℒlogit=1C∑c=1CKL(G,c∥Ik,c),L_logit= 1C _c=1^CD_KL(I_G,c\|I_k,c), (14) where KL(P∥Q)=∑iP(i)ln(P(i)Q(i))D_KL(P\|Q)= _iP(i) ( P(i)Q(i)). Finally, to preserve discriminative capability on local data domains, we incorporate the standard cross-entropy loss between the predicted logits and ground-truth labels: ℒce=∑(i,yi)∈k−yilog(hk(fkm(xi))),L_ce= _( x_i,y_i) _k-1_y_i (h_k(f_k^m(x_i))), (15) where σ(⋅)σ(·) is the softmax function. The total local loss is given as follows: ℒclient=ℒce+λ1ℒfea+λ2ℒlogit.L_client=L_ce+ _1L_fea+ _2L_logit. (16) The overall MFL algorithm is shown in Algorithm 1. 5 Convergence Analysis To analyze the convergence of MFedPBA, we define t as the current communication round, e∈0,1,⋯,Ee∈\0,1,·s,E\ as the number of local iterations, where E denotes the maximum number of local iterations. Thus, (tE+e)(tE+e) represents the e-th iteration in the (t+1)(t+1)-th communication round. We make some assumptions see Appendix B.1. Based on the above assumptions, we have the following lemmas and theorems: Lemma 5.1. Based on Assumption B.1 and B.2, in the local iteration e∈0,1,…,Ee∈\0,1,...,E\ of the (t+1)(t+1)-th training round, the local model loss of any client is bounded by: [ℒ(t+1)E]≤ℒtE+0−(ηc−Lηc22)∑e=0E‖∇ℒtE+e‖22+LEηc22σ2. split&E [L_(t+1)E ]≤\\ &L_tE+0-( _c- Lη^2_c2) _e=0^E\| _tE+e\|_2^2+ LE _c^22σ^2. split (17) Lemma 5.2. After feature prototype learning and logit prototype aggregation are completed on the server, the loss function of any client can be constrained as follows: [ℒ(t+1)E+0]≤ℒ(t+1)E+(Mλ1+λ2)LrEηcGc+2λ1(δs+MGs). split&E[L_(t+1)E+0] _(t+1)E\\ &+(M _1+ _2)L_rE _cG_c+2 _1( _s+MG_s). split (18) Based on Lemma 5.1 and Lemma 5.2, we can further derive the following theorems. Theorem 5.3. The above assumptions, the expectation of the loss of an arbitrary client’s local model before the start of a round of local iteration satisfies: [ℒ(t+1)E+0]≤ℒtE+0−(ηc−Lηc22)∑e=0E‖∇ℒtE+e‖22+LEηc22σ2+ΓdEηc+Γs. splitE[L_(t+1)E+0]& _tE+0-( _c- L _c^22) _e=0^E\| _tE+e\|_2^2\\ &+ LE _c^22σ^2+ _dE _c+ _s. split (19) where Γd=(Mλ1+λ2)LrGc _d=(M _1+ _2)L_rG_c and Γs=2λ1(δs+MGs) _s=2 _1( _s+MG_s) represent the drift constant and server bias constant. Theorem 5.4. The above assumptions, for arbitrary client and any ϵ>0ε>0, if learning rate ηc<min2L,2(ϵ−ΓdE)L(ϵ+Eσ2) _c< \ 2L, 2(ε- _dE)L(ε+Eσ^2) \ and Γs→0 _s→ 0, the following inequality holds: 1T∑t=0T−1∑e=0E 1T _t=0^T-1 _e=0^E [‖ℒtE+e‖22]≤2(ℒt=1−ℒ∗)Tηc(2−Lηc) [ \|L_tE+e \|_2^2 ]≤ 2 (L_t=1-L^* )T _c (2-L _c ) (20) +LEηc2σ2+2(ΓdEηc+Γs)2ηc−Lηc2. + LE _c^2σ^2+2( _dE _c+ _s)2 _c-L _c^2. With this, it is evident that the local model of any client of MFedPBA converges at a non-convex convergence rate (1T)O ( 1T ). See Appendix B for a detailed proof. 6 Experiments 6.1 Experimental Setup Datasets and Model: We used four benchmark datasets for experiments: Caltech101 11 1 https://data.caltech.edu/records/mzrjq-6wc02., Reuters 22 2 https://w.kaggle.com/datasets/nltkdata/reuters., NUS-WIDE 33 3 https://w.kaggle.com/datasets/xinleili/nuswide., and Youtube 44 4 http://archive.ics.uci.edu/ml/datasets.. The data information is shown in Table 1. All datasets are divided into training and test sets in a ratio of 75%/25%. Table 1: Dataset statistics. Acronyms for modality types: I (Image), L (Language), A (Audio), T (Time-series), X1 (Style-one X), X2 (Style-two X). The #\#S: Samples, #\#C: Classes, #\#M: Modalities. Datasets Types #\# S #\# C #\# M Input modalities Caltech101 I 9144 102 3 I1, I2, and I3 Reuters L 18758 6 5 L1, L2, L3 L4, and L5 NUS-WIDE I, L 5000 10 4 I1, I2, I3, and L Youtube I, A, T 2000 10 6 I1, I2, T A1, A2, and A3 To model heterogeneous MFL scenarios, under the assumption that the number of clients K equals the number of modalities M, we consider three client data distribution settings: “M2”, where each client possesses two modalities; “M1+”, where each client has at least one modality; and “M1”, where each client contains exactly one modality. Using the YouTube dataset as an example, the corresponding data distributions are illustrated in Figure 3. We assign heterogeneous models to each modality of every client to simulate realistic model heterogeneity. Detailed descriptions of the local dataset partitions and the corresponding neural architecture configurations are provided in Appendix C.1 & C.2. Figure 3: Example of dataset distribution partitioning. Table 2: Comparison of average performance of different methods in simulations with different data distributions. M#: Data partition (K=M); K#: Number of large clients (in M1+). Best and second-best are bold and underlined respectively. Dataset Local FedProto FedTGP FedPall Harmony FedMVP FedMobile MFedPBA Caltech101 M2 42.65±1.47 38.72±0.42 40.78±1.31 42.29±0.27 45.29±0.38 40.23±0.65 43.51±0.54 49.04±0.21 M1+ 40.83±1.12 36.81±0.35 40.71±1.23 41.8±0.77 43.04±0.64 38.91±1.25 42.29±0.6 46.82±0.54 M1 38.47±0.45 31.57±1.23 36.18±0.72 40.77±0.36 40.1±0.69 38.92±0.62 40.09±0.13 43.64±0.11 K50 29.38±1.63 25.17±1.78 28.77±1.43 30.59±1.26 30.58±0.65 27.74±1.5 30.88±0.81 31.19±0.92 Reuters M2 70.61±0.28 73.39±0.57 69.73±0.88 73.08±1.04 72.18±0.41 74.07±1.08 73.19±0.26 75.16±0.45 M1+ 70.85±0.59 68.54±1.46 69.8±0.59 71.09±1.56 72.28±0.47 68.09±0.82 72.79±0.44 75.01±0.22 M1 67.05±1.04 68.75±0.63 65.81±0.33 66.96±0.22 70.16±1.14 70.18±0.57 69.92±0.95 72.09±0.79 K100 52.88±1.08 46.48±0.93 48.31±1.41 56.55±0.53 53.96±0.51 52.28±1.53 56.64±0.29 57.83±0.97 NUS-WIDE M2 32.18±0.64 29.28±0.46 31.19±0.51 33.38±0.91 32.97±0.28 31.89±0.4 33.86±0.22 35.2±0.46 M1+ 28.72±0.71 27.79±0.55 27.13±0.18 30.99±0.63 31.56±0.61 29.82±0.59 30.95±0.57 32.15±0.36 M1 31.15±0.14 24.96±0.52 29.55±0.73 31.02±0.96 32.19±0.35 32.7±0.51 31.2±0.36 33.25±0.24 K20 24.52±0.5 24.47±0.95 24.15±0.37 26.35±0.44 26.28±0.41 26.61±0.86 27.34±0.37 28.08±0.37 Youtube M2 41.72±0.66 41.12±0.68 45.16±1.02 44.23±0.3 43.8±0.76 45.53±0.52 44.83±0.53 47.92±0.93 M1+ 41.89±0.76 38.82±1.12 44.55±0.68 43.04±1.09 43.47±0.91 44.24±0.36 44.28±0.64 47.21±0.59 M1 40.01±0.41 36.59±0.39 38.62±0.64 39.15±1.5 41.07±0.42 40.04±0.46 39.48±0.99 44.52±0.54 K20 32.01±0.25 30.57±1.49 34.15±0.31 32.59±0.47 33.5±0.49 31.12±0.77 34.22±0.63 35.07±0.92 Baselines: To ensure fair comparison, we re-implemented all baseline methods within a unified HtFLlib framework (Zhang et al., 2025a) and evaluated them under identical experimental settings. Specifically, we include representative prototype-based federated learning methods FedProto (Tan et al., 2022a), FedTGP (Zhang et al., 2024), and FedPall (Zhang et al., 2025c), as well as state-of-the-art multimodal federated learning approaches Harmony (Ouyang et al., 2023), FedMVP (Che et al., 2024), and FedMobile (Liu et al., 2025), with local training (Local) serving as a reference baseline. For consistency, Local and all prototype-based methods adopt the same local multimodal aggregation strategy as our approach. Moreover, to accommodate model heterogeneity, methods that originally require uploading local models are uniformly modified to upload classifier parameters instead. Details of all baselines are provided in Appendix C.3. Implementation details: To ensure reliability, all methods are evaluated across five independent trials, and we report the mean performance alongside the standard deviation. SGD is adopted as the optimizer uniformly, with the number of local training epochs set to 2. Additionally, to accommodate varying dataset scales, the total number of global communication rounds is configured adaptively for each dataset. The details are provided in Appendix C.4. Figure 4: Test accuracy (%) on four datasets under the M2 data distribution setting with model heterogeneity. 6.2 Experimental Result Test Accuracy. Table 2 reports the test accuracies of comparative methods on four multimodal datasets under the M2, M1+, and M1 partition settings. MFedPBA consistently achieves superior performance in all scenarios, with substantial improvements validating the efficacy of the dual-prototype bilateral alignment framework. To further investigate feature space, we visualize the feature prototypes EkmE_k^m of three-modal clients in the Reuters dataset using t-SNE (Van der Maaten and Hinton, 2008), as shown in Figure 5. Despite sharing identical class labels, features extracted from different modalities are clearly separated due to heterogeneous encoders, corroborating our motivation on feature space incompatibility. Compared with Local training, our method exhibits more compact and well-structured clusters, indicating more effective representation learning. Moreover, in several settings, FedProto underperforms Local training, further confirming that naively aggregating misaligned prototypes under model heterogeneity can introduce knowledge contamination and lead to severe performance degradation rather than improvement. Furthermore, we extended experiment to a larger number of clients to assess the scalability of the proposed framework. As summarized in Table 2 (K #Client number), the experimental results demonstrate that MFedPBA consistently maintains its superior performance even in large-scale FL settings. Detailed discussions and per-client performance results are provided in Appendix D. Figure 5: Feature visualization results of Local and MFedPBA. Communication Efficiency. Theoretically, prototype-based FL significantly reduces communication overhead compared to methods that transmit full model parameters. The standard bidirectional communication cost for FedProto is K2CMkdDK2CM_kd_D, our framework introduces only a marginal increase for transmitting logit prototypes, bringing the total complexity to K2C(MkdD+dC)K2C(M_kd_D+d_C). Since the number of classes is typically several orders of magnitude smaller than the total model parameters, this additional overhead remains negligible. Consequently, our approach maintains high communication efficiency without imposing substantial costs relative to methods requiring parameters. We plotted the performance curves of various methods across four datasets under the M2 configuration, as shown in Figure 4, thereby confirming the stable convergence of our proposed framework. We observe that prototype-based baselines, particularly FedTGP, exhibit pronounced fluctuations during the training process. This instability stems from the erratic server prototype updates in multimodal scenarios, which hinder the acquisition of discriminative representations. In contrast, our feature prototypes are regularized by a multi-faceted objective function, allowing them to consolidate global semantic knowledge while preserving critical modality-specific discriminability. Computational Cost. Compared to the baseline FedProto, the incremental client overhead is limited to the KL divergence between local and global logit prototypes, yielding a marginal complexity of (C2)O(C^2). On the server, logit aggregation follows a linear complexity of (KCdC)O(KCd_C), while feature-level operations encompass autoencoder mapping at (∑k|ℳk|CdD2)O( _k|M_k|Cd_D^2), reconstruction loss calculation at (∑k|ℳk|CdD)O( _k|M_k|Cd_D), cross-modal contrastive loss at (CM2dD)O(CM^2d_D), and intra-modality structural alignment at (sMC2)O(T_sMC^2), where sT_s is the number of iterations of the Sinkhorn algorithm. Although these complexities scale with the class count C, they remain manageable within the scope of standard classification tasks. Crucially, our framework strategically offloads the primary computational burden to the resource-rich server, leveraging its superior processing power to facilitate high-precision alignment without straining resource-constrained clients. Figure 7 shows the running time of the client in each round on Reuters, further demonstrating that our computational overhead is acceptable. Figure 6: Time consumption. Figure 7: Ablation results. Ablation Study. Figure 7 presents a comprehensive ablation study conducted under the Caltech101 M2 configuration to evaluate the individual contributions of both server-side and client-side modules. In this analysis, the horizontal axis denotes the exclusion (w/o) of specific algorithmic components, while ”EA” indicates the substitution of our proposed entropy-based weighting with standard equal averaging. The empirical results clearly demonstrate that the removal of any server-side module precipitates a significant drop in overall accuracy, thereby validating the necessity and synergy of the integrated design. Notably, eliminating the reconstruction loss (ℒrecL_rec) triggers the most severe performance degradation. This acute decline occurs because the absence of reconstruction constraints induces representation collapse within the latent space, which subsequently propagates erroneous global knowledge into the local training phase of individual clients. Impact of Feature Dimensions. We systematically evaluate the impact of the feature dimension dDd_D on model performance by varying it across a predefined set of values (e.g., dD∈24,48,64,128,256d_D∈\24,48,64,128,256\), as summarized in Table 3. We observe a clear trade-off: smaller dimensions restrict the model’s ability to capture complex data patterns, leading to underfitting, whereas excessively large dimensions introduce redundant noise and complicate classifier training. Specifically, on the YouTube dataset, most evaluated methods achieve their most stable and superior performance at dD=48d_D=48. Based on this empirical observation, we independently conduct this sensitivity analysis across all datasets, ultimately selecting the dataset-specific optimal dimensions to strike the best balance between representational capacity and trainability. Table 3: The test accuracy (%) on Youtube in the M1+ setting. “Fed” is omitted in the method name due to limited space. dDd_D Local Proto TGP Pall Harm MVP Mobile PBA 24-d 41.19 30.12 41.19 42.56 42.06 43.85 40.91 44.43 48-d 42.34 39.05 43.85 44.63 44.43 45.21 44.57 47.02 64-d 43.02 37.82 45.22 42.27 43.13 43.21 44.24 46.46 128-d 42.64 31.24 45.01 42.15 43.24 44.33 44.33 45.68 256-d 41.95 26.32 43.22 41.75 42.74 42.84 42.54 43.84 Hyper-parameter. We analyze the sensitivity of server epochs S and regularization terms λ1,λ2 _1, _2. As shown in Figure 8 (left). While accuracy improves with larger S, gains diminish significantly from 5050 to 10001000 epochs. Consequently, we set S=10S=10 to balance efficiency and performance. We evaluated the sensitivity of the λ1,λ2 _1, _2 using Youtube in the range 0.001,0.01,0.05,0.1,0.5,1,5,8,10\0.001,0.01,0.05,0.1,0.5,1,5,8,10\. As shown in Figure 8 (right), MFedPBA demonstrates strong robustness across most parameter choices, consistently achieving accuracy above 46 and outperforming all baselines. Under several favorable configurations, the performance further improves to 48.9. In contrast, excessively large loss weights may introduce overly strong alignment constraints, leading to performance degradation. Figure 8: Hyperparameter experiments for λ1 _1,λ2 _2, and S. 7 Conclusion. This paper addresses the challenges of model heterogeneity and modality imbalance in multimodal federated learning by proposing a prototype-guided bidirectional alignment framework for effective knowledge sharing across heterogeneous feature spaces. We analyze the limitations of conventional prototype aggregation under heterogeneous encoders and introduce a dual alignment mechanism that anchors knowledge transfer through semantically aligned logit prototypes while bridging embedding space discrepancies via GW–based feature alignment. Experimental results show that the proposed method effectively mitigates negative transfer and knowledge contamination, and significantly improves generalization and robustness under imbalanced and missing-modality settings. Acknowledgements The research is supported by Seed Funding for Collaborative Research Grants of HKBU (with Grant No. RC-SFCRG/23-24/R2/SCI/06), the National Key Research and Development Program of China (2023YFB2703700), and the GMCC-SYSU Joint Lab for Smart Applications. Impact Statement This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here. References Cao et al. (2026) M. Cao, T. Chen, M. Hu, Z. Qi, Y. Cui, J. Zhou, and X. Xie Two heads are better than one: generalized cross-domain federated learning via dual-prototype. In Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining, p. 59–70. Cited by: §1. Che et al. (2024) L. Che, J. Wang, X. Liu, and F. Ma Leveraging foundation models for multi-modal federated learning with incomplete modality. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, p. 401–417. Cited by: §A.2, §C.3, §1, §2.1, §6.1. Che et al. (2023) L. Che, J. Wang, Y. Zhou, and F. Ma Multimodal federated learning: a survey. Sensors 23 (15), p. 6986. Cited by: §1. Chen et al. (2025) C. Chen, T. Liao, X. Deng, Z. Wu, S. Huang, and Z. Zheng Advances in robust federated learning: a survey with heterogeneity considerations. IEEE Transactions on Big Data 11 (3), p. 1548–1567. Cited by: §1. Chen et al. (2024) H. Chen, Y. Zhang, D. Krompass, J. Gu, and V. Tresp Feddat: an approach for foundation model finetuning in multi-modal heterogeneous federated learning. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 38, p. 11285–11293. Cited by: §2.1. Chen and Zhang (2024) J. Chen and A. Zhang FedMBridge: bridgeable multimodal federated learning. In Forty-first International Conference on Machine Learning, Cited by: §1, §2.1. Collins et al. (2021) L. Collins, H. Hassani, A. Mokhtari, and S. Shakkottai Exploiting shared representations for personalized federated learning. In International conference on machine learning, p. 2089–2099. Cited by: §1. Cuturi (2013) M. Cuturi Sinkhorn distances: lightspeed computation of optimal transport. Advances in neural information processing systems 26. Cited by: §4.1. Deng et al. (2025) Y. Deng, N. He, X. Li, F. Wu, Y. Zhang, and J. Ren Cross-modal federated learning among unimodal devices. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 9 (3), p. 1–26. Cited by: §2.1. Feng et al. (2023) T. Feng, D. Bose, T. Zhang, R. Hebbar, A. Ramakrishna, R. Gupta, M. Zhang, S. Avestimehr, and S. Narayanan Fedmultimodal: a benchmark for multimodal federated learning. In Proceedings of the 29th ACM SIGKDD conference on knowledge discovery and data mining, p. 4035–4045. Cited by: §2.1. Fu et al. (2025) L. Fu, S. Huang, Y. Lai, C. Zhang, H. Dai, Z. Zheng, and C. Chen Federated domain-independent prototype learning with alignments of representation and parameter spaces for feature shift. IEEE Transactions on Mobile Computing 24 (9), p. 9004–9019. Cited by: §1, §2.2. He et al. (2025) W. He, W. Huang, B. Yang, S. Liu, and M. Ye SPMC: self-purifying federated backdoor defense via margin contribution. In Forty-second International Conference on Machine Learning, Cited by: §1. Hu et al. (2024a) M. Hu, Z. Yue, X. Xie, C. Chen, Y. Huang, X. Wei, X. Lian, Y. Liu, and M. Chen Is aggregation the only choice? federated learning via layer-wise model recombination. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, p. 1096–1107. Cited by: §1. Hu et al. (2024b) M. Hu, P. Zhou, Z. Yue, Z. Ling, Y. Huang, A. Li, Y. Liu, X. Lian, and M. Chen FedCross: towards accurate federated learning via multi-model cross-aggregation. In Proceedings of IEEE International Conference on Data Engineering (ICDE), p. 2137–2150. Cited by: §1. Huang et al. (2026) S. Huang, L. Fu, Z. Chen, T. Zhang, X. Li, and Z. Cui ADAN: adversarial distribution alignment network for multi-view semi-supervised classification. IEEE Transactions on Image Processing 35, p. 4861–4876. Cited by: §1. Huang et al. (2024) W. Huang, D. Wang, X. Ouyang, J. Wan, J. Liu, and T. Li Multimodal federated learning: concept, methods, applications and future directions. Information Fusion 112, p. 102576. Cited by: §1. Huang et al. (2022) W. Huang, M. Ye, and B. Du Learn from others and be yourself in heterogeneous federated learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 10143–10153. Cited by: §2.2. Huang et al. (2023) W. Huang, M. Ye, Z. Shi, H. Li, and B. Du Rethinking federated learning with domain shift: a prototype view. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 16312–16322. Cited by: §1, §2.2. Kou et al. (2026) Z. Kou, J. Wu, W. Huang, W. He, M. Xie, C. Wang, Y. Jia, D. Jiang, Y. Liu, X. Geng, and Q. Yang FedHarmony: harmonizing heterogeneous label correlations in federated multi-label learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Cited by: §1. Li et al. (2025) Y. Li, H. Wang, Y. Qi, W. Liu, and R. Li Re-fed+: a better replay strategy for federated incremental learning. IEEE Transactions on Pattern Analysis and Machine Intelligence. Cited by: §1. Li et al. (2024) Y. Li, W. Xu, Y. Qi, H. Wang, R. Li, and S. Guo SR-FDIL: synergistic replay for federated domain-incremental learning. IEEE Trans. Parallel Distributed Syst. 35 (11), p. 1879–1890. Cited by: §1. Liao et al. (2024) T. Liao, L. Fu, J. Chen, Z. Wang, Z. Zheng, and C. Chen A swiss army knife for heterogeneous federated learning: flexible coupling via trace norm. Advances in Neural Information Processing Systems 37, p. 139886–139911. Cited by: §1. Liao et al. (2025) T. Liao, B. Xie, L. Fu, S. Huang, B. Deng, C. Chen, and Z. Zheng Federated domain generalization with decision insight matrix. In Proceedings of the International Joint Conference on Artificial Intelligence, Montréal, QC, Canada, p. 16–22. Cited by: §2.2. Liu et al. (2025) Y. Liu, C. Wang, and X. Yuan FedMobile: enabling knowledge contribution-aware multi-modal federated learning with incomplete modalities. In Proceedings of the ACM on Web Conference 2025, p. 2775–2786. Cited by: §A.2, §C.3, §2.1, §6.1. [25] Y. Ma, W. Dai, G. Jiang, C. Zhou, Y. Zhang, F. Luo, J. Wang, A. Zhang, et al. FedMC: federated manifold calibration. In The Fourteenth International Conference on Learning Representations, Cited by: §1. Mémoli (2011) F. Mémoli Gromov–wasserstein distances and the metric approach to object matching. Foundations of computational mathematics 11 (4), p. 417–487. Cited by: §3.3, §4.1. Meng et al. (2024) L. Meng, Z. Qi, L. Wu, X. Du, Z. Li, L. Cui, and X. Meng Improving global generalization and local personalization for federated learning. IEEE Transactions on Neural Networks and Learning Systems 36 (1), p. 76–87. Cited by: §1. Ouyang et al. (2023) X. Ouyang, Z. Xie, H. Fu, S. Cheng, L. Pan, N. Ling, G. Xing, J. Zhou, and J. Huang Harmony: heterogeneous multi-modal federated learning through disentangled model training. In Proceedings of the 21st Annual International Conference on Mobile Systems, Applications and Services, p. 530–543. Cited by: §A.2, §C.3, §2.1, §6.1. Pan et al. (2024) H. Pan, X. Zhao, L. He, Y. Shi, and X. Lin A survey of multimodal federated learning: background, applications, and perspectives. Multimedia Systems 30 (4), p. 222. Cited by: §2.1. Pan et al. (2025) H. Pan, X. Zhao, Y. Jiang, L. He, B. Wang, and Y. Shu FedVLP: visual-aware latent prompt generation for multimodal federated learning. Computer Vision and Image Understanding 259, p. 104442. Cited by: §1. Peyré et al. (2016) G. Peyré, M. Cuturi, and J. Solomon Gromov-wasserstein averaging of kernel and distance matrices. In International conference on machine learning, p. 2664–2672. Cited by: §3.3. Qi et al. (2023) Z. Qi, L. Meng, Z. Chen, H. Hu, H. Lin, and X. Meng Cross-silo prototypical calibration for federated learning with non-iid data. In Proceedings of the 31st ACM international conference on multimedia, p. 3099–3107. Cited by: §1. Qi et al. (2025) Z. Qi, L. Meng, Z. Li, H. Hu, and X. Meng Cross-silo feature space alignment for federated learning on clients with imbalanced data. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 39, p. 19986–19994. Cited by: §1, §2.2. Rahman and Nguyen (2024) R. Rahman and D. C. Nguyen Multimodal federated learning with model personalization. In OPT 2024: Optimization for Machine Learning, Cited by: §1. Saha et al. (2025) P. Saha, D. Mishra, F. Wagner, K. Kamnitsas, and J. A. Noble Fedpia–permuting and integrating adapters leveraging wasserstein barycenters for finetuning foundation models in multi-modal federated learning. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 39, p. 20228–20236. Cited by: §3.3. Séjourné et al. (2021) T. Séjourné, F. Vialard, and G. Peyré The unbalanced gromov wasserstein distance: conic formulation and relaxation. Advances in Neural Information Processing Systems 34, p. 8766–8779. Cited by: §4.1. Tan et al. (2022a) Y. Tan, G. Long, L. Liu, T. Zhou, Q. Lu, J. Jiang, and C. Zhang Fedproto: federated prototype learning across heterogeneous clients. In Proceedings of the AAAI conference on artificial intelligence, Vol. 36, p. 8432–8440. Cited by: §B.2, Appendix B, §C.3, §1, §3.1, §6.1. Tan et al. (2022b) Y. Tan, G. Long, J. Ma, L. Liu, T. Zhou, and J. Jiang Federated learning from pre-trained models: a contrastive learning approach. Advances in neural information processing systems 35, p. 19332–19344. Cited by: §2.2. Thrasher et al. (2025) J. Thrasher, A. Devkota, P. Siwakoti, R. Chivukula, P. Poudel, C. Hu, A. Tafti, B. Bhattarai, and P. Gyawali Multimodal federated learning in healthcare: a review. Journal of Healthcare Informatics Research, p. 1–30. Cited by: §2.1. Van der Maaten and Hinton (2008) L. Van der Maaten and G. Hinton Visualizing data using t-sne.. Journal of machine learning research 9 (11). Cited by: §6.2. Wang et al. (2024) S. Wang, Z. Qu, Y. Liu, S. Kan, Y. Liang, and J. Wang Fedmmr: multi-modal federated learning via missing modality reconstruction. In 2024 IEEE International Conference on Multimedia and Expo, p. 1–6. Cited by: §1. Xiao et al. (2026) T. Xiao, Y. Li, Y. Qi, Y. Liu, H. Wang, Y. Wang, and R. Li Enhancing privacy in multimodal federated learning with information theory. Advances in Neural Information Processing Systems 38, p. 142199–142215. Cited by: §1. Yi et al. (2023) L. Yi, G. Wang, X. Liu, Z. Shi, and H. Yu Fedgh: heterogeneous federated learning with generalized global header. In Proceedings of the 31st ACM international conference on multimedia, p. 8686–8696. Cited by: §2.2. Yu et al. (2025) S. Yu, S. Liu, S. Wang, C. Tang, Z. Luo, X. Liu, and E. Zhu Sparse low-rank multi-view subspace clustering with consensus anchors and unified bipartite graph. IEEE Transactions on Neural Networks and Learning Systems 36 (1), p. 1438–1452. Cited by: §1. Zhang et al. (2024) J. Zhang, Y. Liu, Y. Hua, and J. Cao Fedtgp: trainable global prototypes with adaptive-margin-enhanced contrastive learning for data and model heterogeneity in federated learning. In Proceedings of the AAAI conference on artificial intelligence, Vol. 38, p. 16768–16776. Cited by: §C.3, §2.2, §6.1. Zhang et al. (2025a) J. Zhang, X. Wu, Y. Zhou, X. Sun, Q. Cai, Y. Liu, Y. Hua, Z. Zheng, J. Cao, and Q. Yang HtFLlib: a comprehensive heterogeneous federated learning library and benchmark. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Cited by: §C.3, §6.1. Zhang et al. (2025b) R. Zhang, X. Chi, W. Zhang, G. Liu, D. Wang, and F. Wang Unimodal training-multimodal prediction: cross-modal federated learning with hierarchical aggregation. IEEE Trans. Mob. Comput. 24 (10), p. 10009–10023. Cited by: §2.1. Zhang et al. (2025c) Y. Zhang, F. Liang, G. Yuan, M. Yang, C. Li, and X. Hu FedPall: prototype-based adversarial and collaborative learning for federated learning with feature drift. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 3111–3120. Cited by: §C.3, §1, §2.2, §6.1. Zhao et al. (2024) J. Zhao, X. Yi, R. Li, Y. Li, H. Wang, Y. Li, Z. Deng, and Z. Xu FedTA: unsupervised federated prototype learning with temperature adaptation. In 2024 IEEE International Conference on High Performance Computing and Communications (HPCC), p. 390–397. Cited by: §2.2. Zhao et al. (2022) Y. Zhao, P. Barnaghi, and H. Haddadi Multimodal federated learning on iot data. In 2022 IEEE/ACM seventh international conference on internet-of-things design and implementation (ioTDI), p. 43–54. Cited by: §2.1. Zheng et al. (2023) T. Zheng, A. Li, Z. Chen, H. Wang, and J. Luo Autofed: heterogeneity-aware federated multimodal learning for robust autonomous driving. In Proceedings of the 29th annual international conference on mobile computing and networking, p. 1–15. Cited by: §2.1. Summary of Appendix In the appendix, the following contents are included: A Algorithm A.1 Algorithm Framework A.2 Privacy Analysis B Convergence Analysis B.1 Assumption B.2 Lemmas & Theorems B.3 Proof of Lemma 5.1 B.4 Proof of Lemma 5.2 B.5 Proof of Theorem 5.3 B.6 Proof of Theorem 5.4 C Details of the Experimental Setup C.1 Dataset Description C.2 Heterogeneous Model C.3 Baseline Methods C.4 Parameter Setting Details D Details of the Experimental Result Appendix A Algorithm A.1 Algorithm Framework We summarize the main steps of the proposed in Algorithm 1. Algorithm 1 MFedPBA Input: total rounds T, local epochs E, server epochs S, total number of clients K, sampled number of clients KCK_C, local learning rate ηc _c, server learning rate ηs _s, hyper-parameter for loss λ1 _1 and λ2 _2. Server executes: 1: Initialize modality-specific encoders Φm(⋅) _m(·) and shared decoder ψ(⋅)ψ(·) 2: for each round t=1⋯Tt=1·s T do 3: Server samples subset KCK_C of clients 4: for each client k∈KCk∈ K_C in parallel do 5: Ek,cmm=1|ℳk|,Ik,c\\E_k,c^m\_m=1^|M_k|,I_k,c\ ← Client updates(G,cm\P_G,c^m\, G,cI_G,c) 6: end for 7: Aggregate logit prototype by Eq. (6) 8: for each server epoch s=1⋯Ss=1·s S do 9: Project client prototype to a unified space by Eq. (7) 10: Aggregate modal feature prototype by Eq. (8) 11: Calculate server loss update by Eq. (12) 12: end for 13: end for Clients updates: 1: for each local epoch e=1⋯Ee=1·s E do 2: Sample mini-batch in ℬB: 3: Calculate sample feature-based prototype Ek,cmE_k,c^m by Eq. (2) and logit-based prototype Ik,cI_k,c by Eq. (3) 4: Calculate local loss by Eq. (16) 5: Update local model: θkt+1←θkt−ηc∇ℒk(θkt;ℬk) _k^t+1← _k^t- _c _k ( _k^t;B_k ) 6: end for 7: return Ek,cmm=1|ℳk|\E_k,c^m\_m=1^|M_k| and Ik,cI_k,c A.2 Privacy Analysis Compared with federated learning paradigms that necessitate gradient or parameter sharing (Ouyang et al., 2023; Che et al., 2024; Liu et al., 2025), the proposed MFedPBA framework affords enhanced privacy preservation by constraining communication exclusively to feature and logit prototypes. Feature prototypes are synthesized via client-specific non-linear encoders whose parameters remain strictly local. Consequently, the inverse reconstruction of raw inputs from embedding vectors presents a mathematically ill-posed problem, thereby significantly impeding data reconstruction attacks. Concurrently, logit prototypes abstract model outputs into class-level decision statistics, revealing minimal information regarding individual data instances. This abstraction renders it arduous for adversaries to distinguish whether a prototype is dominated by a solitary sensitive sample or derived from a diverse aggregate, effectively attenuating the efficacy of inference attacks. Furthermore, the generation of both prototype modalities via class-wise averaging eliminates sample-level granularity, mitigating the risk of single-instance inference. Finally, the server-side prototype optimization introduces an additional transformation layer, ensuring that the global prototypes distributed to clients are not direct replicas of any single client’s representations, thus further reducing the probability of cross-client information leakage. Appendix B Convergence Analysis To analyze the convergence of MFedPBA, we follow (Tan et al., 2022a) and make the following assumptions B.1. As for the iteration notation system, we define t as the current communication round, e∈0,1,⋯,Ee∈\0,1,·s,E\ as the number of local iterations, where E denotes the maximum number of local iterations. Thus, (tE+e)(tE+e) represents the e-th iteration in the (t+1)(t+1)-th communication round. The (t+0)(t+0) denotes that at the beginning of the (t+1)(t+1)-th round. Note that (tE+E)(tE+E) corresponds to the last iteration in round (t+1)(t+1). Moreover, tEtE represents the time step before prototype aggregation, and tE+0tE+0 represents the time step between prototype learning and the first iteration of the current round. Therefore, we can decompose one round of communication into two stages: (1) Local update phase: [tE+0→(t+1)E][tE+0→(t+1)E] indicates that the client has completed the local update. (2) Server learning phase: [(t+1)E→(t+1)E+0][(t+1)E→(t+1)E+0] indicates that the server updates the prototype and distributes it. Here, we provide detailed mathematical expressions to better represent the process of updating local models. We split each client k’s model θk _k into a feature extractor fkmf_k^m parameterized by φk _k and a classifier hkh_k parameterized by wkw_k. Each client is equipped with |ℳk||M_k| modality specific feature extractors, denoted as fkm:ℳk→ℝdDf_k^m:X_M_k ^d_D. The client possesses a single classifier, denoted as hk:ℝdD→ℝdCh_k:R^d_D ^d_C. So the client loss function can be written as ℱk=fkm(φkm)m∘hk(wk)F_k=\f_k^m( _k^m)\_m h_k(w_k), and sometimes we use θk _k to represent (φkmm,wk)(\ _k^m\_m,w_k) for short. Therefore, the local loss function of client k can be written as: ℒ(θk,,y)=ℒce(ℱk(θk,),y)+λ1∑m‖fkm(φkm,m)−G,cm‖22+λ2‖hk(wk,fkmm)−G,c‖22. ( _k; x,y)=L_ce(F_k( _k; x),y)+ _1 _m||f_k^m( _k^m; x^m)-P_G,c^m||_2^2+ _2||h_k(w_k;f_k^m x^m)-I_G,c||_2^2. (21) It is worth noting that λ2 _2 serves as the weighting coefficient for the KL divergence term. Since the KL divergence can be approximated by the quadratic Mean Squared Error (MSE) loss via second-order Taylor expansion near the convergence point, we formulate the objective in Eq. (21) using the MSE form to facilitate the theoretical convergence analysis. Moreover, the global feature prototype G,cmP_G,c^m is obtained by the server via S rounds of training with the loss function ℒserverL_server, while the global logit prototype G,cI_G,c is aggregated at the server. Therefore, the server optimizes the global loss for S rounds can be written as: ℒserver()=ℒrec+ℒcon+ℒalign=1K∑k,m,c‖Pk,cm−Ek,cm‖2+ℒcon()+ℒalign(,).L_server(P)=L_rec+L_con+L_align= 1K _k,m,c\|P_k,c^m-E_k,c^m\|^2+L_con(P)+L_align(P,I). (22) For ease of analysis, we will represent both loss ℒconL_con and ℒalignL_align together as ℒmL_m, i.e., ℒm=ℒcon+ℒalignL_m=L_con+L_align. B.1 Assumption Assumption B.1 (Lipschitz Smoothness). The k-th client’s local model loss function ℒL is Lipschitz continuous with Lipschitz constant L, and L>0L>0 with ℒ(0)=0L(0)=0, i.e., ‖∇ℒt1−∇ℒt2‖2≤L‖θk,t1−θk,t2‖2,∀t1,t2>0,k∈1,2,…,K. \| _t_1- _t_2\|_2≤ L\| _k,t_1- _k,t_2\|_2, ∀ t_1,t_2>0, k∈\1,2,…,K\. (23) which implies the following quadratic bound, ℒt1−ℒt2≤⟨∇ℒt2,(θk,t1−θk,t2)⟩+L2‖θk,t1−θk,t2‖22,∀t1,t2>0,k∈1,2,…,K. _t_1-L_t_2≤ _t_2,( _k,t1- _k,t2) + L2|| _k,t1- _k,t2||_2^2, ∀ t_1,t_2>0, k∈\1,2,…,K\. (24) Assumption B.2 (Unbiased Gradient and Bounded Variance). The random gradient gk,t=∇ℒt(θk,t,ℬk,t)g_k,t= _t ( _k,t;B_k,t ) of each client’s local model is unbiased, where ℬB is a batch of local data, i.e., ℬk,t⊆Nk[gk,t]=∇ℒ(θk,t)=∇ℒt,∀k∈1,2,…,K,E_B_k,t N_k [g_k,t ]= ( _k,t)= _t, ∀ k∈\1,2,…,K\, (25) and the variance of random gradient gk,tg_k,t is bounded by: ℬk,t⊆Nk[‖∇ℒt(θk,t,ℬk,t)−∇ℒt(θk,t)‖22]≤σ2,∀k∈1,2,…,K.E_B_k,t N_k [ \| _t ( _k,t;B_k,t )- _t ( _k,t ) \|_2^2 ]≤σ^2, ∀ k∈\1,2,…,K\. (26) Assumption B.3 (Bounded Gradients). The expectation of the stochastic gradient is bounded by GcG_c: [||∇ℒk||2]≤Gc2,∀k∈1,2,…,K. [|| _k||^2 ]≤ G_c^2, ∀ k∈\1,2,…,K\. (27) Assumption B.4 (Lipschitz Continuity). For client k, feature extraction fkm(φkm)f_k^m( _k^m) and classifier hk(wk)h_k(w_k) are Lipschitz continuous with Lipschitz constant LrL_r, and Lr>0L_r>0: ‖fkm(φk,t1m)−fkm(φk,t2m)‖2 \|f_k^m( _k,t_1^m)-f_k^m( _k,t_2^m)\|_2 ≤Lr‖φk,t1m−φk,t2m‖2, ≤ L_r\| _k,t_1^m- _k,t_2^m\|_2, (28) ‖hk(wk,t1)−∇hk(wk,t2)‖2 \|h_k(w_k,t_1)-∇ h_k(w_k,t_2)\|_2 ≤Lr‖wk,t1−wk,t2‖2. ≤ L_r\|w_k,t_1-w_k,t_2\|_2. Assumption B.5 (Bounded Server Optimization Drift). The server objective function ℒserverL_server is LsL_s-smooth, and the expectation of the stochastic gradient is bounded, i.e., server Lipschitz Smoothness: ‖∇ℒserver,t1−∇ℒserver,t2‖2≤Ls‖θk,t1−θk,t2‖2\| _server,t_1- _server,t_2\|_2≤ L_s\| _k,t_1- _k,t_2\|_2, gradient bounded: [‖∇ℒm‖2]≤Gs2E [|| _m||^2 ]≤ G_s^2, and [‖∇ℒserver‖2]≤δs2E [|| _server||^2 ]≤ _s^2. B.2 Lemmas & Theorems Based on the above assumptions, MFedPBA uses a prototype-based approach for local training updates, Lemma B.6 derived by Tan et al., (Tan et al., 2022a) still holds. Lemma B.6. Let Assumption B.1 and B.2 hold. From the beginning of communication round t+1t+1 to the last local update step, the loss function of an arbitrary client can be bounded as: [ℒ(t+1)E]≤ℒtE+0−(ηc−Lηc22)∑e=0E‖∇ℒtE+e‖22+LEηc22σ2. splitE[L_(t+1)E] _tE+0-( _c- Lη^2_c2) _e=0^E\| _tE+e\|_2^2+ LE _c^22σ^2. split (29) Lemma B.7. Let Assumption B.3, B.4 and B.5 hold. After feature prototype learning and logit prototype aggregation are completed on the server, the loss function of any client can be constrained as follows: [ℒ(t+1)E+0]≤ℒ(t+1)E+(Mλ1+λ2)LrEηcGc+2λ1(δs+MGs). splitE[L_(t+1)E+0] _(t+1)E+(M _1+ _2)L_rE _cG_c+2 _1( _s+MG_s). split (30) Theorem B.8 (One-round deviation). Based on the above assumptions, using Γd=(Mλ1+λ2)LrGc _d=(M _1+ _2)L_rG_c and Γs=2λ1(δs+MGs) _s=2 _1( _s+MG_s) represent the drift constant and server bias constant. The expectation of the loss of an arbitrary client’s local model before the start of a round of local iteration satisfies: [ℒ(t+1)E+0]≤ℒtE+0−(ηc−Lηc22)∑e=0E‖∇ℒtE+e‖22+LEηc22σ2+ΓdEηc+Γs. splitE[L_(t+1)E+0]& _tE+0-( _c- L _c^22) _e=0^E\| _tE+e\|_2^2+ LE _c^22σ^2+ _dE _c+ _s. split (31) Theorem B.9 (Non-convex convergence rate of MFedPBA). Based on the above assumptions, for an arbitrary client and any ϵ>0ε>0, the following inequality holds: 1T∑t=0T−1∑e=0E[‖ℒtE+e‖22] 1T _t=0^T-1 _e=0^EE [ \|L_tE+e \|_2^2 ] ≤2(ℒt=1−ℒ∗)Tηc(2−Lηc)+LEηc2σ2+2(ΓdEηc+Γs)2ηc−Lηc2, ≤ 2 (L_t=1-L^* )T _c (2-L _c )+ LE _c^2σ^2+2( _dE _c+ _s)2 _c-L _c^2, (32) ≤ϵ. ≤ε. s.t.ηc s.t. _c <min2L,(ϵ−ΓdE)+(ϵ−ΓdE)2−2L(ϵ+Eσ2)ΓsL(ϵ+Eσ2). < \ 2L, (ε- _dE)+ (ε- _dE)^2-2L(ε+Eσ^2) _sL(ε+Eσ^2) \. B.3 Proof of Lemma 5.1 Lemma 5.1. Let Assumption B.1 and B.2 hold. From the beginning of communication round t+1t+1 to the last local update step, the loss function of an arbitrary client can be bounded as: [ℒ(t+1)E]≤ℒtE+0−(ηc−Lηc22)∑e=0E‖∇ℒtE+e‖22+LEηc22σ2. splitE[L_(t+1)E] _tE+0-( _c- Lη^2_c2) _e=0^E\| _tE+e\|_2^2+ LE _c^22σ^2. split Proof. For arbitrary clients, we have θt+1=θt−ηcgt _t+1= _t- _cg_t, then ℒtE+1≤ℒtE+0+⟨∇ℒtE+0,(θtE+1−θtE+0)⟩+L2‖θtE+1−θtE+0‖22=ℒtE+0−ηc⟨∇ℒtE+0,gtE+0⟩+L2‖ηcgtE+0‖22. splitL_tE+1& _tE+0+ _tE+0,( _tE+1- _tE+0) + L2\| _tE+1- _tE+0\|_2^2\\ &=L_tE+0- _c _tE+0,g_tE+0 + L2\| _cg_tE+0\|_2^2. split (33) Taking expectation of both sides of the above equation on the random variable ℬB, we have [ℒtE+1] E[L_tE+1] ≤ℒtE+0−ηc[⟨∇ℒtE+0,gtE+0⟩]+Lηc22[‖gtE+0‖22] _tE+0- _cE[ _tE+0,g_tE+0 ]+ L _c^22E[\|g_tE+0\|_2^2] (34) =ℒtE+0−ηc‖∇ℒtE+0‖22+Lηc22[‖gk,tE+0‖22] =L_tE+0- _c\| _tE+0\|_2^2+ L _c^22E[\|g_k,tE+0\|_2^2] ≤ℒtE+0−ηc‖∇ℒtE+0‖22+Lηc22(‖∇ℒtE+0‖22+Var(gk,tE+0)) ≤L_tE+0- _c\| _tE+0\|_2^2+ L _c^22(\| _tE+0\|_2^2+Var(g_k,tE+0)) =ℒtE+0−(ηc−Lηc22)‖∇ℒtE+0‖22+Lηc22Var(gk,tE+0) =L_tE+0-( _c- L _c^22)\| _tE+0\|_2^2+ L _c^22Var(g_k,tE+0) ≤ℒtE+0−(ηc−Lηc22)‖∇ℒtE+0‖22+Lηc22σ2, ≤L_tE+0-( _c- L _c^22)\| _tE+0\|_2^2+ L _c^22σ^2, where Var(x)=[x2]−([x])2Var(x)=E[x^2]-(E[x])^2. Take expectation of θ on both sides. Then, by telescoping of E steps, we have, [ℒ(t+1)E]≤ℒtE+0−(ηc−Lηc22)∑e=0E‖∇ℒtE+e‖22+LEηc22σ2, splitE[L_(t+1)E] _tE+0-( _c- L _c^22) _e=0^E\| _tE+e\|_2^2+ LE _c^22σ^2, split (35) which completes the proof. ∎ B.4 Proof of Lemma 5.2 Lemma 5.2. Let Assumption B.3, B.4 and B.5 hold. After feature prototype learning and logit prototype aggregation are completed on the server, the loss function of any client can be constrained as follows: [ℒ(t+1)E+0]≤ℒ(t+1)E+(Mλ1+λ2)LrEηcGc+2λ1(δs+MGs). splitE[L_(t+1)E+0] _(t+1)E+(M _1+ _2)L_rE _cG_c+2 _1( _s+MG_s). split Proof. We define the temporal states of a client. Time (t+1)E(t+1)E denotes the moment when the client has just completed E local training epochs in the t-th round but has not yet received the new prototypes from the server. Time (t+1)E+0(t+1)E+0 denotes the moment when the client receives the updated prototypes dispatched by the server. First, we have: ℒ(t+1)E+0=ℒ(t+1)E+ℒ(t+1)E+0−ℒ(t+1)EL_(t+1)E+0=L_(t+1)E+L_(t+1)E+0-L_(t+1)E. Therefore, our main goal is to derive an upper bound for the difference in loss before and after aggregation: Δℒ=[ℒ(t+1)E+0−ℒ(t+1)E]. split =E[L_(t+1)E+0-L_(t+1)E]. split (36) Since the model parameter θ does not change during aggregation, the cross-entropy loss ℒceL_ce cancels out. The difference only comes from the changes in the global prototypes in the regularization term. Thus we have: ℒ(t+1)E+0−ℒ(t+1)E=λ1∑m(‖fkm(ϕk,(t+1)Em)−G,t+2m‖−‖fkm(ϕk,(t+1)Em)−G,t+1m‖)+λ2(‖hk(wk,(t+1)E)−G,t+2‖−‖hk(wk,(t+1)E)−G,t+1‖). splitL_(t+1)E+0-L_(t+1)E=& _1 _m (||f_k^m(φ^m_k,(t+1)E)-P^m_G,t+2||-||f_k^m(φ^m_k,(t+1)E)-P^m_G,t+1|| )\\ &+ _2 (||h_k(w_k,(t+1)E)-I_G,t+2||-||h_k(w_k,(t+1)E)-I_G,t+1|| ). split (37) Using the reverse triangle inequality ‖a−b‖2−‖a−c‖2≤‖b−c‖2\|a-b\|_2-\|a-c\|_2≤\|b-c\|_2, we can bound the above expression as follows: ℒ(t+1)E+0−ℒ(t+1)E=λ1∑m‖G,t+2m−G,t+1m‖⏟A+λ2‖G,t+2−G,t+1‖⏟B. splitL_(t+1)E+0-L_(t+1)E= _1 _m\|P^m_G,t+2-P^m_G,t+1\|_A+ _2\|I_G,t+2-I_G,t+1\|_B. split (38) Next, we define the changes in Part A and Part B separately. Since the logical value prototype of Part B uses weighted aggregation, we will begin our analysis with Part B. From Eq. (6), we can see that: G,t+2=∑k=1KqkIk,(t+1)E,G,t+1=∑k=1KqkIk,tE. splitI_G,t+2= _k=1^Kq_kI_k,(t+1)E, _G,t+1= _k=1^Kq_kI_k,tE. split (39) Therefore, we have: ‖G,t+2−G,t+1‖=‖∑k=1KqkIk,(t+1)E−∑k=1KqkIk,tE‖2=‖∑k=1Kqk(Ik,(t+1)E−Ik,tE)‖2≤‖∑k=1Kqk1Nk∑i=1Nk(hk(wk,(t+1)E,k,i)−hk(wk,tE,k,i))‖2≤∑k=1Kqk1Nk∑i=1Nk‖hk(wk,(t+1)E,k,i)−hk(wk,t+E,k,i)‖2≤Lr∑k=1Kqk‖wk,(t+1)E−wk,tE‖2≤Lr∑k=1Kqk‖θk,(t+1)E−θk,tE‖2=Lrηc∑k=1Kqk‖∑e=0E−1gk,tE+e‖2≤Lrηc∑k=1Kqk∑e=0E−1‖gk,tE+e‖2. split\|I_G,t+2-I_G,t+1\|&= \| _k=1^Kq_kI_k,(t+1)E- _k=1^Kq_kI_k,tE \|_2= \| _k=1^Kq_k(I_k,(t+1)E-I_k,tE) \|_2\\ &≤ \| _k=1^Kq_k 1N_k _i=1^N_k(h_k(w_k,(t+1)E; x_k,i)-h_k(w_k,tE; x_k,i)) \|_2\\ &≤ _k=1^Kq_k 1N_k _i=1^N_k \|h_k(w_k,(t+1)E; x_k,i)-h_k(w_k,t+E; x_k,i) \|_2\\ &≤ L_r _k=1^Kq_k \|w_k,(t+1)E-w_k,tE \|_2\\ &≤ L_r _k=1^Kq_k \| _k,(t+1)E- _k,tE \|_2=L_r _c _k=1^Kq_k \| _e=0^E-1g_k,tE+e \|_2\\ &≤ L_r _c _k=1^Kq_k _e=0^E-1 \|g_k,tE+e \|_2. split (40) Take expectations on both sides, since ∑qk=1Σq_k=1, then: λ2‖G,t+2−G,t+1‖≤λ2Lrηc∑k=1Kqk∑e=0E−1‖gk,tE+e‖2≤λ2LrEηcGc. split _2\|I_G,t+2-I_G,t+1\|≤ _2L_r _c _k=1^Kq_k _e=0^E-1 \|g_k,tE+e \|_2≤ _2L_rE _cG_c. split (41) The derivation for Part A is complete. Since the global feature prototype is aggregated using non-convex SGD, the server prototype GmP^m_G is the result of the client uploading and optimizing EkmE_k^m for S rounds. Based on Assumption B.5, we introduce the aggregated mean as an intermediate variable. Let t+2=∑qkEk,t+2E_t+2=Σ q_kE_k,t+2 be the geometric center of the currently uploaded feature prototypes, and t+1=∑qkEk,t+1E_t+1=Σ q_kE_k,t+1 be the geometric center of the features uploaded in the previous round. Here, we omit the modality superscript m for ease of analysis. Therefore, for part A, we have: ‖G,t+2−G,t+1‖ \|P_G,t+2-P_G,t+1\| =‖G,t+2−t+2+t+2−t+1+t+1−G,t+1‖ =\|P_G,t+2-E_t+2+E_t+2-E_t+1+E_t+1-P_G,t+1\| (42) ≤‖G,t+2−t+2‖⏟A1+‖t+2−t+1‖⏟A2+‖t+1−G,t+1‖⏟A3. ≤ \|P_G,t+2-E_t+2\|_A1+ \|E_t+2-E_t+1\|_A2+ \|E_t+1-P_G,t+1\|_A3. We found that Part A2 is completely consistent with the derivation of the logical value prototype, and the change in the mean is limited by the drift of the client’s local parameters. Therefore, we have: ‖t+2−t+1‖≤LrEηcGc.\|E_t+2-E_t+1\|≤ L_rE _cG_c. (43) By solving for the gradient of Eq. (22), we obtain: ∇ℒserver(Gm) _server(P^m_G) =12K∑k=1K∑m=1M2(Pkm−Ekm)+∑m=1M∇ℒm(Gm) = 12KΣ^K_k=1Σ^M_m=12(P^m_k-E^m_k)+Σ^M_m=1 _m(P^m_G) (44) =M(G−avg)+M∇ℒm(mG). =M(P_G-E_avg)+M _m(P^m_G). Through algebraic manipulation, we obtain: G−avg=1M∇ℒserver(mG)−∇ℒm(mG). _G-E_avg= 1M _server(P^m_G)- _m(P^m_G). (45) By applying the L2-norm and the triangle inequality to the above expression, we obtain: ∥G−avg∥2≤∥1M∇ℒserver(mG)∥2+∥∇ℒm(mG)∥2. \|P_G-E_avg\|_2≤\| 1M _server(P^m_G)\|_2+\| _m(P^m_G)\|_2. (46) According to assumption B.5, therefore we have: [‖G−avg‖2] E[\|P_G-E_avg\|_2] ≤[∥1M∇ℒserver(mG)∥2]+[∥∇ℒm(mG)∥2] ≤E[\| 1M _server(P^m_G)\|_2]+E[\| _m(P^m_G)\|_2] (47) ≤δsM+Gs. ≤ _sM+G_s. Accordingly, the derivation process for A1 and A3 is the same as described above. Therefore, considering that there are M modes in total, we can integrate the above process into Eq. (42) to obtain: ‖G,t+2−G,t+1‖ \|P_G,t+2-P_G,t+1\| ≤‖G,t+2−t+2‖⏟A1+‖t+2−t+1‖⏟A2+‖t+1−G,t+1‖⏟A3. ≤ \|P_G,t+2-E_t+2\|_A1+ \|E_t+2-E_t+1\|_A2+ \|E_t+1-P_G,t+1\|_A3. (48) ≤2M(δsM+Gs)+MLrEηcGc=2δs+2MGs+MLrEηcGc. ≤ 2M( _sM+G_s)+ML_rE _cG_c=2 _s+2MG_s+ML_rE _cG_c. Based on the derivation results from Eq. (48) and Eq. (41), substituting them into Eq. (38), and taking expectation of both sides: [ℒ(t+1)E+0]−[ℒ(t+1)E]≤λ1(2δs+2MGs+MLrEηcGc)+λ2LrEηcGc=(Mλ1+λ2)LrEηcGc+2λ1(δs+MGs), splitE[L_(t+1)E+0]-E[L_(t+1)E]&≤ _1(2 _s+2MG_s+ML_rE _cG_c)+ _2L_rE _cG_c\\ &=(M _1+ _2)L_rE _cG_c+2 _1( _s+MG_s), split (49) which completes the proof. ∎ B.5 Proof of Theorem 5.3 Theorem 5.3 Based on the above assumptions, using Γd=(Mλ1+λ2)LrGc _d=(M _1+ _2)L_rG_c and Γs=2λ1(δs+MGs) _s=2 _1( _s+MG_s) represent the drift constant and server bias constant. The expectation of the loss of an arbitrary client’s local model before the start of a round of local iteration satisfies: [ℒ(t+1)E+0]≤ℒtE+0−(ηc−Lηc22)∑e=0E‖∇ℒtE+e‖22+LEηc22σ2+ΓdEηc+Γs. splitE[L_(t+1)E+0]& _tE+0-( _c- L _c^22) _e=0^E\| _tE+e\|_2^2+ LE _c^22σ^2\\ & + _dE _c+ _s. split Proof. According to the expectation equation, the change in one communication round consists of two phases: the server-side phase and the client-side phase. The total change can be expressed as: [ℒ(t+1)E+0]−ℒtE+0≤([ℒ(t+1)E+0]−[ℒ(t+1)E])⏟server+([ℒ(t+1)E]−[ℒtE+0])⏟client. splitE[L_(t+1)E+0]-L_tE+0≤ (E[L_(t+1)E+0]-E[L_(t+1)E] )_server+ (E[L_(t+1)E]-E[L_tE+0] )_client. split (50) Substituting Lemma B.6 into the second term on the right-hand side of Lemma B.7, we have: [ℒ(t+1)E+0]≤[ℒ(t+1)E]+(Mλ1+λ2)LrEηcGc+2λ1(δs+MGs)≤ℒtE+0−(ηc−Lηc22)∑e=0E‖∇ℒtE+e‖22+LEηc22σ2+(Mλ1+λ2)LrEηcGc+2λ1(δs+MGs). splitE[L_(t+1)E+0]& [L_(t+1)E]+(M _1+ _2)L_rE _cG_c+2 _1( _s+MG_s)\\ & _tE+0-( _c- L _c^22) _e=0^E\| _tE+e\|_2^2+ LE _c^22σ^2\\ & +(M _1+ _2)L_rE _cG_c+2 _1( _s+MG_s). split (51) To simplify the notation, we let Γd=(Mλ1+λ2)LrGc _d=(M _1+ _2)L_rG_c represent the drift constant, and Γs=2λ1(δs+MGs) _s=2 _1( _s+MG_s) represent the server bias constant, then we have [ℒ(t+1)E+0]≤ℒtE+0−(ηc−Lηc22)∑e=0E‖∇ℒtE+e‖22+LEηc22σ2+ΓdEηc+Γs, splitE[L_(t+1)E+0]& _tE+0-( _c- L _c^22) _e=0^E\| _tE+e\|_2^2+ LE _c^22σ^2\\ & + _dE _c+ _s, split (52) which completes the proof. ∎ B.6 Proof of Theorem 5.4 Theorem 5.4 Based on the above assumptions, for an arbitrary client and any ϵ>0ε>0, the following inequality holds: 1T∑t=0T−1∑e=0E[‖ℒtE+e‖22] 1T _t=0^T-1 _e=0^EE [ \|L_tE+e \|_2^2 ] ≤2(ℒt=1−ℒ∗)Tηc(2−Lηc)+LEηc2σ2+2(ΓdEηc+Γs)2ηc−Lηc2 ≤ 2 (L_t=1-L^* )T _c (2-L _c )+ LE _c^2σ^2+2( _dE _c+ _s)2 _c-L _c^2 ≤ϵ. ≤ε. s.t.ηc s.t. _c <min2L,(ϵ−ΓdE)+(ϵ−ΓdE)2−2L(ϵ+Eσ2)ΓsL(ϵ+Eσ2). < \ 2L, (ε- _dE)+ (ε- _dE)^2-2L(ε+Eσ^2) _sL(ε+Eσ^2) \. Proof. Transform the form of Theorem B.8 into ∑e=0E‖ℒtE+e‖22≤ℒtE+0−[ℒ(t+1)E+0]+LEηc22σ2+ΓdEηc+Γsηc−Lηc22. _e=0^E \|L_tE+e \|_2^2≤ L_tE+0-E [L_(t+1)E+0 ]+ LE _c^22σ^2+ _dE _c+ _s _c- L _c^22. (53) Take expectations of model θ on both sides, we have: ∑e=0E[‖ℒtE+e‖22]≤[ℒtE+0]−[ℒ(t+1)E+0]+LEηc22σ2+ΓdEηc+Γsηc−Lηc22. _e=0^EE [ \|L_tE+e \|_2^2 ]≤ E [L_tE+0 ]-E [L_(t+1)E+0 ]+ LE _c^22σ^2+ _dE _c+ _s _c- L _c^22. (54) Summing both sides of Eq. (54) over T rounds, since ∑t=1T([ℒtE+0]−[ℒ(t+1)E+0])≤ℒt=0−ℒ∗ _t=1^T (E [L_tE+0 ]-E [L_(t+1)E+0 ] ) _t=0-L^*, for each round: 1T∑t=0T−1∑e=0E[‖ℒtE+e‖22] 1T _t=0^T-1 _e=0^EE [ \|L_tE+e \|_2^2 ] ≤1T∑t=0T−1([ℒtE+0]−[ℒ(t+1)E+0])+LEηc22σ2+ΓdEηc+Γsηc−Kηc22 ≤ 1T _t=0^T-1 (E [L_tE+0 ]-E [L_(t+1)E+0 ] )+ LE _c^22σ^2+ _dE _c+ _s _c- K _c^22 (55) ≤1T(ℒt=0−ℒ∗)+LEηc22σ2+ΓdEηc+Γsηc−Lηc22 ≤ 1T (L_t=0-L^* )+ LE _c^22σ^2+ _dE _c+ _s _c- L _c^22 =2(ℒt=0−ℒ∗)+TLEηc2σ2+2T(ΓdEηc+Γs)T(2ηc−Lηc2) = 2 (L_t=0-L^* )+TLE _c^2σ^2+2T( _dE _c+ _s)T (2 _c-L _c^2 ) =2(ℒt=0−ℒ∗)Tηc(2−Lηc)+LEηc2σ2+2(ΓdEηc+Γs)2ηc−Lηc2. = 2 (L_t=0-L^* )T _c (2-L _c )+ LE _c^2σ^2+2( _dE _c+ _s)2 _c-L _c^2. Given any ϵ>0ε>0 the above equation satisfies 2(ℒt=0−ℒ∗)Tηc(2−Lηc)+LEηc2σ2+2(ΓdEηc+Γs)2ηc−Lηc2≤ϵ. 2 (L_t=0-L^* )T _c (2-L _c )+ LE _c^2σ^2+2( _dE _c+ _s)2 _c-L _c^2≤ε. (56) Then, we can obtain: T≥2(ℒt=0−ℒ∗)ηcϵ(2−Lηc)−ηcE(Lηcσ2+2Γdηc)−2Γs.T≥ 2 (L_t=0-L^* ) _cε (2-L _c )- _cE(L _cσ^2+2 _d _c)-2 _s. (57) Since T>0,ℒt=0−ℒ∗>0T>0,L_t=0-L^*>0, we can further derive: ηcϵ(2−Lηc)−ηcE(Lηcσ2+2Γdηc)−2Γs>0, _cε (2-L _c )- _cE(L _cσ^2+2 _d _c)-2 _s>0, (58) Since Eq. (58) is a quadratic function that opens upwards, we solve it to obtain: ηc<min2L,(ϵ−ΓdE)+(ϵ−ΓdE)2−2L(ϵ+Eσ2)ΓsL(ϵ+Eσ2), _c< \ 2L, (ε- _dE)+ (ε- _dE)^2-2L(ε+Eσ^2) _sL(ε+Eσ^2) \, (59) When Γs _s approaches 0, Eq. (59) can be simplified to: ηc<min2L,2(ϵ−ΓdE)L(ϵ+Eσ2), _c< \ 2L, 2(ε- _dE)L(ε+Eσ^2) \, (60) which completes the proof. ∎ Appendix C Details of the Experimental Setup C.1 Dataset Description This paper focuses on multimodal federated learning under model heterogeneity and imbalanced modality distributions. To evaluate performance in modality-imbalanced scenarios, we select multiple datasets with diverse modalities and classes and construct distributed federated settings, denoted as datasets Caltech101 55 5 https://data.caltech.edu/records/mzrjq-6wc02., Reuters 66 6 https://w.kaggle.com/datasets/nltkdata/reuters., NUS-WIDE 77 7 https://w.kaggle.com/datasets/xinleili/nuswide., and Youtube 88 8 http://archive.ics.uci.edu/ml/datasets.. Detailed descriptions of each dataset are provided below. The Caltech101 dataset is a widely used benchmark dataset introduced by the California Institute of Technology in 2003 and has been extensively adopted in computer vision, pattern recognition, and machine learning research. The dataset is designed to provide a well-annotated and low-noise image collection for object recognition and image classification tasks. It consists of 102 categories, including 101 object classes and one background category (BACKGROUND_Google). To fully exploit the visual information in Caltech101, we adopt a multimodal feature extraction strategy that characterizes images from complementary perspectives. Specifically, we consider the following three modalities: • I1 (Histogram of Oriented Gradients): A 1984-dimensional Histogram of Oriented Gradients feature, which captures local structural and edge information. Images are partitioned into connected regions, within which gradient orientations are computed and aggregated into histograms. The regional histograms are then concatenated and block-normalized to improve robustness to illumination variations and local deformations. • I2 (Gist Descriptor): A 512-dimensional Gabor feature designed to model texture and frequency responses. A bank of multi-scale and multi-orientation Gabor filters is applied to each image, followed by average pooling over a 4×4 spatial grid. The pooled responses from all filters and grid cells are concatenated to form the final representation. • I3 (Local Binary Pattern): A 928-dimensional Local Binary Pattern feature that encodes local texture patterns. For each pixel, binary codes are generated by comparing the center pixel with its neighbors and converted into decimal values. The global image representation is obtained by computing the histogram of LBP codes over the entire image. The Reuters dataset is a widely adopted benchmark in multi-view learning research. It comprises 18,758 news articles collected by Reuters in 1987, which are categorized into six distinct topics. While the original corpus consists solely of English text, the multi-view variant augments these documents with translations into four additional European languages: French, German, Italian, and Spanish. Consequently, each sample is represented by five distinct views corresponding to these languages. Regarding feature representation, the documents are encoded as Bag-of-Words (BoW) vectors weighted by TF-IDF. The specific feature dimensionality for each view is as follows: 21,531 for English, 24,892 for French, 34,251 for German, 15,506 for Italian, and 11,547 for Spanish. The NUS-WIDE dataset, released by the National University of Singapore, is a large-scale multi-label image dataset comprising 8 distinct categories. It stands as a milestone benchmark in the fields of web image retrieval, multimodal learning, and semantic annotation research. The dataset provides both visual content and associated textual metadata, including user-contributed tags, titles, and descriptions from Flickr, enabling comprehensive cross-modal analysis. In this work, we extract four complementary feature modalities from NUS-WIDE, each capturing distinct aspects of visual and structural information: • I1 (Cube Color Histogram): A 64-dimensional color histogram computed in a perceptually robust color space. Images are first transformed from the RGB color space to a color space less sensitive to illumination variations, and color distributions are then quantified to obtain global color representations. • I2 (Color Correlation Map): A 144-dimensional color correlogram that captures both color statistics and their spatial correlations. This representation models not only the occurrence probability of individual colors but also the co-occurrence of identical or different colors at predefined spatial distances. • I3 (Block-wise Color Moments, BOC): A 225-dimensional block-wise color moments feature that preserves spatial layout information. Each image is uniformly partitioned into a 5×5 grid, and low-order color moments are computed within each block across multiple color channels. The resulting block-level features are concatenated to form the final representation. • L (Bag-of-Visual-Words): A 500-dimensional bag-of-visual-words representation that treats an image as a document composed of visual words. Local keypoints are first detected and described using local feature descriptors, which are then clustered to construct a visual vocabulary. Each image is subsequently encoded as a histogram over the learned visual words. The YouTube dataset comprises a large collection of short video clips sourced from the YouTube platform, organized around specific events or actions and spanning 10 distinct categories. As a naturally multimodal data source, it provides synchronized visual, auditory, and temporal information. In this study, we extract six complementary feature modalities to capture diverse aspects of the video content: • I1 (Spatiotemporal Cuboids): A 2,000-dimensional spatiotemporal interest point descriptor that captures local regions exhibiting significant variations across both spatial and temporal dimensions. This representation models local motion patterns and appearance changes using three-dimensional spatiotemporal cuboids, enabling effective characterization of dynamic visual structures. • I2 (Optical Flow Histogram): A 1,024-dimensional temporally pooled histogram of oriented gradients representation that encodes global shape and edge information. HOG features are extracted frame-wise and aggregated via temporal pooling operations to obtain a compact global descriptor while preserving spatial structural cues. • T (Frame-level HOG): A 64-dimensional global optical flow histogram that summarizes pixel-level motion statistics. Optical flow fields are computed between consecutive frames, and the distributions of flow magnitudes and orientations are statistically modeled to form a concise representation of overall motion dynamics. • A1 (Mel-Frequency Features): A 512-dimensional Mel-frequency cepstral coefficient representation that characterizes the spectral properties of audio signals. The audio stream is segmented into frames and processed through windowing, short-time Fourier transform, Mel-scale filtering, and discrete cosine transform, yielding perceptually robust acoustic features. • A2 (Volume Streams): A 64-dimensional energy-based descriptor that captures variations in audio intensity over time. Frame-level energy statistics are computed using squared amplitude or root mean square measures and aggregated to reflect global loudness dynamics. • A3 (Spectrogram-based Features): A 647-dimensional spectrogram-based statistical representation that models the joint time–frequency distribution of the audio signal. Spectrograms are generated via short-time Fourier transform and partitioned along both temporal and frequency axes, with energy statistics aggregated within each region to provide a hierarchical description of audio structure. To accurately simulate complex data distribution scenarios in heterogeneous multimodal federated learning, this study designs three client modality distribution configurations. The “M2” setup refers to each client randomly selecting two modalities from the complete modality set to form its local dataset; the “M1+” setup requires each client to possess at least one modality, while allowing variations in modality composition across different clients; and the “M1” setup restricts each client to holding only a single modality. The number of clients is equal to the number of modalities. In Figures 9, 10, and 11, we visually present the client modality distribution across four datasets. In these visualizations, each circle represents a client, with the circle’s area proportional to the sample size of the client’s data for the corresponding modality, intuitively illustrating the diversity of modality distribution and the imbalance in data volume under heterogeneous settings. To evaluate the performance of the algorithm in larger-scale client environments, we scale the number of clients for the four datasets as follows: dataset Caltech101 to 50 clients, dataset Reuters to 100 clients, and datasets NUS-WIDE and YouTube to 20 clients each. The experiments adopt the ”M1+” modality configuration, with a Dirichlet distribution (concentration parameter α = 0.5) used to partition the data among clients in a non-IID manner. The resulting client data distributions for the scaled scenarios are visualized in Figure 12. (a) Caltech101 (b) Reuters (c) NUS-WIDE (d) Youtube Figure 9: The distribution of these datasets is set in the M2 scenario. A larger circle means a larger sample size. (a) Caltech101 (b) Reuters (c) NUS-WIDE (d) Youtube Figure 10: The distribution of these datasets is set in the M1+ scenario. A larger circle means a larger sample size. (a) Caltech101 (b) Reuters (c) NUS-WIDE (d) Youtube Figure 11: The distribution of these datasets is set in the M1 scenario. A larger circle means a larger sample size. Figure 12: In scenarios involving a larger number of clients, the data distribution across these four datasets within the M1+ scenario. C.2 Heterogeneous Model. After the modality-specific feature preprocessing described in Section C.1, we further assign heterogeneous feature extractors to different modalities in order to obtain modality-specific embeddings. Specifically, the model architecture for each modality is stochastically sampled from a candidate pool, as listed in Table 4, thereby simulating realistic model heterogeneity in federated learning. To further enforce architectural diversity, the depth (number of repeating blocks) and width (hidden dimensions) of the networks are also randomized. Table 4 details the sequential structure of these extractors. The symbol “*” denotes the number of repeated modules, which takes integer values in the range of [2,5][2,5] in our experiments. The hidden dimension of each model is set as a multiple of 32 and is kept constant or gradually reduced as the number of modules increases, balancing representational capacity and model complexity. This randomized model assignment strategy enables a controlled yet realistic evaluation of multimodal federated learning under heterogeneous architectures. Table 4: Heterogeneous model architectures for modality-specific feature extraction. Model Sequentially Connected Feature Extractors Applicable Modality Types DNN [input_dim, *(hidden_dim, Relu), Classifier] Image, Language, Audio, Time-series CNN1D [*(Conv1d, Relu, AdaptiveMaxPool1d), *(hidden_dim, Relu), Classifier] Language, Audio, Time-series CNN2D [*(Conv2d, Relu), *(hidden_dim, Relu), Classifier] Image TextCNN [*(Conv1d, Relu, AdaptiveMaxPool1d), Concat, *(hidden_dim, Relu), Classifier] Language Resmodel [input_dim, *ResBlock(hidden_dim,BatchNorm1d, Relu), Classifier] Audio, Time-series C.3 Baseline Methods To ensure fair comparison, we re-implemented all baseline methods within a unified HtFLlib framework (Zhang et al., 2025a) and evaluated them under identical experimental settings. Specifically, we include representative prototype-based federated learning methods FedProto (Tan et al., 2022a), FedTGP (Zhang et al., 2024), and FedPall (Zhang et al., 2025c), as well as state-of-the-art multimodal federated learning approaches Harmony (Ouyang et al., 2023), FedMVP (Che et al., 2024), and FedMobile (Liu et al., 2025), with local training (Local) serving as a reference baseline. Prior to the main experiments, we conducted a preliminary investigation into local multimodal fusion strategies, comparing feature summation (“sum”) and feature concatenation (“concat”). Results indicated that feature summation (“sum”) yields better performance. Therefore, during local training, for Local, the prototype-based methods, and our proposed MFedPBA, when a client possesses multimodal data, the features extracted by heterogeneous encoders are summed before being sent to the server. In contrast, the multimodal federated learning methods Harmony, FedMVP, and FedMobile follow their original local learning strategies as described in their respective papers. Furthermore, we note that no publicly available model-heterogeneous multimodal federated learning methods are directly applicable for comparative evaluation. To establish a reasonable heterogeneous experimental environment, we selected state-of-the-art homogeneous multimodal federated learning methods as baselines and adapted their communication mechanisms by modifying the uploaded parameters from full local models to classifier parameters only, thereby accommodating model-heterogeneous federated learning scenarios. C.4 Parameter Setting Details The parameters of our method are detailed in Table 5. We ran these experiments on 2 NVIDIA GeForce RTX 4090 GPUs with AMD Ryzen 9 9950X 16-Core Processor (32 threads) for all methods. Table 5: List of Hyperparameters. Descriptions Caltech101 Reuters NUS-WIDE Youtube Total rounds T 400 / 500 200 400 400 Server Server optimizers SGD Server learning rate ηs _s 0.005∼ 0.01 Server training epoch S 10 Client Local training epoch E 2 Training batch size ℬB 12 Local optimizers SGD Weighting parameter λ λ1∈0.01,0.01,0.1 _1∈\0.01,0.01,0.1\, λ2∈0.5,1,5 _2∈\0.5,1,5\ Local learning rate ηc _c 0.005∼ 0.01 0.001∼ 0.005 0.01 0.005∼ 0.01 Feature embedding dimension dDd_D 64 24 32 48 Appendix D Details of the Experimental Result In the main text, we report the test accuracy of each method across four multimodal datasets under three data partitioning schemes: “M2”, “M1+”, and “M1”. In the appendix, we provide detailed client-level experimental results and performance curves for each data partitioning scheme. Regarding the evaluation metrics, the accuracy for an individual client k is defined as the ratio of correctly classified samples (Nk,correctN_k,correct) to the total number of samples on that client i.e., acck=Nk,correctNktestacc_k= N_k,correctN_k^test. To account for the influence of data volume, the aggregated accuracy (Avg) reported in the tables is calculated as the arithmetic mean of client accuracies, expressed as: Avg=∑k=1KNk,correct∑k=1KNktestAvg= _k=1^KN_k,correct _k=1^KN_k^test. The organization of the experimental results corresponding to each setting is as follows: Under the M2 partitioning scheme, the data partition is illustrated in Figure 9. The corresponding test results and performance curves are presented in Table 6 and Figure 13, respectively. Similarly, results for the M1+ scheme are presented in Figure 10, Table 7, and Figure 14, while those for the M1 scheme are provided in Figure 11, Table 8, and Figure 15, respectively. Furthermore, we extend the evaluation to a larger client pool to assess the scalability of the proposed framework. Based on the dataset size, Dataset Caltech101 is distributed across 50 clients, Dataset Reuters is scaled to 100 clients, and Datasets NUS-WIDE and Youtube are partitioned into 20 clients each, following the M1+ partitioning scheme (as illustrated in Figure 12). The models are trained until stable convergence is achieved. The detailed experimental configurations are summarized in the row labeled “K#” in Table 2, and the resulting performance curves are depicted in Figure 16. The experimental results substantiate that MFedPBA consistently maintains superior performance and resilience, even within large-scale federated learning scenarios. Table 6: Performance comparison (%) of all compared methods on Caltech101, Reuters, NUS-WIDE, and Youtube using M2 data partitioning, where the number of clients K is equal to the number of modalities M. The k1k1 represents the client with ID 1. Dataset Local FedProto FedTGP FedPall Harmony FedMVP FedMobile MFedPBA Caltech101 k1 40.48±0.86 37.96±0.75 39.71±1.14 39.81±0.96 42.14±0.17 35.01±0.96 41.89±1.07 47.35±0.19 k2 43.28±0.42 38.56±0.21 39.86±0.94 44.18±0.16 46.44±0.15 41.28±0.42 44.72±0.28 48.19±0.18 k3 44.24±0.46 39.89±0.15 43.52±1.65 43.3±0.15 47.23±0.23 44.8±0.49 43.52±0.14 52.44±0.27 Avg 42.65±1.47 38.72±0.42 40.78±1.31 42.29±0.27 45.29±0.38 40.23±0.65 43.51±0.54 49.04±0.21 Reuters k1 77.22±1.58 82.06±0.32 78.63±0.78 76.92±0.62 76.94±0.91 82.53±1.67 80.8±1.07 82.55±0.79 k2 66.82±1.42 64.49±0.65 62.23±1.46 66.88±0.61 68.12±1.05 65.62±1.11 68.11±0.38 67.83±0.78 k3 76.69±0.98 79.83±1.32 73.33±1.47 79.87±0.23 79.43±2.11 82.44±0.61 79.46±1.48 79.66±1.21 k4 69.86±1.09 73.71±0.51 70.78±0.82 73.77±0.39 69.14±1.06 71.52±1.67 70.87±1.17 75.71±0.72 k5 65.84±1.05 71.12±0.46 66.76±1.34 71.11±0.33 71.48±1.21 73.78±0.58 70.97±1.66 73.29±0.87 Avg 70.6±0.28 73.39±0.57 69.73±0.88 73.08±1.04 72.18±0.41 74.07±1.08 73.19±0.26 75.16±0.45 NUS-WIDE k1 35.28±0.53 32.36±0.59 35.28±0.17 38.97±1.77 38.43±0.95 37.57±0.98 39.26±1.03 38.91±0.78 k2 35.76±1.54 32.46±0.66 34.26±0.48 34.72±0.83 35.68±1.37 32.74±0.46 36.47±1.00 39.03±1.00 k3 29.52±0.83 26.86±0.12 28.42±1.02 29.9±0.18 28.9±0.84 30.08±0.57 30.16±0.43 32.66±0.42 k4 29.31±0.71 26.42±1.47 27.65±0.65 30.48±1.04 30.04±0.22 26.71±0.5 30.5±0.54 31.14±1.53 Avg 32.18±0.64 29.28±0.46 31.19±0.51 33.38±0.91 32.97±0.28 31.89±0.4 33.86±0.22 35.2±0.46 Youtube k1 26.82±0.33 31.99±1.85 36.97±1.33 33.14±0.88 31.61±1.15 35.44±0.33 34.67±0.33 39.46±1.45 k2 23.77±1.12 22.3±4.05 22.55±0.42 23.04±0.42 21.81±1.12 23.53±0.74 22.06±1.27 21.12±0.74 k3 42.24±1.49 35.63±1.32 36.49±1.00 39.94±0.5 43.1±2.28 38.22±0.5 43.68±3.03 46.84±1.32 k4 50.86±1.57 51.55±0.89 55.5±3.31 52.23±1.81 55.15±1.79 58.08±1.66 52.06±0.89 55.5±2.84 k5 45.83±1.74 44.83±1.55 51.01±2.53 53.02±1.88 50.14±1.51 52.3±0.25 50.57±1.99 54.18±0.5 k6 56.28±1.5 53.46±0.99 58.44±2.25 55.41±2.46 53.68±3.07 55.84±1.12 59.52±1.35 60.92±0.2 Avg 41.72±0.66 41.12±0.68 45.16±1.02 44.23±0.3 43.8±0.76 45.53±0.52 44.83±0.53 47.92±0.93 (a) Caltech101 (b) Reuters (c) NUS-WIDE (d) Youtube Figure 13: The test accuracy and convergence process of each method in M2 scenario. Table 7: Performance comparison (%) of all compared methods on Caltech101, Reuters, NUS-WIDE, and Youtube using M1+ data partitioning, where the number of clients K is equal to the number of modalities M. The k1k1 represents the client with ID 1. Dataset Local FedProto FedTGP FedPall Harmony FedMVP FedMobile MFedPBA Caltech101 k1 34.51±0.67 30.74±1.09 34.7±1.53 35.17±0.79 39.34±0.65 29.58±1.00 36.58±0.61 41.18±0.24 k2 40.63±0.87 35.02±0.39 38.57±0.77 41.68±0.83 41.04±1.03 33.69±0.54 41.05±0.58 45.46±0.47 k3 43.49±0.58 40.46±0.24 44.59±2.03 44.53±1.22 45.92±0.56 46.3±1.1 46.45±1.57 50.02±0.59 Avg 40.83±1.12 36.81±0.35 40.71±1.23 41.8±0.77 43.04±0.64 38.91±1.25 42.29±0.6 46.82±0.54 Reuters k1 72.28±1.92 75.00±2.54 74.61±2.58 74.8±1.76 74.71±0.85 61.12±0.88 75.12±1.25 80.82±1.05 k2 82.11±0.77 81.2±0.81 80.04±1.56 81.9±1.17 80.76±0.29 81.63±0.23 83.32±0.18 83.72±0.64 k3 60.57±0.85 53.4±1.25 57.83±1.87 60.12±1.18 61.7±0.67 60.05±0.34 61.31±0.14 62.21±0.7 k4 65.57±0.53 58.42±1.32 61.35±1.94 63.04±2.47 67.64±0.25 64.27±0.67 67.06±0.45 68.34±0.46 k5 73.95±2.65 77.74±1.01 77.39±1.57 78.4±1.14 77.61±0.46 77.26±0.27 78.95±1.55 81.82±0.21 Avg 70.85±0.59 68.54±1.46 69.8±0.59 71.09±1.56 72.28±0.47 68.09±0.82 72.79±0.44 75.01±0.22 NUS-WIDE k1 28.63±0.75 30.21±0.43 27.48±1.11 30.77±1.41 29.82±0.65 28.16±0.9 32.82±0.53 33.02±1.31 k2 28.17±0.24 26.2±0.68 25.67±1.39 29.35±0.42 31.83±1.24 29.51±0.53 29.77±0.55 30.9±0.85 k3 28.7±1.26 25.94±0.81 26.37±2.01 31.01±0.81 32.42±0.59 30.92±0.51 29.61±0.73 31.93±0.33 k4 34.32±2.79 28.75±0.57 28.52±1.91 35.56±0.86 34.21±1.97 32.96±1.2 31.35±3.97 33.71±2.53 Avg 28.72±0.71 27.79±0.55 27.13±0.18 30.99±0.63 31.56±0.61 29.82±0.59 30.95±0.57 32.15±0.36 Youtube k1 33.5±1.02 33.33±0.85 33.17±2.04 35.62±1.02 35.78±1.7 35.46±0.57 40.23±2.19 40.52±1.02 k2 57.18±2.46 52.87±0.5 58.05±2.24 56.7±1.63 57.95±1.68 59.58±0.66 58.14±0.66 60.25±0.44 k3 37.5±1.38 30.56±4.35 37.5±1.56 36.63±1.97 37.15±1.31 34.55±0.3 37.15±0.6 36.81±1.06 k4 40.37±1.79 37.93±4.48 44.54±1.00 42.82±1.79 43.39±3.06 42.24±0.86 44.4±0.75 45.4±0.25 k5 44.32±1.97 40.87±4.2 49.68±0.96 47.51±1.38 47.13±0.66 49.17±0.96 46.49±1.73 50.45±1.33 k6 22.08±1.12 22.51±5.32 29.06±2.15 22.73±2.83 22.73±2.34 27.92±2.34 27.27±2.34 29.22±2.34 Avg 41.89±0.76 38.82±1.12 44.55±0.68 43.04±1.09 43.47±0.91 44.24±0.36 44.28±0.64 47.21±0.59 (a) Caltech101 (b) Reuters (c) NUS-WIDE (d) Youtube Figure 14: The test accuracy and convergence process of each method in M1+ scenario. Table 8: Performance comparison (%) of all compared methods on Caltech101, Reuters, NUS-WIDE, and Youtube using M1 data partitioning, where the number of clients K is equal to the number of modalities M. The X1 represents a client that has the X1 modal enabled. Dataset Local FedProto FedTGP FedPall Harmony FedMVP FedMobile MFedPBA Caltech101 I1 39.47±0.13 38.27±1.4 41.16±0.67 42±0.71 39.84±0.62 38.62±0.5 41.87±0.13 48.73±0.9 I2 36.89±0.28 25.56±2.04 31.47±1.73 37.02±0.56 39.78±1.3 38.09±1.16 38.17±0.73 40.78±0.37 I3 39.07±1.22 30.89±1.34 35.91±0.66 43.29±0.2 40.69±0.83 40.04±1.65 40.22±0.28 41.4±1.05 Avg 38.47±0.45 31.57±1.23 36.18±0.72 40.77±0.36 40.1±0.69 38.92±0.62 40.09±0.13 43.64±0.11 Reuters L1 64.47±1.07 58.87±0.61 60.39±0.93 65.85±0.55 64.39±0.55 63.85±1.59 64.93±0.43 67.2±1.33 L2 69.81±1.4 79.52±0.67 77.58±0.88 74.31±2.03 76.44±0.18 76.41±0.25 75.37±0.38 77.87±0.85 L3 63.15±0.96 66.7±0.71 48.43±0.99 54.55±3.67 67.38±0.43 68.61±1.21 67.31±0.44 68.26±0.33 L4 72.39±1.59 78.29±0.16 76.61±1.64 73.24±1.29 76.19±0.12 75.01±1.12 75.2±0.62 79.83±1.65 L5 65.44±1.55 60.38±0.65 66.04±0.77 66.84±0.98 66.42±0.7 67.04±0.13 66.81±0.33 67.31±0.33 Avg 67.05±1.04 68.75±0.63 65.81±0.33 66.96±0.22 70.16±1.14 70.18±0.57 69.92±0.95 72.09±0.79 NUS-WIDE I1 34.5±0.85 23.15±1.55 35.04±0.18 32.27±3.24 34.4±1.12 33.33±1.21 33.65±1.34 34.29±0.8 I2 36.85±0.49 32.32±1.34 34.08±3.02 36.00±0.49 38.76±1.58 39.4±0.37 36.53±1.76 40.79±0.85 I3 32.37±0.8 22.17±1.63 30.35±0.86 35.46±1.15 33.01±1.29 34.61±1.21 33.55±1.66 35.46±0.68 L 20.87±0.49 22.19±3.02 18.74±0.82 20.34±0.74 22.58±1.61 23.43±0.78 21.09±0.32 22.47±0.49 Avg 31.15±0.14 24.96±0.52 29.55±0.73 31.02±0.96 32.19±0.35 32.7±0.51 31.2±0.36 33.25±0.24 Youtube I1 34.13±0.69 39.29±2.06 32.94±2.75 34.13±2.48 33.33±2.06 34.34±1.74 37.3±0.69 44.44±1.37 I2 34.13±1.37 26.9±2.89 27.78±1.37 33.33±1.19 41.27±3.64 38.89±1.82 34.52±3.15 30.56±1.37 T 22.22±1.82 18.2±0.59 21.83±0.69 21.43±3.15 22.62±2.06 21.37±1.29 20.24±2.38 28.56±1.05 A1 67.06±1.37 60.71±1.19 57.54±3.44 57.14±2.38 64.68±2.48 60.32±1.81 56.4±0.77 71.37±1.2 A2 32.94±3.44 24.81±1.73 32.94±2.48 35.71±2.06 34.13±1.82 33.33±1.19 34.87±1.74 30.75±2.22 A3 49.6±1.37 49.61±1.39 58.73±0.69 53.17±2.48 50.4±0.69 51.98±2.48 53.57±3.15 61.44±0.64 Avg 40.01±0.41 36.59±0.39 38.62±0.64 39.15±1.5 41.07±0.42 40.04±0.46 39.48±0.99 44.52±0.54 (a) Caltech101 (b) Reuters (c) NUS-WIDE (d) Youtube Figure 15: The test accuracy and convergence process of each method in M1 scenario. (a) Caltech101 (b) Reuters (c) NUS-WIDE (d) Youtube Figure 16: The testing accuracy and convergence process of each method in the M1+ scenario, with a large number of clients participating.