Paper deep dive
Probabilistic Federated Learning on Uncertain and Heterogeneous Data with Model Personalization
Ratun Rahman, Dinh C. Nguyen
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 98%
Last extracted: 3/22/2026, 5:53:34 AM
Summary
Meta-BayFL is a personalized probabilistic federated learning framework that integrates meta-learning with Bayesian neural networks (BNNs) to address data uncertainty and heterogeneity. By dynamically selecting optimal learning rates through meta-learning and modeling parameter uncertainty, the framework improves training stability and robustness on non-IID datasets, outperforming existing methods like pFedMe and Ditto.
Entities (5)
Relation Signals (3)
Meta-BayFL → evaluatedon → CIFAR-10
confidence 100% · Extensive experiments on CIFAR-10, CIFAR-100, and Tiny-ImageNet show that Meta-BayFL consistently outperforms
Meta-BayFL → uses → Bayesian Neural Networks
confidence 100% · we propose Meta-BayFL, a personalized probabilistic FL method that combines meta-learning with BNNs
Meta-BayFL → outperforms → pFedMe
confidence 95% · Meta-BayFL consistently outperforms state-of-the-art methods, including both standard and personalized FL approaches (e.g., pFedMe, Ditto, FedFomo)
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Conventional federated learning (FL) frameworks often suffer from training degradation due to data uncertainty and heterogeneity across local clients. Probabilistic approaches such as Bayesian neural networks (BNNs) can mitigate this issue by explicitly modeling uncertainty, but they introduce additional runtime, latency, and bandwidth overhead that has rarely been studied in federated settings. To address these challenges, we propose Meta-BayFL, a personalized probabilistic FL method that combines meta-learning with BNNs to improve training under uncertain and heterogeneous data. The framework is characterized by three main features: (1) BNN-based client models incorporate uncertainty across hidden layers to stabilize training on small and noisy datasets, (2) meta-learning with adaptive learning rates enables personalized updates that enhance local training under non-IID conditions, and (3) a unified probabilistic and personalized design improves the robustness of global model aggregation. We provide a theoretical convergence analysis and characterize the upper bound of the global model over communication rounds. In addition, we evaluate computational costs (runtime, latency, and communication) and discuss the feasibility of deployment on resource-constrained devices such as edge nodes and IoT systems. Extensive experiments on CIFAR-10, CIFAR-100, and Tiny-ImageNet show that Meta-BayFL consistently outperforms state-of-the-art methods, including both standard and personalized FL approaches (e.g., pFedMe, Ditto, FedFomo), with up to 7.42\% higher test accuracy.
Tags
Links
- Source: https://arxiv.org/abs/2603.18083v1
- Canonical: https://arxiv.org/abs/2603.18083v1
Trouble viewing inline? Open PDF directly →
Full Text
90,387 characters extracted from source content.
Expand or collapse full text
Probabilistic Federated Learning on Uncertain and Heterogeneous Data with Model Personalization Ratun Rahman and Dinh C. Nguyen, Ratun Rahman and Dinh C Nguyen are with the Department of Electrical and Computer Engineering, University of Alabama in Huntsville, Huntsville, AL 35899, USA, emails: r0110@uah.edu, dinh.nguyen@uah.edu Abstract Conventional federated learning (FL) frameworks often suffer from training degradation due to data uncertainty and heterogeneity across local clients. Probabilistic approaches such as Bayesian neural networks (BNNs) can mitigate this issue by explicitly modeling uncertainty, but they introduce additional runtime, latency, and bandwidth overhead that has rarely been studied in federated settings. To address these challenges, we propose Meta-BayFL, a personalized probabilistic FL method that combines meta-learning with BNNs to improve training under uncertain and heterogeneous data. The framework is characterized by three main features: (1) BNN-based client models incorporate uncertainty across hidden layers to stabilize training on small and noisy datasets, (2) meta-learning with adaptive learning rates enables personalized updates that enhance local training under non-IID conditions, and (3) a unified probabilistic and personalized design improves the robustness of global model aggregation. We provide a theoretical convergence analysis and characterize the upper bound of the global model over communication rounds. In addition, we evaluate computational costs (runtime, latency, and communication) and discuss the feasibility of deployment on resource-constrained devices such as edge nodes and IoT systems. Extensive experiments on CIFAR-10, CIFAR-100, and Tiny-ImageNet show that Meta-BayFL consistently outperforms state-of-the-art methods, including both standard and personalized FL approaches (e.g., pFedMe, Ditto, FedFomo), with up to 7.42% higher test accuracy. Index Terms: Personalized federated learning, probabilistic learning I Introduction Federated learning (FL) has gained considerable momentum due to its ability to facilitate data training at the network edge while ensuring the preservation of privacy [rahman2025electrical2, rahman2024multimodal]. FL enables local clients to independently train machine learning (ML) models on their local data, subsequently sharing only the model parameters with a centralized server for aggregation without exchanging raw data. This approach significantly improves data privacy and security by keeping local data private within local devices [rahman2024electrical]. Furthermore, FL approaches mitigate communication latency by introducing a decentralized approach and not transmitting only the parameter of local models [nguyen2022latency]. Despite such promising research efforts, traditional FL frameworks struggle with two practical challenges: uncertain and heterogeneous data, limiting training performance in federated settings [gawlikowski2023survey]. 1. Uncertain data. Data in a collaborative environment can be uncertain and the quality of the dataset may vary for each global round and for every client. For example, some client information may be out of date, noisy, or very small in number, which could have a negative impact on the training process when local updates are combined with the global model. This discrepancy increases difficulties in ensuring that the global model remains stable and improves consistently over each training round. Furthermore, uncertainty in data quality can hamper the model's convergence, since the learning algorithm may struggle to establish a generalizable pattern that adequately fits all data quality variations [ye2023adaptive]. This leads to a critical question: a) How can inherent uncertain data be controlled to stabilize the training of the global model in FL? 2. Heterogeneous data. Another significant problem is data heterogeneity where data distributions across clients are naturally non-independent and identically distributed (non-IID). For instance, different clients may have various feature spaces that allow them to be biased towards certain features. These differences create challenges for clients to construct a global model that represents all clients equally, and it can significantly affect the overall effectiveness of FL [t2020personalized, rahman2024improved]. This raises a critical question, b) How can FL effectively handle heterogeneous data distribution in non-IID settings? To address data uncertainty and heterogeneity challenges in FL, several works have been proposed that use advanced learning techniques. Personalized Federated Learning (PFL) has been developed to precisely address heterogeneous data settings. PFL methods ensure that the global model works well across various datasets by tailoring the training process to the features of individual or grouped clients. On the other hand, to address data uncertainty, probabilistic model training methods have been proposed, where probabilistic neural networks use a kernel function to approximate the probability density function of the training data and estimate the probability distribution of the data [specht1990probabilistic]. This allows the model to account for uncertainty in parameters, which is essentially important in training on noisy or unpredictable data. I-A Related Works We summarize the related works in BNN, probabilistic FL, and personalized FL. 1) BNN: One of the most common probabilistic methods is the bayesian neural network (BNN) which uses the Bayes theorem and observed data to update the model parameters given a prior distribution to estimate the posterior distribution [tran2019bayesian]. BNN utilizes possibilities for all neural network layers, making the prediction more accurate for uncertain data than the basic FL method [doan2025bayesian, jospin2022hands, akram2025uncertainty]. However, in the BNN, every client is handled equally and as a result, underperforming clients can significantly impact the global model, degrading the training performance. This also leads to poor generalization for individual clients, causing the international model to be biased towards some and irrelevant to others. As a result, the model becomes overly confident and performs poorly, especially on client datasets that differ significantly from the aggregated training data. Fig. 1: A simple probabilistic model structure in which probabilistic properties link every layer to every layer after it. The likelihood graph shows the likelihood of moving from the current layer to the next. It shows that probabilistic (distributional) parameters spread uncertainty through depth. 2) Personalized FL: Federated distillation and contrastive learning have also been investigated in recent works to address heterogeneity and communication efficiency. Examples of these include efficient federated distillation for multitask time-series classification [xing2022efficient] and federated contrastive learning with feature-based distillation for human activity recognition [xiao2025federated]. [li2020federated] proposed FedProx that utilizes a proximal term to improve the stability of the FL process. Despite their effectiveness in representation sharing, they do not explicitly describe uncertainty or offer probabilistic personalization under noisy and small client datasets. Also methods like pFedMe [t2020personalized] and Ditto [li2021ditto] learned additional personalized models. However, only the desired information that improves the quality of the local model is beneficial for this method, and undesired information leads to poor generalization. Models like Meta-FL [rahman2024improved], has continuously been used in various fields such as air quality [rahman2025multimodal] and load forecasting [rahman2025electrical]. It uses local personalized aggregation to learn the local model, making the local model more accurate for the global aggregation. However, FedAMP and FedPHP performed their pFL without considering any local objectives. FedMFomo downloaded other client models for aggregation, creating a high communication overhead. 3) Personalized Probabilistic FL: [li2021fedmask] proposed Fedmask that learns a sparse binary mask for personalized. However, it is mainly used to reduce computational costs and does not enhance the overall performance. pFedBayes [zhang2022personalized] and pFedBL [yu2025pfedbl] also used the pFL method on the Bayesian layer that introduces weight uncertainty for both the clients and server. The computational becomes very expensive in this method for large-scale data. However, the problem of uncertain and heterogeneous data training in FL settings has been largely under-explored. It is crucial to develop an efficient model training method to overcome the issue of uncertain and heterogeneous data in real-life applications. In our work, we propose Meta-BayFL, a personalized probabilistic FL based on meta-learning that focuses on enhancing the training performance by selecting the optimal results. To the best of our knowledge, this is the first work to study meta-learning on probabilistic FL. I-B Our Key Contributions Motivated by the above limitations and to further improve the handling of the uncertain data, we propose Meta-BayFL, a novel personalized probabilistic FL method with meta-learning-based BNNs on uncertain and heterogeneous data. Our key contributions are summarized as follows: • We introduce a novel notion of model personalization by framing the problem within the probabilistic federated learning (FL) paradigm (Section I). We specifically use meta-learning, a well-established approach for analyzing data in collaborative scenarios with diverse clients. While the structure of FL has been thoroughly researched, the relationship between probabilistic modeling and meta-learning remains largely unexplored. To close this gap, we introduce Meta-BayFL, a personalized probabilistic FL framework that successfully incorporates meta-learning principles. Furthermore, we create an effective computational approach that is designed to meet the problems posed by uncertain and heterogeneous data, improving both adaptability and robustness. • We perform convergence analysis for the proposed Meta-BayFL framework (Section IV) where we considered both meta-learning and the probabilistic approach and the derived upper bound reveals the following key properties: 1) using three different learning rates produces three different loss values and using the best learning rate would produce better performance and 2) the loss value decreases when global communication rounds. • We evaluate our performance with other state-of-the-art methods (Section V). The simulation results show the effectiveness of Meta-BayFL for uncertain and heterogeneous data which outperforms other state-of-the-art methods by up to 7.42% in test accuracy. I-C Paper Structure The rest of the paper is organized as follows. Section I discusses the necessary preliminaries, such as Bayesian neural networks, federated learning, and personalized federated learning. The proposed Meta-BayFL framework is described in detail in Section I, along with the training process, the meta-learning-based personalization strategy, and the probabilistic model formulation. A theoretical convergence analysis of Meta-BayFL and upper bounds on the global model performance over communication rounds are presented in Section IV. In-depth experiments on CIFAR-10, CIFAR-100, and Tiny-ImageNet are used in Section V to assess the suggested approach. These experiments include comparisons with cutting-edge federated and personalized learning techniques as well as an examination of runtime, latency, and communication overhead. The paper is finally concluded in Section VI, which also explores possible future research directions. Code availability: The source code for Meta-BayFL, including all scripts required to reproduce the experimental results, is publicly available at https://github.com/Ratun11/Meta-BayFL. I Preliminaries I-A Bayesian Neural Networks (BNNs) Consider the local training iteration index denoted as t∈t , where t=1,2,3,…,Tt=\1,2,3,…,T\ and n,kt w_n,k^t is the local weight for client n and global round k. The local update such as stochastic gradient descent (SGD) at each local iteration t is expressed as: θt+1=θt−η∇F(θ,χn,kt),θ^t+1=θ^t-η∇ F(θ, _n,k^t), (1) where η>0η>0 is the local learning rate, χ is the data sample from the local dataset, F is the loss value, and θ is the local parameter. For BNN, the local model update and aggregation are based on parameter distributions rather than deterministic values in 1. Instead of directly minimizing the loss function F, clients aim to maximize their Evidence Lower BOund, ELBOELBO function. Therefore, the local update using BNN is given as θt+1=θt+η∇ELBO(θ).θ^t+1=θ^t+η∇ ELBO(θ). (2) A higher ELBOELBO indicates a tighter lower bound on the log marginal likelihood, which is defined as: ELBO(θ)=qθ[logp(D|θ)]−KL(q(θ)||p(θ)).ELBO(θ)=E_q_θ[ p(D|θ)]-KL(q(θ)||p(θ)). (3) The prior distribution, denoted by p(θ)p(θ), represents our assumption and the value of θ prior to any data being observed and is expressed as p(θ)=∏(θ;μ,σ2),p(θ)= (θ;μ,σ^2), (4) where μ is the mean and σ2σ^2 is the variance of the distribution. Also, q(θ)q(θ) is the approximation to the true posterior distribution p(θ|D)p(θ|D) known as the variational distribution which is defined by q(θ))=∏(θ|μv,σv2),q(θ))= (θ| _v, _v^2), (5) where μv _v and σv2 _v^2 are variational mean and variational variance learned during training. To measure the distance between probability distributions, Kullback–Leibler (KL) divergence is introduced which is given as KL[q(θ|D)||p(θ)]=∫q(θ|D)logq(θ|D)p(θ)dθ.KL [q(θ|D)||p(θ) ]= q(θ|D) q(θ|D)p(θ)\,dθ. (6) Fig. 1 describes the probabilistic model architecture where we can see that for each input, we are calculating probabilistic outputs across all the hidden layers. I-B Federated Learning Consider an FL environment where the goal is to minimize the loss value of all the clients, which is usually expressed as min∈ℝdF():=1N∑n=1Nfn(), _ w∈R^dF( w):= 1N _n=1^Nf_n( w), (7) where w represents the model parameter, ℝdR^d denotes the d-dimensional real space in which the model parameters w reside local loss function, N is the total number of clients participating in federated learning, fk()f_k( w) is the local loss function computed by the i−thi-th client using its data. Given the local dataset as DnD_n the local loss function is denoted by: fn()=1|Dn|∑(xi,yi)∈Dnℒ(yi,f(xi)),f_n( w)= 1|D_n| _(x_i,y_i)∈ D_nL(y_i,f_ w(x_i)), (8) where |Dn||D_n| is the number of data points in k client's dataset and ℒ(yi,f(xi))L(y_i,f_ w(x_i)) is the loss for a single data point where yiy_i is the true label and f(xi)f_ w(x_i) is the model's prediction for input xix_i, parameterized by w. Each client performs SGD locally by iteratively updating its model parameters in the direction that minimizes its own loss function, often represented as: xi,new=xi,old−η∇fi(xi,old),x_i,new=x_i,old-η∇ f_i(x_i,old), (9) where η is the learning rate. After performing a certain number of local updates or epochs, each client sends their model updates or gradients to a central server. To improve the global model, the server aggregates these local updates n,kt w_n,k^t. The global model at the server is constructed by using the model averaging technique, i.e., g,k=1N∑i=1Ni,k. w_g,k= 1N _i=1^N w_i,k. (10) The central server sends the updated global model parameters back to the clients, and the process repeats for several iterations or until convergence. The server broadcasts the updated global model parameters global w_global back to the clients in the next round. I-C Personalized Federated Learning For PFL, every client has their own personalized model where they train their model based on their distinct data distribution. So, the basic FL equation in 7 becomes minx∈ℝdF(x):=1||∑n=1||fi(x)+xn, _x∈R^dF(x):= 1|N| _n=1^|N|f_i(x)+x_n, (11) where xnx_n is the personalized term for client n∈n . The local SGD update can be expressed as: n,kt+1=n,kt−ηk∇F(n,kt,χn,kt)+vn,k, w_n,k^t+1= w_n,k^t- _k∇ F( w_n,k^t, _n,k^t)+v_n,k, (12) where χ is the local dataset non-IID sample, and vn,kv_n,k is the n client's personalized value for global round k. Each client updates its personalized model as follows: n,kt+1=n,kt−ηk[∇F(n,kt,ζn,kt)+λ(n,kt−n,kt)], x_n,k^t+1= x_n,k^t- _k [∇ F( x_n,k^t, _n,k^t)+λ( x_n,k^t- w_n,k^t) ], (13) where η is the personalized learning rate, ζ is a local data sample, and λ>0λ>0 is the parameter that controls the interpolation of global and personalized models. Federated averaging is carried out at the server like 10. I Proposed Probabilistic FL Framework with Model Personalization We consider an FL system where a set of clients denoted by as N collaborate to train a shared ML model with a single server, as shown in Fig. 2. For each global round k∈k , each client n∈n trains a BBN model represented by n,kt w_n,k^t where local BNN iteration is denoted as t∈t . I-A Model Training Concept Assuming the number of available learning rates as j, we can separate our system model into multiple steps as follows. Fig. 2: Our proposed Meta-BayFL FL algorithm where there are K clients connected to the global server with adaptive model aggregation using a personalized learning approach. Each client creates local models using BNN and their local data and different temporary learning rates, and chooses the best rates for local model training. The server aggregates the local models and distributes the updated global model to all the clients. Step 1: In every k, the global model's weight (g,k)( w_g,k) is distributed across all n. All n receives g,k w_g,k as well as their local dataset denoted by DnkD_n^k for local model raining. We denote local model's weight as n,k w_n,k, and so initially, n,k=g,k w_n,k= w_g,k. Step 2: Every n creates a small testing dataset from DnkD_n^k denoted as dnk⊂Dnkd_n^k⊂ D_n^k and is used to build the optimal local model using meta-learning. Step 3: Under the federated setting, the prior p(n,kt)p( w_n,k^t), posterior q(n,kt)q( w_n,k^t) distribution, and KL divergence can be expressed as p(n,kt)=∏(n,kt;μ,σ2),p( w_n,k^t)= ( w_n,k^t;μ,σ^2), (14) q(n,kt)=∏(n,kt|μv,σv2),q( w_n,k^t)= ( w_n,k^t| _v, _v^2), (15) and KL[q(n,kt|dnk)||p(n,kt)]=∫q(n,kt|dnk)logq(n,kt|dnk)p(θ)dn,kt. splitKL [q( w_n,k^t|d_n^k)||p( w_n,k^t) ]\\ = q( w_n,k^t|d_n^k) q( w_n,k^t|d_n^k)p(θ)\,d w_n,k^t. split (16) Step 4: We calculate ELBOELBO using equation 3 as follows. ELBO(n,kt)=q(n,kt)[logp(dnk|n,kt)]−KL[q(n,kt)||p(n,kt)]. splitELBO( w_n,k^t)=E_q( w_n,k^t)[ p(d_n^k| w_n,k^t)]\\ -KL[q( w_n,k^t)||p( w_n,k^t)]. split (17) Step 5: The local model is updated for every temporary local round ttempt_temp using different learning rates ηi _i where i∈ji∈ j for all j as follows. n,kttemp+1=n,kttemp+ηi∇ELBO(n,kttemp). w_n,k^t_temp+1= w_n,k^t_temp+ _i∇ ELBO( w_n,k^t_temp). (18) After ttempt_temp temporary local rounds, we get a temporary local model for learning rate ηi _i. Let us denote the temporary local model as n,ki w_n,k^i. For all learning rates j, we will get the total j number of temporary local models in every n and every k. Step 6: After calculating all the temporary local models, we calculate the temporary loss value based on those models on dataset dnkd_n^k as follows fn,k(n,ki)=1|dnk|∑(xi,yi)∈dnkℒ(yi,fn,ki(xi)).f_n,k( w_n,k^i)= 1|d_n^k| _(x_i,y_i)∈ d_n^kL(y_i,f_ w_n,k^i(x_i)). (19) We calculate the loss value for every learning rate ηi _i where i∈ji∈ j. The learning rate that provides the lowest fnf_n value is considered the optimal learning rate denoted by ηn,kbestη^best_n,k for the local model training, i.e., ηn,kbest:=argmini∈jfn,k(n,ki).η^best_n,k:= *arg\,min_i∈ jf_n,k( w_n,k^i). (20) Step 7: We use ηn,kbestη^best_n,k for the local model training on client n and global round k. So for local round t, n,kt+1=n,kt+ηn,kbest∇ELBO(n,kt). w_n,k^t+1= w_n,k^t+η^best_n,k∇ ELBO( w_n,k^t). (21) After T local epochs, we finally get the updated local model n,k=n,kT w_n,k= w_n,k^T. The local model is sent to the server. Step 8: The server collects all the local models' weights for the global model aggregation as follows: g,k+1=1N∑i=1Ni,k. w_g,k+1= 1N _i=1^N w_i,k. (22) Finally, the updated global model g,k+1 w_g,k+1 is distributed across all clients k for the K+1K+1 global round. After K global rounds, we finally get the optimal global model ∗ w^*. In our approach, we used 80% of our data for local model training and expressed them as Trainloader (Dnk)(D_n^k) and the rest 20% for testing as Testloader (Dnk′)(D_n^ k). To evaluate the performance, we used 20% of the training data expressed as Dataloader, dnkd_n^k where dnk⊂Dnkd_n^k⊂ D_n^k and used to find the optimal learning rate ηn,kbestη^best_n,k. Algorithm 1 Proposed FL algorithm 1: Input: The set of global communication rounds K, local training round T, number of clients N, set of available learning rates η1,2,…,j _1,2,...,j, the temporary round for meta-learning as TtempT_temp 2: Initialization: Initialize global model 0 w_0 3: for each global communication round k∈k do 4: Send g,k w_g,k to N sampled clients 5: for each sampled client n∈n do 6: Receive the global weight as local weight n,k=g,k w_n,k= w_g,k 7: for each local training epoch t∈t do 8: Compute the prior p(n,k)p( w_n,k) distribution, posterior q(n,k)q( w_n,k) distribution, and KL divergence using equation 14, 15, and 16 9: Calculate ELBOELBO using the variables above and equation 17 10: for each learning rates ηj _j where i∈ji∈ j do 11: Update the temporary local model weight using ηi _i for all local temp rounds ttempt_temp as n,ki w_n,k^i 12: Perform test on n,ki w_n,k^i using equation 19 on data dnkd_n^k 13: Calculate loss(ηj)=fi(w,Dnk′)loss( _j)=f_i(w,D_n^ k) on Dnk′D_n^ k 14: Save the ηi _i that has the lowest lossloss value as ηn,kbestη^best_n,k using equation 20 15: Return ηn,kbestη^best_n,k 16: end for 17: Aggregate the local model on DnkD_n^k using ηn,kbestη^best_n,k, n,kt+1⟵n,kt+ηn,kbest∇ELBO(n,kt) w_n,k^t+1 w_n,k^t+η^best_n,k∇ ELBO( w_n,k^t) 18: end for 19: Send n,K=n,KT w_n,K= w_n,K^T to server 20: end for 21: The server updates the global parameter by averaging: g,k+1=1N∑n∈n,k w_g,k+1= 1N _n w_n,k 22: The server broadcasts it to all clients for the next round of training 23: end for 24: Output: Optimal global model ∗ w^* The proposed PFL approach is summarized in Algorithm 1. Here, for every global round k, we send global weight to all clients in line 4. Then we calculate prior distribution, posterior distribution, KL divergence, and ELBO in every local round t explained in lines 8-9. Lines 10-16 focus on the meta-learning functionalities where total j learning rates (ηi _i where i∈ji∈ j) are used on dnkd_n^k before local model aggregation. The learning rate that produces the lowest loss value is used to aggregate the local model in line 17. We send the optimal local weight to the server in line 19. The server collects all the local weights from every client n calculates federated averaging in line 21 and saves the updated global weight for the next global rounds. Among the three learning rate schedules analyzed in Theorem 1, we adopt the meta-learned adaptive learning rate in all experiments, as it empirically yields faster convergence and improved stability under non-IID and noisy settings. The remaining schedules are included in the analysis to establish general convergence guarantees and to serve as theoretical baselines. I-B Convergence Analysis To support our convergence analysis, we introduce two virtual sequences: ¯kt=1N∑n∈kt,¯kt=1N∑n∈kt. w_k^t= 1N _n w_k^t, 10000\ 10000\ 10000\ 10000\ 10000\ 10000\ 10000\ 10000\ 10000\ 10000\ 10000\ 10000\ 10000\ 10000\ 10000\ 10000\ 10000\ 10000\ 10000\ 10000\ x_k^t= 1N _n x_k^t. (23) Subsequently, each client updates its personalized model as kt+1=kt−ηtgt,gt=∇f(kt)+bt+nt x_k^t+1= x_k^t- _tg_t, 10000\ 10000\ 10000\ 10000\ 10000\ 10000\ 10000\ 10000\ 10000\ 10000\ 10000\ 10000\ 10000\ 10000\ 10000\ 10000\ 10000\ 10000\ 10000\ 10000\ g_t=∇ f( x_k^t)+b_t+n_t (24) where for zero-mean noise nt=0En_t=0 and bias bkb_k, gtg_t is a gradient oracle and ηt _t is the sequence of step sizes. If there is no bias, bt=0b_t=0, it becomes the SGD setting and for no noise, nt=0n_t=0, it becomes the classic gradient descent algorithm. It is easy to observe that, ¯kt+1=kt−ηk∇F(kt,χkt)+ηkBk−ηkNk, w_k^t+1= w_k^t- _k∇ F( w_k^t, _k^t)+ _kB_k- _kN_k, (25) To facilitate the analysis, we use the following common assumptions: Assumption 1. (L−L-smoothness). Each local loss function FnF_n (n∈n ) is L-smooth (L>0L>0), i.e. Fn(′)−Fn()≤⟨′−,∇F(⟩+L2||′−||,∀′,F_n( w )-F_n( w)≤ w - w,∇ F( w + L2|| w - w||,∀ w , w (26) Assumption 2. ((M,σ2M,σ^2)-bounded noise). There exists constant M, σ2>=0σ^2>=0 such that ‖n(,ξ)‖2≤M‖∇Fn()+b()‖2+σ2,∀∈ℝd.E||n( w,ξ)||^2≤ M||∇ F_n( w)+b( w)||^2+σ^2,∀ w ^d. (27) Assumption 3. ((m,ζm,ζ)-bounded bias). There exists constants 0≤m<10≤ m<1 and ζ2≥0ζ^2≥ 0 such that ||b(||2≤m||∇Fn()||2+ζ2,∀∈ℝd.||b( w||^2≤ m||∇ F_n( w)||^2+ζ^2,∀ w ^d. (28) Assumption 4. Finite parameter space: The parameter space Θ is finite, i.e, Θ=χk1,χk2,χk3,…,χkt = _k^1, _k^2, _k^3,…, _k^t. Also, χkt′∈Θ _k^ t∈ There exist 0<LH<∞0<L_H<∞ such that ‖∇F(,χk1)−∇F(,χk2)‖2 ||∇ F( w, _k^1)-∇ F( w, _k^2)||_2 ≤LH‖χk1−χk2‖2∀χk1,χk2∈Θ, ≤ L_H|| _k^1- _k^2||_2∀ _k^1, _k^2∈ , (29) ∀x∈ ∀ x∈ X Sampling variance is bounded by σ2σ^2, such that [||∇f(∗,ξ)−∇F(kt,χkt)||22|χn]≤σ2,∀χk∈ΘE[||∇ f( w*,ξ)-∇ F( w_k^t, _k^t)||_2^2| _n]≤σ^2,∀ _k∈ (30) Assumption 5. The variance of stochastic gradients on local model training at each client is bounded: ‖∇F(n,kt,χn,kt)−∇F(n,kt)‖2≤σg2E||∇ F( w_n,k^t, _n,k^t)-∇ F( w_n,k^t)||^2≤ _g^2 Lemma 1. Under Assumption 4, there exist a constant C1>0C_1>0 such that for any δ>0δ>0, with probability 1−δ1-δ we have ‖πt∇F(k,χkt)−π1∇F(k,χkt′)‖22≤C1logDt+log1δDt, ||E_ _t∇ F( w_k, _k^t)-E_ _1∇ F( w_k, _k^ t)||_2^2≤ C_1 logDt+log 1δDt, (31) ∀x∈,∀t>0∀ x∈ X,∀ t>0 Lemma 2. Let F be L-smooth, kt+1 x_k^t+1 and kt x_k^t as in 25 with Assumption 2 and 3. For any stepsize η≤1(M+1)Lη≤ 1(M+1)L, it holds ξ[F(+)−F()|]≤ _ξ[F( w_k^t+1)-F( w_k^t)| w_k^t]≤ η(m−1)2‖∇F()‖2+ η(m-1)2||∇ F( w_k^t)||^2+ (32) η2ζ2+η2L2σ2 η2ζ^2+ η^2L2σ^2 when M=m=ζ2=cM=m=ζ^2=c for any constant c, we recover the standard descent lemma. Lemma 3. Under Assumption 4, there exists a constant C1>0C_1>0 such that for any δ>0δ>0, with probability at least 1−δ1-δ we have ||∇F(,χt)− ||E∇ F( w,χ^t)- π1∇F(,χt′)||22≤C1logDt+log1δDt, _ _1∇ F( w,χ^ t)||_2^2≤ C_1 logDt+log 1δDt, (33) ∀x∈,∀t>0. ∀ x∈ X,∀ t>0. Lemma 4. Under the assumption p=Psp=P_s and series ∑l≥1Nle−l2 _l≥ 1N_le^-l^2 converges. v¯[B(s,kn)|X]≥1−δ,∀s∈S,ϵ,δ∈(0,1) v[B(s, k n)|X]≥ 1-δ,∀ s∈ S,ε,δ∈(0,1) (34) with probability at least 1−ϵ1-ε with respect to PsnP_s^n, and where k=k(ϵ,δ,v(s))=inf[j≥1|∑l≥jNle−l2≤ϵδv(s)]k=k(ε,δ,v(s))=inf [j≥ 1| _l≥ jN_le^-l^2≤ε δ v(s) ] is independent of n and nonincreasing with the positive parameters ϵ,δ,andv(s)ε,δ,andv(s). Lemma 5. Let Assumption 5 hold, the expected upper bound of the variance of the stochastic gradient on local model training is given as ‖gkt−g¯kt‖2≤σg2N2.E||g_k^t- g_k^t||^2≤ _g^2N^2. (35) Lemma 6. The expected upper bound of the divergence of n,kt w_n,k^t is given as [1N∑n∈‖¯kt−n,kt‖2]≤4ηkTB2, [ 1N _n E \| w_k^t- w_n,k^t \|^2 ]≤ 4 _kTB^2, (36) for some positive B. Lemma 7. The expected upper bound of [‖kt+1¯−∗‖2]E[|| w_k^t+1- w^*||^2] is given as ‖¯kt+1−∗‖2 || w_k^t+1- w^*||^2 ≤‖(1−μηk)‖¯kt−∗‖2+1N∑n∈‖¯kt− ≤||(1-μ _k)|| w_k^t- w^*||^2+ 1N _n || w_k^t- (37) n,kt||2+14ηk1N∑n∈||n,kt−¯kt||2+ w_n,k^t||^2+ 14 _k 1N _n || w_n,k^t- w_k^t||^2+ 12min(Θ1,Θ2,Θ3)+ηk2‖gkt−g¯kt‖2 12 ( _1, _2, _3)+ _k^2||g_k^t- g_k^t||^2 Theorem 1. Under some Assumptions, for any δ>0δ>0, we have the probability at least 1−δ1-δ, for any T>0T>0, the following bound on the expected gradient of the final output under the true parameter χkt′ _k^ t (i) If the step size (η) satisfies ηk=aK,∀k≤ _k= a K,∀ k , for some constant a<Lha< KL_h, then [‖∇F(z,χkt′)‖22]≤[2(F(1,χkt′)−minx∈F(,χkt′))a]+[A1+A2log+A3log2]+Lhaσ2E[||∇ F(z_K, _k^ t)||_2^2]≤ [ 2(F( w_1, _k^ t)- _x∈ XF( w, _k^ t))a K ]+ [ A_1K+ A_2logKK+ A_3log^2KK ]+ L_haσ^2 K, where A1=C1(logD−logδ)LhD,A2=C1(logD−logδ)LhD+C1LhD,A3=C1LhDA_1= C_1(logD-logδ)L_hD,A_2= C_1(logD-logδ)L_hD+ C_1L_hD,A_3= C_1L_hD. (i) If the step size (η) satisfies ηk=ak,∀k≤ _k= ak,∀ k , for some constant a<1Lha< 1L_h, then [‖∇F(z,χkt′)‖22]≤[2(F(1,χkt′)−minx∈F(,χkt′))a+6C1+π2C1(logD−logδ)6D+π2Lhaσ26]1logE[||∇ F(z_K, _k^ t)||_2^2]≤ [ 2(F( w_1, _k^ t)- _x∈ XF( w, _k^ t))a+ 6C_1+π^2C_1(logD-logδ)6D+ π^2L_haσ^26 ] 1logK. (i) if the step size (η) satisfies ηk=ak,∀k≤ _k= a k,∀ k , for some constant a<1Lha< 1L_h, then [‖∇F(z,χkt)‖22]≤[2(F(1,χkt′)−minx∈F(,χkt′))a+3C1(logD−logδ)+4C1D+Lhaσ2]+Lhaσ2log.E[||∇ F(z_K, _k^t)||_2^2]≤ [ 2(F( w_1, _k^ t)- _x∈ XF( w, _k^ t))a K+ 3C_1(logD-logδ)+4C_1D K+ L_haσ^2 K ]+ L_haσ^2logK K. Remark 1. Different η values produced different loss values as shown in Theorem 1. The theorem shows that in (i) the η value aK a K for constant a<Lha< KL_h, (i) the η value ηk=ak _k= ak for constant a<1Lha< 1L_h, and (i) the ηk=ak _k= a k for constant a<1Lha< 1L_h has produces three different [‖∇F(z,χkt′)‖22]E[||∇ F(z_K, _k^ t)||_2^2] values. In basic FL, there are only one η value and that produce one [‖∇F(z,χkt′)‖22]E[||∇ F(z_K, _k^ t)||_2^2] value. Theorem 2. Based on the above Lemmas and Theorem 1, the convergence bound of our approach after K global communication rounds is given as [F(K)]−F∗ [F( w_K) ]-F^* (38) ≤L2(K+L/μ)[16ΦK15μ2+(Lμ+1)‖0−∗‖2]. ≤ L2(K+L/μ) [ 16 _K15μ^2+ ( Lμ+1 )E|| w_0- w^*||^2 ]. Remark 2. The upper bound of convergence achieved in Theorem 2 shows that the convergence of the Meta-BayFL algorithm is strongly influenced by the number of total communication rounds. IV Detailed Proofs of Convergence IV-A Proof of Lemma 2 By the quadratic upper bound in Assumption 1 and Assumption 2: F(kt+1) ( w_k^t+1) ≤F(kt)−ηk(∇F(kt),gt)+η2L2(||gt−gt||2+ ≤ F( w_k^t)- _k(∇ F( w_k^t),Eg_t)+ η^2L2(E||g_t-Eg_t||^2+E ||gt||2) 10000\ 10000\ 10000\ 10000\ ||Eg_t||^2) =F(kt)−ηk(∇F(kt),∇f(kt)+bt+nt)+ =F( w_k^t)- _k(∇ F( w_k^t),∇ f( x_k^t)+b_t+n_t)+ η2L2(‖nt‖2+‖∇f(kt)+bt+nt‖2) 10000\ 10000\ 10000\ 10000\ η^2L2(E||n_t||^2+E||∇ f( x_k^t)+b_t+n_t||^2) ≤F(kt)−ηk(∇F(kt),∇f(kt)+bt+nt)+η2L2((M+1) ≤ F( w_k^t)- _k(∇ F( w_k^t),∇ f( x_k^t)+b_t+n_t)+ η^2L2((M+1) ||∇f(kt)+bt+nt||2+σ2) 10000\ 10000\ 10000\ 10000\ E||∇ f( x_k^t)+b_t+n_t||^2+σ^2) By the choice of stepsize, η≤1(M+1)Lη≤ 1(M+1)L, and Assumption 3: F(kt+1) ( w_k^t+1) (39) ≤F(kt)+ηk2(−2(∇F(kt),∇f(kt)+bt+nt)+ ≤ F( w_k^t)+ _k2(-2(∇ F( w_k^t),∇ f( x_k^t)+b_t+n_t)+ ||∇f(kt)+bt+nt||2)+η2L2σ2 10000\ 10000\ 10000\ 10000\ ||∇ f( x_k^t)+b_t+n_t||^2)+ η^2L2σ^2 =F(kt)+ηk2(−||∇F(kt||2+||bt+nt||2)+η2L2σ2 =F( w_k^t)+ _k2(-||∇ F( w_k^t||^2+||b_t+n_t||^2)+ η^2L2σ^2 =F(kt)+ηk2(m−1)‖∇F(kt)‖2+η2ζ2+η2L2σ2 =F( w_k^t)+ _k2(m-1)||∇ F( w_k^t)||^2+ η2ζ^2+ η^2L2σ^2 This concludes the proof. IV-B Proof of Lemma 3 The Hellignger distance between θ1 _1 and θ2 _2 d(θ1,θ2)=12∫(f(y;θ1−f(y;θ2))2d( _1, _2)= 12 _ Y( f(y; _1- f(y; _2))^2 (40) There exists a constant A such that ||θ1=θ2||≤Ad(θ1,θ2)|| _1= _2||≤ Ad( _1, _2), where ||.||||.|| is the Euclidean norm. Let Bkt=B(θc,kDt)B_k^t=B(θ^c, k Dt) be a ball centered at θcθ^c with radius kDt k Dt under distance d. Since Θ is finite, we can directly apply Lemma 4. So for t≤T,ϵ,δ∈(0,1)t≤ T,ε,δ∈(0,1) with probability at least 1−6δπ2t21- 6δπ^2t^2 with respect to ℙθctP_θ^c^t, we have πt(Bk(t)t)≥1−ϵ, _t(B_k(t)^t)≥ 1-ε, (41) where k(t)=inf[j≥1|∑i≥j|Θ|e−i2≤6δπ2t2ϵπ0(θc)]k(t)=inf [j≥ 1| _i≥ j| |e^-i^2≤ 6δπ^2t^2 ε _0(θ^c) ]. Note that ∑i≥je−i2≤e−1e−j2 _i≥ je^-i^2≤ ee-1e^-j^2, we can set k(t) to be the solution of next equation. e−1|Θ|e−k(t)2=6δπ2t2ϵπ0(θc) ee-1| |e^-k(t)^2= 6δπ^2t^2 ε _0(θ^c) (42) we get k(t)=loge|Θ|π2t26δ(e−1)ϵ,π0(θc)k(t)= log e| |π^2t^26δ(e-1) ε, _0(θ^c). Now the bias in the gradient estimator can be bonded as follows. ‖πt∇xF(,χ)−πt∇xF(,χ′)‖22 ||E_ _t _xF( w,χ)-E_ _t _xF( w,χ^ )||_2^2 (43) =‖∫(∇xF(,χ)−∇xF(,χ′))πt(θ)θ‖22 =|| ( _xF( w,χ)- _xF( w,χ^ )) _t(θ)dθ||_2^2 ≤∫‖(∇xF(,χ))−(∇xF(,χ′))22‖πt(θ)θ ≤ ||( _xF( w,χ))-( _xF( w,χ^ ))_2^2|| _t(θ)dθ ≤LH2‖χ−χ′‖22πt(θ)dθ ≤ L_H^2||χ-χ^ ||_2^2 _t(θ)dθ =F(kt)−ηk(∇F(kt),∇f(kt)+bt+nt)+η2L2(||nt||2+ =F( w_k^t)- _k(∇ F( w_k^t),∇ f( x_k^t)+b_t+n_t)+ η^2L2(E||n_t||^2+ ||∇f(kt)+bt+nt||2) 10000\ 10000\ 10000\ 10000\ E||∇ f( x_k^t)+b_t+n_t||^2) =∫Bk(t)tLH2‖χ−χ′‖22πt(θ)θ+∫(Bk(t)t)′LH2‖χ−χ′‖22πt(θ)θ = _B_k(t)^tL_H^2||χ-χ^ ||_2^2 _t(θ)dθ+ _(B_k(t)^t)^ L_H^2||χ-χ^ ||_2^2 _t(θ)dθ ≤A2LH2k(t)2Dt∫Bk(t)tπt(θ)θ+LH2maxχ∈Θ‖χ−χ′‖22∫(Bk(t)t)′πt(θ)θ ≤ A^2L_H^2 k(t)^2Dt _B_k(t)^t _t(θ)dθ+L_H^2 _χ∈ ||χ-χ^ ||_2^2 _(B_k(t)^t)^ _t(θ)dθ ≤A2LH2k(t)2Dt+LH2maxχ∈Θ‖χ−χ′‖22ϵ ≤ A^2L_H^2 k(t)^2Dt+L_H^2 _χ∈ ||χ-χ^ ||_2^2ε Here, D is the data batch size. Note that ϵ=1Dtε= 1Dt and k(t)=loge|Θ|π2t2Dt6δ(e−1)π0(χ′)k(t)= log e| |π^2t^2 Dt6δ(e-1) _0(χ ), we have ‖πt∇xF(,χ)−πt∇xF(,χ′)‖22 ||E_ _t _xF( w,χ)-E_ _t _xF( w,χ^ )||_2^2 (44) ≤A2LH2k(t)2Dt+LH2maxχ∈Θ‖χ−χ′‖22ϵ 10000\ 10000\ 10000\ 10000\ 10000\ 10000\ 10000\ 10000\ 10000\ 10000\ 10000\ 10000\ 10000\ ≤ A^2L_H^2 k(t)^2Dt+L_H^2 _χ∈ ||χ-χ^ ||_2^2ε ≤2A2LH2maxθ∈Θ‖χ−χ′‖22loge|Θ|π2t2Dt6δ(e−1)π0(χ′)Dt 10000\ 10000\ 10000\ 10000\ 10000\ 10000\ 10000\ 10000\ 10000\ 10000\ 10000\ 10000\ 10000\ ≤ 2A^2L_H^2 _θ∈ ||χ-χ^ ||_2^2 log e| |π^2t^2 Dt6δ(e-1) _0(χ )Dt =O(logDt+log1δDt) 10000\ 10000\ 10000\ 10000\ 10000\ 10000\ 10000\ 10000\ 10000\ 10000\ 10000\ 10000\ 10000\ =O ( logDt+log 1δDt ) Let ℰt E_t denote the event that the above inequality holds, and ℰtc E_t^c denote that the above inequality does not hold. ℙ(ℰtc)≤6δπ2t2P( E_t^c)≤ 6δπ^2t^2 (45) Therefore, ℙ(∩t=1∞ℰt) ( _t=1^∞ E_t) (46) =1−ℙ(∪t=1∞ℰtc) 10000\ 10000\ 10000\ 10000\ 10000\ 10000\ 10000\ 10000\ 10000\ 10000\ 10000\ 10000\ 10000\ =1-P( _t=1^∞ E_t^c) ≥1−∑t=1∞ℙ(ℰtc))(Union bound) 10000\ 10000\ 10000\ 10000\ 10000\ 10000\ 10000\ 10000\ 10000\ 10000\ 10000\ 10000\ 10000\ ≥ 1- _t=1^∞P( E_t^c)) 10000\ 10000\ 10000\ 10000\ 10000\ 10000\ 10000\ (Union bound) ≥1−∑t=1∞6δπ2t2 10000\ 10000\ 10000\ 10000\ 10000\ 10000\ 10000\ 10000\ 10000\ 10000\ 10000\ 10000\ 10000\ ≥ 1- _t=1^∞ 6δπ^2t^2 =1−δ 10000\ 10000\ 10000\ 10000\ 10000\ 10000\ 10000\ 10000\ 10000\ 10000\ 10000\ 10000\ 10000\ =1-δ IV-C Proof of Lemma 5 From Assumption 5, we have ‖gkt−g¯kt‖2 ||g_k^t- g_k^t||^2 =‖1N∑n∈(∇F(n,kt,χn,kt)−∇F(n,kt))‖2 =E \| 1N _n (∇ F( w_n,k^t, _n,k^t)-∇ F( w_n,k^t) ) \|^2 (47) =1N2∑n∈‖(∇F(n,kt,χn,kt)−∇F(n,kt))‖2 = 1N^2 _n E \| (∇ F( w_n,k^t, _n,k^t)-∇ F( w_n,k^t) ) \|^2 ≤σg2N2. ≤ _g^2N^2. IV-D Proof of Lemma 6 We know that in every global communication round, each client performs T rounds of local SGDs where there always exits t′≤t ≤ t such that t−t′≤Tt-t ≤ T and n,kt′=¯kt′ w_n,k^t = w_k^t , ∀n∈∀ n . By using the fact that ‖X−X‖2=‖X‖2−‖X‖2E||X-EX||^2=||X||^2-||EX||^2 and ¯kt=n,kt w_k^t=E w_n,k^t, we have: 1N∑n∈‖¯kt−n,kt‖2 1N _n E \| w_k^t- w_n,k^t \|^2 (48) =1N∑n∈‖n,kt−¯kt‖2 10000\ 10000\ 10000\ 10000\ 10000\ 10000\ 10000\ 10000\ = 1N _n E \| w_n,k^t- w_k^t \|^2 =1N∑n∈‖(n,kt−¯kt′)−(¯kt−¯kt′)‖2 10000\ 10000\ 10000\ 10000\ 10000\ 10000\ 10000\ 10000\ = 1N _n E \|( w_n,k^t- w_k^t )-( w_k^t- w_k^t ) \|^2 ≤1N∑n∈‖n,kt−¯kt′‖2 10000\ 10000\ 10000\ 10000\ 10000\ 10000\ 10000\ 10000\ ≤ 1N _n E \| w_n,k^t- w_k^t \|^2 ≤1N∑n∈‖(∑t=t′t−1(n,kt−¯kt′))‖2 10000\ 10000\ 10000\ 10000\ 10000\ 10000\ 10000\ 10000\ ≤ 1N _n E \| ( _t=t ^t-1( w_n,k^t- w_k^t ) ) \|^2 =1N∑n∈‖(∑t=t′t−1ηk∇F(n,kt,χn,kt))‖2 10000\ 10000\ 10000\ 10000\ 10000\ 10000\ 10000\ 10000\ = 1N _n E \| ( _t=t ^t-1 _k∇ F( w_n,k^t, _n,k^t) ) \|^2 ≤1N∑n∈‖(∑t=1t−t′ηk∇F(n,kt,χn,kt))‖2, 10000\ 10000\ 10000\ 10000\ 10000\ 10000\ 10000\ 10000\ ≤ 1N _n E \| ( _t=1^t-t _k∇ F( w_n,k^t, _n,k^t) ) \|^2, where the last inequality holds since the learning rate ηk _k is decreasing. Using the fact that ‖∑t=1Uzt‖2≤U∑t=1U‖zt‖2|| _t=1^Uz^t||^2≤ U _t=1^U||z^t||^2, t−t′≤Tt-t ≤ T and assume that ηkt′≤2ηk _k^t ≤ 2 _k and ‖∇F(n,kt,χn,kt)‖2≤B2||∇ F( w_n,k^t, _n,k^t)||^2≤ B^2 for positive constant B, we have 1N∑n∈‖¯kt−n,kt‖2 1N _n E \| w_k^t- w_n,k^t \|^2 (49) ≤1N∑n∈(∑t=1t−t′ηk2(t−t′)‖∇F(n,kt,χn,kt)‖2) ≤ 1N _n (E _t=1^t-t _k^2(t-t ) \|∇ F( w_n,k^t, _n,k^t) \|^2 ) ≤1N∑n∈(∑t=1t−t′ηk2T‖∇F(n,kt,χn,kt)‖2) ≤ 1N _n (E _t=1^t-t _k^2T \|∇ F( w_n,k^t, _n,k^t) \|^2 ) ≤1N∑n∈((ηkt′)2T∑t=1t−t′B2) ≤ 1N _n (( _k^t )^2T _t=1^t-t B^2 ) ≤1N∑n∈(ηkt′)2TB2≤4ηkTB2. ≤ 1N _n ( _k^t )^2TB^2≤ 4 _kTB^2. IV-E Proof of Theorem 1 The local SGD update at client n is followed as: ¯kt+1= w_k^t+1= kt−ηk∇F(kt,χkt′)+ηk[π1∇F(kt,χkt)− w_k^t- _k∇ F( w_k^t, _k^ t)+ _k[E_ _1∇ F( w_k^t, _k^t)- (50) ∇F(kt,χ¯kt′)]−ηk[∇f(kt,ξkt)−π1∇F(kt,χ¯kt)], ∇ F( w_k^t, χ_k^ t)]- _k[∇ f( w_k^t, _k^t)-E_ _1∇ F( w_k^t, χ_k^t)], where F is a local loss function, η>0η>0 is the local learning rate, and χ is a sample uniformly chosen from the local dataset. Now if we will consider π1∇F(kt,χkt)−∇F(kt,χ¯kt′)E_ _1∇ F( w_k^t, _k^t)-∇ F( w_k^t, χ_k^ t) as bias BkB_k and ∇f(kt,ξkt)−π1∇F(kt,χ¯kt)∇ f( w_k^t, _k^t)-E_ _1∇ F( w_k^t, χ_k^t) as noise NkN_k, the equation becomes: ¯kt+1=kt−ηk∇F(kt,χkt′)+ηkBk−ηkNk, w_k^t+1= w_k^t- _k∇ F( w_k^t, _k^ t)+ _kB_k- _kN_k, (51) By Lemma 3, we know [‖Bt‖22]≤C1logDt+log1δDtE[||B_t||_2^2]≤ C_1 logDt+log 1δDt (52) By Assumption 4, we have [‖Nt‖22]≤σ2E[||N_t||_2^2]≤σ^2 (53) By proof of Lemma 2, we know that [F(kt+1,χkt′)−F(kt,χkt′)] [F( w_k^t+1, _k^ t)-F( w_k^t, _k^ t)] (54) ≤−ηk2‖∇F(kt,χkt′)‖22+ηk2C1logDt+log1δDt+ηk22Lhσ2 ≤- _k2||∇ F( w_k^t, _k^ t)||_2^2+ _k2C_1 logDt+log 1δDt+ _k^22L_hσ^2 and after multiplying by 2, 2[F(kt+1,χkt′)−F(kt,χkt′)] 2E[F( w_k^t+1, _k^ t)-F( w_k^t, _k^ t)] (55) ≤−ηk‖∇F(kt,χkt′)‖22+ηkC1logDt+log1δDt+ηk2Lhσ2 ≤- _k||∇ F( w_k^t, _k^ t)||_2^2+ _kC_1 logDt+log 1δDt+ _k^2L_hσ^2 after rearranging, ηk‖∇F(kt,χkt′)‖22 _k||∇ F( w_k^t, _k^ t)||_2^2 (56) ≤−2[F(kt+1,χkt′)−F(kt,χkt′)]+ηkC1logDt+log1δDt+ ≤-2E[F( w_k^t+1, _k^ t)-F( w_k^t, _k^ t)]+ _kC_1 logDt+log 1δDt+ ηk2Lhσ2 10000\ 10000\ 10000\ 10000\ _k^2L_hσ^2 noting that F(kt,χkt′)≤minxinF(,χkt′)F( w_k^t, _k^ t)≤ _xin XF( w, _k^ t) etak‖∇F(kt,χkt′)‖22 eta_k||∇ F( w_k^t, _k^ t)||_2^2 (57) ≤2(F(1,χkt′)−minx∈F(,χkt′))+ηkC1logDt+log1δDt+ ≤ 2(F( w_1, _k^ t)- _x∈ XF( w, _k^ t))+ _kC_1 logDt+log 1δDt+ ηk2Lhσ2 10000\ 10000\ 10000\ 10000\ _k^2L_hσ^2 summing over k from 1 to K, ∑k=1ηk‖∇F(kt,χkt′)‖22 _k=1^K _k||∇ F( w_k^t, _k^ t)||_2^2 (58) ≤2(F(1,χkt′)−minxinF(,χkt′))+C1∑k=1ηklogDt+log1δDt+ ≤ 2(F( w_1, _k^ t)- _xin XF( w, _k^ t))+C_1 _k=1^K _k logDt+log 1δDt+ σ2∑k=1ηk2 10000\ 10000\ 10000\ 10000\ σ^2 _k=1^K _k^2 Dividing both sides by ∑k=1ηk _k=1^K _k, 1∑k=1ηk∑k=1ηk‖∇F(kt,χkt′)‖22 1 _k=1^K _k _k=1^K _k||∇ F( w_k^t, _k^ t)||_2^2 (59) ≤1∑k=1ηk[2(F(1,χkt′)−minxinF(,χkt′))+ ≤ 1 _k=1^K _k [2(F( w_1, _k^ t)- _xin XF( w, _k^ t))+ C1∑k=1ηklogDt+log1δDt+Lhσ2∑k=1ηk2] 10000\ 10000\ 10000\ 10000\ C_1 _k=1^K _k logDt+log 1δDt+L_hσ^2 _k=1^K _k^2 ] noting that 1∑k=1ηk∑k=1ηk‖∇F(kt,χkt′)‖22=‖∇F(zT,χkt′)‖22 1 _k=1^K _k _k=1^K _k||∇ F( w_k^t, _k^ t)||_2^2=E||∇ F(z_T, _k^ t)||_2^2, ‖∇F(zT,χkt′)‖22 ||∇ F(z_T, _k^ t)||_2^2 (60) ≤1∑k=1ηk[2(F(1,χkt′)−minxinF(,χkt′))+ ≤ 1 _k=1^K _k [2(F( w_1, _k^ t)- _xin XF( w, _k^ t))+ C1∑k=1ηklogDt+log1δDt+Lhσ2∑k=1ηk2] 10000\ 10000\ 10000\ 10000\ C_1 _k=1^K _k logDt+log 1δDt+L_hσ^2 _k=1^K _k^2 ] (i) if the step size (η) satisfies ηk=aK,∀k≤ _k= a K,∀ k , for some constant a<Lha< KL_h. Note that ∑k=11t≤log+1 _k=1^K 1t≤ logK+1 and ∑k=1logk≤log(log+1) _k=1^K logkk≤ log(logK+1). Then Θ1≜[‖∇F(z,χkt′)‖22] _1 [||∇ F(z_K, _k^ t)||_2^2] (61) ≤2(F(1,χkt′)−minx∈F(,χkt′))a+ ≤ 2(F( w_1, _k^ t)- _x∈ XF( w, _k^ t))a K+ C1(logD−logδ)(log+1)LhD+C1log(log+1)LhD 10000\ 10000\ 10000\ 10000\ C_1(logD-logδ)(logK+1)L_hDK+ C_1logK(logK+1)L_hDK =2(F(1,χkt′)−minx∈F(,χkt′))a+C1(logD−logδ)LhD+ = 2(F( w_1, _k^ t)- _x∈ XF( w, _k^ t))a K+ C_1(logD-logδ)L_hDK+ C1(logD−logδ)logLhD+C1log2LhD+Lhaσ2 10000\ 10000\ 10000\ 10000\ C_1(logD-logδ)logKL_hDK+ C_1log^2KL_hDK+ L_haσ^2 K (i) if the step size (η) satisfies ηk=ak,∀k≤ _k= ak,∀ k , for some constant a<1Lha< 1L_h. Let M=∑k=11kM_K= _k=1^K 1k. Note that ∑k=1logk2<∑k=1∞logk2=π26(12lnA−γ−ln2π)<1 _k=1^K logkk^2< _k=1^∞ logkk^2= π^26(12lnA-γ-ln2π)<1 (62) where the Glaisher-Kinkelin constant A≈1.28A≈ 1.28 and the Euler-Mascheroni constant γ≈0.58γ≈ 0.58. Then we have Θ2≜[‖∇F(z,χkt′)‖22] _2 [||∇ F(z_K, _k^ t)||_2^2] (63) ≤2(F(1,χkt′)−minx∈F(,χkt′))aM+ ≤ 2(F( w_1, _k^ t)- _x∈ XF( w, _k^ t))aM_K+ C1M∑k=1logDk+log1δDk2+∑k=1Lhaσ2Mt2 10000\ 10000\ 10000\ 10000\ C_1M_K _k=1^K logDk+log 1δDk^2+ _k=1^K L_haσ^2M_Kt^2 ≤[2(F(1,χkt′)−minx∈F(,χkt′))a+ ≤ [ 2(F( w_1, _k^ t)- _x∈ XF( w, _k^ t))a+ 6C1+π2C1(logD−logδ)+π2Lhaσ266D]1log 10000\ 10000\ 10000\ 10000\ 6C_1+π^2C_1(logD-logδ)+ π^2L_haσ^266D ] 1logK (i) if the step size (η) satisfies ηk=ak,∀k≤ _k= a k,∀ k , for some constant a<1Lha< 1L_h. Let Qt=∑k=11kQ_t= _k=1^K 1 k. Note that ∑k=1∞1kk=ζ(1.5)≈2.61≤3 _k=1^∞ 1k k=ζ(1.5)≈ 2.61≤ 3, ∑t=1∞logkk<4,∑k=11k≥ _t=1^∞ logkk k<4, _k=1^K 1 k≥ K, where ζ(.)ζ(.) is the Riemann’s zeta function. Then we have Θ3≜[‖∇F(z,χkt′)‖22] _3 [||∇ F(z_K, _k^ t)||_2^2] (64) ≤2(F(1,χkt′)−minx∈F(,χkt′))aQ+ ≤ 2(F( w_1, _k^ t)- _x∈ XF( w, _k^ t))aQ_K+ C1(logD−logδ)DQ∑k=11kk+C1DQ∑k=1logkk+ 10000\ 10000\ 10000\ 10000\ C_1(logD-logδ)DQ_K _k=1^K 1k k+ C_1DQ_K _k=1^K logkk k+ Lhaσ2Q∑k=11k 10000\ 10000\ 10000\ 10000\ L_haσ^2Q_K _k=1^K 1k ≤[2(F(1,χkt′)−minx∈F(,χkt′))a+ ≤[ 2(F( w_1, _k^ t)- _x∈ XF( w, _k^ t))a K+ 3C1(logD−logδ)+4C1D+Lhaσ2]+Lhaσ2log 10000\ 10000\ 10000\ 10000\ 3C_1(logD-logδ)+4C_1D K+ L_haσ^2 K]+ L_haσ^2logK K IV-F Proof of Theorem 38 From the SGD update rule ¯kt+1=¯kt−ηkgkt+v¯kt w_k^t+1= w_k^t- _kg_k^t+ v_k^t and ‖a+b‖2≤2‖a‖2+2‖b‖2||a+b||^2≤ 2||a||^2+2||b||^2 for two real valued vectors a and b, we have ‖¯kt+1−∗‖2 || w_k^t+1- w^*||^2 =‖¯kt−ηkgkt+v¯kt−∗‖2 =|| w_k^t- _kg_k^t+ v_k^t- w^*||^2 (65) ≤‖¯kt−ηkgkt−∗‖2⏟(A)+‖v¯kt‖2 ≤ || w_k^t- _kg_k^t- w^*||^2_(A)+|| v_k^t||^2 We now focus on the bounding term (A)(A) in 65. We have ‖¯kt−ηkgkt−∗‖2=‖¯kt−ηkgkt−∗−ηkg¯kt+ηkg¯kt‖2 || w_k^t- _kg_k^t- w^*||^2=|| w_k^t- _kg_k^t- w^*- _k g_k^t+ _k g_k^t||^2 (66) =||(¯kt−∗−ηkg¯kt||2+2ηk⟨¯kt−∗−ηkg¯kt,g¯kt−gkt⟩+ =||( w_k^t- w^*- _k g_k^t||^2+2 _k w_k^t- w^*- _k g_k^t, g_k^t-g_k^t + ηk2‖gkt−g¯kt‖2 10000\ 10000\ 10000\ 10000\ _k^2||g_k^t- g_k^t||^2 =||(¯kt−∗−ηkg¯kt||2⏟(B)+ηk2‖gkt−g¯kt‖2, = ||( w_k^t- w^*- _k g_k^t||^2_(B)+ _k^2||g_k^t- g_k^t||^2, where ⟨¯kt−∗−ηkg¯kt,g¯kt−gkt⟩=0 w_k^t- w^*- _k g_k^t, g_k^t-g_k^t =0. We now focus on bounding term (B)(B). We have ||(¯kt−∗−ηkg¯kt||2 ||( w_k^t- w^*- _k g_k^t||^2 (67) =‖¯kt−∗‖2+ηk2‖g¯kt‖2−2ηk1N∑n∈⟨¯kt−∗,∇F(n,kt)⟩ =|| w_k^t- w^*||^2+ _k^2|| g_k^t||^2-2 _k 1N _n w_k^t- w^*,∇ F( w_n,k^t) ≤‖¯kt−∗‖2+ηk21N∑n∈‖∇F(n,kt)‖2− ≤|| w_k^t- w^*||^2+ _k^2 1N _n ||∇ F( w_n,k^t)||^2- 2ηk1N∑n∈⟨¯kt−n,kt+n,kt−∗,∇F(n,kt)⟩ 10000 10000 10000 100002 _k 1N _n w_k^t- w_n,k^t+ w_n,k^t- w^*,∇ F( w_n,k^t) ≤‖¯kt−∗‖2+2ηk2LN∑n∈(F(n,kt)−F∗)− ≤|| w_k^t- w^*||^2+2 _k^2 LN _n (F( w_n,k^t)-F^*)- 2ηk1N∑n∈⟨¯kt−n,kt,∇F(n,kt)⟩ 10000 10000 10000 100002 _k 1N _n w_k^t- w_n,k^t,∇ F( w_n,k^t) −2ηk1N∑n∈⟨n,kt−∗,∇F(n,kt)⟩, 10000\ 10000\ 10000\ 10000\ -2 _k 1N _n w_n,k^t- w^*,∇ F( w_n,k^t) , where in the first inequality we applied ‖∑n∈zn‖2≤N∑n∈‖zn‖2|| _n z_n||^2≤ N _n ||z_n||^2 , and in the second inequality we applied L-smoothness ‖∇F(n,kt)‖2≤2L(F(n,kt)−F∗)||∇ F( w_n,k^t)||^2≤ 2L(F( w_n,k^t)-F^*) . For the third term in 67, by using the Cauchy–Schwarz inequality and arithmetic and geometric means (AM-GM) inequality: 2⟨a,b⟩≤1ε‖a‖2+ε‖b‖22 a,b ≤ 1 ||a||^2+ ||b||^2 for ε>0 >0, we have −2⟨¯kt−n,kt,∇F(n,kt)⟩ -2 w_k^t- w_n,k^t,∇ F( w_n,k^t) (68) =2⟨n,kt−¯kt,∇F(n,kt)⟩ 10000\ 10000\ 10000\ 10000\ 10000\ 10000\ 10000\ =2 w_n,k^t- w_k^t,∇ F( w_n,k^t) ≤1ηk‖n,kt−¯kt‖2+ηk‖∇F(n,kt)‖2 10000\ 10000\ 10000\ 10000\ 10000\ 10000\ 10000\ ≤ 1 _k|| w_n,k^t- w_k^t||^2+ _k||∇ F( w_n,k^t)||^2 ≤1ηk‖n,kt−¯kt‖2+2ηkL(F(n,kt)−F∗). 10000\ 10000\ 10000\ 10000\ 10000\ 10000\ 10000\ ≤ 1 _k|| w_n,k^t- w_k^t||^2+2 _kL(F( w_n,k^t)-F^*). For the last term in 67, by using μ-strong convexity, we have −⟨n,kt−∗,∇F(n,kt)⟩ - w_n,k^t- w^*,∇ F( w_n,k^t) (69) ≤−(F(n,kt)−F∗)−μ2‖n,kt−∗‖2. ≤-(F( w_n,k^t)-F^*)- μ2|| w_n,k^t- w^*||^2. Therefore, 67 can be rewritten as ||(¯kt−∗−ηkg¯kt||2 ||( w_k^t- w^*- _k g_k^t||^2 (70) ≤‖¯kt−∗‖2+2ηk2LN∑n∈(F(n,kt)−F∗)+ ≤|| w_k^t- w^*||^2+2 _k^2 LN _n (F( w_n,k^t)-F^*)+ ηk1N∑n∈(1ηk‖n,kt−¯kt‖2+2ηkL(F(n,kt)−F∗))− 10000\ 10000\ 10000\ 10000\ _k 1N _n ( 1 _k|| w_n,k^t- w_k^t||^2+2 _kL(F( w_n,k^t)-F^*) )- 2ηk1N∑n∈(F(n,kt)−F∗)−μηk1N∑n∈μ2‖n,kt−∗‖2 10000 10000 10000 100002 _k 1N _n (F( w_n,k^t)-F^*)-μ _k 1N _n μ2|| w_n,k^t- w^*||^2 ≤‖¯kt−∗‖2+2ηk(2ηkL−1)1N∑n∈(F(n,kt)−F∗)+ ≤|| w_k^t- w^*||^2+2 _k(2 _kL-1) 1N _n (F( w_n,k^t)-F^*)+ 1N∑n∈‖¯kt−n,kt‖2−μηk1N∑n∈‖n,kt−∗‖2 10000\ 10000\ 10000\ 10000\ 1N _n || w_k^t- w_n,k^t||^2-μ _k 1N _n || w_n,k^t- w^*||^2 =(1−μηk)‖¯kt−∗‖2+2ηk(2ηkL−1)1N =(1-μ _k)|| w_k^t- w^*||^2+2 _k(2 _kL-1) 1N ∑n∈(F(n,kt)−F∗)+1N∑n∈‖¯kt−n,kt‖2, 10000\ 10000\ 10000\ 10000\ _n (F( w_n,k^t)-F^*)+ 1N _n || w_k^t- w_n,k^t||^2, where we used the fact: 1N∑n∈‖n,kt−∗‖2=‖¯kt−∗‖2 1N _n || w_n,k^t- w^*||^2=|| w_k^t- w^*||^2. We assume ηk≤14L _k≤ 14L, it holds ηkL≤14⟹2ηkL−1≤−12 _kL≤ 14 2 _kL-1≤- 12. Thus ||(¯kt−∗−ηkg¯kt||2 ||( w_k^t- w^*- _k g_k^t||^2 (71) ≤(1−μηk)‖¯kt−∗‖2+1N∑n∈‖¯kt−n,kt‖2− ≤(1-μ _k)|| w_k^t- w^*||^2+ 1N _n || w_k^t- w_n,k^t||^2- 121N∑n∈(F(n,kt)−F∗) 10000\ 10000\ 10000\ 10000\ 12 1N _n (F( w_n,k^t)-F^*) ||(¯kt−∗−ηkg¯kt||2 ||( w_k^t- w^*- _k g_k^t||^2 (72) ≤(1−μηk)‖¯kt−∗‖2+1N∑n∈‖¯kt−n,kt‖2− ≤(1-μ _k)|| w_k^t- w^*||^2+ 1N _n || w_k^t- w_n,k^t||^2- 12[‖∇F(z,χkt′)‖22]⏟(C) 10000\ 10000\ 10000\ 10000\ 12E[||∇ F(z_K, _k^ t)||_2^2]_(C) We can bound (C)(C) using equation 61, 63, and 64 [‖∇F(z,χkt′)‖22]=min(Θ1,Θ2,Θ3)E[||∇ F(z_K, _k^ t)||_2^2]= ( _1, _2, _3) (73) where the first inequality results from the convexity of Fn(.)F_n(.), the second inequality is derived from the AM-GM inequality, and the third inequality results from the smoothness of Fn(.)F_n(.). Therefore, 72 is further expressed as ||(¯kt−∗−ηkg¯kt||2 ||( w_k^t- w^*- _k g_k^t||^2 (74) ≤(1−μηk)‖¯kt−∗‖2+1N∑n∈‖¯kt−n,kt‖2+ ≤(1-μ _k)|| w_k^t- w^*||^2+ 1N _n || w_k^t- w_n,k^t||^2+ 14ηk1N∑n∈‖n,kt−¯kt‖2+12min(Θ1,Θ2,Θ3) 10000\ 10000\ 10000\ 10000\ 14 _k 1N _n || w_n,k^t- w_k^t||^2+ 12 ( _1, _2, _3) By plugging LABEL:equa:subsubC-final into 65 and taking expectation we obtain ‖¯kt+1−∗‖2 || w_k^t+1- w^*||^2 (75) ≤||(1−μηk)||¯kt−∗||2+1N∑n∈‖¯kt−n,kt‖2+ ≤||(1-μ _k)|| w_k^t- w^*||^2+ 1N _n || w_k^t- w_n,k^t||^2+ 14ηk1N∑n∈‖n,kt−¯kt‖2+12min(Θ1,Θ2,Θ3)+ 10000\ 10000\ 10000\ 10000\ 14 _k 1N _n || w_n,k^t- w_k^t||^2+ 12 ( _1, _2, _3)+ ηk2‖gkt−g¯kt‖2 10000\ 10000\ 10000\ 10000\ _k^2||g_k^t- g_k^t||^2 From Lemmas 5, 6, and 7, we have ‖¯kt+1−∗‖2 || w_k^t+1- w^*||^2 (76) ≤(1−μηk)‖¯kt−∗‖2+4(1+1ηk)ηkTB2+ηk2σg2N2 ≤(1-μ _k)E|| w_k^t- w^*||^2+4 (1+ 1 _k ) _kTB^2+ _k^2 _g^2N^2 Let us define Ykt=‖¯kt−∗‖2Y_k^t=E|| w_k^t- w^*||^2 and Φk=4(ηk+1ηk2)TB2+σg2N2 _k=4 ( _k+1 _k^2 )TB^2+ _g^2N^2, from 76 we have ∑t=1TYkt+1≤∑t=0T−1(1−μηk)Ykt+ηk2Φk, _t=1^TY_k^t+1≤ _t=0^T-1(1-μ _k)Y_k^t+ _k^2 _k, (77) By Yk=∑t=0T−1YktY_k= _t=0^T-1Y_k^t, 77 is rewritten as Ykt+1≤(1−μηk)Ykt+ηk2Φk,Y_k^t+1≤(1-μ _k)Y_k^t+ _k^2 _k, (78) We define a diminishing stepsize ηk=4θk+ω _k= 4θk+ω for some θ>14μθ> 14μ and ω>0ω>0. By defining m=maxθ2Φk4θμ−1,(ω+1)Y0m= \ θ^2 _k4θμ-1,(ω+1)Y_0\, we prove that Yk≤mk+ωY_k≤ mk+ω by induction. Due to 4θμ>14θμ>1, from 78 we have Yk+1 Y_k+1 =(1−4θμk+ω)mk+ω+16θ2(k+ω)2Φk = (1- 4θμk+ω ) mk+ω+ 16θ^2(k+ω)^2 _k (79) ≤k+ω−1(k+ω)2m+16θ2(k+ω)2Φk ≤ k+ω-1(k+ω)^2m+ 16θ^2(k+ω)^2 _k ≤k+ω−1(k+ω)2m+16θ2(k+ω)2Φk−4θμ−1(k+ω)2 ≤ k+ω-1(k+ω)^2m+ 16θ^2(k+ω)^2 _k- 4θμ-1(k+ω)^2 ≤k+ω−1(k+ω)2m−4θμ−1(k+ω)2 ≤ k+ω-1(k+ω)^2m- 4θμ-1(k+ω)^2 ≤k+ω−4θμ(k+ω)2m ≤ k+ω-4θμ(k+ω)^2m ≤k+ω−4θμ(k+ω)2−(4θμ)2m ≤ k+ω-4θμ(k+ω)^2-(4θμ)^2m 1k+ω+4θμm 1k+ω+4θμm ≤1k+ω+1m ≤ 1k+ω+1m We choose θ=4μθ= 4μ and ω=Lμω= Lμ , it follows that m m =maxθ2Φk4θμ−1,(ω+1)Y0 = \ θ^2 _k4θμ-1,(ω+1)Y_0\ (80) ≤θ2Φk4θμ−1+(ω+1)Y0 ≤ θ^2 _k4θμ-1+(ω+1)Y_0 =16Φk15μ2+(Lμ+1)Y0 = 16 _k15μ^2+ ( Lμ+1 )Y_0 By using the L-smoothness of F(.)F(.), we have [F(¯k)]−F∗ [F( w_k) ]-F^* (81) ≤L2Yk≤L2m(k+ω) ≤ L2Y_k≤ L2 m(k+ω) ≤L2(k+L/μ)[16Φk15μ2+(Lμ+1)‖0−∗‖2] ≤ L2(k+L/μ) [ 16 _k15μ^2+ ( Lμ+1 )E|| w_0- w^*||^2 ] Finally, by applying 81 recursively, the convergence bound of our approach after K global communication rounds can be given as [F(K)]−F∗ [F( w_K) ]-F^* (82) ≤L2(K+L/μ)[16ΦK15μ2+(Lμ+1)‖0−∗‖2], ≤ L2(K+L/μ) [ 16 _K15μ^2+ ( Lμ+1 )E|| w_0- w^*||^2 ], which completes the proof. Interpretation of the bound: In terms of the step size, stochastic gradient variance, and data-heterogeneity factors, the derived upper limit describes the predicted optimality gap of the global model following T communication rounds, i.e., [F(wT)]−F(w⋆)E[F(w_T)]-F(w ) (or an analogous global loss gap). The bound lowers with T (often showing a O(1/T)O(1/T) dependency) under conventional smoothness and bounded-variance assumptions, indicating better global performance as the number of rounds rises. We emphasize that the constraint is not claimed to be tight; rather, it acts as a convergence guarantee for Meta-BayFL under probabilistic personalized updates. Even though Theorem 1 provides convergence guarantees for different learning rate schedules, practical observations show that the meta-learned adaptive schedule is used in real life because it consistently produces faster convergence and stronger stability in experimental situations that are not IID. V Simulations and Performance Evaluation V-A Environment Settings For our Meta-BayFL approach, we only consider non-IID data. Clients have different data sizes, data points, number of features, and batch sizes. The total number of clients is five, and we have only considered the non-IID data. For our simulation, we have used three datasets: i) CIFAR-10 [krizhevsky2010convolutional], i) CIFAR-100 [krizhevsky2009learning], and i) Tiny-ImageNet [le2015tiny] datasets to calculate the performance of our proposed method. Non-i.i.d. Type U |||U| ConvNet ResNet20 ResNet32 ResNet44 ResNet56 Step CIFAR-10 10K 65.5± 0.52 67.5± 0.52 62.7± 0.47 55.5± 0.42 57.0± 0.55 CIFAR-100 50K 65.4± 0.48 68.2± 0.48 62.3± 0.33 55.1± 0.41 51.0± 0.59 Tiny-ImageNet 100K 65.5± 0.60 67.1± 0.61 62.3± 0.53 54.5± 0.41 59.0± 0.42 Dirichlet CIFAR-10 10K 63.9± 0.55 68.2± 0.46 67.7± 0.55 57.0± 0.48 57.0± 0.42 CIFAR-100 50K 63.5± 0.51 68.6± 0.53 66.5± 0.41 62.0± 0.27 56.9± 0.47 Tiny-ImageNet 100K 64.0± 0.55 68.2± 0.52 56.0± 0.62 61.6± 0.76 57.3± 0.42 Table I: Meta-BayFL on non-i.i.d CIFAR-10 with different unlabeled data U. rows correspond to non-IID types (Step/Dirichlet) and choices of unlabeled data source and size; entries report accuracy (mean ± std) for each backbone. Out-of-domain unlabeled data (e.g., CIFAR-100/Tiny-ImageNet) is sufficient to maintain or improve performance, supporting the practicality of the approach. Data CIFAR-10 CIFAR-100 Tiny-ImageNet FedAVG BayFL FedAVG BayFL FedAVG BayFL |D||D| = 100% ϵε = 0.0 66.86% 67.20% 71.25% 71.23% 73.11% 73.59% ϵε = 0.0001 66.71% 66.81% 71.22% 71.20% 73.05% 73.50% ϵε = 0.001 65.69% 66.69% 69.50% 71.05% 72.20% 73.44% ϵε = 0.01 63.10% 66.52% 67.98% 70.60% 70.66% 73.20% ϵε = 0.1 59.23% 65.65% 64.40% 70.00% 67.84% 72.19% |D||D| = 50% ϵε = 0.0 58.28% 58.31% 59.98% 60.03% 60.45% 60.51% ϵε = 0.0001 56.73% 57.14% 58.72% 59.49% 58.66% 59.99% ϵε = 0.001 53.47% 56.71% 56.50% 59.33% 56.84% 59.33% ϵε = 0.01 51.70% 55.78% 54.67% 58.20% 55.45% 58.70% ϵε = 0.1 50.01% 54.65% 52.07% 57.01% 52.77% 57.55% |D||D| = 25% ϵε = 0.0 49.39% 49.78% 48.32% 49.83% 51.22% 51.20% ϵε = 0.0001 48.31% 49.88% 48.44% 49.49% 50.12% 50.92% ϵε = 0.001 47.47% 49.01% 46.05% 49.33% 47.78% 50.20% ϵε = 0.01 45.70% 48.78% 43.68% 48.19% 45.80% 49.66% ϵε = 0.1 42.99% 48.65% 40.70% 47.02% 43.31% 49.19% |D||D| = 10% ϵε = 0.0 42.40% 45.78% 42.32% 44.83% 41.23% 47.02% ϵε = 0.0001 42.18% 44.88% 42.44% 45.49% 40.87% 46.44% ϵε = 0.001 41.47% 44.51% 41.50% 45.33% 40.05% 46.04% ϵε = 0.01 40.70% 44.38% 40.67% 43.70% 40.01% 45.50% ϵε = 0.1 39.91% 44.05% 38.06% 44.00% 37.98% 44.99% Table I: Difference in performance with different data size |D||D| and noise level ϵε across various datasets for 5 non-IID Dirichlet clients. Each block fixes |D||D| and sweeps ϵε; values compare FedAVG vs. BayFL on each dataset. The BayFL advantage widens as data becomes smaller and noisier, showing stronger robustness in challenging federated environments. Fig. 3: Performance comparison on the CIFAR-10 non-IID dataset between basic CNN, FedAVG (FL with CNN), and BayFL (FL with BNN) approaches. A total of 5 clients with various data distributions and batch sizes are used for FL approaches with 10 local epochs in every global round. V-B Non-IID Data Distribution Setup We focus on CIFAR-10, where 10,000 training photos are stored as unlabeled data on the server. The remaining photos are given to ten clients in two non-identifiable scenarios. According to the strategy proposed by Cao et al. [cao2019learning], in the 'Step' scenario, each client receives 10 images for each of the eight minor classes and 1,960 images for each of the two major classes. In the 'Dirichlet' scenario, suggested in Hsu et al. [hsu2019measuring], we apply a heterogeneous partitioning method for N clients across C classes. For each class c∈Cc∈ C, we create an N-dimensional vector qcq_c using a Dirichlet distribution with parameter 0.1 and distribute the photos to client n based on the fraction of qc[n]q_c[n]. This results in clients having differing total amounts of images. V-C Simulation Results Comparison with baseline. First, we compare the performance between basic CNN, FedAVG (basic FL with CNN), and BayFL (FL with BNN) for 5 clients non-IID data in Fig. 3. The figure shows that both FL (FedAVG and BayFL) significantly outperform CNN. The accuracy results between FedAVG and BayFL are extremely close however, in contrast to FedAVG, BayFL is extremely smooth in the non-IID scenario. Thus, we may say that BNN is more capable than CNN at handling non-IID datasets for the FL settings. However, in the initial epochs, the performance of the BNN approach was worse than that of the CNN approach. It is because in order to begin using Bayesian approaches, one must first create a prior distribution p(n,k(t))p( w_n,k^(t)) on the weights that represent the degree of uncertainty over their accurate values [shridhar2019comprehensive]. The posterior distributions q(n,k(t)|D)q( w_n,k^(t)|D) are still in the early stages of training and have not yet converged to more peaked distributions around the most likely weights. Because of the initial high level of uncertainty, more conservative projections may result from poorly calibrated data, which usually reduces initial accuracy. Effects on unlabeled data. We investigate the situation in which the unlabeled data comes from a different domain or task. The setup is intended to imitate scenarios in which (a) the server has little knowledge of the client's data and (b) the server is unable to collect unlabeled data that accurately represents the test data. In Table I, we replace the unlabeled data with CIFAR-100 and Tiny-ImageNet. The findings reveal that accuracy remains equivalent to, or even surpasses, that of CIFAR-10, implying that out-of-domain unlabeled data are sufficient for Meta-BayFL. Effects on challenging environment. However, a distinction is beginning to emerge between the probabilistic approach (BayFL) and the deterministic approach (FedAVG) in Table I when we employ FedAVG and BayFL on small and noisy different datasets. The results demonstrate that when the dataset size or noise level drops, BayFL consistently surpasses FedAVG, and the performance difference widens. BayFL outperforms FedAVG by 4.14% on CIFAR-10, 5.94% on CIFAR-100, and 7.01% on Tiny-ImageNet when |D|=10%|D|=10\% and ϵ=0.1ε=0.1, respectively. Similarly, for |D|=25%|D|=25\% and ϵ=0.1ε=0.1, BayFL retains a lead of 5.66%, 6.32%, and 5.88%, respectively. These findings show that BayFL manages uncertainty and limited data more effectively offering it a viable option for federated learning in challenging environments. Variables CIFAR-10 CIFAR-100 Tiny-ImageNet ϵε 0.001 BayFL, lrlr = 0.0001 66.69% 71.05% 73.44% BayFL, lrlr = 0.001 66.90% 71.66% 73.75% BayFL, lrlr = 0.01 66.02% 70.76% 73.41% Meta-BayFL 72.25% 73.06% 77.00% ϵε 0.01 BayFL, lrlr = 0.0001 65.40% 70.05% 71.34% BayFL, lrlr = 0.001 65.60% 70.33% 71.30% BayFL, lrlr = 0.01 65.12% 70.35% 71.25% Meta-BayFL 71.67% 72.88% 76.32% ϵε 0.1 BayFL, lrlr = 0.0001 62.48% 68.80% 70.98% BayFL, lrlr = 0.001 62.48% 69.20% 71.05% BayFL, lrlr = 0.01 62.75% 69.02% 70.90% Meta-BayFL 70.55% 71.39% 75.81% Table I: Difference in performance with different noise level ϵε and BayFL with different learning rates lrlr across various datasets for 5 non-IID Dirichlet clients. For each ϵε, we compare fixed learning rates against Meta-BayFL. Meta-learning is consistently best and degrades more gracefully under increasing noise. Advantage of meta-learning. For our meta-learning approach, Table I shows the comparison of results with and without meta-learning for different noise values across all datasets (CIFAR-10, CIFAR-100, and Tiny-ImageNet). Meta-BayFL consistently beats BayFL for all datasets and noise levels (ϵε = 0.001, 0.01, 0.1). As noise grows, accuracy decreases for all models, but Meta-BayFL retains a greater accuracy, proving its robustness and resilience to noise. Among BayFL models, a learning rate of 0.001 outperforms 0.0001 and 0.01 in most circumstances. The accuracy reduction is particularly dramatic at ϵ=0.1ε=0.1, when the basic BayFL models struggle, but Meta-BayFL still performs competitively. Overall, Meta-BayFL outperforms other approaches, notably in terms of noise management and accuracy stability. Environment Method Acc (%) Time/Round (s) Simulated FedAvg / FedProx 66.85 8.60 Meta-BayFL (MC Dropout) 67.10 8.72 Meta-BayFL (BNN) 69.48 11.00 Raspberry Pi FedAvg / FedProx 66.10 27.8 Meta-BayFL (MC Dropout) 66.40 29.4 Meta-BayFL (BNN) 68.90 37.6 Table IV: Ablation study comparing uncertainty approximation methods under non-IID CIFAR-10 (Dirichlet α=0.5α=0.5, 5 clients, ResNet-56 backbone, same training budget). Results are reported for both simulated execution and real Raspberry Pi (CPU-only) deployment. Runtime denotes average client-side training time per communication round. Ablation Study on Uncertainty Approximation. We conduct an ablation comparing complete Bayesian neural networks with Monte Carlo dropout under identical non-IID scenarios in Table IV in order to investigate the trade-off between computational efficiency and uncertainty modeling fidelity. The full BNN-based Meta-BayFL achieves higher accuracy and more stable convergence, justifying its application in extremely heterogeneous and noisy federated situations, while MC dropout shortens runtime. Due to Bayesian parameterization, Meta-BayFL has a little longer runtime than FedAvg/FedProx, but it achieves a notably greater accuracy for strongly non-IID data. The Raspberry Pi results confirm consistent behavior on realistic CPU-only edge hardware by showing the same relative accuracy and runtime ordering as the simulated setting. Clients CIFAR-10 CIFAR-100 Tiny-ImageNet 5 72.25% 73.06% 77.00% 10 73.10% 74.25% 78.02% 15 73.55% 74.80% 78.40% 20 74.02% 75.10% 78.85% Table V: Performance of Meta-BayFL under different numbers of non-IID Dirichlet clients with noise level ϵ=0.001ε=0.001. Each row changes client count while keeping the noise fixed; entries are final test accuracy. Meta-BayFL scales positively with more clients (more diversity), with the expected trade-off of higher system overhead discussed in Section V. Scalability with number of clients. Table V shows Meta-BayFL's performance with varying client counts (5, 10, 15, and 20) with a fixed noise level (ϵ=0.001ε=0.001). We see a noticeable improvement in accuracy as the number of clients grows across all datasets. This tendency may be ascribed to the increasing data diversity and richer representation that result from more clients contributing to the global model, which enhances generalization performance. For example, using CIFAR-10, accuracy increases from 72.25% with 5 clients to 74.02% with 20 clients. Similarly, CIFAR-100 and Tiny-ImageNet exhibit consistent gains, achieving 75.10% and 78.85% accuracy, respectively, with 20 clients. These findings illustrate Meta-BayFL's capacity to scale to a larger federated population while retaining robust performance in the face of uncertain and diverse data. However, we observe that increasing the number of clients may result in greater communication costs and synchronization delay, implying a trade-off between accuracy improvements and system overhead. Table VI: Comparison with state-of-the-art approaches. Method CIFAR-10 CIFAR-100 Tiny-ImageNet Loss Value Accuracy Loss Value Accuracy Loss Value Accuracy FedAVG [li2019convergence] 4.35 45.50% 4.42 37.95% 4.43 36.55% BNN [jospin2022hands] 2.53 52.22% 2.89 48.48% 2.75 50.18% pFedBayes [zhang2022personalized] 1.94 64.27% 2.33 65.31% 2.10 63.51% pFedBe [yu2025pfedbl] 1.92 64.86% 1.99 64.03% 2.13 63.11% Fedmask [li2021fedmask] 1.67 66.25% 1.88 65.50% 1.85 66.33% Meta-BayFL 1.28 69.48% 1.33 70.20% 1.47 73.75% Comparison with existing approaches. Finally, we have compared all our approaches with all existing methods for both noisy and small data in Table VI. For comparison, we have selected FedAVG [li2019convergence] as the baseline FL method, BNN [jospin2022hands] for the probabilistic approach, FedBE [chen2020fedbe] and FedFomo [zhang2020personalized] for federated probabilistic approach, and Fedmask [li2021fedmask] for PFL with BNN approach to compare with our Meta-BayFL approach across all datasets (CIFAR-10, CIFAR-100, and Tiny-ImageNet) in terms of both accuracy and loss value. Despite the fact that both FedMask and Meta-BayFL use Bayesian components, Meta-BayFL performs better due to the synergistic combination of meta-learned personalization and Bayesian uncertainty modeling, which allows for client-adaptive and uncertainty-aware optimization in the presence of heterogeneous data. The meta-learning component allows the model to generalize more effectively among varied federated clients, while the Bayesian framework provides robust uncertainty estimation, mitigating the impact of noise and small sample numbers. Meta-BayFL outperforms Fedmask, the second-best performing approach, by 3.23% in CIFAR-10, 4.70% in CIFAR-100, and 7.42% in Tiny-ImageNet, respectively. This suggests that Meta-BayFL is a better alternative for federated learning in realistic noisy and small data scenarios. V-D Strengths and Limitations. The proposed Meta-BayFL framework has a number of noteworthy advantages. First, it successfully tackles client heterogeneity and data uncertainty, two significant issues in actual federated learning systems, by combining Bayesian neural networks with meta-learning-based personalization. Second, a thorough convergence analysis backs up the proposed approach and provides theoretical guarantees on performance and stability throughout communication rounds. Third, even in small, noisy, and non-IID data settings, comprehensive experimental results on CIFAR-10, CIFAR-100, and Tiny-ImageNet show consistent performance improvements over the state-of-the-art federated and personalized learning baselines. Despite these benefits, our study has a number of limitations that identify areas for further research. One drawback of Meta-BayFL is that, in contrast to deterministic federated learning, the use of Bayesian neural networks results in higher model parameterization and communication overhead. This could restrict direct deployment on IoT devices with extremely limited resources without the need for additional model compression or lightweight uncertainty approximations. Lastly, standard federated learning communicates a single set of deterministic model parameters per communication round, which results in a communication cost. On the other hand, Meta-BayFL increases the per-round communication overhead by about two times by transmitting the Bayesian neural network's mean and variance parameters. Although the load might be decreased by using Bayesian compression and parameter quantization approaches, a promising direction for future research. VI Conclusion In this paper, we propose a novel Meta-BayFL methodology to handle uncertain and heterogeneous data based on an innovative design of BNN and pFL. The model increases client diversity by allowing clients with different data and batch sizes to participate in the FL training and create the best local model by fine-tuning their learning rates. We also analyze enhancement in performance on small data by using BNN and on noisy data by using pFL. Finally, we show that Meta-BayFL outperforms other state-of-the-art pFL methods in both classification and regression tasks on CIFAR-10, CIFAR-100, and Tiny-ImageNet data respectively over non-IID settings. One major disadvantage of Meta-BayFL is that it may lead to high computation on clients due to personalized training. Developmental Trends and Open Challenges. Current federated learning research focuses on personalized and uncertainty-aware models for diverse edge settings. However, significant hurdles remain in lowering communication overhead, scaling to large and dynamic client populations, and enabling implementation on resource-constrained IoT devices, motivating further research into lightweight uncertainty modeling and compression-aware personalization. Wireless federated learning systems may be vulnerable to communication-layer challenges like jamming in addition to data heterogeneity and unpredictability. Bayesian game-theoretic formulations under partial knowledge have been used in previous works to study jamming mitigation. Such security techniques complement probabilistic personalized federated learning and are an essential factor to consider for realistic deployments. References