Paper deep dive
DP-FedAdamW: An Efficient Optimizer for Differentially Private Federated Large Models
Jin Liu, Yinbin Miao, Ning Xi, Junkang Liu
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 7/20/2026, 3:28:33 PM
Summary
The paper introduces DP-FedAdamW, a differentially private federated learning optimizer based on AdamW. It addresses three key challenges in applying AdamW to DPFL: variance amplification of second-moment estimators due to non-IID data and privacy noise, systematic bias in second-moment estimation caused by DP mechanisms, and exacerbated client drift. DP-FedAdamW stabilizes variance via block-wise aggregation, removes bias through an unbiased second-moment correction, and mitigates drift via local-global alignment. Theoretical analysis proves linear acceleration and tighter DP guarantees, with empirical results showing superior performance on Transformers and ResNet-18 compared to state-of-the-art baselines.
Entities (9)
Relation Signals (6)
DP-FedAdamW → isbasedon → AdamW
confidence 95% · We propose DP-FedAdamW, the first AdamW-based optimizer for DPFL.
DP-FedAdamW → outperforms → SOTA
confidence 90% · DP-FedAdamW outperforms the state-of-the-art (SOTA) by 5.83%.
DP-FedAdamW → solves → Client Drift
confidence 90% · aligning local updates to the global descent to curb client drift.
DP-FedAdamW → solves → Second-moment estimator
confidence 90% · It restores AdamW under DP by stabilizing second-moment variance, removing DP-induced bias...
Differential Privacy → causes → Client Drift
confidence 85% · DP amplify AdamW sensitivity to local overfitting, worsening client drift.
Differential Privacy → causes → Second-moment estimator
confidence 85% · DP perturbations bias the second-moment estimator
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Balancing convergence efficiency and robustness under Differential Privacy (DP) is a central challenge in Federated Learning (FL). While AdamW accelerates training and fine-tuning in large-scale models, we find that directly applying it to Differentially Private FL (DPFL) suffers from three major issues: (i) data heterogeneity and privacy noise jointly amplify the variance of second-moment estimator, (ii) DP perturbations bias the second-moment estimator, and (iii) DP amplify AdamW sensitivity to local overfitting, worsening client drift. We propose DP-FedAdamW, the first AdamW-based optimizer for DPFL. It restores AdamW under DP by stabilizing second-moment variance, removing DP-induced bias, and aligning local updates to the global descent to curb client drift. Theoretically, we establish an unbiased second-moment estimator and prove a linearly accelerated convergence rate without any heterogeneity assumption, while providing tighter $(\varepsilon,\delta)$-DP guarantees. Our empirical results demonstrate the effectiveness of DP-FedAdamW across language and vision Transformers and ResNet-18. On Tiny-ImageNet (Swin-Base, $\varepsilon=1$), DP-FedAdamW outperforms the state-of-the-art (SOTA) by 5.83\%. The code is available in Appendix.
Tags
Links
- Source: https://arxiv.org/abs/2602.19945v1
- Canonical: https://arxiv.org/abs/2602.19945v1
Trouble viewing inline? Open PDF directly →
Full Text
113,038 characters extracted from source content.
Expand or collapse full text
DP-FedAdamW: An Efficient Optimizer for Differentially Private Federated Large Models Jin Liu Yinbin Miao Ning Xi School of Cyber Engineering, Xidian University, Xi’an, China jinliu9787,ybmiao,nxi@xidian.edu.cn Junkang Liu College of Intelligence and Computing, Tianjin University, Tianjin, China junkangliukk@gmail.com Abstract Balancing convergence efficiency and robustness under Differential Privacy (DP) is a central challenge in Federated Learning (FL). While AdamW accelerates training and fine-tuning in large-scale models, we find that directly applying it to Differentially Private FL (DPFL) suffers from three major issues: (i) data heterogeneity and privacy noise jointly amplify the variance of second-moment estimator, (i) DP perturbations bias the second-moment estimator, and (i) DP amplify AdamW’s sensitivity to local overfitting, worsening client drift. We propose DP-FedAdamW, the first AdamW-based optimizer for DPFL. It restores AdamW under DP by stabilizing second-moment variance, removing DP-induced bias, and aligning local updates to the global descent to curb client drift. Theoretically, we establish an unbiased second-moment estimator and prove a linearly accelerated convergence rate without any heterogeneity assumption, while providing tighter (ε,δ)( ,δ)-DP guarantees. Our empirical results demonstrate the effectiveness of DP-FedAdamW across language and vision Transformers and ResNet-18. On Tiny-ImageNet (Swin-Base, ε=1 =1), DP-FedAdamW outperforms the state-of-the-art (SOTA) by 5.83%. The code is available in Appendix. 1 Introduction Figure 1: Illustration of our DP-FedAdamW. It aggregates the mean of block-wise Bias-Corrected second moment estimates and performs local–global alignment to stabilize DPFL optimization. Figure 2: An illustration of local update in DP-FedAdamW, which corrects client drift caused through global update guidance. As data volumes surge and privacy concerns grow, centralized learning methods are facing increasing limitations in terms of security and collaboration. Federated Learning (FL) [49] enables decentralized training without sharing raw data, reducing privacy risks [7]. However, FL model updates remain vulnerable to attacks [71, 12, 72]. Differential Privacy (DP) [11] provides formal protection by adding noise, but this increases gradient variance and hinders convergence, especially under non-IID data [22]. The emergence of large-scale models such as Swin Transformer [47], BERT [26], and GPT [2] has become dominant in vision and language tasks. These deep, parameter-heavy models are highly sensitive to optimization settings, which not only affects training stability but also impacts overall performance. Most DPFL algorithms are built upon the SGD optimizer and adopt the DPSGD [1] mechanism for privacy preservation [23, 52, 73, 24]. While effective for small-scale networks, DPSGD becomes inefficient for large Transformer models as the combination of gradient clipping and injected noise severely distorts gradients, amplifying instability and slowing convergence. In the pursuit of more efficient optimizers, adaptive methods like AdamW [48] have become state-of-the-art (SOTA) in deep learning. By decoupling weight decay from adaptive moment estimation, AdamW provides enhanced stability and generalization, making it a compelling choice for various machine learning tasks. These advantages make AdamW a promising candidate for DPFL, where privacy and robustness are crucial. However, our experiments reveal that AdamW fails to deliver its usual benefits in the DPFL setting. In some cases, its performance is comparable to or even worse than SGD. As illustrated in Table 2, when AdamW is applied locally without any modifications (referred to as DP-LocalAdamW), its performance on the Swin-Tiny model (CIFAR-100, Dirichlet distribution α=0.1α=0.1) is worse than DP-FedAvg-LS. These findings suggest that AdamW struggles with the challenges posed by privacy constraints and decentralized FL. Challenges. We identify three main challenges that contribute to the unexpected results. (i) Second-moment estimator variance amplification. Non-IID client data and DP noise jointly inflate the variance of the second-moment estimator, leading to unstable adaptive scaling in AdamW. (i) Bias in second-moment estimator. Gradient clipping and noise injection introduce a systematic bias in the second-moment estimator of AdamW. (i) Client crift. Under non-IID data, DP clipping and noise amplify AdamW’s sensitivity to local overfitting, further worsening client drift. All challenges are explored in more detail in Section 4. These findings raise a key question: How can AdamW maintain effectiveness in the presence of differential privacy and federated heterogeneity? To answer this, we propose DP-FedAdamW, a new differentially private adaptive optimizer that restores the effectiveness of AdamW under non-IID data and privacy noise. DP-FedAdamW (i) aggregates second-moment estimates in a block-wise manner to stabilize variance and improve communication efficiency, (i) eliminates DP-induced bias through an unbiased second-moment correction, and (i) enforces local–global update alignment to mitigate client drift, achieving more stable and accurate convergence. Our contributions are summarized as follows: • Empirical importance of AdamW and challenges in DPFL. We empirically demonstrate the effectiveness of AdamW in DPFL. Our analysis reveals three key challenges: high variance in second-moment estimator, bias in second-moment estimator, and high client drift. • DP-FedAdamW: A principled adaptive optimizer for DPFL. We propose DP-FedAdamW, a communication-efficient DPFL algorithm. It integrates global update estimation into local updates to suppress client drift, aggregates block-wise means of second-moment statistics across clients to stabilize variance, and introduces a Bias-Corrected mechanism to eliminate second-moment estimation bias caused by DP. • Theoretical and empirical guarantees. We derive the first convergence guarantee for a DPFL adaptive optimizer that overcomes gradient heterogeneity, achieving a linear speedup rate of ((LΔσl2)/(SKTϵ2)+(LΔ)/T+σ2Gg2/s2R2)O( (L _l^2)/(SKTε^2)+(L )/T+σ^2G_g^2/s^2R^2). Experiments on vision and language benchmarks demonstrate that DP-FedAdamW achieves a better privacy–utility trade-off and consistently outperforms SOTA DPFL baselines in convergence. 2 Related work Table 1: Comparison with existing DPFL optimizers. Method DP-FedAvg DP-SCAFFOLD DP-FedSAM DP-LocalAdamW DP-FedAdamW Optimizer Type SGD SGD SAM AdamW AdamW Scalability to Large Models Limited Limited Moderate Moderate High Client Drift Strong Weakened Slightly weakened Slightly weakened Weakened Communication Cost 1× 2× 2× 1× 1× Differentially Private Federated Learning (DPFL). We adopt sample-level DP for FL, ensuring privacy for each training example against any observer of the updates (the server or third parties). The most prevalent DP-FedAvg [52] applies per-sample clipping and Gaussian noise locally before aggregation. DP-SCAFFOLD [52] introduces control variates to alleviate client drift under privacy constraints. DP-FedAvg-LS [34] improves performance using Laplacian smoothing [53]. Beyond SGD, DP-FedSAM [62] uses Sharpness-Aware Minimization (SAM [21]) to find flatter minima, improving robustness to DP noise. Despite progress in refined clipping and personalized DP [68, 44], optimization remains the dominant bottleneck. However, these algorithms all employ the SGD optimizer for local updates and do not consider the use of adaptive optimizers. In the optimization of large-scale models, the AdamW optimizer has been shown to outperform SGD significantly in both convergence speed and final performance [76, 77, 75, 35, 8, 9, 4, 3, 5, 45, 46, 17, 14, 19, 18, 15, 66, 20, 13, 16, 36, 60, 61, 64, 63, 33, 31, 32, 50, 55, 56, 54]. Adaptive optimization. Adaptive optimizers such as Adam [27], AdamW [48], AMSGrad [59], Amsgrad [58] adapt per-parameter step sizes from gradient history, achieving faster and more stable convergence than vanilla SGD and becoming standard for large models (e.g., Transformers) [74, 29]. Among these, AdamW applies weight decay directly on parameters, preventing undesired interaction between regularization and moment adaptation, thus improving convergence and generalization [48, 70, 39, 38, 37]. Adaptive optimization in Federated Learning. A line of work brings adaptive updates into FL. FedOpt [57] introduces server-side adaptivity, instantiating Adam and Yogi at the aggregator. FAFED [69] stabilizes training by aggregating clients’ first- and second-moment estimates of Adam. FedAMS [6] highlights that averaging the second-moment is pivotal to avoid divergence. FedLADA [65] aggregates the second-moment estimate to reduce communication cost. However, these studies mainly evaluate on CNNs. However, these algorithms do not take into account the impact of noise introduced by DP. FedBCGD [38] proposes an accelerated block coordinate gradient descent framework for FL. FedSWA [37] improves generalization under highly heterogeneous data by stochastic weight averaging. FedAdamW [42] introduces a communication-efficient AdamW-style optimizer tailored for federated large models. FedNSAM [40] studies the consistency relationship between local and global flatness in FL. FedMuon [41] accelerates federated optimization via matrix orthogonalization. DP-FedPGN [43] develops a penalizes gradient norms to encourage globally flatter minima in DP-FL. Our contributions. To the best of our knowledge, adaptive optimization has not yet been systematically explored in the context of DPFL. Existing DPFL optimizers still struggle to train large models in a stable and efficient manner. To address this gap, we propose a method that stabilizes second-moment statistics via block-wise aggregation, removes DP-induced bias in the second moment, and aligns local updates with an estimated global descent direction in a communication-efficient way. We summarize representative DPFL methods and their characteristics in Table 1. 3 DPFL framework Consider a general DPFL system with N clients. The FL objective is to minimize the following population risk: f()=1N∑i=1Nfi(),fi():=ξi∼i[Fi(;ξi)],f( θ)= 1N\!Σ _i=1^Nf_i( θ),f_i( θ):=E_ _i _i [F_i( θ; _i) ], (1) where fif_i is the loss function of client i. ξi _i is drawn from distribution iP_i in client i, and [⋅]E[·] denotes expectation with respect to the sample ξi _i. In communication round t, the server selects S clients and distributes the global model t θ^t. The client i computes the gradient for each sample j in local k iterations (with a mini-batch ik=⌊sR⌋D_i^k= sR , s is data sampling rate, R is number of client samples): gij=∇fi(it,k;ξij)g_ij=∇ f_i( θ_i^t,k; _ij), and performs gradient clipping and Gaussian noise addition to satisfy DP: g¯ij=gij/max(1,‖gij‖2/C), g_ij=g_ij/ (1,\|g_ij\|_2/C), (2) g~it,k←1sR∑j∈ikg¯ij+CsR(0,σ2C2I), g_i^t,k← 1sR _j _i^k g_ij+ CsRN\! (0,σ^2C^2I ), (3) where C is the clipping threshold, σ is the noise factor. After K local steps, the server updates the global model t+1 θ^t+1 by adding the averaged aggregate local update to t θ^t. (a) Variance evolution of v (b) Client drift comparison Figure 3: Training on CIFAR-100, Swin-Tiny, σ=1σ=1, α=0.1α=0.1. (a) Non-IID (DPFL) causes high variance in second-moment estimator across clients of DP-LocalAdamW. (b) DP-LocalAdamW suffers from more severe client drift than FedAvg and LocalAdamW. 4 Motivation and challenges Prior work [48, 78] shows that AdamW can substantially improve convergence and generalization. However, our key observation is that, in DPFL, AdamW fails to exhibit its typical advantages. In fact, in transformer-based models, AdamW sometimes performs worse than SGD, indicating the benefits of AdamW are diminished when privacy constraints and the FL setup introduce additional complexities. Challenge 1: Variance amplification of the second-moment estimator under non-IID and DP. In FL, non-IID data disperses client gradients, while DP noise further increases the variance of noisy gradients. This high variance accumulates in AdamW’s exponentially weighted moving averages of the first-moment m and second-moment v: it,k m_i^t,k ←β1it,k−1+(1−β1)g~it,k, ← _1 m_i^t,k-1+(1- _1)\, g_i^t,k, (4) it,k v_i^t,k ←β2it,k−1+(1−β2)g~it,k⊙g~it,k, ← _2 v_i^t,k-1+(1- _2)\, g_i^t,k g_i^t,k, where β1=0.9 _1=0.9, β2=0.999 _2=0.999 are the exponential decay rates. ⊙ represents the elementwise (Hadamard) product. Var(vt) (v^t) ≈(1−β2)21−β22Var(g~t⊙g~t). ≈ (1- _2)^21- _2^2\,Var\! ( g^t g^t ). (5) Since g~t⊙g~t g^t g^t magnifies noise and β2≈1 _2\!≈\!1 induces slow averaging, the variance of v grows rapidly and dominates the optimization noise under DP and heterogeneity. Figure 3(a) shows the variance of v under non-IID+DP remains higher and converges much slower than in IID or non-DP cases. (a) Histogram of t m^t (b) Histogram of t v^t Figure 4: Histogram for DP-LocalAdamW, CIFAR-10, Swin-Tiny, σ=1σ=1, α=0.1α=0.1. (a) The distribution centers of t m^t are aligned with or without DP, but the variance is larger with DP. (b) The distribution of t v^t shows a significant difference, with the center of the distribution shifting approximately by σ2C2/(sR)2 σ^2C^2/(sR)^2. Challenge 2: Systematic bias in the second-moment estimator introduced by DP. Under DP clipping and noise, directly using the perturbed gradient to estimate the second moment produces a systematic bias. This bias is additive and distinct from the initialization bias addressed by Adam’s exponential moving average, so that correction does not remove it in Figure 4. Recall the definition of it,k v_i^t,k from Eq. 4, Eq. 2, Eq. 3, we have [g~it,k⊙g~it,k]=[g¯it,k⊙g¯it,k]+σ2C2/(sR)2I, \! [ g_i^t,k g_i^t,k ]=E\! [ g_i^t,k g_i^t,k ]+σ^2C^2/(sR)^2\,I, (6) [vit,k]=[vit,k]w/oDP+(1−β2k)σ2C2/(sR)2I. \! [v_i^t,k ]=E\! [v_i^t,k ]_w/o\ DP+ (1- _2^\,k )\,σ^2C^2/(sR)^2\,I. Challenge 3: Client drift caused by local adaptivity and DP mechanism. In FL, each client optimizes its local objective f(⋅)f(·), creating a gap between local and global optima. AdamW accelerates this by pushing clients toward their local optima, increasing client drift. In DPFL, gradient clipping and noise injection reduce effective sample sizes, making updates noisier and amplifying overfitting. This exacerbates client drift, hindering global model convergence. Figure 3(b) shows that DP-FedAdamW results in higher client drift compared to FedAvg and LocalAdamW, where LocalAdamW refers to Federated AdamW without DP. Algorithm 1 DP-FedAdamW 1: Initial initial model 0 θ^0; rounds T; local steps K; step size η; weight decay λ; AdamW (β1,β2,ϵ)( _1, _2,ε); clip norm C; noise multiplier σ; S clients per round 2: for t=1t=1 to T do 3: for selected clients i=1,…,Si=1,…,S in parallel do 4: it,0←t θ_i^t,0← θ^t; it,0← m_i^t,0← 0; it,0←¯t v_i^t,0← v^t 5: for k=1k=1 to K do 6: sample ik⊂iD_i^k _i of size ⌊sR⌋ sR 7: for each sample j∈ikj _i^k do 8: gij←∇fi(it,k;ξij)g_ij←∇ f_i( θ_i^t,k; _ij) 9: g¯ij←gij/max(1,‖gij‖2/C) g_ij← g_ij/ (1,\|g_ij\|_2/C) 10: end for 11: g~it,k←1sR∑j∈ikg¯ij+CsR(0,σ2C2I) g_i^t,k← 1sR _j _i^k g_ij+ CsRN\! (0,σ^2C^2I ) 12: it,k←β1it,k−1+(1−β1)g~it,k m_i^t,k← _1 m_i^t,k-1+(1- _1)\, g_i^t,k 13: it,k←β2it,k−1+(1−β2)g~it,k⊙g~it,k v_i^t,k← _2 v_i^t,k-1+(1- _2)\, g_i^t,k g_i^t,k 14: ^it,k←it,k/(1−β1k) m_i^t,k← m_i^t,k/(1- _1^k) 15: ^it,k←it,k/(1−β2k) v_i^t,k← v_i^t,k/(1- _2^k) 16: ϑit,k←1/(^it,k−(σCsR)2+ϵ) _i^t,k← 1/( v_i^t,k- ( σ CsR )^2+ε) 17: it,k+1←it,k−η(^it,k⊙ϑit,k+γGt−λit,k) θ_i^t,k+1\!←\! θ_i^t,k\!-\!η( m_i^t,k\! \! _i^t,k\!+\!γ _G^t\!-\!λ θ_i^t,k) 18: end for 19:Send (it,K−it,0,¯i=Block_mean(it,K))( θ^t,K_i\!\!-\!\! θ^t,0_i, v_i\!=\! Block\_mean( v^t,K_i)) 20: end for 21: Gt+1=−1SKη∑i=1S(it,K−it,0) _G^t+1= -1SKη _i=1^S( θ^t,K_i- θ^t,0_i) 22: t+1=t+1S∑i=1S(it,K−it,0) θ^t+1= θ^t+ 1S _i=1^S( θ^t,K_i- θ^t,0_i) 23: ¯t+1=1S∑i=1S¯i v^t+1= 1S _i=1^S v_i 24: Broadcast (t+1,¯t+1,Gt+1)( θ^t+1, v^t+1, _G^t+1) to clients 25: end for 5 Methodology To restore the effectiveness of AdamW in DPFL, we propose Differentially Private Federated AdamW (DP-FedAdamW). DP-FedAdamW is built upon a unified perspective: the moment statistics of AdamW become distorted under non-IID data and DP. Our design corrects this distortion along three complementary axes: parameter-block aggregation to stabilize second-moment variance, explicit removal of DP-induced bias in the second moment, and local–global update alignment to reduce client drift. 5.1 Second-moment aggregation To address Challenge 1, we stabilize the training process by aggregating the second-moment estimates of AdamW. However, directly aggregating per-parameter second moments requires communicating an additional vector with the same dimensionality as the model parameters, which almost doubles the communication overhead. We therefore introduce a block-wise aggregation scheme that communicates only one statistic per parameter block. We adopt a parameter-block averaging strategy, where each block shares a single learning rate to capture local curvature effectively. Based on this idea, we partition the parameters into B blocks and compute the mean of second-moments within each block (line 19 in Algorithm 1): ¯b=1|b|∑i∈bvi,b=1,…,B. v_b= 1| v_b| _i∈ v_bv_i, b=1,…,B. (7) Block-wise averaging strategy. We design the blocks to align with the model architecture. The attention-related parameters are partitioned at the granularity of attention heads, while the remaining modules are partitioned at the granularity of whole layers. For each parameter block, we compute the mean of its second-moment estimates and transmit only this block-wise statistic: • Head-wise Q, K, V. According to the number of attention heads in the model, each head-specific slice of the query (Q), key (K), and value (V) weight matrix is treated as an independent parameter block. For every such Q/K/V block, we compute and transmit the mean of its second-moment estimates. • attn.proj and MLP layers. Each attention output projection layer (attn.proj) and each MLP layer is regarded as a single parameter block. For each layer, we aggregate the second-moment estimates of all parameters within the layer and transmit the block-wise mean. • Embedding and output layers. Similarly, each Embedding layer and each output (e.g., classification) layer is treated as one parameter block. For these blocks, we also compute and transmit the mean of the corresponding second-moment estimates. CNNs (e.g., ResNet): For convolutional networks, we define each convolutional layer or residual block as one parameter block. This reduces the communication cost from per-parameter second moments to B block-level values while preserving adaptive behavior across different components of the network. 5.2 Unbiased second-moment correction To address Challenge 2, we propose a Bias-Corrected (BC). The correction term subtracts the variance contribution of the noise from the second-moment estimate of DP-AdamW, restoring the scaling behavior of non-private Adam and removing the influence of DP on the second-moment estimation. The update rule is given by (line 16 in Algorithm 1): ϑit,k=1/(^it,k−(σCsR)2+ϵ) _i^t,k=1/( v_i^t,k- ( σ CsR )^2+ε) (8) Here, ^it,k v_i^t,k denotes the exponential moving average of the (noisy) second moment of the gradient for parameter i, σ is the noise multiplier, C is the clipping norm, and sRsR is the batch size. The term σCsR σ CsRcorresponds to the variance of the Gaussian noise added for differential privacy, and subtracting it removes the constant bias introduced by DP noise in the second-moment estimate. 5.3 Local-global alignment To mitigate Challenge 3 (client drift under non-IID data with DP), we augment the local AdamW update with an explicit alignment toward the global descent direction. Concretely, we modify the local update rule (line 17 in Algorithm 1) as it,k+1=it,k−η(^it,k⊙ϑit,k−λit,k+γGt), θ_i^t,k+1= θ_i^t,k-η ( m^t,k_i _i^t,k-λ θ_i^t,k+γ _G^t ), (9) where Gt=−1SKη∑i=1S(it,K−it,0) _G^t= -1SKη _i=1^S( θ_i^t,K\!-\! θ_i^t,0) is an empirical estimate of the global update over S participating clients. Under non-IID data, the adaptive local gradient ^it,k⊙ϑit,k m^t,k_i _i^t,k is biased toward each client’s own objective. DP clipping distorts the magnitude and direction of these gradients, and DP noise adds high-variance perturbations, pushing the local parameters it,k θ_i^t,k toward different client-specific optima and amplifying the client drift identified in Challenge 3. The alignment term γGtγ _G^t softly regularizes each local step toward the global descent direction. It is nearly inactive when a client’s update already follows the global trend, but when non-IID data and DP operations drive the local direction away, the alignment term counteracts this deviation and steers the trajectory back toward the global path. As illustrated in Figure 2, this local–global alignment tightens the spread of client models, reduces cross-client update variance at aggregation, and yields a smoother, more stable global optimization trajectory under strong non-IID and DP. 6 Theoretical analysis 6.1 Convergence analysis In this part, we give the convergence theoretical analysis of our proposed DP-FedAdamW algorithm. Firstly, we give some standard assumptions for the non-convex function f. Assumption 1 (Smoothness). (Smoothness) The non-convex fif_i is a L-smooth function for all i∈[m]i∈[m], i.e., ‖∇fi(1)−∇fi(2)‖≤L‖1−2‖\|∇ f_i( θ_1)-∇ f_i( θ_2)\|≤ L\| θ_1- θ_2\|, for all 1,2∈ℝd θ_1, θ_2 ^d. Assumption 2 (Bounded Stochastic Gradient). it=∇fi(it,ξit) g_i^t=∇ f_i( θ_i^t, _i^t) computed by using a sampled mini-batch data ξit _i^t in the local client i is an unbiased estimator of ∇fi∇ f_i with bounded variance, i.e., ξit[it]=∇fi(it)E_ _i^t[ g_i^t]=∇ f_i( θ_i^t) and ξit‖git−∇fi(it)‖2≤σl2E_ _i^t\|g_i^t-∇ f_i( θ_i^t)\|^2≤ _l^2, for all it∈ℝd θ_i^t ^d. Assumption 3 (Bounded Stochastic Gradient I). Each element of stochastic gradient it g_i^t is bounded, i.e., ‖it‖∞=‖fi(it,ξit)‖∞≤Gg\| g_i^t\|_∞=\|f_i( θ_i^t, _i^t)\|_∞≤ G_g, for all it∈ℝd θ_i^t ^d and any sampled mini-batch data ξit _i^t. Theorem 1 (Convergence for non-convex functions). Under Assumptions A.1, A.2, and A.3, if we take g0=0g^0=0, then DP-FedAdamW converges as follows 1T∑t=0T−1[‖∇f(t)‖2]≲(LΔσl2SKTϵ2+LΔT+σ2Gg2s2R2). 1T\! _t=0^T-1\!E\! [\|∇ f( θ^t)\|^2 ] \! ( L _l^2SKTε^2+ L T+ σ^2G_g^2s^2R^2 ). (10) Here G0:=1N∑i=1N‖∇fi(0)‖2G_0:= 1N _i=1^N \|∇ f_i ( θ^0 ) \|^2,Δ=f(0)−f⋆ =f ( θ^0 )-f , S is the number of participating clients per round, σ is DP noise level, σl _l is noise of stochastic gradient, K is the number of local iterations, and T is the total number of communication rounds. The proof is provided in Appendix 11.1. The convergence rate of DP-FedAdamW is faster than that of DP-LocalAdamW’s (LΔ(σl2+σg2)SKTϵ2+LΔT+σ2Gg2s2R2)O ( L ( _l^2+ _g^2)SKTε^2+ L T+ σ^2G_g^2s^2R^2 ), and we do not need bounded heterogeneity assumption. This is due to the suppression of local drift by the global update estimation Gt _G^t. Unlike prior adaptive FL optimizers [65] which rely on heterogeneity assumption, our convergence result holds without such restrictions, making it more widely applicable. 6.2 Privacy analysis Differential privacy (DP) [10] provides a formal framework for quantifying disclosure risk. We conduct privacy accounting under Rényi DP (RDP) [51], which gives tight bounds under composition and subsampling. Following [52], we finally convert RDP to (ε,δ)( ,δ)-DP [51]. Definition 1. (Sample-level DP, [22]). For any two neighboring datasets D and ′D that differ in a single sample or record (either by addition or removal), a randomized mechanism ℳ:n→M:X^n\!→\!Y satisfies (ε,δ)( ,δ)-DP at the sample level if, for every possible output set O⊆O , the following holds: Pr[ℳ()∈O]≤eεPr[ℳ(′)∈O]+δ,Pr[M(D)∈ O]≤ e Pr [M (D )∈ O ]+δ, (11) where ε is the privacy budget, δ is the failure probability. Definition 2. (Rényi DP, [51]). For any order ζ∈(1,∞)ζ∈(1,∞), ℳM satisfies (ζ,εT)(ζ, _T)-RDP if for any two neighboring datasets D, ′D that differ by a single sample, it has: Dζ[ℳ()∥ℳ(′)]:=1ζ−1log[(pℳ()(Y)pℳ(′)(Y))ζ]≤εR,\!\!\!D_ζ [M(D)\|M (D ) ]\!:=\! 1ζ\!-\!1 [ ( p_M(D) (Y )p_M (D ) (Y ) )^ζ\! ]\!≤\! _R, (12) The expectation E is taken over the output Y∼ℳ(′)Y (D ). We adopt standard RDP accounting following [52], the cumulative privacy over K local steps and T rounds is: Theorem 2 (Privacy guarantee). For DP-FedAdamW, T θ^T is (ε,δ)( ,δ)-DP towards a third party ε=(sTKlog(2/δ)log(2T/δ)σ), =O ( s TK (2/δ) (2T/δ)σ ), -2.84526pt (13) Towards the server, accumulative (εs,δs)( _s, _s)-DP is εs=εNl,δs=δ2(1l+1), _s= Nl, _s= δ2 ( 1l+1 ), (14) where N is the number of clients, l is clients sample rate, and s is client data sample rate. Table 2: Averaged test accuracy (%) and privacy budget comparison on CIFAR-10 and CIFAR-100 using ResNet-18 and ViT-Base. Assuming DP-FedAvg is (ε,δ)( ,δ)-DP, all other methods share this budget except DP-FedSAM, which is approximately (2ε,δ)(2 ,δ)-DP. Method CIFAR-10 (ResNet-18) CIFAR-100 (ResNet-18) CIFAR-10 (ViT-Base) CIFAR-100 (ViT-Base) Privacy Budget α=0.6α=0.6 α=0.1α=0.1 α=0.6α=0.6 α=0.1α=0.1 α=0.6α=0.6 α=0.1α=0.1 α=0.6α=0.6 α=0.1α=0.1 DP-FedAvg 71.69±0.3771.69_± 0.37 60.38±0.2160.38_± 0.21 31.13±0.5731.13_± 0.57 25.04±0.7225.04_± 0.72 90.51±0.3790.51_± 0.37 78.85±0.2178.85_± 0.21 45.86±0.5745.86_± 0.57 13.41±0.7213.41_± 0.72 (ε,δ)( ,δ) DP-SCAFFOLD 71.79±0.4471.79_± 0.44 59.04±0.3959.04_± 0.39 31.52±0.3631.52_± 0.36 24.94±0.5824.94_± 0.58 90.88±0.5290.88_± 0.52 80.69±0.6480.69_± 0.64 62.45±0.5762.45_± 0.57 64.69±0.7664.69_± 0.76 (ε,δ)( ,δ) DP-FedAvg-LS 72.27±0.5772.27_± 0.57 60.84±0.4560.84_± 0.45 35.91±0.5435.91_± 0.54 28.68±0.5228.68_± 0.52 91.48±0.4791.48_± 0.47 88.48±0.5088.48_± 0.50 67.76±0.5867.76_± 0.58 65.85±0.6365.85_± 0.63 (ε,δ)( ,δ) DP-FedSAM 67.39±0.4167.39_± 0.41 57.04±0.4657.04_± 0.46 30.36±0.5030.36_± 0.50 24.03±0.6624.03_± 0.66 91.76±0.5591.76_± 0.55 89.35±0.6589.35_± 0.65 70.24±0.6370.24_± 0.63 66.27±0.6566.27_± 0.65 (2ε,δ)(2 ,δ) DP-LocalAdamW 71.73±0.4771.73_± 0.47 60.55±0.5360.55_± 0.53 34.92±0.6934.92_± 0.69 28.70±0.8128.70_± 0.81 91.16±0.4791.16_± 0.47 89.85±0.5389.85_± 0.53 68.21±0.6968.21_± 0.69 65.24±0.8165.24_± 0.81 (ε,δ)( ,δ) DP-FedAdamW 74.81±0.19 [rgb]1,0,0 [named]pgfstrokecolorrgb1,0,074.81_± 0.19 64.65±0.24 [rgb]1,0,0 [named]pgfstrokecolorrgb1,0,064.65_± 0.24 39.56±0.42 [rgb]1,0,0 [named]pgfstrokecolorrgb1,0,039.56_± 0.42 33.55±0.50 [rgb]1,0,0 [named]pgfstrokecolorrgb1,0,033.55_± 0.50 92.02±0.49 [rgb]1,0,0 [named]pgfstrokecolorrgb1,0,092.02_± 0.49 90.30±0.24 [rgb]1,0,0 [named]pgfstrokecolorrgb1,0,090.30_± 0.24 71.59±0.42 [rgb]1,0,0 [named]pgfstrokecolorrgb1,0,071.59_± 0.42 67.56±0.50 [rgb]1,0,0 [named]pgfstrokecolorrgb1,0,067.56_± 0.50 (ε,δ)( ,δ) Table 3: Averaged test accuracy (%) and privacy budget comparison on CIFAR-100 and Tiny-ImageNet using Swin-Tiny and Swin-Base. Method CIFAR-100 (Swin-T) CIFAR-100 (Swin-B) Tiny-ImageNet (Swin-T) Tiny-ImageNet (Swin-B) Privacy Budget α=0.6α=0.6 α=0.1α=0.1 α=0.6α=0.6 α=0.1α=0.1 α=0.6α=0.6 α=0.1α=0.1 α=0.6α=0.6 α=0.1α=0.1 DP-FedAvg 36.96±0.5036.96_± 0.50 28.07±0.3128.07_± 0.31 65.61±0.2765.61_± 0.27 60.77±0.3260.77_± 0.32 18.82±0.9418.82_± 0.94 10.12±1.3610.12_± 1.36 52.04±0.6652.04_± 0.66 36.19±0.7436.19_± 0.74 (ε,δ)( ,δ) DP-SCAFFOLD 41.68±0.6441.68_± 0.64 33.52±0.6533.52_± 0.65 68.03±0.5568.03_± 0.55 65.34±0.4965.34_± 0.49 21.91±0.7821.91_± 0.78 14.54±0.7214.54_± 0.72 53.78±0.7953.78_± 0.79 43.55±0.6043.55_± 0.60 (ε,δ)( ,δ) DP-FedAvg-LS 41.95±0.5341.95_± 0.53 36.72±0.6536.72_± 0.65 68.80±0.5268.80_± 0.52 66.08±0.5666.08_± 0.56 23.03±0.6423.03_± 0.64 15.99±0.6815.99_± 0.68 55.83±0.6655.83_± 0.66 47.46±0.5747.46_± 0.57 (ε,δ)( ,δ) DP-FedSAM 44.19±0.4844.19_± 0.48 38.43±0.6238.43_± 0.62 70.44±0.5770.44_± 0.57 66.28±0.6466.28_± 0.64 22.75±0.6522.75_± 0.65 16.86±0.7716.86_± 0.77 56.12±0.7456.12_± 0.74 48.08±0.6248.08_± 0.62 (2ε,δ)(2 ,δ) DP-LocalAdamW 42.04±0.6342.04_± 0.63 35.51±0.5835.51_± 0.58 68.33±0.3368.33_± 0.33 65.28±0.3965.28_± 0.39 23.41±0.8723.41_± 0.87 15.81±0.7315.81_± 0.73 54.58±0.6054.58_± 0.60 46.52±0.6446.52_± 0.64 (ε,δ)( ,δ) DP-FedAdamW 46.77±0.38 [rgb]1,0,0 [named]pgfstrokecolorrgb1,0,046.77_± 0.38 39.36±0.55 [rgb]1,0,0 [named]pgfstrokecolorrgb1,0,039.36_± 0.55 71.58±0.30 [rgb]1,0,0 [named]pgfstrokecolorrgb1,0,071.58_± 0.30 67.45±0.26 [rgb]1,0,0 [named]pgfstrokecolorrgb1,0,067.45_± 0.26 23.96±0.50 [rgb]1,0,0 [named]pgfstrokecolorrgb1,0,023.96_± 0.50 19.19±0.55 [rgb]1,0,0 [named]pgfstrokecolorrgb1,0,019.19_± 0.55 58.44±0.54 [rgb]1,0,0 [named]pgfstrokecolorrgb1,0,058.44_± 0.54 50.76±0.51 [rgb]1,0,0 [named]pgfstrokecolorrgb1,0,050.76_± 0.51 (ε,δ)( ,δ) Table 4: Averaged test accuracy (%) on RoBERTa-Base. Method SST-2 QQP QNLI MNLI DP-FedAvg 49.08±0.1249.08_± 0.12 63.18±0.5063.18_± 0.50 49.46±0.4449.46_± 0.44 35.45±0.2035.45_± 0.20 DP-SCAFFOLD 52.35±0.1852.35_± 0.18 66.87±0.4266.87_± 0.42 52.62±0.5252.62_± 0.52 38.74±0.3238.74_± 0.32 DP-FedAvg-LS 68.38±0.5068.38_± 0.50 72.59±0.4872.59_± 0.48 69.22±0.6869.22_± 0.68 50.46±0.5550.46_± 0.55 DP-FedSAM 70.95±0.6270.95_± 0.62 75.66±0.5375.66_± 0.53 70.54±0.6370.54_± 0.63 56.88±0.5656.88_± 0.56 DP-LocalAdamW 91.04±0.5391.04_± 0.53 82.33±0.5982.33_± 0.59 86.07±0.4786.07_± 0.47 75.20±0.4475.20_± 0.44 DP-FedAdamW 91.17±0.32 [rgb]1,0,0 [named]pgfstrokecolorrgb1,0,091.17_± 0.32 83.34±0.39 [rgb]1,0,0 [named]pgfstrokecolorrgb1,0,083.34_± 0.39 86.33±0.34 [rgb]1,0,0 [named]pgfstrokecolorrgb1,0,086.33_± 0.34 78.68±0.38 [rgb]1,0,0 [named]pgfstrokecolorrgb1,0,078.68_± 0.38 Table 5: Effect of privacy budgets (ε ) on test accuracy (%) using Swin-Base, α=0.1α=0.1. Method CIFAR-10 CIFAR-100 Tiny-ImageNet ε=1 =1 ε=2 =2 ε=3 =3 ε=1 =1 ε=2 =2 ε=3 =3 ε=1 =1 ε=2 =2 ε=3 =3 DP-FedAvg 66.52±0.8966.52_± 0.89 72.63±0.5572.63_± 0.55 88.02±0.7688.02_± 0.76 19.66±1.5219.66_± 1.52 42.98±0.6042.98_± 0.60 57.73±0.5557.73_± 0.55 20.65±0.7720.65_± 0.77 37.48±0.5437.48_± 0.54 45.32±0.4745.32_± 0.47 DP-SCAFFOLD 66.39±0.6866.39_± 0.68 72.81±0.5372.81_± 0.53 88.07±0.6688.07_± 0.66 22.38±0.6722.38_± 0.67 44.52±0.6844.52_± 0.68 59.85±0.5259.85_± 0.52 20.93±0.6620.93_± 0.66 38.40±0.5138.40_± 0.51 47.06±0.3047.06_± 0.30 DP-FedAvg-LS 71.57±1.0271.57_± 1.02 75.02±0.5975.02_± 0.59 88.96±0.5488.96_± 0.54 25.77±0.6125.77_± 0.61 49.54±0.5949.54_± 0.59 62.46±0.5362.46_± 0.53 26.89±0.4626.89_± 0.46 46.57±0.4446.57_± 0.44 48.83±0.2148.83_± 0.21 DP-FedSAM 53.51±1.3953.51_± 1.39 59.33±0.8659.33_± 0.86 76.82±0.9776.82_± 0.97 12.33±0.6412.33_± 0.64 28.32±0.4928.32_± 0.49 46.93±0.5046.93_± 0.50 10.88±0.9510.88_± 0.95 26.76±0.7826.76_± 0.78 41.69±0.4541.69_± 0.45 DP-LocalAdamW 66.57±1.3566.57_± 1.35 73.04±0.5873.04_± 0.58 88.83±0.7188.83_± 0.71 24.69±0.8824.69_± 0.88 47.26±0.5947.26_± 0.59 62.18±0.6162.18_± 0.61 28.40±0.7228.40_± 0.72 46.62±0.3746.62_± 0.37 48.55±0.2848.55_± 0.28 DP-FedAdamW 77.50±0.54 [rgb]1,0,0 [named]pgfstrokecolorrgb1,0,077.50_± 0.54 80.37±0.58 [rgb]1,0,0 [named]pgfstrokecolorrgb1,0,080.37_± 0.58 89.98±0.55 [rgb]1,0,0 [named]pgfstrokecolorrgb1,0,089.98_± 0.55 30.06±0.41 [rgb]1,0,0 [named]pgfstrokecolorrgb1,0,030.06_± 0.41 52.11±0.54 [rgb]1,0,0 [named]pgfstrokecolorrgb1,0,052.11_± 0.54 65.47±0.42 [rgb]1,0,0 [named]pgfstrokecolorrgb1,0,065.47_± 0.42 34.23±0.50 [rgb]1,0,0 [named]pgfstrokecolorrgb1,0,034.23_± 0.50 50.95±0.26 [rgb]1,0,0 [named]pgfstrokecolorrgb1,0,050.95_± 0.26 52.53±0.22 [rgb]1,0,0 [named]pgfstrokecolorrgb1,0,052.53_± 0.22 (a) ResNet-18, α=0.6α=0.6 (b) ResNet-18, α=0.1α=0.1 (c) Swin-Tiny, α=0.6α=0.6 (d) Swin-Tiny, α=0.1α=0.1 Figure 5: Test accuracy (%) on CIFAR-100 using ResNet-18 and Swin-Tiny under the Dirichlet α=0.6α=0.6 and α=0.1α=0.1 settings. Table 6: Ablation on the main components of DP-FedAdamW on Swin-Base. We report test accuracy (%) on CIFAR-100 and Tiny-ImageNet under non-IID setting (α=0.1α=0.1). “Agg” denotes block-wise second-moment aggregation, “BC” the DP bias correction, and “Align” the local–global alignment term. Method CIFAR-100 Tiny-ImageNet DP-LocalAdamW 65.2865.28 46.5246.52 w/o Agg 66.4166.41 47.3547.35 w/o BC 67.0267.02 48.1148.11 w/o Align 66.3766.37 47.8247.82 DP-FedAdamW (full) 67.42 [rgb]1,0,0 [named]pgfstrokecolorrgb1,0,067.42 50.73 [rgb]1,0,0 [named]pgfstrokecolorrgb1,0,050.73 Table 7: Ablation study of moment aggregation strategies of DP-LocalAdamW on CIFAR-100 with Swin-Base under α=0.1α=0.1. Strategy CIFAR-100 Tiny-ImageNet Comm(↑) NoAgg 66.21±0.1166.21_± 0.11 49.12±0.1249.12_± 0.12 5.75.7M Agg- v 67.52±0.1267.52_± 0.12 50.88±0.1250.88_± 0.12 11.411.4M Agg-mean- v 67.45±0.10 [rgb]1,0,0 [named]pgfstrokecolorrgb1,0,067.45_± 0.10 50.76±0.10 [rgb]1,0,0 [named]pgfstrokecolorrgb1,0,050.76_± 0.10 5.7M Table 8: Impact of γ using Swin-Base (α=0.1α=0.1). γ 0.000.00 0.250.25 0.50 [rgb]1,0,0 [named]pgfstrokecolorrgb1,0,00.50 0.750.75 1.001.00 CIFAR-100 (%) 66.3766.37 67.0167.01 67.45 [rgb]1,0,0 [named]pgfstrokecolorrgb1,0,067.45 67.1067.10 66.5266.52 Tiny-ImageNet (%) 47.8247.82 49.3649.36 50.76 [rgb]1,0,0 [named]pgfstrokecolorrgb1,0,050.76 50.2150.21 49.1849.18 Table 9: Comparison with DP variants of federated adaptive optimizers under the same DP settings (Swin-Base, α=0.1α=0.1, σ=1σ=1). Method CIFAR-100 Tiny-ImageNet DP-FedOpt(Adam) [57] 64.54±0.5664.54_± 0.56 44.81±0.6244.81_± 0.62 DP-FAFED [69] 65.75±0.5465.75_± 0.54 46.10±0.6746.10_± 0.67 DP-FedLADA [65] 66.03±0.6166.03_± 0.61 47.85±0.5747.85_± 0.57 DP-FedAdamW (ours) 67.45±0.26 [rgb]1,0,0 [named]pgfstrokecolorrgb1,0,067.45_± 0.26 50.76±0.51 [rgb]1,0,0 [named]pgfstrokecolorrgb1,0,050.76_± 0.51 7 Experiments 7.1 Experimental setup DPFL Algorithms. We conduct experiments on six DPFL algorithms: DP-FedAvg [52], DP-SCAFFOLD [52], DP-FedAvg-LS [34], DP-FedSAM [62], DP-LocalAdamW, and our DP-FedAdamW. In all performance evaluation experiments, the DP noise multiplier σ is fixed. Under the same clipping norm and communication rounds, DP-FedSAM requires roughly twice the privacy budget of other methods, as it injects independent DP noise into two gradient evaluations at each local step. All methods are implemented using PyTorch and executed on NVIDIA RTX 4090 GPUs. Datasets and data partition. We evaluate DP-FedAdamW on seven datasets covering both vision and language tasks. For image classification, we adopt CIFAR-10, CIFAR-100 [28], and Tiny-ImageNet [30]. For natural language understanding, we consider four GLUE [67] benchmarks: SST-2, QQP, QNLI, and MNLI. We simulate non-IID client data using a Dirichlet partition [25], where each client’s distribution is sampled from Dir(α) with α∈0.1,0.6,0.8α∈\0.1,0.6,0.8\. Model architectures. We study three representative architecture families. (1) convolutional neural networks(CNNs): GNResNet-18 evaluated on CIFAR-10/100. (2) vision transformers: ViT-Base for CIFAR-10/100 and Swin-Tiny/Base for CIFAR-10/100 and Tiny-ImageNet. (3) language transformer: RoBERTa-Base assessed on the GLUE. Configurations. For DP-FedAvg, DP-SCAFFOLD, DP-FedAvg-LS, and DP-FedSAM, the learning rate η is selected from 10−2, 3×10−2, 5×10−2, 10−1, 3×10−1\10^-2,\,3× 10^-2,\,5× 10^-2,\,10^-1,\,3× 10^-1\ with a weight decay of 0.0010.001. For DP-LocalAdamW, DP-FedAdamW, η is selected from 10−4, 3×10−4, 5×10−4, 8×10−4, 10−3\10^-4,\,3× 10^-4,\,5× 10^-4,\,8× 10^-4,\,10^-3\ with weight decay 0.010.01 or 0.0010.001, β1=0.9 _1=0.9, β2=0.999 _2=0.999. We apply cosine learning rate decay, and set DP-FedAdamW to γ=0.5γ=0.5, weight decay λ=0.01λ=0.01. For ResNet-18, ronds T=300T=300, batch size 5050, local step K=50K=50. For Transformers, ronds T=100T=100, batch size 1616, local step K=20K=20. The noise multiplier is σ=1σ=1. The privacy failure probability is set to δ=10−5δ=10^-5 for CIFAR10/100 and δ=10−6δ=10^-6 for Tiny-ImageNet. Please refer to Appendix 9.4 for more details of the experimental settings. All results are averaged over 5 runs with std reported. 7.2 Performance on vision datasets ResNet-18 on CIFAR-10/100. Table 2 summarizes the results on CIFAR-10/100 under different heterogeneity levels (α=0.6α=0.6 and α=0.1α=0.1). DP-FedAdamW achieves the best performance across all settings, reaching 74.81%74.81\% and 64.65%64.65\% on CIFAR-10, and 39.56%39.56\% and 33.55%33.55\% on CIFAR-100. When the data heterogeneity is severe (α=0.1α=0.1), DP-FedAdamW surpasses the SOTA baselines (DP-FedAvg-LS for CIFAR-10 and DP-LocalAdamW for CIFAR-100) by 3.81%3.81\% and 4.85%4.85\%, respectively. ViT-Base on CIFAR-10/100. We fine-tune ViT-Base on CIFAR-10/100, and the results are reported in Table 2 under α=0.6α=0.6 and α=0.1α=0.1. DP-FedAdamW consistently achieves the best performance. On CIFAR-100 with α=0.1α=0.1, it outperforms the strongest baseline DP-FedSAM by 1.29%1.29\%. Recall that, under our DP configuration, DP-FedSAM is trained with a much weaker privacy guarantee (it uses twice the privacy budget of the other methods). Swin-Tiny/Base on CIFAR-10/100/Tiny-ImageNet. We fine-tune Swin-Tiny and Swin-Base on CIFAR-10, CIFAR-100, and Tiny-ImageNet, and report the results in Tables 3, 10. DP-FedAdamW consistently attains the highest accuracy across all settings. For example, on CIFAR-100 with Swin-Tiny at α=0.6α=0.6, it reaches 46.77%46.77\%, outperforming the strongest baseline DP-FedSAM by 2.58%2.58\%. on Tiny-ImageNet with Swin-Base at α=0.1α=0.1, it achieves 50.85%50.85\%, exceeding DP-FedSAM by 2.77%2.77\%. Figure 5 (c,d) shows that DP-FedAdamW achieves advantage on Swin-Tiny. 7.3 Performance on language datasets RoBERTa-Base on GLUE. We further evaluate our method on four GLUE tasks (SST-2, QQP, QNLI, MNLI) by fine-tuning RoBERTa-Base under α=0.8α=0.8, and report the results in Table 4. DP-FedAdamW attains the best accuracy on all tasks, reaching 91.17%91.17\%, 83.34%83.34\%, 86.33%86.33\%, and 78.68%78.68\%, respectively. on the MNLI, DP-FedAdamW outperforms the baseline DP-LocalAdamW by 3.48%3.48\%, showing that the benefits of DP-FedAdamW extend from vision benchmarks to language understanding in the DPFL setting. 7.4 Discussion for DP with AdamW in FL Performance under different privacy budgets ε . Table 5 reports the results of Swin-Base on CIFAR-10, CIFAR-100, and Tiny-ImageNet under ε=1,2,3 =1,2,3 with α=0.1α=0.1. As expected, all methods benefit from larger ε (weaker privacy). Notably, the performance gap between DP-FedAdamW and other methods is most pronounced when ε is small, i.e., under stronger privacy constraints. For instance, on CIFAR-10, DP-FedAdamW achieves 77.50%77.50\% at ε=1 =1, surpassing the best baseline by 5.93%5.93\%. These results show that DP-FedAdamW achieves a superior utility–privacy trade-off. Noise multiplier σ. Results are provided in Appendix 10. 7.5 Ablation study Effect of each component. DP-FedAdamW differs from DP-LocalAdamW in three aspects: (i) block-wise second-moment aggregation (Agg); (i) DP bias correction on the second-moment (BC); (i) local–global alignment term (Align). We construct several variants by removing one component at a time, while keeping all other settings fixed. Table 9 shows that each component yields a gain, and their combination achieves the best performance. Bias correction (BC) improves robustness to DP noise, the local–global alignment stabilizes optimization under non-IID and DP, and block-wise aggregation provides an additional improvement with negligible communication overhead. Impact of aggregation strategy. Table 7 shows that our block-wise averaging strategy, Agg-mean- v, achieves the best balance between accuracy and communication cost. Agg-mean- v achieves comparable gains with only (B)O(B) communication, where B is the number of blocks, demonstrating its scalability and ability to stabilize updates even under strict communication budgets. Effect of the alignment coefficient γ. Table 8 varies γ on Swin-Base (α=0.1α=0.1). Increasing γ from 0.00.0 to 0.50.5 improves accuracy on CIFAR-100 and Tiny-ImageNet, indicating that a moderate alignment term effectively counteracts client drift and stabilizes optimization under DP noise. Further increasing γ beyond 0.50.5 leads to a slight degradation, as overly strong alignment over-constrains local updates. We therefore set γ=0.5γ=0.5 as the default. Comparison with DP variants of federated adaptive optimizers. As shown in Table 9, on Swin-Base with Dirichlet α=0.1α=0.1, DP noise σ=1σ=1, simply adding DP to existing federated adaptive optimizers yields limited gains. DP-FAFED and DP-FedLADA slightly outperform DP-FedOpt(Adam) on both CIFAR-100 and Tiny-ImageNet. Our DP-FedAdamW consistently achieves the best performance, with roughly 1.42%1.42\% on CIFAR-100 and 2.91%2.91\% on Tiny-ImageNet compared to the SOTA DP baseline. 8 Conclusion In this work, we revisited AdamW for differentially private federated learning and revealed that a naïve adoption leads to amplified second-moment variance, systematic DP-induced bias, and pronounced client drift under small-batch local training. Building on these insights, we introduced DP-FedAdamW, which integrates block-wise second-moment aggregation, explicit DP bias correction, and a local–global alignment mechanism, with non-convex convergence and sample-level privacy guarantees. Experiments on vision (CIFAR-10/100, Tiny-ImageNet with ResNet/ViT/Swin) and language (GLUE with RoBERTa) benchmarks demonstrate DP-FedAdamW consistently surpasses SOTA DPFL baselines under the same privacy budgets, especially in highly non-IID regimes. References [1] M. Abadi, A. Chu, I. Goodfellow, H. B. McMahan, I. Mironov, K. Talwar, and L. Zhang (2016) Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, p. 308–318. Cited by: §1. [2] J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al. (2023) Gpt-4 technical report. arXiv preprint arXiv:2303.08774. Cited by: §1. [3] W. An, Y. Liu, F. Shang, H. Liu, and L. Jiao (2024) DEs-inspired accelerated unfolded linearized admm networks for inverse problems. IEEE Transactions on Neural Networks and Learning Systems 36 (3), p. 5319–5333. Cited by: §2. [4] W. An, Y. Liu, F. Shang, and H. Liu (2024) Robust and faster zeroth-order minimax optimization: complexity and applications. Advances in Neural Information Processing Systems 37, p. 37050–37069. Cited by: §2. [5] W. An, Y. Yue, Y. Liu, F. Shang, and H. Liu (2022) A numerical des perspective on unfolded linearized admm networks for inverse problems. In Proceedings of the 30th ACM International Conference on Multimedia, p. 5065–5073. Cited by: §2. [6] X. Chen, X. Li, and P. Li (2020) Toward communication efficient adaptive gradient method. In Proceedings of the 2020 ACM-IMS on Foundations of Data Science Conference, p. 119–128. Cited by: §2. [7] Y. Chen, W. Huang, and M. Ye (2024) Fair federated learning under domain skew with local consistency and domain diversity. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 12077–12086. Cited by: §1. [8] Q. Ding, H. Ding, J. Wang, Y. Xian, T. Li, N. Li, T. Fang, and J. Chen (2026) Enhancing news classification: domain-specific guided pretraining based on adaptive selective masking. Knowledge-Based Systems, p. 115516. Cited by: §2. [9] Q. Ding, N. Li, H. Ding, J. Wang, T. Li, Y. Chen, Y. Xian, and J. Chen (2025) Traffic prediction and load balancing routing algorithm based on deep q-network for sd-iot. Advanced Engineering Informatics 68, p. 103596. Cited by: §2. [10] C. Dwork, A. Roth, et al. (2014) The algorithmic foundations of differential privacy. Foundations and Trends® in Theoretical Computer Science 9 (3–4), p. 211–407. Cited by: §6.2. [11] C. Dwork (2006) Differential privacy. In International colloquium on automata, languages, and programming, p. 1–12. Cited by: §1. [12] M. Fan, C. Chen, C. Wang, X. Li, and W. Zhou (2025) Refiner: data refining against gradient leakage attacks in federated learning. In 34th USENIX Security Symposium (USENIX Security 25), p. 3005–3024. Cited by: §1. [13] C. Feng, Z. Liu, Z. Zhi, I. Bogunovic, C. Gerner-Beuerle, and M. R. D. Rodrigues (2025-02) PROSAC: provably safe certification for machine learning models under adversarial attacks. In The 39th Annual AAAI Conference on Artificial Intelligence (AAAI) [Oral], External Links: Document Cited by: §2. [14] C. Feng and I. Patras (2023-06) MaskCon: Masked Contrastive Learning for Coarse-Labelled Dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), External Links: Document Cited by: §2. [15] C. Feng, N. Sebe, G. Tzimiropoulos, M. R. D. Rodrigues, and I. Patras (2025-10) Unveiling open-set noise: theoretical insights into label noise. In The 33rd ACM International Conference on Multimedia (ACM M), External Links: Document Cited by: §2. [16] C. Feng, M. Shen, A. Balashankar, C. Gerner-Beuerle, and M. R. D. Rodrigues (2026-04) Noisy but valid: robust statistical evaluation of LLMs with imperfect judges. In The Fourteenth International Conference on Learning Representations (ICLR), External Links: Link Cited by: §2. [17] C. Feng, G. Tzimiropoulos, and I. Patras (2022-11) SSR: An Efficient and Robust Framework for Learning with Unknown Label Noise. In 33rd British Machine Vision Conference (BMVC), External Links: Link Cited by: §2. [18] C. Feng, G. Tzimiropoulos, and I. Patras (2024-10) CLIPCleaner: Cleaning Noisy Labels with CLIP. In The 32nd ACM International Conference on Multimedia (ACM M), External Links: Document Cited by: §2. [19] C. Feng, G. Tzimiropoulos, and I. Patras (2024-07) NoiseBox: Towards More Efficient and Effective Learning with Noisy Labels. IEEE Transactions on Circuits and Systems for Video Technology. External Links: Document, ISSN 1558-2205 Cited by: §2. [20] C. Feng, Z. Zhi, Z. Huang, J. Ge, L. Xiao, N. Sebe, G. Tzimiropoulos, and I. Patras (2026-06) Deconstructing the failure of ideal noise correction: a three-pillar diagnosis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Cited by: §2. [21] P. Foret, A. Kleiner, H. Mobahi, and B. Neyshabur (2021) Sharpness-aware minimization for efficiently improving generalization. In ICLR, Cited by: §2. [22] J. Fu, Y. Hong, X. Ling, L. Wang, X. Ran, Z. Sun, W. H. Wang, Z. Chen, and Y. Cao (2024) Differentially private federated learning: a systematic review. arXiv preprint arXiv:2405.08299. Cited by: §1, Definition 1. [23] R. C. Geyer, T. Klein, and M. Nabi (2017) Differentially private federated learning: a client level perspective. arXiv preprint arXiv:1712.07557. Cited by: §1. [24] X. Gu, M. Li, and L. Xiong (2025) \dp-BREM\:\differentially-private\ and \byzantine-robust\ federated learning with client momentum. In 34th USENIX Security Symposium (USENIX Security 25), p. 3065–3082. Cited by: §1. [25] T. H. Hsu, H. Qi, and M. Brown (2019) Measuring the effects of non-identical data distribution for federated visual classification. arXiv preprint arXiv:1909.06335. Cited by: §7.1. [26] J. D. M. C. Kenton, L. K. Toutanova, et al. (2019) Bert: pre-training of deep bidirectional transformers for language understanding. In Proceedings of naacL-HLT, Vol. 1. Cited by: §1. [27] D. P. Kingma (2014) Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980. Cited by: §2. [28] A. Krizhevsky (2009) Learning multiple layers of features from tiny images. Technical report University of Toronto. Note: Technical Report Cited by: §7.1, §9.2, Table 12, Table 12. [29] F. Kunstner, A. Milligan, R. Yadav, M. Schmidt, and A. Bietti (2024) Heavy-tailed class imbalance and why adam outperforms gradient descent on language models. In Advances in Neural Information Processing Systems, A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (Eds.), Vol. 37, p. 30106–30148. External Links: Link Cited by: §2. [30] Y. Le and X. Yang (2015) Tiny imagenet visual recognition challenge. Note: Stanford CS231N Cited by: §7.1, §9.2, Table 12. [31] Y. Li, K. Li, X. Yin, Z. Yang, J. Dong, Z. Dong, C. Yang, Y. Tian, and Y. Lu (2025) Sepprune: structured pruning for efficient deep speech separation. arXiv preprint arXiv:2505.12079. Cited by: §2. [32] Y. Li, S. Meng, C. Yang, W. Feng, J. Liu, Z. An, Y. Wang, and Y. Tian (2026) A comprehensive survey of interaction techniques in 3d scene generation. Authorea Preprints. Cited by: §2. [33] Y. Li, C. Yang, H. Zeng, Z. Dong, Z. An, Y. Xu, Y. Tian, and H. Wu (2025) Frequency-aligned knowledge distillation for lightweight spatiotemporal forecasting. In Proceedings of the IEEE/CVF International Conference on Computer Vision, p. 7262–7272. Cited by: §2. [34] Z. Liang, B. Wang, Q. Gu, S. Osher, and Y. Yao (2024) Differentially private federated learning with laplacian smoothing. Applied and Computational Harmonic Analysis 72, p. 101660. Cited by: §2, §7.1. [35] B. Liao, Z. Zhao, H. Li, Y. Zhou, Y. Zeng, H. Li, and P. Liu (2025) Convex relaxation for robust vanishing point estimation in manhattan world. In Proceedings of the Computer Vision and Pattern Recognition Conference, p. 15823–15832. Cited by: §2. [36] D. Liu, Q. Shen, and J. Liu (2026) The health-wealth gradient in labor markets: integrating health, insurance, and social metrics to predict employment density. Computation 14 (1), p. 22. Cited by: §2. [37] J. Liu, Y. Liu, F. Shang, H. Liu, J. Liu, and W. Feng (2025) Improving generalization in federated learning with highly heterogeneous data via momentum-based stochastic controlled weight averaging. In Forty-second International Conference on Machine Learning, Cited by: §2. [38] J. Liu, F. Shang, Y. Liu, H. Liu, Y. Li, and Y. Gong (2024) Fedbcgd: communication-efficient accelerated block coordinate gradient descent for federated learning. In Proceedings of the 32nd ACM International Conference on Multimedia, p. 2955–2963. Cited by: §2. [39] J. Liu, F. Shang, Y. Tian, H. Liu, and Y. Liu (2025) Consistency of local and global flatness for federated learning. In Proceedings of the 33rd ACM International Conference on Multimedia, M ’25, New York, NY, USA, p. 3875–3883. External Links: ISBN 9798400720352, Link, Document Cited by: §2. [40] J. Liu, F. Shang, Y. Tian, H. Liu, and Y. Liu (2025) Consistency of local and global flatness for federated learning. In Proceedings of the 33rd ACM International Conference on Multimedia, p. 3875–3883. Cited by: §2. [41] J. Liu, F. Shang, J. Zhou, H. Liu, Y. Liu, and J. Liu (2025) FedMuon: accelerating federated learning with matrix orthogonalization. arXiv preprint arXiv:2510.27403. Cited by: §2. [42] J. Liu, F. Shang, K. Zhu, H. Liu, Y. Liu, and J. Liu (2025) FedAdamW: a communication-efficient optimizer with convergence and generalization guarantees for federated large models. arXiv preprint arXiv:2510.27486. Cited by: §2. [43] J. Liu, Y. Tian, F. Shang, Y. Liu, H. Liu, J. Zhou, and D. Ding (2025) DP-fedpgn: finding global flat minima for differentially private federated learning via penalizing gradient norm. arXiv preprint arXiv:2510.27504. Cited by: §2. [44] J. Liu, J. Lou, L. Xiong, J. Liu, and X. Meng (2024) Cross-silo federated learning with record-level personalized differential privacy. In Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security, p. 303–317. Cited by: §2. [45] Y. Liu, F. Shang, W. An, H. Liu, and Z. Lin (2022) Kill a bird with two stones: closing the convergence gaps in non-strongly convex optimization by directly accelerated svrg with double compensation and snapshots. In International Conference on Machine Learning, p. 14008–14035. Cited by: §2. [46] Y. Liu, F. Shang, W. An, J. Liu, H. Liu, and Z. Lin (2023) A single-loop accelerated extra-gradient difference algorithm with improved complexity bounds for constrained minimax optimization. Advances in Neural Information Processing Systems 36, p. 61699–61711. Cited by: §2. [47] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo (2021) Swin transformer: hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, p. 10012–10022. Cited by: §1. [48] I. Loshchilov, F. Hutter, et al. (2017) Fixing weight decay regularization in adam. arXiv preprint arXiv:1711.05101 5 (5), p. 5. Cited by: §1, §2, §4. [49] B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas (2017) Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, p. 1273–1282. Cited by: §1. [50] L. Meng, Z. Qi, L. Wu, X. Du, Z. Li, L. Cui, and X. Meng (2025) Improving global generalization and local personalization for federated learning. IEEE Transactions on Neural Networks and Learning Systems 36 (1), p. 76–87. External Links: Document Cited by: §2. [51] I. Mironov (2017) Rényi differential privacy. In Proc. IEEE computer security foundations symposium (CSF), p. 263–275. Cited by: §6.2, Definition 2. [52] M. Noble, A. Bellet, and A. Dieuleveut (2022) Differentially private federated learning on heterogeneous data. In International conference on artificial intelligence and statistics, p. 10110–10145. Cited by: §1, §2, §6.2, §6.2, §7.1. [53] S. Osher, B. Wang, P. Yin, X. Luo, F. Barekat, M. Pham, and A. Lin (2022) Laplacian smoothing gradient descent. Research in the Mathematical Sciences 9 (3), p. 55. Cited by: §2. [54] X. Qi, M. Li, S. Zhou, W. Feng, and Z. Qi (2025) Federated learning for science: a survey on the path to a trustworthy collaboration ecosystem. Authorea Preprints. Cited by: §2. [55] X. Qi, T. Xu, C. Dang, Z. Qi, L. Meng, and H. Yu (2026) Federated learning in oncology: bridging artificial intelligence innovation and privacy protection. Information Fusion, p. 104154. Cited by: §2. [56] Z. Qi, L. Meng, Z. Chen, H. Hu, H. Lin, and X. Meng (2023) Cross-silo prototypical calibration for federated learning with non-iid data. In Proceedings of the 31st ACM international conference on multimedia, p. 3099–3107. Cited by: §2. [57] S. J. Reddi, Z. Charles, M. Zaheer, Z. Garrett, K. Rush, J. Konečnỳ, S. Kumar, and H. B. McMahan (2021) Adaptive federated optimization. In International Conference on Learning Representations, Cited by: §2, Table 9. [58] S. J. Reddi, S. Kale, and S. Kumar (2018) On the convergence of adam and beyond. In International Conference on Learning Representations, Cited by: §2. [59] S. J. Reddi, S. Kale, and S. Kumar (2019) On the convergence of adam and beyond. arXiv preprint arXiv:1904.09237. Cited by: §2. [60] Q. Shen and J. Zhang (2025-12) AI-enhanced disaster risk prediction with explainable shap analysis: a multi-class classification approach using xgboost. Note: Preprint, Version 1, posted December 31, 2025 External Links: Document, Link Cited by: §2. [61] Q. Shen and J. Zhang (2026) MFTFormer: meteorological-frequency-temporal transformer with block-aligned fusion for traffic flow prediction. Research Square. Note: Preprint, doi:10.21203/rs.3.rs-8770196/v1 Cited by: §2. [62] Y. Shi, Y. Liu, K. Wei, L. Shen, X. Wang, and D. Tao (2023) Make landscape flatter in differentially private federated learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 24552–24562. Cited by: §2, §7.1. [63] W. Sun, Z. Qi, and Q. Shen (2025) High-recall deep learning: a gated recurrent unit approach to bank account fraud detection on imbalanced data. Note: Research Square Preprint External Links: Document, Link Cited by: §2. [64] W. Sun, Q. Shen, Y. Gao, Q. Mao, T. Qi, and S. Xu (2025) Objective over architecture: fraud detection under extreme imbalance in bank account opening. Computation 13 (12), p. 290. Cited by: §2. [65] Y. Sun, L. Shen, H. Sun, L. Ding, and D. Tao (2023) Efficient federated learning via local adaptive amended optimizer with linear speedup. IEEE Transactions on Pattern Analysis and Machine Intelligence 45 (12), p. 14453–14464. Cited by: §2, §6.1, Table 9. [66] Z. Sun, C. Feng, I. Patras, and G. Tzimiropoulos (2024-06) LAFS: Landmark-based Facial Self-supervised Learning for Face Recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), External Links: Document Cited by: §2. [67] A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. Bowman (2018) GLUE: a multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP workshop BlackboxNLP: Analyzing and interpreting neural networks for NLP, p. 353–355. Cited by: §7.1, §9.2. [68] C. Wei, W. Li, G. Chen, and W. Chen (2025) DC-sgd: differentially private sgd with dynamic clipping through gradient norm distribution estimation. IEEE Transactions on Information Forensics and Security. Cited by: §2. [69] X. Wu, F. Huang, Z. Hu, and H. Huang (2023) Faster adaptive federated learning. In Proceedings of the AAAI conference on artificial intelligence, Vol. 37, p. 10379–10387. Cited by: §2, Table 9. [70] S. Xie and Z. Li (2024) Implicit bias of adamw: ℓ∞ _∞-norm constrained optimization. In International Conference on Machine Learning, p. 54488–54510. Cited by: §2. [71] R. Xu, S. Gao, C. Li, J. Joshi, and J. Li (2024) Dual defense: enhancing privacy and mitigating poisoning attacks in federated learning. Advances in Neural Information Processing Systems 37, p. 70476–70498. Cited by: §1. [72] X. Xu, Z. Li, Y. Han, B. Wang, J. Liu, and W. Wang (2025) From risk to resilience: towards assessing and mitigating the risk of data reconstruction attacks in federated learning. In 34th USENIX Security Symposium (USENIX Security 25), p. 3141–3160. Cited by: §1. [73] Y. Yang, B. Hui, H. Yuan, N. Gong, and Y. Cao (2023) \privatefl\: Accurate, differentially private federated learning via personalized data transformation. In 32nd USENIX Security Symposium (USENIX Security 23), p. 1595–1612. Cited by: §1. [74] Y. Zhang, C. Chen, T. Ding, Z. Li, R. Sun, and Z. Luo (2024) Why transformers need adam: a hessian perspective. Advances in neural information processing systems 37, p. 131786–131823. Cited by: §2. [75] Z. Zhao and B. M. Chen (2023) Benchmark for evaluating initialization of visual-inertial odometry. In 2023 42nd Chinese Control Conference (C), p. 3935–3940. Cited by: §2. [76] Z. Zhao, H. Yang, B. Liao, Y. Zeng, S. Yan, Y. Gu, P. Liu, Y. Zhou, H. Li, and J. Civera (2026) Advances in global solvers for 3d vision. arXiv preprint arXiv:2602.14662. Cited by: §2. [77] Z. Zhao (2024) Balf: simple and efficient blur aware local feature detector. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, p. 3362–3372. Cited by: §2. [78] P. Zhou, X. Xie, Z. Lin, and S. Yan (2024) Towards understanding convergence and generalization of adamw. IEEE transactions on pattern analysis and machine intelligence 46 (9), p. 6486–6493. Cited by: §4. DP-FedAdamW: An Efficient Optimizer for Differentially Private Federated Large Models Supplementary Material 9 More Implementation Detail 9.1 More Results Swin-Tiny/Base on CIFAR-10. Table 10 reports the averaged test accuracy on CIFAR-10 for six different DPFL methods evaluated on Swin-Tiny and Swin-Base, under two data heterogeneity levels (Dirichlet α=0.6α=0.6 and α=0.1α=0.1). Overall, DP-FedAdamW consistently achieves the best performance, while Swin-Base is markedly more accurate than Swin-Tiny and all methods degrade when the data becomes more heterogeneous (smaller α). For Swin-Base with α=0.1α=0.1, DP-FedAdamW attains 90.68%90.68\% test accuracy, outperforming DP-FedSAM (89.82%89.82\%) and DP-FedAvg (88.23%88.23\%), even though DP-FedSAM needs roughly twice the privacy budget. Table 10: Averaged test accuracy (%) comparison on CIFAR-10. Assuming DP-FedAvg is (ε,δ)( ,δ)-DP, all other methods share this budget except DP-FedSAM, which is approximately (2ε,δ)(2 ,δ)-DP. Method Swin-Tiny Swin-Base Privacy Budget α=0.6α=0.6 α=0.1α=0.1 α=0.6α=0.6 α=0.1α=0.1 DP-FedAvg 82.93±0.5382.93_± 0.53 78.24±0.5978.24_± 0.59 89.56±0.4789.56_± 0.47 88.23±0.4488.23_± 0.44 (ε,δ)( ,δ) DP-SCAFFOLD 82.85±0.5082.85_± 0.50 78.53±0.5478.53_± 0.54 90.07±0.4890.07_± 0.48 89.08±0.6289.08_± 0.62 (ε,δ)( ,δ) DP-FedAvg-LS 83.94±0.6283.94_± 0.62 79.67±0.6879.67_± 0.68 90.97±0.5290.97_± 0.52 90.14±0.5690.14_± 0.56 (ε,δ)( ,δ) DP-FedSAM 84.05±0.5184.05_± 0.51 80.36±0.5780.36_± 0.57 91.25±0.5491.25_± 0.54 89.82±0.6089.82_± 0.60 (2ε,δ)(2 ,δ) DP-LocalAdamW 83.73±0.5883.73_± 0.58 79.28±0.6479.28_± 0.64 90.84±0.3990.84_± 0.39 89.08±0.4189.08_± 0.41 (ε,δ)( ,δ) DP-FedAdamW 84.23±0.32 [rgb]1,0,0 [named]pgfstrokecolorrgb1,0,084.23_± 0.32 81.23±0.39 [rgb]1,0,0 [named]pgfstrokecolorrgb1,0,081.23_± 0.39 91.79±0.34 [rgb]1,0,0 [named]pgfstrokecolorrgb1,0,091.79_± 0.34 90.68±0.38 [rgb]1,0,0 [named]pgfstrokecolorrgb1,0,090.68_± 0.38 (ε,δ)( ,δ) Noise multiplier results. To further analyze the impact of privacy noise on algorithm performance, we evaluate different noise multipliers σ∈0.5,1.0,1.5,2.0σ∈\0.5,1.0,1.5,2.0\ on CIFAR-100 using the Swin-Base model under Dirichlet α=0.6α=0.6 and α=0.1α=0.1. As shown in Table 11, increasing the noise multiplier consistently degrades test accuracy for all methods, while DP-FedAdamW maintains the best performance across all σ. Although σ=0.5σ=0.5 yields the highest accuracy, it also consumes a much larger privacy budget. In contrast, setting σ=1σ=1 significantly strengthens the formal privacy guarantee while only causing a small drop in accuracy, whereas larger noise levels (σ≥1.5σ≥1.5) lead to a much more pronounced loss in utility. Therefore, we adopt σ=1σ=1 as the default noise multiplier in our main experiments, as it offers a favorable privacy–utility trade-off. Table 11: Test accuracy (%) on CIFAR-100 with Swin-Base under different noise multipliers σ. Method α=0.6α=0.6 α=0.1α=0.1 σ=0.5σ=0.5 σ=1σ=1 σ=1.5σ=1.5 σ=2σ=2 σ=0.5σ=0.5 σ=1σ=1 σ=1.5σ=1.5 σ=2σ=2 DP-FedAvg 67.23±0.3367.23_± 0.33 65.61±0.2765.61_± 0.27 63.78±0.3463.78_± 0.34 60.22±0.6460.22_± 0.64 62.49±0.2662.49_± 0.26 60.77±0.3260.77_± 0.32 57.85±0.4857.85_± 0.48 54.34±0.7954.34_± 0.79 DP-SCAFFOLD 70.41±0.2570.41_± 0.25 68.03±0.5568.03_± 0.55 64.64±0.4864.64_± 0.48 63.14±0.5863.14_± 0.58 66.72±0.3866.72_± 0.38 65.34±0.4965.34_± 0.49 61.73±0.4461.73_± 0.44 58.23±0.6658.23_± 0.66 DP-FedAvg-LS 70.93±0.4070.93_± 0.40 68.80±0.5268.80_± 0.52 66.52±0.4366.52_± 0.43 64.77±0.6364.77_± 0.63 68.36±0.3968.36_± 0.39 66.08±0.5666.08_± 0.56 62.87±0.5262.87_± 0.52 59.98±0.5859.98_± 0.58 DP-FedSAM 72.55±0.2272.55_± 0.22 70.44±0.5770.44_± 0.57 67.45±0.3367.45_± 0.33 65.75±0.6965.75_± 0.69 67.94±0.5167.94_± 0.51 66.28±0.6466.28_± 0.64 62.21±0.6062.21_± 0.60 60.84±0.7160.84_± 0.71 DP-LocalAdamW 70.74±0.1370.74_± 0.13 68.33±0.3368.33_± 0.33 66.46±0.5366.46_± 0.53 64.10±0.6064.10_± 0.60 67.35±0.3167.35_± 0.31 65.28±0.3965.28_± 0.39 60.55±0.4360.55_± 0.43 58.78±0.7458.78_± 0.74 DP-FedAdamW 73.62±0.36 [rgb]1,0,0 [named]pgfstrokecolorrgb1,0,073.62_± 0.36 71.58±0.30 [rgb]1,0,0 [named]pgfstrokecolorrgb1,0,071.58_± 0.30 68.52±0.45 [rgb]1,0,0 [named]pgfstrokecolorrgb1,0,068.52_± 0.45 66.79±0.51 [rgb]1,0,0 [named]pgfstrokecolorrgb1,0,066.79_± 0.51 69.22±0.37 [rgb]1,0,0 [named]pgfstrokecolorrgb1,0,069.22_± 0.37 67.45±0.26 [rgb]1,0,0 [named]pgfstrokecolorrgb1,0,067.45_± 0.26 63.28±0.42 [rgb]1,0,0 [named]pgfstrokecolorrgb1,0,063.28_± 0.42 61.05±0.61 [rgb]1,0,0 [named]pgfstrokecolorrgb1,0,061.05_± 0.61 9.2 Datasets Vision datasets. We evaluate our methods on three widely used image classification benchmarks: CIFAR-10, CIFAR-100, and Tiny-ImageNet in Table 12. CIFAR-10 and CIFAR-100 [28] each contain 60,00060,000 natural images of size 32×3232× 32, split into 50,00050,000 training and 10,00010,000 test examples. CIFAR-10 comprises 10 coarse-grained object categories, whereas CIFAR-100 has finer labeling with 100 fine-grained categories organized into 20 superclasses. Tiny-ImageNet [30] is a downsampled subset of ImageNet with 200 classes, where each class contains 500 training images and 50 validation images of size 64×6464× 64. Table 12: Summary of the vision datasets used in our experiments. Dataset #Classes Image Size Train / Test CIFAR-10 [28] 10 32×3232× 32 50,000 / 10,000 CIFAR-100 [28] 100 32×3232× 32 50,000 / 10,000 Tiny-ImageNet [30] 200 64×6464× 64 100,000 / 10,000 Language datasets. For natural language understanding, we experiment on four tasks from the GLUE [67] benchmark: SST-2, QQP, QNLI, and MNLI (Table 13. SST-2 is a binary sentiment classification task over movie reviews. QQP is a paraphrase identification task that determines whether two Quora questions express the same semantic intent. QNLI is a binary classification task that predicts whether a candidate sentence contains the answer to a given question. MNLI is a three-way natural language inference benchmark (entailment, contradiction, neutral) covering multiple genres of text. We follow the standard GLUE data splits and evaluation protocols. Table 13: Summary of GLUE datasets used in our experiments. Dataset Task Type #Classes Train/Test SST-2 Sentiment Classification (binary) 2 67,349 / 1,821 QQP Duplicate Question Detection (binary) 2 363,846 / 390,965 QNLI Question-Answer NLI (binary) 2 104,743 / 5,463 MNLI Natural Language Inference (entailment) 3 392,702 / 9,815 9.3 Models We study three representative architecture families, covering five specific models in total, as summarized in Table 14. (1) ResNet-18. We adopt a ResNet-18 with Group Normalization (GN), where all Batch Normalization layers are replaced by GN to avoid dependence on batch statistics, which is more suitable for federated and differentially private training. To adapt CIFAR-10/100, we follow standard practices and modify the stem by replacing the original 7×77×7 convolution with a 3×33×3 kernel and removing the initial downsampling layers (i.e., the stride-2 convolution and max-pooling layer). (2) ViT-Base. We use the standard Vision Transformer Base (ViT-Base) architecture, which splits an input image into non-overlapping 16×1616×16 patches and linearly embeds them into a sequence of tokens with a prepended class token. The token sequence is processed by 12 Transformer encoder layers with multi-head self-attention and MLP blocks, and the final class token representation is fed to a linear classifier to produce the output logits. (3) Swin-Tiny. Swin-Tiny is a hierarchical Vision Transformer that uses shifted window self-attention. The model consists of four stages with depths [2,2,6,2][2,2,6,2], embedding dimensions [96,192,384,768][96,192,384,768], and attention heads [3,6,12,24][3,6,12,24]. Images are partitioned into 4×44×4 patches, and each stage applies window-based attention with window size 77 and an MLP ratio of 44, using LayerNorm and relative positional bias. DropPath regularization with depth-scaled rates is used, and the final features are aggregated by global pooling and passed to a linear classifier. (4) Swin-Base. Swin-Base is a larger variant of Swin-Tiny, sharing the same hierarchical shifted-window design. It is configured with four stages of depths [2,2,18,2][2,2,18,2], embedding dimensions [128,256,512,1024][128,256,512,1024], and attention heads [4,8,16,32][4,8,16,32], using a patch size of 4×44×4, window size 77, and MLP ratio 44. Similar to Swin-Tiny, the final stage features are pooled globally and fed into a linear classifier to generate predictions. (5) RoBERTa-Base. We use the RoBERTa-Base architecture, which follows the standard Transformer encoder design with 12 layers, 12 attention heads, and a hidden size of 768. It employs byte-pair encoding (BPE) tokenization and is pretrained with a masked language modeling objective on large-scale corpora using dynamic masking. The final representation corresponding to the classification token is fed into a task-specific linear classifier. Table 14: Overview of the model architectures used in our experiments. “Depth” denotes the total number of layers (blocks for ResNet/Swin, encoder layers for ViT/RoBERTa), and “Stages” denotes the number of blocks per stage for hierarchical models. Model Family Depth Stages Width / Dim Heads ResNet-18 (GN) CNN 18 [2,2,2,2][2,2,2,2] [64,128,256,512][64,128,256,512] – ViT-Base ViT Transformer 12 – 768 hidden, 3072 MLP 12 / layer Swin-Tiny Swin Transformer 12 [2,2,6,2][2,2,6,2] [96,192,384,768][96,192,384,768] [3,6,12,24][3,6,12,24] Swin-Base Swin Transformer 24 [2,2,18,2][2,2,18,2] [128,256,512,1024][128,256,512,1024] [4,8,16,32][4,8,16,32] RoBERTa-Base NLP Transformer 12 – 768 hidden, 3072 FFN 12 / layer Algorithm 2 DP-LocalAdamW 1: Initial initial model 0 θ^0; rounds T; local steps K; step size η; weight decay λ; AdamW (β1,β2,ϵ)( _1, _2,ε); clip norm C; noise multiplier σ; S clients per round 2: for t=1t=1 to T do 3: for selected clients i=1,…,Si=1,…,S in parallel do 4: it,0←t θ_i^t,0← θ^t; it,0← m_i^t,0← 0; it,0← v_i^t,0← 0 5: for k=1k=1 to K do 6: sample ik⊂iD_i^k _i of size ⌊sR⌋ sR 7: for each sample j∈ikj _i^k do 8: gij←∇fi(it,k;ξij)g_ij←∇ f_i( θ_i^t,k; _ij) 9: g¯ij←gij/max(1,‖gij‖2/C) g_ij← g_ij/ (1,\|g_ij\|_2/C) 10: end for 11: g~it,k←1sR∑j∈ikg¯ij+CsR(0,σ2C2I) g_i^t,k← 1sR _j _i^k g_ij+ CsRN\! (0,σ^2C^2I ) 12: it,k←β1it,k−1+(1−β1)g~it,k m_i^t,k← _1 m_i^t,k-1+(1- _1)\, g_i^t,k 13: it,k←β2it,k−1+(1−β2)g~it,k⊙g~it,k v_i^t,k← _2 v_i^t,k-1+(1- _2)\, g_i^t,k g_i^t,k 14: ^it,k←it,k/(1−β1k) m_i^t,k← m_i^t,k/(1- _1^k) 15: ^it,k←it,k/(1−β2k) v_i^t,k← v_i^t,k/(1- _2^k) 16: it,k+1←it,k−η(^it,k/(^it,k+ϵ)−λit,k) θ_i^t,k+1← θ_i^t,k-η\,( m_i^t,k/( v_i^t,k+ε)-λ θ_i^t,k) 17: end for 18: Clients send (it,K−it,0)( θ_i^t,K- θ_i^t,0) to server 19: end for 20: t+1←t+1S∑i=1S(it,K−it,0) θ^t+1← θ^t+ 1S _i=1^S( θ_i^t,K- θ_i^t,0) 21: Broadcast t+1 θ^t+1 to clients 22: end for 9.4 Complete Configuration Federated setup. In all experiments, we fine-tune the models using parameter-efficient Low-Rank Adaptation ( LoRA). We then detail the federated learning configurations adopted in our experiments. Table 15 specifies the settings for all vision benchmarks, including the number of clients N, communication rounds T, local update steps K, client participation rate l, local batch size sRsR, DP noise multiplier σ, failure probability δ, and LoRA hyperparameters (r,αL)(r, _L). Table 17 reports the corresponding configuration for the GLUE benchmarks with RoBERTa-Base and LoRA, using the same federated parameters while additionally documenting the maximum sequence length and dropout rate. Table 15: Federated configurations for vision models across CIFAR-10/100 and Tiny-ImageNet. We report the number of clients N, communication rounds T, local steps K, client participation rate l, local batch size sRsR, DP noise multiplier σ, failure probability δ, and LoRA hyperparameters (r,αL)(r, _L) and dropout rate. Dataset Model N T K l sRsR σ δ r (LoRA) αL _L (LoRA) Dropout CIFAR-10 ResNet-18 (GN) 50 300 50 0.2 50 1.0 10−510^-5 16 32 0.1 ViT-Base 50 100 20 0.1 16 1.0 10−510^-5 16 32 0.1 Swin-Tiny/Base 50 100 20 0.1 16 1.0 10−510^-5 16 32 0.1 CIFAR-100 ResNet-18 (GN) 50 300 50 0.2 50 1.0 10−510^-5 16 32 0.1 ViT-Base 50 100 20 0.1 16 1.0 10−510^-5 16 32 0.1 Swin-Tiny/Base 50 100 20 0.1 16 1.0 10−510^-5 16 32 0.1 Tiny-ImageNet Swin-Tiny/Base 50 100 20 0.1 16 1.0 10−610^-6 16 32 0.1 Table 16: Federated configurations for RoBERTa-Base with LoRA on GLUE tasks. We report the number of clients N, communication rounds T, local steps K, client participation rate l, local batch size sRsR, DP noise multiplier σ, failure probability δ, and LoRA hyperparameters (r,αL)(r, _L), as well as the maximum sequence length and dropout rate. Dataset Model N T K l sRsR σ δ r (LoRA) αL _L (LoRA) Length_max Dropout SST-2 RoBERTa-Base 4 100 20 1.0 16 1.0 10−510^-5 16 32 128 0.1 QQP 4 100 20 1.0 16 1.0 10−610^-6 16 32 128 0.1 QNLI 4 100 20 1.0 16 1.0 10−510^-5 16 32 128 0.1 MNLI 4 100 20 1.0 16 1.0 10−610^-6 16 32 128 0.1 Other setup. DP-FedAvg, DP-SCAFFOLD, DP-FedAvg-LS, and DP-FedSAM adopt learning rates selected from 10−2,3×10−2,5×10−2,10−1,3×10−110^-2,3×10^-2,5×10^-2,10^-1,3×10^-1 with a fixed weight decay of 0.0010.001. For DP-FedSAM, the SAM perturbation ρ is selected via grid search over 0.01, 0.05, 0.1, 0.5\0.01,\,0.05,\,0.1,\,0.5\. For DP-LocalAdamW and DP-FedAdamW, the learning rate is chosen from 10−4,2×10−4,3×10−4,5×10−4,8×10−4,10−310^-4,2×10^-4,3×10^-4,5×10^-4,8×10^-4,10^-3, combined with weight decay in 0.01,0.001\0.01,0.001\ and standard AdamW parameters (β1=0.9,β2=0.999)( _1=0.9, _2=0.999). We employ cosine learning-rate decay throughout, and DP-FedAdamW additionally uses parameter γ=0.5γ=0.5 and weight decay λ=0.01λ=0.01. For all visual fine-tuning tasks, we initialize from official ImageNet-22K pre-trained weights to ensure consistency across methods. Across all tasks, the gradient clipping threshold is chosen by grid search over 0.05, 0.1, 0.2, 0.3, 0.5\0.05,\,0.1,\,0.2,\,0.3,\,0.5\, and we find that larger clipping values can trigger gradient explosion. To obtain a simple and robust configuration across all benchmarks, we therefore adopt a unified clipping threshold of C=0.1C=0.1. Table 17: Hyperparameter configuration of ResNet-18 (CIFAR-10/100) across different algorithms. Method Local Optimizer η β1 _1 β2 _2 Weight Decay DP-FedAvg SGD 10−110^-1 - - 0.001 DP-SCAFFOLD SGD 10−110^-1 - - 0.001 DP-FedAvg-LS SGD 10−110^-1 - - 0.001 DP-FedSAM SAM 10−110^-1 - - 0.001 DP-LocalAdamW AdamW 3×10−43×10^-4 0.9 0.999 0.01 DP-FedAdamW AdamW 3×10−43×10^-4 0.9 0.999 0.01 10 DP-LocalAdamW Algorithm For completeness, we provide in Algorithm 2 the full local training procedure of DP-LocalAdamW. 11 Theoretical Analysis Details 11.1 Proof of Theorem 1 and Convergence Analysis Assumption A.1 (Smoothness). (Smoothness) The non-convex fif_i is a L-smooth function for all i∈[m]i∈[m], i.e., ‖∇fi(1)−∇fi(2)‖≤L‖1−2‖\|∇ f_i( θ_1)-∇ f_i( θ_2)\|≤ L\| θ_1- θ_2\|, for all 1,2∈ℝd θ_1, θ_2 ^d. Assumption A.2 (Bounded Stochastic Gradient). it=∇fi(it,ξit) g_i^t=∇ f_i( θ_i^t, _i^t) computed by using a sampled mini-batch data ξit _i^t in the local client i is an unbiased estimator of ∇fi∇ f_i with bounded variance, i.e., ξit[it]=∇fi(it)E_ _i^t[ g_i^t]=∇ f_i( θ_i^t) and ξit‖git−∇fi(it)‖2≤σl2E_ _i^t\|g_i^t-∇ f_i( θ_i^t)\|^2≤ _l^2, for all it∈ℝd θ_i^t ^d. Assumption A.3 (Bounded Stochastic Gradient I). Each element of stochastic gradient it g_i^t is bounded, i.e., ‖it‖∞=‖fi(it,ξit)‖∞≤Gg\| g_i^t\|_∞=\|f_i( θ_i^t, _i^t)\|_∞≤ G_g, for all it∈ℝd θ_i^t ^d and any sampled mini-batch data ξit _i^t. Assumption A.4 (Bounded Heterogeneity). The gradient dissimilarity between clients is bounded: ‖∇fi(1)−∇fi(2)‖2≤σg2\|∇ f_i( θ_1)-∇ f_i( θ_2)\|^2≤ _g^2, for all ∈ℝd θ ^d. In this section, we give the theoretical analysis of our proposed DP-FedAdamW algorithm. Firstly we state some standard assumptions for the non-convex function F. We use the common gradient bound condition in our proof. Then we can upper bound GϑG_ as: Gϑ G_ \ =‖ϑit,k‖∞=‖1(1−β1k)2/(1−β2t)it,k+ϵ‖∞ = \\| _i^t,k\|_∞ \= \\| 1 (1- _1^k)^2/(1- _2^t) v_i^t,k+ε\|_∞ \ =‖1(1−β1k)2/(1−β2t)(β2it,k−1+(1−β2)it,k⋅it,k)+ϵ‖∞. = \\| 1 (1- _1^k)^2/(1- _2^t)( _2 v^t,k-1_i+ (1- _2 ) g^t,k_i· g^t,k_i)+ε\|_∞ \. (1−β1k)2/(1−β2t)(β2it,k−1+(1−β2)it,k⋅it,k (1- _1^k)^2/(1- _2^t)( _2 v^t,k-1_i+ (1- _2 ) g^t,k_i· g^t,k_i is bounded as: ‖(1−β1k)2/(1−β2t)(β2it,k−1+(1−β2)it,k⋅it,k‖∞≤Gg.\| (1- _1^k)^2/(1- _2^t)( _2 v^t,k-1_i+ (1- _2 ) g^t,k_i· g^t,k_i\|_∞≤ G_g. (15) Thus we bound GϑG_ as 1Gg≤Gϑ≤1ϵ 1G_g≤ G_ ≤ 1ε. 11.2 Main Lemmas Lemma 1. Suppose X1,⋯,Xτ⊂ℝd \X_1,·s,X_τ \ ^d be random variables that are potentially dependent. If their marginal means and variances satisfy [Xi]=μiE [X_i ]= _i and [∥Xi−E [\|X_i- . μi∥2]≤σ2 . _i\|^2 ]≤σ^2, then it holds that [‖∑i=1τXi‖2]≤‖∑i=1τμi‖2+τ2σ2.E [ \| _i=1^τX_i \|^2 ]≤ \| _i=1^τ _i \|^2+τ^2σ^2. If they are correlated in the Markov way such that [Xi∣Xi−1,⋯X1]=μiE [X_i X_i-1,·s X_1 ]= _i and [∥Xi−μi∥2∣E [ \|X_i- _i \|^2 . μi]≤σ2 . _i ]≤σ^2, i.e., the variables Xi−μi \X_i- _i \ form a martingale. Then the following tighter bound holds: [‖∑i=1τXi‖2]≤2[‖∑i=1τμi‖2]+2τσ2.E [ \| _i=1^τX_i \|^2 ]≤ 2E [ \| _i=1^τ _i \|^2 ]+2τσ^2. Lemma 2. Given vectors v1,⋯,vN∈ℝdv_1,·s,v_N ^d and v¯=1N∑i=1Nvi v= 1N _i=1^Nv_i, if we sample ⊂1,⋯,NS⊂\1,·s,N\ uniformly randomly such that ||=S|S|=S, then it holds that [‖1S∑i∈vi‖2]=‖v¯‖2+N−S(N−1)1N∑i=1N‖vi−v¯‖2.E [ \| 1S _i v_i \|^2 ]=\| v\|^2+ N-SS(N-1) 1N _i=1^N \|v_i- v \|^2. Proof. Letting i∈I\i \ be the indicator for the event i∈ri _r, we prove this lemma by direct calculation as follows: [‖1S∑i∈vi‖2] [ \| 1S _i v_i \|^2 ] =[‖1S∑i=1Nvii∈‖2] =E [ \| 1S _i=1^Nv_iI\i \ \|^2 ] =1S2[(∑i‖vi‖2i∈+2∑i<jvi⊤vji,j∈)] = 1S^2E [ ( _i \|v_i \|^2I\i \+2 _i<jv_i v_jI\i,j \ ) ] =1SN∑i=1N‖vi‖2+1S2S(S−1)N(N−1)2∑i<jvi⊤vj = 1SN _i=1^N \|v_i \|^2+ 1S^2 S(S-1)N(N-1)2 _i<jv_i v_j =1SN∑i=1N‖vi‖2+1S2S(S−1)N(N−1)(‖∑i=1Nvi‖2−∑i=1N‖vi‖2) = 1SN _i=1^N \|v_i \|^2+ 1S^2 S(S-1)N(N-1) ( \| _i=1^Nv_i \|^2- _i=1^N \|v_i \|^2 ) =N−S(N−1)1N∑i=1N‖vi‖2+N(S−1)S(N−1)‖v¯‖2 = N-SS(N-1) 1N _i=1^N \|v_i \|^2+ N(S-1)S(N-1)\| v\|^2 =N−S(N−1)1N∑i=1N‖vi−v¯‖2+‖v¯‖2. = N-SS(N-1) 1N _i=1^N \|v_i- v \|^2+\| v\|^2. ∎ 11.3 Basic Assumptions and Notations Let ℱ0=∅F^0= and ℱit,k:=σ(it,j0≤j≤k∪ℱt)F_i^t,k:=σ ( \ θ_i^t,j \_0≤ j≤ k ^t ) and ℱt+1:=σ(∪iℱit,K)F^t+1:=σ ( _iF_i^t,K ) for all t≥0t≥ 0 where σ(⋅)σ(·) indicates the σ-algebra. Let t[⋅]:=¯[⋅∣ℱt]E_t[·]:= E [· ^t ] be the expectation, conditioned on the filtration ℱtF^t, with respect to the random variables t,ξit,k1≤i≤N,0≤k<K \S^t, \ _i^t,k \_1≤ i≤ N,0≤ k<K \ in the t-th iteration. We also use [⋅]E[·] to denote the global expectation over all randomness in algorithms. Through out the proofs, we use ∑i _i to represent the sum over i∈1,…,Ni∈\1,…,N\, while ∑i∈t _i ^t denotes the sum over i∈ti ^t. Similarly, we use ∑k _k to represent the sum of k∈0,…,K−1k∈\0,…,K-1\. For all t≥0t≥ 0, we define the following auxiliary variables to facilitate proofs: ℰt _t :=[‖∇f(t)⊙1SK∑i∈St∑k=1Kϑit,k−ΔGt+1‖2], =E [ \|∇ f ( θ^t ) 1SK _i∈ S^t _k=1^K _i^t,k- ^t+1_G \|^2 ], Ut U_t :=1NK∑i∑k[‖it,k−t‖]2, = 1NK _i _kE [ \| θ_i^t,k- θ^t \| ]^2, ξit,k _i^t,k :=[it,k+1−it,k∣ℱit,k], =E [ θ_i^t,k+1- θ_i^t,k _i^t,k ], Ξt _t :=1N∑i=1N[‖ξit,0‖2]. = 1N _i=1^NE [ \| _i^t,0 \|^2 ]. Throughout the Appendix, we let Δ:=f(0)−f⋆,G0:=1N∑i‖∇fi(0)‖2,−1:=0 :=f ( θ^0 )-f ,G_0:= 1N _i \|∇ f_i ( θ^0 ) \|^2, θ^-1:= θ^0 and ℰ−1:=E_-1:= [‖∇f(0)−g0‖2]E [ \|∇ f ( θ^0 )-g^0 \|^2 ]. We will use the following foundational lemma for all our algorithms. 11.4 Proof of Theorem 1 Lemma 3. Under Assumption A.1 , if γL≤124γ L≤ 124, the following holds all t≥0t≥ 0 : [f(t+1)]≤[f(t)]−11γ24[‖∇f(t)‖2]+13γ24ℰt+13γ24σ2Gg2s2R2.E [f ( θ^t+1 ) ] [f ( θ^t ) ]- 11γ24E [ \|∇ f ( θ^t ) \|^2 ]+ 13γ24E_t+ 13γ24 σ^2G_g^2s^2R^2. Proof. Since f is L-smooth, we have f(t+1) f ( θ^t+1 ) ≤f(t)+⟨∇f(t),t+1−t⟩+L2‖t+1−t‖2 ≤ f ( θ^t )+ ∇ f ( θ^t ), θ^t+1- θ^t + L2 \| θ^t+1- θ^t \|^2 ≤f(t)+γ⟨∇f(t),ΔGt+1⟩+Lγ22‖ΔGt+1‖2 ≤ f ( θ^t )+γ ∇ f ( θ^t ), ^t+1_G + Lγ^22 \| ^t+1_G \|^2 =f(t)−γ‖∇f(t)⊙1SK∑i∈St∑k=1Kϑit,k‖2+γ⟨∇f(t),∇f(t)⊙1SK∑i∈St∑k=1Kϑit,k−ΔGt+1⟩ =f ( θ^t )-γ \|∇ f ( θ^t ) 1SK _i∈ S^t _k=1^K _i^t,k \|^2+γ ∇ f ( θ^t ),∇ f ( θ^t ) 1SK _i∈ S^t _k=1^K _i^t,k- ^t+1_G +Lγ22‖ΔGt+1‖2. + Lγ^22 \| ^t+1_G \|^2. Since t+1=t−γΔGt+1 θ^t+1= θ^t-γ ^t+1_G, using Young’s inequality, we further have f(t+1) ( θ^t+1 ) ≤f(t)−(γGg2−γ2)‖∇f(t)‖2+γ2‖∇f(t)⊙1SK∑i∈St∑k=1Kϑit,k−ΔGt+1‖2 ≤ f ( θ^t )-(γ G_g^2- γ2) \|∇ f ( θ^t ) \|^2+ γ2 \|∇ f ( θ^t ) 1SK _i∈ S^t _k=1^K _i^t,k- ^t+1_G \|^2 +Lγ2(‖∇f(t)⊙1SK∑i∈St∑k=1Kϑit,k‖2+‖∇f(t)⊙1SK∑i∈St∑k=1Kϑit,k−ΔGt+1‖2) +Lγ^2 ( \|∇ f ( θ^t ) 1SK _i∈ S^t _k=1^K _i^t,k \|^2+ \|∇ f ( θ^t ) 1SK _i∈ S^t _k=1^K _i^t,k- ^t+1_G \|^2 ) ≤f(t)−(γGg2−γ2−Lγ2Gg2)‖∇f(t)‖2+13γ24‖∇f(t)⊙1SK∑i∈St∑k=1Kϑit,k−ΔGt+1‖2+13γ24σ2Gg2s2R2 ≤ f ( θ^t )-(γ G_g^2- γ2-Lγ^2G_g^2) \|∇ f ( θ^t ) \|^2+ 13γ24 \|∇ f ( θ^t ) 1SK _i∈ S^t _k=1^K _i^t,k- ^t+1_G \|^2+ 13γ24 σ^2G_g^2s^2R^2 ≤f(t)−11γ24‖∇f(t)‖2+13γ24‖∇f(t)⊙1SK∑i∈St∑k=1Kϑit,k−ΔGt+1‖2+13γ24σ2Gg2s2R2, ≤ f ( θ^t )- 11γ24 \|∇ f ( θ^t ) \|^2+ 13γ24 \|∇ f ( θ^t ) 1SK _i∈ S^t _k=1^K _i^t,k- ^t+1_G \|^2+ 13γ24 σ^2G_g^2s^2R^2, where the last inequality is due to γL≤124γ L≤ 124, −(γGg2−γ2−Lγ2Gg2)≤−11γ24-(γ G_g^2- γ2-Lγ^2G_g^2)≤- 11γ24. Taking the global expectation, we finish the proof. ∎ Lemma 4. If γL≤γ6γ L≤ γ6, the following holds for t≥1t≥ 1 : ℰt≤(1−8γ9)ℰt−1+4γ2L2γ[‖∇f(t−1)‖2]+2γ2σl2SKϵ2+4γϵ2L2Ut.E_t≤ (1- 8γ9 )E_t-1+ 4γ^2L^2γE [ \|∇ f ( θ^t-1 ) \|^2 ]+ 2γ^2 _l^2SKε^2+4 γε^2L^2U_t. Additionally, it holds for t=0t=0 that ℰ0≤(1−γ)ℰ−1+2γ2σl2SK+4γL2U0.E_0≤(1-γ)E_-1+ 2γ^2 _l^2SK+4γ L^2U_0. Proof. For t>1t>1, ℰt= _t= [‖1SK∑i∈St∑k=1K∇f(t)⊙ϑit,k−ΔGt+1‖2] [ \| 1SK _i∈ S^t _k=1^K∇ f ( θ^t ) _i^t,k- ^t+1_G \|^2 ] = = [‖(1−γ)(1SK∑i∈St∑k=1K∇f(t)⊙ϑit,k−ΔGt)+γ(1SK∑i∈St∑k=1K∇f(t)⊙ϑit,k−1SK∑i∈St∑k=1Kgit,k⊙ϑit,k)‖2] [ \|(1-γ) ( 1SK _i∈ S^t _k=1^K∇ f ( θ^t ) _i^t,k- ^t_G )+γ ( 1SK _i∈ S^t _k=1^K∇ f ( θ^t ) _i^t,k- 1SK _i∈ S^t _k=1^Kg_i^t,k _i^t,k ) \|^2 ] ≤ ≤ [‖(1−γ)(1SK∑i∈St∑k=1K∇f(t)⊙ϑit,k−ΔGt)‖2]+γ21ϵ2[‖∇f(t)−1SK∑i∈St∑k=1Kgit,k‖2] [ \|(1-γ) ( 1SK _i∈ S^t _k=1^K∇ f ( θ^t ) _i^t,k- ^t_G ) \|^2 ]+γ^2 1ε^2E [ \|∇ f ( θ^t )- 1SK _i∈ S^t _k=1^Kg_i^t,k \|^2 ] +2γ[⟨(1−γ)(1SK∑i∈St∑k=1K∇f(t)⊙ϑit,k−ΔGt),1SK∑i∈St∑k=1K∇f(t)⊙ϑit,k−1SK∑i∈St∑k=1Kgit,k⊙ϑit,k⟩]. +2 [ (1-γ) ( 1SK _i∈ S^t _k=1^K∇ f ( θ^t ) _i^t,k- ^t_G ), 1SK _i∈ S^t _k=1^K∇ f ( θ^t ) _i^t,k- 1SK _i∈ S^t _k=1^Kg_i^t,k _i^t,k ]. Note that ∇F(it,k;ξit,k)0≤k<K \∇ F ( θ_i^t,k; _i^t,k ) \_0≤ k<K are sequentially correlated. Applying the AM-GM inequality and Lemma 1, we have ℰt≤(1+γ2)[‖(1−γ)(∇f(t)−ΔGt)‖2]+2γϵ2L2Ut+2γ2ϵ2(σl2SK+L2Ut).E_t≤ (1+ γ2 )E [ \|(1-γ) (∇ f ( θ^t )- ^t_G ) \|^2 ]+2 γε^2L^2U_t+2 γ^2ε^2 ( _l^2SK+L^2U_t ). Using the AM-GM inequality again and Assumption A.1, we have ℰt _t ≤(1−γ)2(1+γ2)[(1+γ2)ℰt−1+(1+2γ)L2[‖t−t−1‖2]]+2γ2σl2SKϵ2+4γϵ2L2Ut ≤(1-γ)^2 (1+ γ2 ) [ (1+ γ2 )E_t-1+ (1+ 2γ )L^2E [ \| θ^t- θ^t-1 \|^2 ] ]+ 2γ^2 _l^2SKε^2+4 γε^2L^2U_t ≤(1−γ)ℰt−1+2γL2[‖t−t−1‖2]+2γ2σl2ϵ2SK+4γϵ2L2Ut ≤(1-γ)E_t-1+ 2γL^2E [ \| θ^t- θ^t-1 \|^2 ]+ 2γ^2 _l^2ε^2SK+4 γε^2L^2U_t ≤(1−8γ9)ℰt−1+4γ2L2γ[‖∇f(t−1)‖2]+2γ2σl2ϵ2SK+4γϵ2L2Ut, ≤ (1- 8γ9 )E_t-1+4 γ^2L^2γE [ \|∇ f ( θ^t-1 ) \|^2 ]+ 2γ^2 _l^2ε^2SK+4 γε^2L^2U_t, where we plug in ‖t−t−1‖2≤2γ2(‖∇f(t−1)‖2+‖ΔGt−∇f(t−1)‖2) \| θ^t- θ^t-1 \|^2≤ 2γ^2 ( \|∇ f ( θ^t-1 ) \|^2+ \| ^t_G-∇ f ( θ^t-1 ) \|^2 ) and use γL≤γ6γ L≤ γ6 in the last inequality. Similarly for t=0t=0, ℰ0 _0 ≤(1+γ2)[‖(1−γ)(∇f(0)−g0)‖2]+2γϵ2L2U0+2γ2ϵ2(σl2SK+L2U0) ≤ (1+ γ2 )E [ \|(1-γ) (∇ f ( θ^0 )-g^0 ) \|^2 ]+2 γε^2L^2U_0+2 γ^2ε^2 ( _l^2SK+L^2U_0 ) ≤(1−γ)ℰ−1+2γ2σl2SKϵ2+4γϵ2L2U0. ≤(1-γ)E_-1+ 2γ^2 _l^2SKε^2+4 γε^2L^2U_0. ∎ Lemma 5. If ηLK≤1γη LK≤ 1γ, the following holds for t≥0t≥ 0 : Ut≤2eK2Ξt+Kη2γ21ϵ2σl2(1+2K3L2η2γ2).U_t≤ 2eK^2 _t+Kη^2γ^2 1ε^2 _l^2 (1+2K^3L^2η^2γ^2 ). Proof. Recall that ξit,k:=[it,k+1−it,k∣ℱit,k]=−η((1−γ)ΔGt+γ∇fi(it,k)⊙ϑit,k) _i^t,k:=E [ θ_i^t,k+1- θ_i^t,k _i^t,k ]=-η ((1-γ) ^t_G+γ∇ f_i ( θ_i^t,k ) _i^t,k ). Then we have [‖ξit,j−ξit,j−1‖2] [ \| _i^t,j- _i^t,j-1 \|^2 ] ≤1ϵ2η2L2γ2[‖it,j−it,j−1‖2] ≤ 1ε^2η^2L^2γ^2E [ \| θ_i^t,j- θ_i^t,j-1 \|^2 ] ≤1ϵ2η2L2γ2(η2γ2σl2+[∥ξit,j−1∥2). ≤ 1ε^2η^2L^2γ^2 (η^2γ^2 _l^2+E [ \| _i^t,j-1 \|^2 ) .. For any 1≤j≤k−1≤K−21≤ j≤ k-1≤ K-2, using ηL≤1γK≤1γ(k+1)η L≤ 1γ K≤ 1γ(k+1), we have [‖ξit,j‖2] [ \| _i^t,j \|^2 ] ≤(1+1k)[‖ξit,j−1‖2]+(1+k)[‖ξit,j−ξit,j−1‖2] ≤ (1+ 1k )E [ \| _i^t,j-1 \|^2 ]+(1+k)E [ \| _i^t,j- _i^t,j-1 \|^2 ] ≤(1+2k)[‖ξit,j−1‖2]+(k+1)1ϵ2L2η4γ4σl2 ≤ (1+ 2k )E [ \| _i^t,j-1 \|^2 ]+(k+1) 1ε^2L^2η^4γ^4 _l^2 ≤e2[‖ξit,0‖2]+41ϵ2k2L2η4γ4σl2. ≤ e^2E [ \| _i^t,0 \|^2 ]+4 1ε^2k^2L^2η^4γ^4 _l^2. where the last inequality is by unrolling the recursive bound and using (1+2k)k≤e2 (1+ 2k )^k≤ e^2. By Lemma 1 , it holds that for k≥2k≥ 2, [‖it,k−t‖2] [ \| θ_i^t,k- θ^t \|^2 ] ≤2[‖∑j=0k−1ξit,j‖2]+21ϵ2kη2γ2σl2 ≤ 2E [ \| _j=0^k-1 _i^t,j \|^2 ]+2 1ε^2kη^2γ^2 _l^2 ≤2k∑j=0k−1[‖ξit,k‖2]+21ϵ2kη2γ2σl2 ≤ 2k _j=0^k-1E [ \| _i^t,k \|^2 ]+2 1ε^2kη^2γ^2 _l^2 ≤2e2k2[‖ξit,0‖2]+21ϵ2kη2γ2σl2(1+4k3L2η2γ2). ≤ 2e^2k^2E [ \| _i^t,0 \|^2 ]+2 1ε^2kη^2γ^2 _l^2 (1+4k^3L^2η^2γ^2 ). This is also valid for k=0,1k=0,1. Summing up over i and k, we finish the proof. ∎ Lemma 6. If 288e(ηKL)2((1−γ)2+e(γγLR)2)≤1288e(η KL)^2 ((1-γ)^2+e(γ LR)^2 )≤ 1, then it holds for t≥0t≥ 0 that ∑t=0R−1Ξt≤172eK2L2∑t=−1R−2(ℰt+[‖∇f(t)‖2])+2η2γ21ϵ2eRG0. _t=0^R-1 _t≤ 172eK^2L^2 _t=-1^R-2 (E_t+E [ \|∇ f ( θ^t ) \|^2 ] )+2η^2γ^2 1ε^2eRG_0. Proof. Note that ξit,0=−η((1−γ)ΔGt+γ∇fi(t)⊙ϑit,k) _i^t,0=-η ((1-γ) ^t_G+γ∇ f_i ( θ^t ) _i^t,k ), 1N∑i=1N‖ξit,0‖2≤2η2((1−γ)2‖ΔGt‖2+γ21N∑i=1N‖∇fi(t)‖2). 1N _i=1^N \| _i^t,0 \|^2≤ 2η^2 ((1-γ)^2 \| ^t_G \|^2+γ^2 1N _i=1^N \|∇ f_i ( θ^t ) \|^2 ). Using Young’s inequality, we have for any q>0q>0 that [‖∇fi(t)‖2] [ \|∇ f_i ( θ^t ) \|^2 ] ≤(1+q)[‖∇fi(t−1)‖2]+(1+q−1)L2[‖t−t−1‖2] ≤(1+q)E [ \|∇ f_i ( θ^t-1 ) \|^2 ]+ (1+q^-1 )L^2E [ \| θ^t- θ^t-1 \|^2 ] ≤(1+q)[‖∇fi(t−1)‖2]+2(1+q−1)γ2L2(ℰt−1+[‖∇f(t−1)‖2]) ≤(1+q)E [ \|∇ f_i ( θ^t-1 ) \|^2 ]+2 (1+q^-1 )γ^2L^2 (E_t-1+E [ \|∇ f ( θ^t-1 ) \|^2 ] ) ≤(1+q)t[∥∇fi(0)∥2]+2qγ2L2∑j=0t−1(ℰj+[∥∇f(j)∥2)(1+q)t−j. ≤(1+q)^tE [ \|∇ f_i ( θ^0 ) \|^2 ]+ 2qγ^2L^2 _j=0^t-1 (E_j+E [ \|∇ f ( θ^j ) \|^2 )(1+q)^t-j .. Take q=1tq= 1t and we have [∥∇fi(t)∥2]≤e[∥∇fi(0)∥2]+2e(t+1)γ2L2∑j=0t−1(ℰj+[∥∇f(j)∥2).E [ \|∇ f_i ( θ^t ) \|^2 ]≤ eE [ \|∇ f_i ( θ^0 ) \|^2 ]+2e(t+1)γ^2L^2 _j=0^t-1 (E_j+E [ \|∇ f ( θ^j ) \|^2 ) .. (16) Note that this inequality is valid for t=0t=0. Therefore, using (16), we have ∑t=0R−1Ξt≤ _t=0^R-1 _t≤ ∑t=0R−12η2[(1−γ)2‖ΔGt‖2+γ21ϵ21N∑i=1N‖∇fi(t)‖2] _t=0^R-12η^2E [(1-γ)^2 \| ^t_G \|^2+γ^2 1ε^2 1N _i=1^N \|∇ f_i ( θ^t ) \|^2 ] ≤ ≤ ∑t=0R−12η2(2(1−γ)2(ℰt−1+[‖∇f(t−1)‖2])+γ21ϵ21N∑i=1N[‖∇fi(t)‖2]) _t=0^R-12η^2 (2(1-γ)^2 (E_t-1+E [ \|∇ f ( θ^t-1 ) \|^2 ] )+γ^2 1ε^2 1N _i=1^NE [ \|∇ f_i ( θ^t ) \|^2 ] ) ≤ ≤ ∑t=0R−14η2(1−γ)2(ℰt−1+[‖∇f(t−1)‖2]) _t=0^R-14η^2(1-γ)^2 (E_t-1+E [ \|∇ f ( θ^t-1 ) \|^2 ] ) +2η2γ21ϵ2∑t=0R−1(eN∑i=1N[‖∇fi(0)‖2]+2e(t+1)(γL)2∑j=0t−1(ℰj+[‖∇f(j)‖2])) +2η^2γ^2 1ε^2 _t=0^R-1 ( eN _i=1^NE [ \|∇ f_i ( θ^0 ) \|^2 ]+2e(t+1)(γ L)^2 _j=0^t-1 (E_j+E [ \|∇ f ( θ^j ) \|^2 ] ) ) ≤ ≤ 4η2(1−γ)2∑t=0R−1(ℰt−1+[‖∇f(t−1)‖2]) 4η^2(1-γ)^2 _t=0^R-1 (E_t-1+E [ \|∇ f ( θ^t-1 ) \|^2 ] ) +2η2γ21ϵ2(eRG0+2e(γLR)2∑t=0R−2(ℰt+[‖∇f(t)‖2])). +2η^2γ^2 1ε^2 (eRG_0+2e(γ LR)^2 _t=0^R-2 (E_t+E [ \|∇ f ( θ^t ) \|^2 ] ) ). Rearranging the equation and applying the upper bound of η, we finish the proof. ∎ Theorem 3 (Convergence for non-convex functions). Under Assumption A.1 and A.2 and A.3 , if we take g0=0g^0=0, γ=min,SKLΔϵ2σl2R for any constant c∈(0,1],γ=min124LGg,γ6L, γ= \, SKL ε^2 _l^2R \ for any constant c∈(0,1], γ= \ 124LG_g, γ6L \, ηKL≲min1,1γγLR,(LΔG0γ3R)1/2,1(γN)1/2,1(γ3NK)1/4. η KL \1, 1γ LR, ( L G_0γ^3R )^1/2, 1(γ N)^1/2, 1 (γ^3NK )^1/4 \. then DP-FedAdamW converges as 1R∑t=0R−1[‖∇f(t)‖2]≲LΔσl2SKR+LΔR+σ2Gg2s2R2. 1R _t=0^R-1E [ \|∇ f ( θ^t ) \|^2 ] L _l^2SKR+ L R+ σ^2G_g^2s^2R^2. Here G0:=1N∑i=1N‖∇fi(0)‖2G_0:= 1N _i=1^N \|∇ f_i ( θ^0 ) \|^2. Proof. Combining Lemma 3 and 4, we have ℰt≤ _t≤ (1−8γ9)ℰt−1+4(γL)2γ[‖∇f(t−1)‖2]+2γ2σl2SKϵ2 (1- 8γ9 )E_t-1+4 (γ L)^2γE [ \|∇ f ( θ^t-1 ) \|^2 ]+ 2γ^2 _l^2SKε^2 +4γϵ2L2(2eK2Ξt+Kη2γ21ϵ2σl2(1+2K3L2η2γ2), +4 γε^2L^2 (2eK^2 _t+Kη^2γ^2 1ε^2 _l^2 (1+2K^3L^2η^2γ^2 ) ., and ℰ0≤(1−γ)ℰ−1+2γ2σl2SKϵ2+4γ1ϵ2L2(2eK2Ξ0+Kη2γ21ϵ2σl2(1+2K3L2η2γ2)).E_0≤(1-γ)E_-1+ 2γ^2 _l^2SKε^2+4γ 1ε^2L^2 (2eK^2 _0+Kη^2γ^2 1ε^2 _l^2 (1+2K^3L^2η^2γ^2 ) ). Summing over t from 0 to R−1R-1 and applying Lemma 6, ∑t=0R−1ℰt≤ _t=0^R-1E_t≤ (1−8γ9)∑t=−1R−2ℰt+4(γL)2γ∑t=0R−2[‖∇f(t)‖2]+2γ2σl2SKϵ2R (1- 8γ9 ) _t=-1^R-2E_t+4 (γ L)^2γ _t=0^R-2E [ \|∇ f ( θ^t ) \|^2 ]+2 γ^2 _l^2SKε^2R +4γ1ϵ2L2(2eK2∑t=0R−1Ξt+RKη2γ21ϵ2σl2(1+2K3L2η2γ2)) +4γ 1ε^2L^2 (2eK^2 _t=0^R-1 _t+RKη^2γ^2 1ε^2 _l^2 (1+2K^3L^2η^2γ^2 ) ) ≤ ≤ (1−7γ9)∑t=−1R−2ℰt+(4(γL)2γ+γ9)∑t=−1R−2[‖∇f(t)‖2]+16γ31ϵ4(eηKL)2RG0 (1- 7γ9 ) _t=-1^R-2E_t+ (4 (γ L)^2γ+ γ9 ) _t=-1^R-2E [ \|∇ f ( θ^t ) \|^2 ]+6γ^3 1ε^4(eη KL)^2RG_0 +2γ2σl2SKϵ2R+4γ31ϵ4(ηKL)2(1K+2(ηKLγ)2)σl2R + 2γ^2 _l^2SKε^2R+4γ^3 1ε^4(η KL)^2 ( 1K+2(η KLγ)^2 ) _l^2R ≤ ≤ (1−7γ9)∑t=−1R−2ℰt+2γ9∑t=−1R−2[‖∇f(t)‖2]+16γ31ϵ4(eηKL)2RG0+4γ2σl2SKϵ2R. (1- 7γ9 ) _t=-1^R-2E_t+ 2γ9 _t=-1^R-2E [ \|∇ f ( θ^t ) \|^2 ]+6γ^3 1ε^4(eη KL)^2RG_0+ 4γ^2 _l^2SKε^2R. Here in the last inequality we apply 4γ1ϵ4(ηKL)2(1K+2(ηKLγ)2)≤2NK and γL≤γ6.4γ 1ε^4(η KL)^2 ( 1K+2(η KLγ)^2 )≤ 2NK and γ L≤ γ6. Therefore, ∑t=0R−1ℰt≤97γℰ−1+27[∑t=−1R−2‖∇f(t)‖2]+14471ϵ4(eγηKL)2G0R+36γσl27SKϵ2R. _t=0^R-1E_t≤ 97γE_-1+ 27E [ _t=-1^R-2 \|∇ f ( θ^t ) \|^2 ]+ 1447 1ε^4(eγη KL)^2G_0R+ 36γ _l^27SKε^2R. Combine this inequality with Lemma 3 and we get 1γ[f(t)−f(0)]≤−17∑t=0R−1[‖∇f(t)‖2]+3956γℰ−1+7871ϵ4(eγηKL)2G0R+39γσl214SKϵ2R. 1γE [f ( θ^t )-f ( θ^0 ) ]≤- 17 _t=0^R-1E [ \|∇ f ( θ^t ) \|^2 ]+ 3956γE_-1+ 787 1ε^4(eγη KL)^2G_0R+ 39γ _l^214SKε^2R. Finally, noticing that g0=0g^0=0 implies ℰ−1≤2L(f(0)−f∗)=2LΔE_-1≤ 2L (f ( θ^0 )-f^* )=2L , we obtain 1R∑t=0R−1[‖∇f(t)‖2] 1R _t=0^R-1E [ \|∇ f ( θ^t ) \|^2 ] ≲LΔγLR+ℰ−1γT+(γηKL)21ϵ4G0+γσl2SKϵ2+σ2Gg2s2T2 L γ LR+ E_-1γ T+(γη KL)^2 1ε^4G_0+ γ _l^2SKε^2+ σ^2G_g^2s^2T^2 ≲LΔT+LΔγT+γσl2SKϵ2+(γηKL)2G01ϵ4+σ2Gg2s2T2 L T+ L γ T+ γ _l^2SKε^2+(γη KL)^2G_0 1ε^4+ σ^2G_g^2s^2T^2 ≲LΔT+LΔσl2SKTϵ2+σ2Gg2s2T2. L T+ L _l^2SKTε^2+ σ^2G_g^2s^2T^2. ∎