Paper deep dive
FL-OA: A Byzantine-Robust Federated Learning Framework with Outsourced Auditing for Intelligent Devices
Hongliang Zhang, Zhongyuan Yu, Fenghua Xu, Teng Hu, Jian Meng, Jiguo Yu
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Federated learning (FL) enables multiple intelligent devices to collaboratively train a high-accuracy model without sharing raw data. However, due to its distributed nature, FL is vulnerable to Byzantine attacks. Existing defense methods rely on strong assumptions, such as the proportion of malicious devices not exceeding 50\%, or the server having an additional root dataset that matches the training task. Moreover, they show limited efficacy as they overlook $(i)$ the divergence among benign updates and $(ii)$ the curse of dimensionality involved in comparing two high-dimensional updates. To solve these concerns, we propose FL-OA, a Byzantine-robust federated learning framework utilizing outsourced auditing. In FL-OA, the server collaborates with third-party organization that holds an additional root dataset to perform outsourced auditing, thereby enabling the server to achieve robust aggregation without strong assumptions. Additionally, FL-OA introduces a gradient ascent step and a correction term during local training to mitigate the divergence among benign updates, and designs a parameter importance indicator to extract critical parameters for auditing, alleviating the curse of dimensionality. We further provide a detailed theoretical analysis of FL-OA. Extensive experiments demonstrate that FL-OA outperforms existing defense methods against Byzantine attacks.
Tags
Links
- Source: https://arxiv.org/abs/2608.01095v1
- Canonical: https://arxiv.org/abs/2608.01095v1
Trouble viewing inline? Open PDF directly →
Full Text
97,897 characters extracted from source content.
Expand or collapse full text
JOURNAL OF L A T E X CLASS FILES, VOL. 14, NO. 8, AUGUST 20211 FL-OA: A Byzantine-Robust Federated Learning Framework with Outsourced Auditing for Intelligent Devices Hongliang Zhang, Zhongyuan Yu, Fenghua Xu, Teng Hu, Jian Meng, Jiguo Yu, Fellow, IEEE Abstract—Federated learning (FL) enables multiple intelligent devices to collaboratively train a high-accuracy model without sharing raw data. However, due to its distributed nature, FL is vulnerable to Byzantine attacks. Existing defense methods rely on strong assumptions, such as the proportion of malicious devices not exceeding 50%, or the server having an additional root dataset that matches the training task. Moreover, they show limited efficacy as they overlook (i) the divergence among benign updates and (i) the curse of dimensionality involved in comparing two high-dimensional updates. To solve these concerns, we propose FL-OA, a Byzantine-robust federated learning framework utilizing outsourced auditing. In FL-OA, the server collaborates with third-party organization that holds an additional root dataset to perform outsourced auditing, thereby enabling the server to achieve robust aggregation without strong assumptions. Additionally, FL-OA introduces a gradient ascent step and a correction term during local training to mitigate the divergence among benign updates, and designs a parameter importance indicator to extract critical parameters for auditing, alleviating the curse of dimensionality. We further provide a detailed theoretical analysis of FL-OA. Extensive experiments demonstrate that FL-OA outperforms existing defense methods against Byzantine attacks. Index Terms—Federated learning, Byzantine attacks, Out- sourced auditing, Divergence, Curse of dimensionality. I. INTRODUCTION With the rapid development of internet of things, the intelligent device market has expanded rapidly. They offer advanced intelligence and communication capabilities. By collecting large amounts of data generated by these devices, their service provider (i.e., server) can train artificial in- This work was partially supported by NSF of China under Grants 62272256 and 62202250, and the Shandong Province Youth Innovation Team Project under Grant 2024KJH032. (Corresponding author: Jiguo Yu) H. Zhang is with the Key Laboratory of Computing Power Network and Information Security, Ministry of Education, Shandong Computer Science Center, Qilu University of Technology (Shandong Academy of Sciences), Jinan, 250353, China, Email: b1043123004@stu.qlu.edu.cn. Z. Yu is with the College of computer science and technol- ogy, China University of Petroleum, Qingdao, 266580, China, Email: yuzhy24601@gmail.com. F. Xu is with the Cyber Security Institute, University of Science and Technology of China, Hefei, 230026, China, Email: nstlxfh@gmail.com. T. Hu is with Institute of Computer Application, China Academy of Engi- neering Physics, Mianyang, 621900, China, Email: mailhuteng@foxmail.com. J. Meng is with Inspur Software Group Ltd., Jinan, 250101, China, Email: mengjian@inspur.com J. Yu is with School of Computer Science and Engineering, University of Electronic Science and Technology of China, Chengdu, 611731, China, and also with the Big Data Institute, Qilu University of Technology, Jinan, 250353, China, Email: jiguoyu@sina.com. telligence (AI) models. However, such data contains sensi- tive user information, which poses potential privacy risks. Intelligent Devices ... Local modelLocal modelLocal model Server PhoneLaptop Desktop computer Local dataset Local dataset Local dataset Upload model update Send latest global model Fig. 1. A typical FL architecture. The server, operated by a tech- nology company (e.g., Huawei or Apple), coordinates global model training, while multiple intelligent devices (e.g., cell phones, laptops, and desktops) collaboratively train the model in edge scenarios. To this end, federated learning (FL) has emerged as a distributed computing paradigm that enables multiple intelligent devices to collaboratively train a global AI model, and has been widely ap- plied in scenarios such as health- care, autonomous driving, and financial systems. As shown in Fig. 1, each device trains its lo- cal model using its own local data and submits its model up- date to the server without ex- plicitly sharing raw training data. However, due to the distributed nature of FL, its training pro- cess is inherently vulnerable to Byzantine attacks, in which some devices controlled by an adversary (malicious devices) send malicious updates to the server, degrading the perfor- mance of the global model [1] [2]. To resist Byzantine attacks in FL, several Byzantine-robust strategies [3]–[13] have been proposed. These strategies are mainly divided into two angles [14]. The first angle leverages statistical knowledge to audit model updates submitted by devices [3]–[7]. Nevertheless, these methods typically assume that the majority of devices are benign. If this assumption is not met, these methods fail to effectively resist Byzantine attacks. The second type assumes that the server holds an additional root dataset, and uses that dataset as a benchmark to identify malicious updates [8]–[13]. Compared to the first angle, the second is more accurate in auditing malicious be- havior from devices and does not require the assumption about the proportion of benign devices. However, due to privacy and regulations, constructing the additional dataset required by the second type is difficult in real-world scenarios, as the server cannot directly obtain data samples from intelligent devices [15]. This inspires the question: Can the server collaborate with a third-organization holding an additional root dataset to outsource the auditing process, enabling Byzantine-robust aggregation without directly acquiring that dataset? Besides the above mentioned, in the practical world, the Byzantine-robust strategies are confronted with the follow- ing challenges. i) Divergence of model updates. In dis- tributed scenarios, FL data is usually heterogeneous, i.e., Non-Independent and Identically Distributed (Non-IID) [16]– arXiv:2608.01095v1 [cs.LG] 2 Aug 2026 JOURNAL OF L A T E X CLASS FILES, VOL. 14, NO. 8, AUGUST 20212 [19]. Existing study [20] observes that Non-IID data leads to different optimal solutions across devices, which causes the divergence among their model updates. However, this divergence interferes with the differences calculated using cosine similarity or Euclidean distance, making it difficult for defense strategies (e.g., [3]–[13], [21]–[23]) to distinguish whether the difference comes from divergence or Byzantine attacks. i) Curse of dimensionality. We notice that existing defense strategies (e.g., [3]–[13], [21]–[23]) evaluate the dif- ferences among model updates using metrics such as cosine similarity or Euclidean distance. However, some studies [24] [25] observe that calculating the difference between two high- dimensional vectors leads to a curse of dimensionality, i.e., the maximum difference between two vectors decreases with increasing dimensionality of vectors. With the rapid expansion of model architectures in recent years, model updates have be- come higher-dimensional [26], making this curse phenomenon more pronounced. Therefore, both the divergence of model updates and the curse of dimensionality should be carefully addressed in Byzantine-robust FL defenses. To tackle the above issues, this paper proposes a Byzantine- robust FL framework utilizing outsourced auditing, named FL-OA. In this framework, the server collaborates with a third-party organization that holds an additional root dataset to defend against Byzantine attacks from malicious devices. Notably, collaboration between two service providers is rea- sonable in practical FL scenarios. Moreover, to mitigate the divergence among benign updates from different devices, we introduce both a gradient ascent step and a correction term into the local optimization process. In addition, to address the curse of dimensionality in auditing, we design a parameter importance indicator to extract critical parameters of each model update for auditing analysis. The main contributions of this paper are summarized as follows: • We propose FL-OA, a Byzantine-robust federated learn- ing framework. It eliminates the server’s reliance on assumptions regarding the root dataset and the proportion of malicious devices through outsourced auditing. • We theoretically analyze FL-OA to provide theoretical support for its effectiveness. • We evaluate the performance of FL-OA against vari- ous Byzantine attacks under different data distribution settings across multiple datasets. Experimental results demonstrate that FL-OA outperforms existing schemes. I. RELATED WORK In this section, we introduce the Byzantine-robust FL works, which are divided into statistical knowledge-based and the additional root dataset-based approaches. 1) Statistical Knowledge-based Approaches:Statistical knowledge-based approaches typically assume that malicious updates submitted by Byzantine devices are geometrically far away from those of benign devices [3]–[7], [21]–[23]. Specifically, Krum selects a single model update from all submitted updates as the trusted one [3]. It calculates the sum of Euclidean distances between each model update and the others to determine a score, and then selects the update with the lowest score as the global update. Similarly, the work in [7] propose Median, which removes the updates with the largest and smallest Euclidean distances from the mean point, and then takes the median of the remaining updates as the global update. However, their effectiveness is built on the assumption that the majority of devices are benign, which limits their applicability. 2) Additional Root Dataset-based Approaches: Additional Root dataset-based approaches typically assume the server holds a root dataset to identify malicious updates [8]–[13], thereby avoiding the assumption that the majority of devices are benign. Specifically, FLTrust is the first to propose main- taining an additional root dataset on the server and generate a trusted root model update [8]. More specifically, the server calculates a trust score by comparing the device update with the root model update, and then uses this score to determine whether the update is benign or malicious. Similarly, the work in [9] uses the root update as a reference for scoring submitted model updates, and then determines the aggregation weight of each device according to the resulting scores. However, these works [8] [9] rely on the assumption that the server holds a clean root dataset, hindering their real-world applicability. To this end, the work in [10] propose FL-Auditor, which uses a third-party auditor to review model updates from devices, thereby assisting the server obtain a more robust global model. The core idea of FL-Auditor is that the server cooperates with a third-party that holds the root dataset to audit, avoiding the requirement that the server itself owns the root dataset. Although this design has inspired our work, in practical scenarios, the data distribution of the root dataset differs from that of the device’s local dataset. This causes the root update deviate from benign updates, thereby increasing the difficulty of detecting malicious updates. In addition, these works [8]– [13] struggle to the divergence among model updates caused by Non-IID data, thereby limiting their defense effectiveness. To this end, existing survey [14] reviews local objective regu- larization as an effective approach to mitigating the divergence of model updates. For example, the work in [27] incorporates a model-contrastive term into the FL objective to align the local model with the global model. Meanwhile, the work in [28] introduces a proximity term (prox term) during local training to adjust the direction of the local model closer to the global model. Similarly, the work in [20] propose a correction term to constrain the direction of model updates at each device. Although these works [20] [27] [28] cannot defend against Byzantine attacks, they inspire us to mitigate the divergence of benign updates, thereby contributing to robust aggregation. 100200300400500 Dimensionality 0 1 2 R(d) Euclidean distance Cosine similarity Fig. 2. The relative maximum difference with respect to di- mensionality. Notably, existing works [3]– [13], [21]–[23] confront the curse of dimensionality with increasing model scales, regardless of whether they rely on statistical knowl- edge or an additional root dataset. Specifically, when model updates have high dimensionality, the Eu- clidean distance or cosine similar- ity computed by these works yields the curse of dimensional- ity. To empirically validate this issue, we conduct the following JOURNAL OF L A T E X CLASS FILES, VOL. 14, NO. 8, AUGUST 20213 experiment. The relative maximum difference is defined as follows: R(d) = log D max (d)− D min (d) D min (d) ,(1) where D max (d) and D min (d) denote the maximum and mini- mum pairwise distances, respectively, in a d-dimensional space. From Fig. 2, it is observed that R(d) gradually decreases with increasing dimensionality, reflecting the curse of dimensional- ity in high-dimensional spaces. Thus, mitigating the curse is crucial for auditing model updates. I. PROBLEM FORMULATION In this section, we provide the system architecture, potential threats, and design goals for FL-OA. A. System Architecture As shown in Fig. 3, FL-OA consists of a Task Server (TS), an Outsourced Server (OS), and intelligent devices. The effectiveness of FL-OA is predicated on the assumption that the task server cooperates with an outsourced server that maintains the additional root datasets required for training tasks. The concrete roles in FL-OA are elaborated as follows. • The Task Server (TS) is the central server (service provider) in the FL system, responsible for publishing training tasks to the entire FL system. • The Outsourced Server (OS) is a third-party server (ser- vice provider) that maintains the additional root dataset required by TS’s task. It is responsible for auditing the model updates from devices, thereby providing guidance for the TS to perform robust aggregation. • Intelligent devices are responsible for performing local training based on their own private datasets. In traditional FL process, each device k ∈K cooperatively assist the TS for training a global model using its own private dataset D k , where D = S k∈K D k is the union of the training samples held by all devices. Formally, FL aims to seek the optimal global model parameters W ⋆ ∈ R d by minimizing the global objective function L(·;·), denoted as: W ⋆ = arg min W L(W;D) = 1 |K| P k∈K L k (W;D k ),(2) where L k (W;D k ) = 1 |D k | l(W;D k ),(3) where l(·;·) denotes the empirical loss function, L k (·;·) de- notes the local objective function of device k. Each device trains a local model by minimizing the empirical loss over its private dataset D k . After completing local training, the device uploads its model update to the TS. At the t-th round, let u t k be the model update of device k, which is calculated as the difference between the local model parameters and the current global model parameters. The TS then aggregates the model updates from devices to compute the global model. B. Potential Threats In practice, since the training process is initiated by the TS, it is assumed to execute its task honestly. In addition, it is reasonable in practice for OS to honestly execute the audit ... Local modelLocal model Poisoned model ② Upload model update ① Local Training Local dataset Task Server Outsourced Server ③ Audit and Aggregate Local dataset Local dataset Root dataset ④ Send latest global model Device Controlled by me DeviceDevice Fig. 3. System architecture of FL-OA. rule due to legal regulations and company reputation. However, the TS cannot guarantee that all intelligent devices follow the protocol. Thus, we assume that the intelligent devices consist of both benign and malicious devices. Specifically, each benign device honestly performs local training and uploads its model update. Instead, malicious devices disturb the global model by tampering their training data or model update, thereby degrading the performance of the global model. C. Design Goals Considering the above threats, FL-OA is designed to achieve the following goals in defending against Byzantine attacks: • Consistency. FL-OA should mitigate the divergence among benign updates during local training under various data distributions. • Fidelity. FL-OA should not sacrifice the accuracy of the global model in the absence of Byzantine attacks. Specifically, without Byzantine attacks, FL-OA should learn a global model whose accuracy is close to that learned by averaging the model updates from all devices. • Robustness. FL-OA should learn a global model with higher accuracy than existing defense schemes under var- ious Non-IID data when subjected to Byzantine attacks. IV. DESIGN OF OUR FRAMEWORK In this section, we introduce the high-level idea of FL-OA, and then describe the concrete construction. A. High-Level Idea of FL-OA The FL-OA is formally presented in Algorithm 1, where the TS, OS, and intelligent devices jointly execute the fed- erated protocol. Specifically, FL-OA iteratively performs the following steps: • Step I. Select and Synchronize. During the t-th round, TS randomly selects a subset of devices, denoted by K t for local training. The main reason is that selecting partial devices reduces the communication overhead of TS. It then synchronizes the latest global model parameters W t to the selected devices and OS. • Step I. Local Training. Each selected device k ∈ K t locally optimizes the received global model using its private dataset D k , and then sends its model update to TS. Meanwhile, OS trains its root model on the clean root dataset D root . JOURNAL OF L A T E X CLASS FILES, VOL. 14, NO. 8, AUGUST 20214 Algorithm 1: High-Level Idea of FL-OA Input: Device set K, initial global model parameters W init , the local iterations E, the number of rounds T , local learning rate η, weighting coefficients β and α, the proportion of selected coordinates κ, and global learning rate μ. Output: Final global model parameters W final . 1 Initialize T,E, W init ; 2 for each round t∈0,· ,T − 1 do 3Step I: Select and synchronize. 4TS selects the set of devices K t ⊂K; 5TS sends W t to devices in K t and OS; 6Step I: Local Training. 7// Device: 8for each selected device k ∈K t do 9u t k ← LocalTraining(W t ,D k ,η,E,β,α); 10Send u t k to TS; 11// OS: u t root ← LocalTraining(W t ,D root ,η,E,β,α); 12Step I: Audit Updates. 13// TS: Send u t k to OS; 14// OS: s t k , e u t k ← Update Audit(u t k ,κ, u t root ); 15Send s t k and e u t k to TS; 16Step IV : Robust Aggregation. 17// TS: W t+1 ← W t + μ 1 P k∈K t s t k P k∈K t s t k e u t k ; 18 W final ← W T ; 19 return Final global model parameters W final . • Step I. Audit Updates. Upon receiving model updates, TS forwards them to OS for auditing. OS then evaluates these updates using its root model update as a reference and returns the audit results to TS. • Step IV. Robust Aggregation. TS aggregates the model up- dates using the audit results provided by OS to calculate the updated global model parameters W t+1 . This process continue until the predefined number of rounds is reached, yielding the final global model parameters W final . In the following, we describe the Steps I, I, and IV in detail. B. Local Training In this step, the selected devices and the OS separately execute Algorithm 2. The devices use it to obtain improved model updates, while the OS uses it to generate the root model update for auditing the devices’ updates in Step I. For clarity, we describe the local training process only from the perspective of a device, as the OS follows the same procedure. Specifically, data heterogeneity causes the divergence among model updates from devices, making it difficult for existing defense schemes [3]–[13], [21]–[23] to distinguish malicious updates from benign ones. To address this issue, we design a correction term into local training for enhancing the consistency of model updates. More specifically, after receiving the global model parameters W t , each selected device iteratively updates its local model as follows: • Stage I. At the e-th local iteration, each selected de- vice randomly samples mini-batch D t k,(e) from its dataset D k and computes the stochastic gradientg t k,(e+1),1 by ∇L k (w t k,(e) ;D t k,(e) ). Using this gradient, the device per- forms a gradient ascent step with learning rate η to obtain the intermediate parameters: e w t k,(e+1) = w t k,(e) + ηg t k,(e+1),1 . Algorithm 2: LocalTraining Input: Local datasetD k , local learning rate η, local iteration E, global model parameters W t , and weighting coefficients β and α. Output: Model update u t k . 1 w t k,(0) ← W t ; 2 for each local iteration e ∈ 0, 1,· ,E− 1 do 3Randomly sample D t k,(e) ⊂D k ; 4g t k,(e+1),1 ←∇L k (w t k,(e) ;D t k,(e) ); 5 e w t k,(e+1) ← w t k,(e) + ηg t k,(e+1),1 ; 6g t k,(e+1),2 ←∇L k ( e w t k,(e+1) ;D t k,(e) ); 7 e g t k,(e+1) ← (1− β)g t k,(e+1),1 + βg t k,(e+1),2 ; 8w t k,(e+1) ← w t k,(e) −η( e g t k,(e+1) − b g t−1 k + 1 α (w t k,(e) −W t )); 9 b g t k ← b g t−1 k − 1 α (w t k,(E) − W t ); 10 u t k ← w t k,(E) − α b g t k − w t k,(0) ; Notably, existing works [29] [30] have observed that ap- plying vanilla gradient descent to local objective function causes the global model to fall into a sharp valley, which increases the inconsistency of updates among devices. Thus, according to the analysis of works [29] [30], introducing a gradient ascent step during the local training can guide the optimization process toward flat minima. • Stage I. The selected device reuses the same mini- batch D t k,(e) to compute a second gradientg t k,(e+1),2 by∇L k ( e w t k,(e+1) ;D t k,(e) ).By combining the two gradi- ents, we calculate the intermediate gradient e g t k,(e+1) by (1− β)g t k,(e+1),1 + βg t k,(e+1),2 .Notably, the work in [31] points out that the intermediate gradient e g t k,(e+1) leverages the first-order approximation to efficiently implement the corresponding gradient to fit well in the gradient descent process. Moreover, the work in [31] further applies gradi- ent normalization to regularize the gradients g t k,(e+1),1 and g t k,(e+1),2 . However, since the magnitude of model up- dates across devices carries important information that reflects the characteristics of their local data, normalizing gradients during local training would lead to the loss of such information. Thus, we directly combine the two gradients g t k,(e+1),1 and g t k,(e+1),2 without applying normalization. • Stage I. The selected device updates its model parame- ters by the following formula: w t k,(e+1) = w t k,(e) − η( e g t k,(e+1) − b g t−1 k + 1 α (w t k,(e) − W t )), where the deviation(w t k,(e) − W t )can be regarded as a proximal term to align local model with the global model. This term facilitates a balance between local model parameters of each device and global model parameters. However, as noted in work [32], because local objec- tives differ across devices, their resulting local solutions inevitably exhibit distinct local offsets. Specifically, the local offset refers to the deviation (w t k,(E) − W t )accumu- lated in each round t. This indicates that the inconsistency of model updates has not been eliminated. To this end, we introduce b g t−1 k as the correction term to counteract the local offset. This correction term could be considered as a momentum that accumulates previous local offset (see JOURNAL OF L A T E X CLASS FILES, VOL. 14, NO. 8, AUGUST 20215 the Theorem 2). At the beginning of federated training, each device’s b g −1 i is initialized to 0. It is updated in line 9 of Algorithm 2 to capture the historical local offset patterns between both local and global models. After repeating the above stage E times, each device up- dates its correction term as b g t k := b g t−1 k − 1 α (w t k,(E) − W t ), where the term(w t k,(E) − W t ) can be considered as local offset. To prevent the correction term itself from introducing additional bias, we subtract the updated correction term b g t k from the local model parametersw t k,(E) , thereby obtaining bias-corrected local model parameters, i.e.,w t k,(E) − α b g t k . Finally, the model update of device k is computed as u t k := w t k,(E) − α b g t k − w t k,(0) . Upon executing Algorithm 2, the selected devices submit their model updates to TS. In parallel, OS also executes Algorithm 2 on its clean root dataset D root to generate the root model update u t root , which is retained locally rather than uploaded to TS. The local training step enhances the consistency of model updates under Non-IID data, facilitating the identification of deviation caused by malicious updates. The system then proceeds to Step I (i.e., Audit Updates). C. Audit Updates TS receives the set of model updates u t k k∈K t and for- wards them to OS for auditing. The audit operation at OS consists of Critical Parameter Extraction, Trust Score Calcu- lation, and Gradient Normalization, as outlined in Algorithm 3. Each device’s model update is a high-dimensional vector characterized by both direction and magnitude. Malicious devices may manipulate the direction of their model updates, thereby compromising the global model update. Without using fundamental trust, it is difficult to determine which model updates can represent the global update direction. Therefore, OS employs cosine similarity to measure the directional dif- ference between its root update and the device model update. However, computing the cosine similarity between two high- dimensional vectors suffers from the curse of dimensionality. To address this issue, OS extracts key parameters from each model update for audit analysis, thereby alleviating the curse of dimensionality. In addition, since malicious devices may manipulate the magnitude of their model updates to amplify their impact [33], OS normalizes each model update before auditing. The detailed procedure is described below. Stage I. Critical Parameter Extraction. To eliminate the curse of dimensionality caused by computing the distance between two high-dimensional vectors, many dimensionality reduction methods are proposed, such as PCA [34] and t- SNE [35]. However, they project high-dimensional parameters into a lower-dimensional space, which may obscure anomalous information in certain parameters caused by malicious updates. Thus, we focus on extracting critical parameters from model updates for auditing analysis. Based on this consideration, we introduce the Parameter Importance Indicator (PII) to measure the importance of each parameter within the model updates and select the critical parameters based on their PII values. Specifically, PIF combines the absolute magnitude of each parameter and its deviation from other devices at the same coordinate, highlighting parameters with large magnitudes that Algorithm 3: UpdateAudit Input: Model update u t k , root model update u t root , and the proportion of selected coordinates κ, Output: Trust score s t k , normalized model update e u t k . 1 // Dimensionality Reduction. 2 for each selected device k ∈K t do 3PII t k ← ParamImpInd(u t k k∈K ,κ) by Eq. (4); 4 M t k ← Top(PII t k ,κ); 5 // Trust Score Calculation. 6 for each selected device k ∈K t do 7 s t k ← Relu cos (u t k ⊙ M t k ), u t root by Eq. (6); 8 // Gradient Normalization. 9 for each selected device k ∈K t do 10 e u t k ← ∥u t root ∥ ∥u t k ∥ u t k ; 11 return Trust score s t k , normalized model update e u t k . deviate significantly from the majority of devices. Formally, for the j-th parameter of device k at round t, its PII t k [j] is defined as: PII t k [j] =|u t k [j]| + |u t k [j]|− med |u t i [j]| i∈K med |u t i [j]| i∈K + ε ,(4) where |u t k [j]| denotes the absolute value of the j-th param- eter in the model update of device k at the t-th round, med |u t i [j]| i∈K is the median absolute magnitude at the j-th parameter across all devices, and ε is a small constant to prevent division by zero. Based on formula (4), each model update u t k yields a corresponding PII t k vector. A parameter with a large PIF value has a substantial influence on the model update and deviates significantly from the parameters of most devices at the same coordinate, indicating that it is more likely to have been maliciously manipulated. Thus, the critical parameter coordinates in each model update are selected ac- cording to their PII t k vectors. Formally, we define the function top(PII t k ,κ) to map the PII t k vector to a binary mask vector M t k ∈ 0, 1 d , where the coordinates corresponding to the largest κ proportion of values in PII t k are set to 1, and the others are set to 0. Hence, the coordinates with mask value 1 are marked as the critical parameter coordinates in device k’s model update at round t. Accordingly, (u t k ⊙ M t k ) selects the parameters used for auditing analysis from the update u t k according to the mask M t k , thereby achieving dimensionality reduction, where ⊙ denotes the Hadamard product. Stage I. Trust Score Calculation. To ensure that benign updates contribute more to the global model, we compute a trust score for each device based on the update derived from the root dataset. The trust score is derived using a similarity- based evaluation method, whose core idea has been adopted in related FL works [8] [9] [10]. However, these methods suffer from the challenges posed by data heterogeneity and the curse of dimensionality. To this end, FL-OA introduces a correction term during local training and focuses on critical parameters during the auditing step, thereby mitigating these challenges. Specifically, after obtaining the mask vector M t k , OS computes a trust score for each selected device. For device k, the trust score s t k is defined as the cosine similarity between the root model update u t root and its model update u t k over the critical JOURNAL OF L A T E X CLASS FILES, VOL. 14, NO. 8, AUGUST 20216 parameters. Formally, s t k is computed as follows: s t k = cos (u t k ⊙ M t k ), u t root ,(5) where cos(·,·) denotes the cosine similarity. However, the cosine similarity may be negative. Specifically, malicious updates may be in the opposite direction to the root model update, resulting in a negative trust score (i.e., s t k < 0). To solve the negative problem, Relu(·) function is used to clip s t k . Formally, the computation of the score s t k is modified as: s t k = Relu cos (u t k ⊙ M t k ), u t root ,(6) where Relu(x) = x if x > 0 and Relu(x) = 0 otherwise. Stage I. Gradient Normalization. Some malicious devices may amplify their influence by submitting model updates with excessively large magnitudes. To counter this behavior, we adopt a unified normalization operation that standardizes the magnitude of each update to match the norm of the root update ∥u t root ∥. This design ensures that all updates have the same magnitude effect during aggregation, thereby preventing malicious devices from dominating the optimization of the global model by manipulating update magnitudes. Similar normalization strategies have been adopted in existing studies [8] [9] [10]. Formally, the model update is normalized as: e u t k := ∥u t root ∥ ∥u t k ∥ u t k , where e u t k is the normalized model update. After Step I, OS obtains the trust score s t k and the normalized model update e u t k for each selected device, and sends them to TS to guide the aggregation process. D. Robust Aggregation In the step, after receiving trust scores s t k and normalized model updates e u t k , TS aggregates the normalized model up- dates to obtain the global model. Formally, the global model is updated as: W t+1 = W t + μu t g , (7) where μ is the global learning rate, u t g represents the global update, computed as: u t g = 1 P k∈K t s t k P k∈K t s t k e u t k .The above pro- cess is repeated in each round until the predefined number of training rounds is reached. V. ANALYSES This section presents the convergence guarantees of FL-OA and analyzes the correction term in depth. Specifically, we prove that the deviation between the global model parameters W t learned by FL-OA (under Byzantine attacks) and the optimal global model parameters W ∗ is bounded. In addition, we derive the relationship between the local offset and the correction term. In the following, we give three assumptions, which are commonly adopted in convergence analyses [8] [36] [37] [38], followed by our theorems. Assumption 1. The functionL(W) is γ-strongly convex (γ>0) and differentiable over the parameter space Φ with L 1 - Lipschitz continuous gradient (L 1 >0). Meanwhile, the em- pirical loss function l(W) is L 2 -Lipschitz probabilistically. Formally, for any W t 1 , W t 2 ∈ Φ, we have the following: L(W t 1 )≥L(W t 2 ) +∇L(W t 2 )⊙ (W t 1 − W t 2 ) + γ 2 ∥W t 1 − W t 2 ∥ 2 , (8) ∇L(W t 2 )−∇L(W t 1 )≤ L 1 ∥W t 2 − W t 1 ∥, (9) where∇ is gradient, x⊙y denotes the inner product of vectors x and y, and ∥·∥ represents ℓ 2 norm. For any δ ∈ (0, 1) and ̄ δ = 1− δ 3 , there exists an L 2 such that: Pr n sup W t 1 ̸=W t 2 ∥∇l(W t 1 )−∇l(W t 2 )∥ ∥W t 1 −W t 2 ∥ ≤ L 2 o ≥ ̄ δ.(10) Assumption 2. The local dataset D k of each device k and the root datasetD root owned by OS are independently sampled from the training dataset distribution χ. Assumption 3. The gradient of the experical loss function l(W ⋆ ;D) at the optimal global model parameter W ⋆ is bounded. For any W∈ Φ, h(W;D) =∇l(W;D)−∇l(W ⋆ ;D) is bounded. Moreover, for any unit vector v, there exist positive constants σ 1 and γ 1 such that ∇l(W ⋆ ;D) ⊙ v is sub-exponential. And there exist positive constants σ 2 and γ 2 such that for any W ∈ Φ with W ⋆ ̸= W and vector v, ((h(W;D)−E[h(W;D)])⊙v) ∥W−W ⋆ ∥ is sub-exponential with σ 2 and γ 2 . Then, let G 1 to denote exp (σ 2 1 ξ 2 /2) and G 2 to denote exp (σ 2 2 ξ 2 /2), for any |ξ| ≤ 1 γ 1 , any |ξ| ≤ 1 γ 2 and B = v : ∥v∥ = 1, we have the following: sup v∈B E[exp (ξ (∇l (W ⋆ ;D)⊙ v))]≤ G 1 ,(11) sup v∈B E h exp ξ((h(W;D)−E[h(W;D)])⊙v) ∥W−W ⋆ ∥ i ≤ G 2 .(12) Theorem 1. Suppose Assumptions 1-3 hold, the deviation between the global model parameters W t learned by FL-OA and the optimal global model parameters W ∗ is bounded. Formally, in the t-th round, the relationship between W t and W ∗ satisfies the following formula: ∥W t − W ∗ ∥≤ (1− τ ) t ∥W init − W ∗ ∥ + 12μ△ 1 τ , (13) where τ = 1− ( p (1 + μ 2 L 2 1 − μγ) + 2μL 1 + 24μ△ 2 ), △ 1 = √ 2σ 1 p (d log 6 + log(3/δ))/|D root |, and △ 2 = σ 2 p 2/|D root | q d log 18L 3 δ 2 + d 2 log |D root | d + log( 6δ 2 2 r √ D root γ 2 σ 1 δ ), L 3 = maxL 1 ,L 2 , and |D root | denotes the size of the root dataset, d is dimension of W. Proof: See Appendix I. Discussion 1. After undergoing several rounds, the deviation between the global model learned through FL-OA and the opti- mal global model will be confined to a relatively narrow range. In other words, FL-OA is theoretically robust to Byzantine attacks. Theorem 2. Let P E−1 e=0 λ (e) = P E−1 e=0 η α (1− η α ) E−e = 1− (1− η α ) E = λ, after E local iterations in Algorithm 2, the local offset for each selected device k ∈K t is denoted as: w t k,(E) − W t = αλ b g t−1 k − α P E−1 e=0 λ (e) e g t k,(e) ,(14) and the update for correction term b g t k can be rewritten as: b g t k = (1− λ) b g t−1 k + P E−1 e=0 λ (e) e g t k,(e) . (15) Discussion 2. The proof details can be referred to the Ap- pendix I. Following Theorem 2, let b g −1 k = 0, we can obtain w 0 k,(E) − W 0 =−α P E−1 e=0 λ (e) e g 0 k,(e) , where −α P E−1 e=0 λ (e) is the same for each selected device, and e g 0 k,(e) is different for each selected device, which shows that the inconsistency of model updates is independent the local learning rate η and the importance α. In addition, we observe that for η < α, the JOURNAL OF L A T E X CLASS FILES, VOL. 14, NO. 8, AUGUST 20217 previous e g t k,(e) is weakened in the local offset with increasing E. Moreover, formula (14) indicates that the local offset is transferred to a exponential average of previous local gradients when applying the prox term. From formula (15), we notice that b g t k performs as a momentum term of the historical updates before round t, which shows that the correction term is considered as an estimation of the local offset. VI. EVALUATION In this section, we describe the experimental settings, and examine the consistency, fidelity, and robustness of FL-OA. A. Experimental Settings 1) Datasets and Model Architecture: Similar to prior works [21] [23] [8] [15] [19], we evaluate the performance of FL- OA using CIFAR10 and CIFAR100. Additionally, we use the ResNet18 architecture [39] as the local model for experiments on CIFAR10 [40] and CIFAR100 [40], where ResNet18 is a deep residual network consisting of convolutional layers, average pooling layers, and fully connected layers. Since ResNet18 contains more than 2.00×10 7 parameters, its model updates can be regarded as vectors in a high-dimensional parameter space. 2) Non-IID and IID Settings: To evaluate the FL-OA performance, we simulate both Non-IID and IID setting in our experiments, respectively. Furthermore, we extract a random subset from each dataset’s training samples to establish the root dataset D root based on the following settings. • Non-IID setting. For the Non-IID setting, we employ the Dirichlet distribution [41] to simulate Non-IID data. Notably, the Dirichlet distribution has been widely ap- plied for simulating data partitioning in FL. Formally, let DIR(ι) be the data distribution, where ι controls the level of data heterogeneity. A smaller ι corresponds to a higher level of Non-IID. Specifically, we extract q j ∼ DIR(ι) for class j from the Dirichlet distribution. Each component q j k of q j determines the percentage of the examples of class j assigned to device k. This partitioning ensures heterogeneous data distributions and sample sizes among different devices. Moreover, under the Non-IID setting, the construction of the clean root datasetD root still follows the Dirichlet distribution. Specifically, the entire datasetD is partitioned according to DIR(ι) into (|K|+1) subsets. Among them, |K| subsets are allocated to the devices, while the remaining subset is assigned to the OS as D root . This design ensures statistical consistency between the root dataset and the devices’ datasets. For example, under the CIFAR10 with|K| = 100 devices and DIR(0.1), the constructed root dataset contains 382 sam- ples, with class proportions approximately [0.02, 0.79, 0.02, 0.04, 0.02, 0.02, 0.01, 0.02, 0.01, 0.05]. Notably, we do not adopt a fixed partitioning for the root dataset. Instead, its construction is adaptively adjusted based on the parameter ι in DIR(ι) to accommodate different degrees of data heterogeneity. • IID setting. In the IID setting, we randomly shuffle the training samples and distribute them equally to each device and OS. 3) Setting of Byzantine Attacks: Since the number of ma- licious devices affects FL performance to varying degrees, we set different percentages of malicious devices in our experiments. Formally, let Att be percentage of malicious devices, computed as Att = |K b | |K| , where |K b | denotes the number of malicious devices. Since FL-OA does not rely on the assumption that the majority of devices are benign, we set Att = 50% by default. In addition, malicious devices maintain consistent adversarial behavior throughout the fed- erated training. We consider three types of attacks to simulate Byzantine attack behaviors. (i) Gaussian Attacks [42]: Some devices are unwilling to contribute their own model updates. To fulfill the task of TS, they upload model updates generated by Gaussian noise. Specifically, each parameter in the model update is replaced with a randomly selected value from a Gaussian distribution. (i) Neurotoxin Attacks [43]: Malicious devices project their updates onto the coordinate set defined by the top 75% largest-magnitude parameters of the previous- round global model, and sets the update in other coordinates to zero. (i) Focused-Flip Attacks [44]: Malicious devices flip the signs of a small portion (set to 50%) of parameters in their model updates, while the signs of the remaining parameters remain unchanged. 4) Implementation Details: We adopt the following hyper- parameter settings. Specifically, the total number of devices is set to 100, and 20% of the devices are randomly selected in each round of local training. In addition, the batch size is set to 50, the number of local iterations E is set to 5, the local learning rate η is set to 0.1, and the global learning rate μ is set to 1. The weighting coefficients α and β are set to 0.1, and the proportion of selected coordinates κ is set as 30%. The number of communication rounds T is set to 500. Furthermore, all benign devices use the same hyper-parameter configuration, and are initialized with the same model parameters W init . Unless otherwise specified, the above settings are used as the default configuration. 5) Baselines: To evaluate the effectiveness of FL-OA, we compare it with five defense schemes: MKrum [3], FLTrust [8], FL-Auditor [10], AlignIns [38], and FLgym [45]. 6) Evaluation Metrics: Since Byzantine attacks are in- tended to degrade the performance of the global model, we adopt the accuracy of the global model on datasets as the met- ric to evaluate the defense effectiveness of different schemes. A higher accuracy indicates better performance of the corre- sponding scheme. Moreover, we define an additional metric, denoted as Acc 50%−0% to measure the accuracy difference between the setting with 50% malicious devices and the setting without malicious devices. A smaller value of Acc 50%−0% indicates that the scheme exhibits less accuracy degradation when subjected to Byzantine attacks, thereby demonstrating greater robustness. B. Experimental Results 1) Consistency of FL-OA: To evaluate the consistency of model updates in FL, we introduce divergence as a evaluation metric, which is defined as: Div := P k∈K t ∥ e u t k − ̄ u t ∥ 2 2 |K t | , (16) JOURNAL OF L A T E X CLASS FILES, VOL. 14, NO. 8, AUGUST 20218 0100200300400500 Communication Rounds 0.0 1.0 2.0 3.0 4.0 5.0 6.0 Divergence FL-OA FedAvg FedDyn FedProx FL-Auditor AlignIns (a) CIFAR10, IID. 0100200300400500 Communication Rounds 1.0 2.0 3.0 4.0 Divergence FL-OA FedAvg FedDyn FedProx FL-Auditor AlignIns (b) CIFAR10, DIR(0.6). 0100200300400500 Communication Rounds 1.0 2.0 3.0 Divergence FL-OA FedAvg FedDyn FedProx FL-Auditor AlignIns (c) CIFAR10, DIR(0.3). 0100200300400500 Communication Rounds 1.0 2.0 Divergence FL-OA FedAvg FedDyn FedProx FL-Auditor AlignIns (d) CIFAR10, DIR(0.1). 0100200300400500 Communication Rounds 0 10 Divergence FL-OA FedAvg FedDyn FedProx FL-Auditor AlignIns (e) CIFAR100, IID. 0100200300400500 Communication Rounds 10 Divergence FL-OA FedAvg FedDyn FedProx FL-Auditor AlignIns (f) CIFAR100, DIR(0.6). 0100200300400500 Communication Rounds 10 Divergence FL-OA FedAvg FedDyn FedProx FL-Auditor AlignIns (g) CIFAR100, DIR(0.3). 0100200300400500 Communication Rounds 0 10 Divergence FL-OA FedAvg FedDyn FedProx FL-Auditor AlignIns (h) CIFAR100, DIR(0.1). Fig. 4. Divergence comparison between FL-OA and existing schemes in both IID and Non-IID settings without attacks. where ̄ u t = μ 1 P k∈K t s t k P k∈K t s t k e u t k . A lower divergence value Div indicates a higher degree of consistency among model updates. We evaluate the divergence of FL-OA across four data distributions, namely IID, DIR(0.6), DIR(0.3), and DIR(0.1). For comparison, we evaluate several non-robust FL schemes, including FedAvg [46], FedDyn [47], and FedProx [28]. We also include several defense schemes, namely FL- Auditor [10] and AlignIns [38]. Notably, these experiments are conducted in the absence of attacks, i.e., Att = 0%. Fig. 4 illustrates the divergence results of different schemes on CIFAR10 and CIFAR100 under both IID and Non-IID settings. It can be observed that, for all schemes, the diver- gence among model updates still exists under IID data. This is because, although the class distributions across devices are identical, the feature distributions of samples held by devices may still differ [48]. In other words, even samples belonging to the same class may exhibit differences in their fine-grained features. Furthermore, we observe that FL-OA yields lower divergence than the other schemes across various data dis- tributions on both CIFAR10 and CIFAR100. This indicates that the gradient ascent step and the correction term in FL- OA effectively mitigate the divergence among model updates. Notably, FedAvg exhibits the largest divergence. In contrast, the other non-robust FL schemes achieve lower divergence than FedAvg because their regularization terms help improve the consistency of model updates. In addition, since AlignIns and FL-Auditor do not enhance the update consistency during local training, their Div values are similar to those of FedAvg. Based on these results, FL-OA is better positioned to defend against Byzantine attacks, as further demonstrated in the following experiments. In addition, as shown in Fig. 4, we observe that the divergence on CIFAR10 and CIFAR100 first grow and then fall during federated training. At the beginning of training, because of differences in local data distributions, each device optimizes its model toward its own optimum, leading to sig- nificant differences among model updates in terms of direction and magnitude. Consequently, the divergence among device’ model updates to increase rapidly. As the global model gradu- ally approaches the local optima, the direction and magnitude of each device’s model updates gradually decrease, thereby reducing the divergence among them. Thus, the divergence exhibits a trend of first increasing and then decreasing during the training process. Furthermore, we observe that the max- imum divergence decreases as data heterogeneity increases. This is because, under the IID setting, each device is trained to fit all classes, so most model parameters are involved in the update process. In contrast, under the Non-IID settings, each device usually focuses on the dominant classes in its local data, while parameters associated with the missing classes are rarely updated. Thus, only a smaller subset of parameters is actively involved in the updates. Therefore, greater data heterogeneity leads to a lower maximum divergence among devices’ model updates. 2) Fidelity of FL-OA: To highlight the fidelity of FL- OA, we construct a version of FL-OA without the defense mechanism as a reference (i.e., FL-OA with only standard average aggregation of model updates). For the sake of clarity, we simplify the “FL-OA without defense” to the “Baseline” below. To this end, we evaluate the performance of both FL- OA and Baseline in the absence of Byzantine attacks. If FL- OA’s performance is close to that of the Baseline, FL-OA is considered to meet the fidelity requirement. Fig. 5 presents the accuracy and loss of the two schemes on the training and test sets, respectively. We observe that, overall, FL-OA and Baseline exhibit similar accuracy and loss across dif- ferent data distributions, indicating that FL-OA preserves the model learning performance well. Thus, Fig. 5 demonstrates the fidelity of FL-OA. Notably, the performance of FL-OA is significantly higher than that of the Baseline under the DIR(0.1) setting on both CIFAR10 and CIFAR100, which still shows that FL-OA satisfies the fidelity. This is because the DIR (0.1) setting exacerbates the inconsistency of model updates, and some devices whose updates deviate from the majority are removed during the auditing. In other words, these removed devices are considered anomalous. Therefore, model updates with large deviations do not participate in the aggregation, which causes the performance of FL-OA to be higher than Baseline. Furthermore, we notice that the test accuracy of FL- OA and Baseline on CIFAR100 is significantly lower than the training accuracy, which is called over-fitting [49]. This is because the model learns the noise in the training samples, rather than the essential pattern of the samples. 3) Robustness of FL-OA: To evaluate the performance of FL-OA under Byzantine attacks, we compare its perfor- mance under Gaussian and Neurotoxin attacks across different JOURNAL OF L A T E X CLASS FILES, VOL. 14, NO. 8, AUGUST 20219 0100200300400500 Communication Rounds 25.0 50.0 75.0 100.0 Accuracy (%) Train Accuracy: FL-OA Train Accuracy: Baseline Test Accuracy: FL-OA Test Accuracy: Baseline 0.0 1.0 2.0 3.0 Loss Train Loss: FL-OA Train Loss: Baseline Test Loss: FL-OA Test Loss: Baseline (a) CIFAR10, IID. 0100200300400500 Communication Rounds 25.0 50.0 75.0 100.0 Accuracy (%) Train Accuracy: FL-OA Train Accuracy: Baseline Test Accuracy: FL-OA Test Accuracy: Baseline 0.0 1.0 2.0 3.0 Loss Train Loss: FL-OA Train Loss: Baseline Test Loss: FL-OA Test Loss: Baseline (b) CIFAR10, DIR(0.6). 0100200300400500 Communication Rounds 25.0 50.0 75.0 Accuracy (%) Train Accuracy: FL-OA Train Accuracy: Baseline Test Accuracy: FL-OA Test Accuracy: Baseline 0.0 1.0 2.0 3.0 Loss Train Loss: FL-OA Train Loss: Baseline Test Loss: FL-OA Test Loss: Baseline (c) CIFAR10, DIR(0.3). 0100200300400500 Communication Rounds 25.0 50.0 75.0 Accuracy (%) Train Accuracy: FL-OA Train Accuracy: Baseline Test Accuracy: FL-OA Test Accuracy: Baseline 0.0 1.0 2.0 3.0 Loss Train Loss: FL-OA Train Loss: Baseline Test Loss: FL-OA Test Loss: Baseline (d) CIFAR10, DIR(0.1). 0100200300400500 Communication Rounds 0.0 25.0 50.0 75.0 100.0 Accuracy (%) Train Accuracy: FL-OA Train Accuracy: Baseline Test Accuracy: FL-OA Test Accuracy: Baseline 2.0 4.0 Loss Train Loss: FL-OA Train Loss: Baseline Test Loss: FL-OA Test Loss: Baseline (e) CIFAR100, IID. 0100200300400500 Communication Rounds 0.0 25.0 50.0 75.0 100.0 Accuracy (%) Train Accuracy: FL-OA Train Accuracy: Baseline Test Accuracy: FL-OA Test Accuracy: Baseline 0.0 1.0 2.0 3.0 Loss Train Loss: FL-OA Train Loss: Baseline Test Loss: FL-OA Test Loss: Baseline (f) CIFAR100, DIR(0.6). 0100200300400500 Communication Rounds 0.0 25.0 50.0 75.0 100.0 Accuracy (%) Train Accuracy: FL-OA Train Accuracy: Baseline Test Accuracy: FL-OA Test Accuracy: Baseline 0.0 1.0 2.0 3.0 Loss Train Loss: FL-OA Train Loss: Baseline Test Loss: FL-OA Test Loss: Baseline (g) CIFAR100, DIR(0.3). 0100200300400500 Communication Rounds 0.0 25.0 50.0 75.0 Accuracy (%) Train Accuracy: FL-OA Train Accuracy: Baseline Test Accuracy: FL-OA Test Accuracy: Baseline 2.0 4.0 Loss Train Loss: FL-OA Train Loss: Baseline Test Loss: FL-OA Test Loss: Baseline (h) CIFAR100, DIR(0.1). Fig. 5. Convergence of the proposed FL-OA over CIFAR10 and CIFAR100 in both IID and Non-IID settings. IIDDIR(0.6)DIR(0.1) 40 50 60 70 80 90 100 Accuracy (%) Baseline (Att=0%) Baseline (Att=10%) Baseline (Att=20%) Baseline (Att=30%) FL-OA (Att=0%) FL-OA (Att=10%) FL-OA (Att=20%) FL-OA (Att=30%) (a) CIFAR10, Gaussian attacks. IIDDIR(0.6)DIR(0.1) 40 50 60 70 80 90 100 Accuracy (%) Baseline (Att=0%) Baseline (Att=10%) Baseline (Att=20%) Baseline (Att=30%) FL-OA (Att=0%) FL-OA (Att=10%) FL-OA (Att=20%) FL-OA (Att=30%) (b) CIFAR10, Neurotoxin attacks. IIDDIR(0.6)DIR(0.1) 0 10 20 30 40 50 60 70 Accuracy (%) Baseline (Att=0%) Baseline (Att=10%) Baseline (Att=20%) Baseline (Att=30%) FL-OA (Att=0%) FL-OA (Att=10%) FL-OA (Att=20%) FL-OA (Att=30%) (c) CIFAR100, Gaussian attacks. IIDDIR(0.6)DIR(0.1) 0 10 20 30 40 50 60 70 Accuracy (%) Baseline (Att=0%) Baseline (Att=10%) Baseline (Att=20%) Baseline (Att=30%) FL-OA (Att=0%) FL-OA (Att=10%) FL-OA (Att=20%) FL-OA (Att=30%) (d) CIFAR100, Neurotoxin attacks. Fig. 6. Test accuracy of FL-OA for different numbers of malicious devices over CIFAR10 and CIFAR100 (IID and Non-IID). proportions of malicious devices. Specifically, we conduct experiments on the CIFAR10 and CIFAR100 with malicious device proportions of 0%, 10%, 30%, and 50%. FL-OA is compared with a baseline variant, i.e., FL-OA using only av- erage aggregation of model updates. The experimental results are shown in Fig. 6. We observe that under both Gaussian and neurotoxin attacks, the performance of the baseline declines significantly, whereas FL-OA maintains the relatively stable performance, demonstrating its strong robustness. In particular, under neurotoxin attacks, the accuracy of the global model learned by FL-OA is nearly identical to that achieved without attacks. This is because TS can effectively mitigate the impact of malicious model updates during the aggregation process. In contrast, the Gaussian attack has a more significant impact on the performance of FL-OA. This is because Gaussian attacks perturb all parameters in the model updates, thereby causing more comprehensive damage to the global model, whereas Neurotoxin mainly poisons only a subset of critical parameters, making its impact relatively limited. In addition, we compare the performance of FL-OA with that of five existing schemes (i.e., Krum, FLTrust, FL-Auditor, AlignIns, and FLgym). Specifically, we evaluate their test accuracy on CIFAR10 and CIFAR100 when subjected to Gaus- sian, Neurotoxin, and Focused-Flip attacks across different data distributions. The proportion of malicious devices is set to 0%, 10%, 30%, and 50%, respectively. The results are reported in Tables I to I. It can be observed that FL-OA achieves the highest test accuracy in the absence of malicious devices. This indicates that the gradient ascent step and correction term introduced by FL-OA enhance the consistency of model updates and further improve the performance of the global model. As the proportion of malicious devices increases, the accuracy of FL-OA gradually declines, which is a reasonable phenomenon. This is because fewer benign samples remain available for effective model training. When the proportion of malicious devices reaches 50%, the performance of Krum degrades significantly. This is because Krum is a statistical knowledge-based approach whose efficacy relies on the as- sumption that benign devices are in the majority. Once this assumption no longer holds, the updates selected by Krum may derive from malicious devices, leading to a significant decline in global model performance. Furthermore, under the DIR(0.1) setting, the test accuracy of both FLTrust and FL- Auditor also drops noticeably. This is because both schemes are root dataset-based approach, the data distribution of the root dataset differs from that of the device’s local dataset, which weakens their ability to distinguish malicious updates. Moreover, AlignIns and FLgym achieve relatively high test accuracy on both Neurotoxin and focused-flip attacks. This is because AlignIns adopts a multi-granularity evaluation that offers strong robustness, while FLgym integrates a sliding window-based weight recovery mechanism, allowing some be- nign updates that are mistakenly filtered out to still participate in aggregation. Overall, FL-OA outperforms the other five schemes across different data distributions and attack settings. However, since FL-OA improves the local training process under Non-IID data, the test accuracy of the model cannot be directly used to compare its defense capability with that of the baseline schemes. To provide a more intuitive evaluation of robustness, we calculate the difference in test accuracy for each scheme when the proportion of malicious devices is 50% and 0%, denoted as Acc 50%−0% . A smaller value of Acc indicates stronger robustness. As highlighted in green in Tables I to I, FL-OA generally achieves lower Acc values than the base- line methods, demonstrating its superior robustness against Byzantine attacks. In particular, under the DIR(0.1) setting, Krum shows large Acc values across all attack types, because severe inconsistency among benign model updates weakens its defense effectiveness. Furthermore, although the performance degradation of AlignIns and FLgym under Non-IID settings is similar to that under the IID setting, their overall performance decline remains significant. 4) Ablation Study: To further evaluate the contribution of each component in FL-OA, we conduct ablation studies on JOURNAL OF L A T E X CLASS FILES, VOL. 14, NO. 8, AUGUST 202110 TABLE I TEST ACCURACY % (TOP-1) COMPARISON BETWEEN FL-OA AND EXISTING SCHEMES UNDER GAUSSIAN ATTACKS. DatasetAtt KrumFLTrustFL-AuditorAlignInsFLgymFL-OA —IID— DIR(0.6) DIR(0.1)—IID— DIR(0.6) DIR(0.1)—IID— DIR(0.6) DIR(0.1)—IID— DIR(0.6) DIR(0.1)—IID— DIR(0.6) DIR(0.1)—IID— DIR(0.6) DIR(0.1) CIFAR10 0%80.9479.5874.7581.5179.4573.9781.7479.6773.9982.8680.6476.0581.2081.7778.8485.7584.5579.45 10%75.7468.2550.8376.1469.3151.6676.9670.1454.6182.2980.6375.7881.4480.0076.9385.0283.1576.04 30%65.5748.4642.0565.1550.8242.7053.9155.6244.8981.1380.3272.3279.6477.9574.2783.6481.7473.35 50%46.8641.4437.7649.9246.2534.4044.6042.6433.8170.8868.0660.1666.1066.8463.4673.6171.1962.97 Acc 50%−0% ↓34.0838.47 36.9931.59 33.20 39.57 37.1437.0340.18 11.98 12.58 15.89 15.10 14.93 15.3812.1413.3616.48 CIFAR100 0%38.9438.7637.0738.0638.5837.7338.3938.9337.4941.3942.3340.1743.9642.6440.2955.5054.0951.61 10%35.2434.1432.2034.2434.1428.6532.1532.4129.7635.8736.5335.2642.1441.5240.4555.2353.7451.13 30%31.4023.8719.2727.6221.5314.0527.4622.0514.0530.7831.8432.4738.1135.2333.2752.6451.0649.02 50%27.1618.4713.7416.1614.1611.4523.1513.9812.1426.0726.5926.3930.1528.7525.8842.8942.4039.39 Acc 50%−0% ↓11.7820.0223.3321.90 24.4226.2815.24 24.95 25.3515.3215.7413.7813.81 13.8914.4112.61 11.6912.22 TABLE I TEST ACCURACY % (TOP-1) COMPARISON BETWEEN FL-OA AND EXISTING SCHEMES UNDER NEUROTOXIN ATTACKS. DatasetAtt KrumFLTrustFL-AuditorAlignInsFLgymFL-OA —IID— DIR(0.6) DIR(0.1)—IID— DIR(0.6) DIR(0.1)—IID— DIR(0.6) DIR(0.1)—IID— DIR(0.6) DIR(0.1)—IID— DIR(0.6) DIR(0.1)—IID— DIR(0.6) DIR(0.1) CIFAR10 0%80.9479.5874.7581.5179.4573.9781.7479.6773.9982.8680.6476.0581.2081.7778.6485.7584.5579.45 10%80.4179.5774.9181.3379.8873.9181.4779.6973.1679.3478.1273.2081.9280.9078.9885.6284.1579.15 30%78.1876.2773.1479.1176.7070.4078.1075.3371.9073.3372.7465.5975.5873.4970.4582.4181.0677.38 50%68.0964.4659.7973.1761.5863.9872.0065.5763.1366.3765.6961.7368.6368.0261.5076.3075.0272.09 Acc 50%−0% ↓12.8515.12 14.96 8.3417.87 9.999.74 14.1010.86 16.4914.95 14.3212.5713.7517.149.459.53 7.36 CIFAR100 0%38.9438.7637.0738.0638.5837.7338.3938.9337.4941.3942.3340.1743.9642.6440.2955.5054.0951.61 10%38.7838.3136.9838.5738.6037.7138.4438.9637.1736.5036.9135.2443.6541.6040.0454.5753.7251.93 30%31.5132.3430.5435.4335.4332.2636.4633.4132.4933.4831.8734.2533.7434.5935.0351.9051.3645.30 50%25.7424.0122.2626.2624.4620.4128.1426.0124.5725.8823.2922.9430.7627.9826.7547.8444.8141.24 Acc 50%−0% ↓ 13.2014.7514.8111.8014.1217.3210.25 12.9212.92 15.51 19.04 17.2313.2014.6613.547.66 9.28 10.37 TABLE I TEST ACCURACY % (TOP-1) COMPARISON BETWEEN FL-OA AND EXISTING SCHEMES UNDER FOCUSED-FLIP ATTACKS. DatasetAtt KrumFLTrustFL-AuditorAlignInsFLgymFL-OA —IID— DIR(0.6) DIR(0.1)—IID— DIR(0.6) DIR(0.1)—IID— DIR(0.6) DIR(0.1)—IID— DIR(0.6) DIR(0.1)—IID— DIR(0.6) DIR(0.1)—IID— DIR(0.6) DIR(0.1) CIFAR10 0%80.9479.5874.7581.5179.4573.9781.7479.6773.9982.8680.6476.0581.2081.7778.6485.7584.5579.45 10%78.9077.9372.6879.8278.8473.1481.2779.0573.4579.5177.3272.6480.7481.2276.4785.4884.4679.29 30%71.3470.4868.5472.9672.1069.6575.4472.3170.4774.3772.0465.5576.4974.2570.9581.2680.1576.27 50%63.4162.5757.8667.5065.7859.8369.2067.5661.5066.6664.4660.6468.0663.9460.7773.9272.7368.04 Acc 50%−0% ↓ 17.5317.0116.8914.0113.67 14.14 12.54 12.11 12.4916.2016.1815.4113.1417.8317.8711.8311.82 11.41 CIFAR100 0%38.9438.7637.0738.0638.5837.7338.3938.9337.4941.3942.3340.1743.9642.6440.2955.5054.0951.61 10%34.5034.2933.1137.2937.0732.4637.9437.8632.9840.1540.0739.4641.4541.0240.2055.3654.1751.44 30%30.9328.4425.0428.6322.2228.8329.7228.5825.5735.4635.1534.3836.7936.6234.6051.9550.6248.05 50%25.7221.3716.6621.8020.9219.1724.8724.4919.7025.7324.6623.2529.1727.1525.2245.2443.0542.30 Acc 50%−0% ↓ 13.22 17.39 20.4116.2617.6618.5613.52 14.44 17.7915.6617.6716.9214.7915.4915.0710.2611.049.31 TABLE IV ABLATION STUDY OF FL-OA WITH GAUSSIAN AND NEUROTOXIN ATTACKS ON CIFAR100. Methods Gaussian (Att = 50%)Neurotoxin (Att = 50%) —IID— DIR(0.6) DIR(0.1)—IID— DIR(0.6) DIR(0.1) RseNet18 w/o GAS40.8537.9934.4743.1040.5637.24 w/o CT40.5740.3638.3842.5042.2540.68 (w/o GAS & CT)33.3432.2029.2735.2933.4531.77 FL-OA (κ = 100%)41.6238.1835.6446.5742.0538.27 FL-OA (κ = 70%)42.2840.8037.4547.8443.0640.56 FL-OA (κ = 30%)42.9842.4039.3947.8244.8141.24 ResNet50 w/o GAS42.2842.1938.5245.9242.3440.25 w/o CT40.4938.5437.9940.4439.2936.95 (w/o GAS & CT)35.8635.0633.4335.0635.8733.23 FL-OA (κ = 100%)44.8443.6041.4946.8245.6343.94 FL-OA (κ = 70%)46.1044.7642.4547.5946.2644.55 FL-OA (κ = 30%)46.3145.5843.9247.8545.7043.02 Note: “w/o GAS” and “w/o CT” denote FL-OA without the gradient ascent step and without the correction term in local training, respectively. FL-OA (κ) indicates that the top-κ% parameters are selected for auditing. CIFAR100 using ResNet18 and ResNet50 under two attack scenarios, where 50% of the devices launch Gaussian attacks and Neurotoxin attacks, respectively. The results are reported in Table IV. (i) Ablation on local training. Specifically, we remove the gradient ascent step (GAS) and the correction term (CT) from the local training of FL-OA to evaluate their effects on model performance. The results show that “w/o GAS & CT” achieves the worst performance. This is because, without the gradient ascent step and the correction term, local training cannot enhance the consistency among benign updates, thereby weakening the defense capability of FL-OA. In contrast, when GAS and CT are introduced separately, FL-OA consistently achieves better performance, indicating that each of them is beneficial to the robustness of FL-OA. (i) Ablation on critical parameters. To evaluate the role of critical parameters, we compare the performance of FL-OA when using all param- eters (κ = 100%) and when using only critical parameters (κ = 70% and κ = 30%). The results show that FL-OA performs better when only the 70% or 30% critical parameters are used than when all parameters are used. This indicates that the parameter importance indicator PII can identify the critical parameters in updates, thereby improving the effectiveness of FL-OA in defending against malicious updates. Moreover, the ablation studies conducted with ResNet18 and ResNet50 yield consistent conclusions with those presented above, indicating that the proposed components generalize well across different model architectures. 5) Performance of FL-OA under High Percentages of Ma- licious Devices: In FL-OA, the root model update u t root is employed as a reference for evaluating each model updates submitted by each device. Thus, different from defense meth- ods that rely on the assumption that benign devices constitute the majority, the effectiveness of FL-OA is not constrained by the percentage of malicious devices. To evaluate the perfor- mance of FL-OA under high percentage of malicious devices, we conduct experiments under focused-flip attacks by setting the percentage of malicious devices to 50%, 60%, and 70%, JOURNAL OF L A T E X CLASS FILES, VOL. 14, NO. 8, AUGUST 202111 TABLE V PERFORMANCE OF FL-OA UNDER HIGH PERCENTAGES OF MALICIOUS DEVICES. Dataset Att Test Accuracy (%) BaselineFL-OA IIDDIR(0.6)DIR(0.1)IIDDIR(0.6)DIR(0.1) CIFAR10 0%85.4285.7779.5885.7584.5779.45 50%54.2361.3355.4373.9272.7368.04 60% 47.1042.5436.4071.1871.4465.67 70%19.6823.8215.7466.7464.2361.03 CIFAR100 0%55.1154.8052.0955.5054.0951.61 50%34.7234.3230.2245.3643.0542.30 60%12.0714.5312.2537.7636.4828.71 70%7.748.747.9926.7424.9419.52 respectively, using a setting without any malicious devices (0%) as a reference. In addition, we construct a version of FL-OA that aggregates model updates using only the standard averaging method. For simplicity, this variant is referred to as “Baseline.” As shown in Table V, FL-OA exhibits stronger defensive capability than Baseline when the percentage of malicious devices is high. For example, under the IID set- ting on CIFAR10, when the percentage of malicious devices reaches 50%, the model accuracy of FL-OA decreases by only approximately 12% relative to the attack-free setting, whereas that of Baseline decreases by approximately 31%. When the percentage further increases to 70%, the accuracy of FL- OA decreases by approximately 19%, while that of Baseline decreases by approximately 33%. These results demonstrate that although increasing the percentage of malicious devices degrades the model accuracy of FL-OA, it remains effective in defending against malicious devices even when malicious devices constitute the majority. The accuracy degradation of FL-OA is attributed to the decrease in the number of benign samples available for model training as the percentages of malicious devices increases. 6) Visualization of the Distribution of Critical Parameters: To further validate the rationality of critical parameter ex- traction and the effectiveness of the Parameter Importance Indicator, we conduct experiments on CIFAR100 for analysis, involving three malicious devices and seven benign devices. Specifically, we visualize the top 500 parameters in benign and malicious updates sorted by importance, as shown in Fig. 7(a). It can be observed that the importance values of malicious updates are higher than those of benign updates. This is because malicious updates tend to exhibit larger magnitudes on certain parameters and deviate more significantly from the majority of model updates, thereby resulting in higher PII values. In addition, we compute the cosine similarity between benign and malicious updates based on the critical parameters, as well as that based on all parameters. Fig. 7(b) shows that the similarity based on the critical parameters differs significantly between benign and malicious updates. In contrast, Fig. 7(c) shows that the similarity based on all parameters fails to reveal a clear distinction between the two. These results indicate that critical parameter extraction can alleviate the curse of dimensionality and enhance the separability between benign and malicious updates. 7) The Impact of α and β on FL-OA: FL-OA intro- duces a gradient ascent step and a correction term into local 055110165220275330385440495 Parameter Index (Sorted by Importance) 0.0 0.2 0.4 0.6 0.8 1.0 1.2 Importance Value Malicious Benign (a) 1.000.360.580.410.270.630.340.220.170.29 0.361.000.240.550.310.470.620.180.260.33 0.580.241.000.380.440.210.530.140.190.25 0.410.550.381.000.290.360.480.270.320.16 0.270.310.440.291.000.520.230.110.280.35 0.630.470.210.360.521.000.390.240.150.30 0.340.620.530.480.230.391.000.200.310.27 0.220.180.140.270.110.240.201.000.840.92 0.170.260.190.320.280.150.310.841.000.81 0.290.330.250.160.350.300.270.920.811.00 (b) 1.000.540.570.610.460.550.720.580.390.53 0.541.000.520.350.640.560.490.590.670.51 0.570.521.000.740.440.580.550.320.630.57 0.610.350.741.000.530.690.470.540.560.36 0.460.640.440.531.000.510.660.420.590.77 0.550.560.580.690.511.000.570.620.380.55 0.720.490.550.470.660.571.000.520.710.48 0.580.590.320.540.420.620.521.000.560.54 0.390.670.630.560.590.380.710.561.000.58 0.530.510.570.360.770.550.480.540.581.00 (c) Fig. 7. (a) The distribution of parameter importance values. (b) Pairwise sim- ilarity of critical parameters in model updates between benign and malicious groups. (c) Pairwise similarity of all parameters in model updates between benign and malicious groups. 0.050.10.20.50.8 0.01 0.05 0.1 0.3 0.5 35.0636.9237.7536.8836.34 37.9038.7838.4238.6038.21 38.1439.3939.9240.2439.76 38.4039.2139.6939.2839.18 36.0137.1537.0637.4736.07 36 37 38 39 40 Test Accuracy (%) (a) Gaussian Attacks 0.050.10.20.50.8 0.05 0.1 0.2 0.3 0.5 37.6838.3739.5138.0637.01 38.7340.6239.1739.2539.30 39.1441.8941.4341.1741.72 39.2941.2741.5340.5840.32 37.9538.4438.1438.4237.89 38 39 40 41 Test Accuracy (%) (b) Neurotoxin Attacks 0.050.10.20.50.8 0.01 0.05 0.1 0.3 0.5 37.7238.8339.7738.8137.79 37.3539.7040.7039.6638.25 39.5442.3042.7342.6341.67 39.8741.1842.7741.0540.28 38.6040.2740.8939.2339.84 38 39 40 41 42 Test Accuracy (%) (c) Focused-Flip Attacks Fig. 8. Test accuracy (%) of FL-OA under the coefficients α and β. training. The effect of the correction term is controlled by the coefficient α, while that of the gradient ascent step is scaled by β. Thus, it is necessary to analyze the sensitivity of FL-OA to the coefficients α and β. To this end, we evaluate the performance of FL-OA under three attack sce- narios, including Gaussian attacks, Neurotoxin attacks, and Focused-Flip attacks, where 50% of the devices are mali- cious. The experiments are conducted on CIFAR100 under the DIR(0.1) data distribution. Specifically, the coefficient α is selected from 0.05, 0.1, 0.2, 0.5, 0.8, and β is selected from 0.01, 0.05, 0.1, 0.3, 0.5. All other experimental settings are kept at their default values. As shown in Fig. 8, the results indicate that when α = 0.8, the effect of the correction term is excessively amplified, leading to a decline in the test accuracy of FL-OA. This is because an overly large α emphasizes the consistency between the local model and the global model, weakening the classification capability of the model. This observation suggests that, defense schemes should not focus solely on the consistency among benign updates while neglecting the classification performance of the model itself when defending against Byzantine attacks. Notably, the performance of FL-OA varies only slightly across different values of α, indicating that FL-OA is relatively insensitive to α. Furthermore, we observe that when β is too small, local training tends to overemphasize the gradient ascent step, making the local model more prone to under-fitting and causing a noticeable drop in accuracy. Overall, FL-OA is sensitive to smaller values of β. Therefore, β should be set to a relatively larger value to enhance the consistency among model updates while maintaining training convergence. 8) Performance of FL-OA on Two-Class Non-IID Data: In our default experimental setup, we employ the Dirichlet distribution for data partitioning, which ensures overlapping JOURNAL OF L A T E X CLASS FILES, VOL. 14, NO. 8, AUGUST 202112 TABLE VI TEST ACCURACY (TOP-1) OF FL-OA ON TWO-CLASS NON-IID DATA AND DIR(0.1) UNDER CIFAR10. Root datasetRoot2Root10DIR(0.1) Att0%10%20%30%0%10%20%30%0%10%20%30% Gaussian attacks 53.17% 50.04%41.85%26.27% 58.64% 58.19%52.93%47.44% 79.45% 76.04%75.46%73.35% Neurotoxin attacks51.60%48.19%43.33%58.36%58.12%57.65%79.15%78.69%77.38% classes between devices and the OS. However, in real-world scenarios, a significant distribution shift may exist between the root dataset and the devices’ data. In extreme cases, their class sets may be completely disjoint. To evaluate the robustness of FL-OA under such extreme distribution shift, we conducted experiments on two-class Non-IID Data [50]. Specifically, we sort the data by class to create highly heterogeneous distri- butions on CIFAR10. The train set is evenly divided among the 100 devices and OS. We further design two root dataset configurations, denoted as Root2 and Root10, respectively. • Root2: The sorted data is divided into 202 partitions. Each device and OS are randomly assigned two partitions from two different classes. • Root10: The sorted data is partitioned into 210 parti- tions. Each device receives 2 partitions from 2 different classes, while the OS allocates 10 partitions covering all 10 classes, thereby providing the server with a more comprehensive view. We conduct experiments under the Root2 and Root10 settings, considering both Gaussian attacks and Neurotoxin attacks, with the proportion of malicious devices Att ∈ 0, 10%, 20%, 30%. For comparison, we performed exper- iments with 100 devices under the DIR(0.1) distribution. The experiment results are reported in Table VI. The results show that the performance of FL-OA declines significantly under the two-class Non-IID setting. Even in the absence of attacks, the global model accuracy on CIFAR10 dropped below 60%, indicating that the correction term is ineffective when the participant class distributions are completely dis- joint. Furthermore, Gaussian and Neurotoxin attacks further exacerbate the performance degradation of FL-OA in both Root2 and Root10 settings, highlighting the limitations of our framework in such distribution shift. In contrast, FL-OA maintains stable performance under the Dirichlet distribution with DIR(0.1). These results indicate that our framework is well-suited for scenarios where data classes among devices overlap significantly. 9) Effectiveness on More Datasets: To further validate the effectiveness of FL-OA on more datasets, we conduct experiments on SVHN [51], CINIC [52], and Tiny-ImageNet [53], using the ResNet18 architecture. In these experiments, 50% of the devices are set to be malicious and launch Gaussian attacks, while other hyper-parameters are kept at default settings. We compare the test accuracy of FL-OA with that of existing schemes, as reported in Table VIII. The results show that FL-OA achieves the highest test accuracy under both IID and DIR(0.1) settings. For example, under the DIR(0.1) setting, FL-OA attains the test accuracies of 72.22%, 43.43%, and 22.47% on the SVHN, CINIC, and Tiny- TABLE VII TEST ACCURACY (%↑) OF FL-OA AND EXISTING SCHEMES ON MORE DATASETS. Methods SVHNCINICTiny-ImageNet —IID—DIR(0.1)—IID—DIR(0.1)—IID—DIR(0.1) Krum57.5651.0536.8330.3818.8415.08 FLTrust62.5058.3141.4332.9717.1215.44 FL-Auditor59.8856.7543.6238.5217.1714.10 AlignIns75.1362.8942.9840.4320.3018.84 FLgym75.5465.4844.2739.6823.6320.29 FL-OA78.4172.2248.7243.4326.1122.47 TABLE VIII Acc 50%−0% (%↓) OF FL-OA AND EXISTING SCHEMES ON MORE DATASETS. Methods SVHNCINICTiny-ImageNet —IID—DIR(0.1)—IID—DIR(0.1)—IID—DIR(0.1) Krum24.0626.2012.2014.387.628.89 FLTrust18.9520.529.2512.786.138.21 FL-Auditor21.6718.118.488.977.978.59 AlignIns7.4012.818.207.225.814.25 FLgym8.291.356.245.704.865.04 FL-OA5.358.625.868.822.113.16 100120140160200 Number of devices 0 20 40 60 80 Accuracy (%) 71.19% 0.12% 0.74% 2.11% 5.67% (a) CIFAR10. 100120140160200 Number of devices 0 20 40 Accuracy (%) 42.40% 0.92% 1.19% 1.69% 3.88% (b) CIFAR100. Fig. 9.Scalability of FL-OA against Gaussian attacks over CIFAR10 and CIFAR100 with data distribution DIR(0.1). ImageNet datasets, respectively. In addition, we compare FL- OA with existing schemes in terms of the Acc 50%−0% metric. The results show that FL-OA demonstrates superior defensive performance compared to existing schemes. Specifically, under the DIR(0.1) setting on the three datasets, FL-OA achieves the lowest Acc 50%−0% values of 8.62, 8.82, and 3.16, respectively. Compared with AlignIns, these Acc values are reduced by 21.42%, 0.03%, and 57.62%, respectively. These results fur- ther verify the robustness of FL-OA across different datasets. 10) Scalability of FL-OA: In practical applications, FL is often deployed in environments with varying numbers of devices. Thus, scalability is crucial for determining whether FL-OA can maintain stable performance as the participant population changes. To evaluate the scalability of FL-OA, we conduct experiments on CIFAR10 and CIFAR100 under the DIR(0.6) setting, where 50% of the devices are malicious. The number of devices is set to 100, 120, 140, 160, and 200, respectively. Fig. 9 illustrates the scalability of FL-OA under Gaussian attacks. The results show that, as the number of devices increases from 100 to 200, the performance of FL-OA decreases by 5.67% on CIFAR10 and by 3.88% on CIFAR100. This is mainly because a larger number of devices leads to a more dispersed data distribution and less data available to each device, making it more difficult for local models to fit their data. Notably, when the number of devices increases to 160, the performance degradation of FL-OA remains relatively small, indicating that FL-OA exhibits good scalability. 11) Stability of FL-OA: As observed from Fig. 10, the performance of FL-OA exhibits some fluctuations. Thus, it JOURNAL OF L A T E X CLASS FILES, VOL. 14, NO. 8, AUGUST 202113 IIDDIR(0.6)DIR(0.1) 50.0 60.0 70.0 80.0 Test Accuracy (%) (a) Gaussian attacks IIDDIR(0.6)DIR(0.1) 50.0 60.0 70.0 80.0 Test Accuracy (%) (b) Neurotoxin attacks Fig. 10. Stability of FL-OA under both Gaussian and Neurotoxin attacks on CIFAR10. TABLE IX TIME AND COMMUNICATION COMPLEXITY IN FL-OA. NotationTask ServerOutsourced ServerDevice Communication complexityO(3nT ·|X| +|Y|) O(nT ·|X| +|Y|) O(2T ·|X|) Time complexityO(Tn· T agg ) O(Tn· T aud + T · T train ) O(T · T train ) is necessary to evaluate its stability through multiple ran- domized experiments. To this end, we conduct experiments on CIFAR10, where 50% of the device are malicious, under three data distribution settings: IID, DIR(0.6), and DIR(0.1). Each experiment is repeated 10 times. The boxplots in Fig. 10 visually demonstrate the stability of FL-OA. It can be observed that FL-OA exhibits only slight performance fluctuations under attack scenarios, and no outliers are observed in the these results, indicating that FL-OA has good stability. 12) Complexity Analysis: To evaluate the computational overhead of FL-OA, we analyze the time and communication complexity of each entity, as shown in Table IX. Specifi- cally, for TS, the time complexity is O(Tn · T agg ), where n · T agg is the time required to aggregate the updates from n devices, and T denotes the number of communication rounds. In each round, TS receives model updates from n selected devices and sends the global model back to them, resulting in a communication complexity of O(2nT · |X|), where |X| denotes the amount of parameters uploaded by a device in one round. In addition, OS sends trust scores and normalized model updates to TS, which incurs an extra communication cost ofO(nT·|X|+|Y|), where|Y| denotes the number of trust scores. Thus, the communication complexity of TS is O(3nT · |X| + |Y|). For OS, the time complexity is O(Tn · T aud + T · T train ), where n · T aud is the time required for OS to audit n devices, and T train denotes time spent by the model training in one communication round. Thus, the communication complexity of OS is represented as O(nT·|X|+|Y|). For each device, the time and communication complexity are O(T· T train ) and O(2T·|X|), respectively. We can observe that the time complexity of TS is significantly reduced due to OS helping the TS to audit model updates. Although the introduction of OS increases the communication overhead of TS byO(nT·|X|+|Y|), this overhead is affordable in practice. Specifically, |Y| denotes the trust scores transmit- ted between OS and TS, which are negligible compared to the parameters of model update, i.e., |Y|≪|X|. Furthermore, transmitting model updates between TS and OS is feasible in the real world. Therefore, the additional communication overhead introduced by OS is acceptable for FL-OA. To further evaluate the transmission delay introduced by FL-OA between the TS and the OS, we conduct measure- ments in practical Wide-Area Network (WAN) scenarios. In the experiments, the TS and OS are deployed at two loca- TABLE X MEASURED PEAK MEMORY FOOTPRINT (MB) FOR DIFFERENT SCHEMES. /FedAvgKrumFLTrustFL-AuditorAlignInsFLgymFL-OA TS1274.51279.01318.41317.22608.61294.31278.5 OS---1365.7--2622.4 0.000.250.500.751.001.251.501.75 Time (s) FL-OA FLgym AlignIns FL-Auditor FLTrust Krum FedAvg 0.972 0.874 1.302 0.931 1.092 1.583 0.702 Aggregation time Auditing time (a) Server-side runtime 0255075100125150175200 Time (s) 5 10 15 20 The Number of Local iterations 42.59 49.86 84.46 98.18 125.14 146.07 165.42 195.73 FedAvg FL-OA (b) Device-side runtime Fig. 12. The runtime for different schemes in FL. (a) Server-side runtime for different schemes. (b) Device-side runtime for FedAvg and FL-OA. tions with a physical distance of approximately 12.51 km. 20 Mbps50 Mbps100 Mbps200 Mbps 0.0 2.5 5.0 7.5 10.0 Transmission Delay (min) 20 devices30 devices40 devices Fig. 11. The transmission de- lay between the TS and the OS in FL-OA. To emulate WAN scenarios under different bandwidth conditions, we limit both the uplink and downlink transmission bandwidths between the TS and the OS to 20 Mbps, 50 Mbps, 100 Mbps, and 200 Mbps, respectively. Additionally, the size of each model update is approx- imately 41.8 MB. We record the total transmission delay required for the TS to transmit model updates from 20, 30, and 40 selected devices to the OS for outsourced auditing. The experimental results are shown in Fig. 11. We observe that the transmission delay gradually decreases as the transmission bandwidth between the TS and the OS increases. Since each model update is approximately 41.8 MB, transmitting model updates from 20, 30, and 40 devices involves approximately 836 MB, 1254 MB, and 1672 MB of data, respectively. Thus, under practical WAN scenarios with tens-to-hundreds of Mbps bandwidth, the transmission delay between the TS and the OS reaches the minute level. To evaluate the memory overhead of FL-OA, we measure the peak memory footprint on the servers and compare it with that of other schemes, as shown in Table X. The experiments are conducted on the CIFAR10 dataset, where 30 clients are selected to participate in training in each round, and the other hyper-parameters are set to their default values. Notably, FedAvg, Krum, FLTrust, AlignIns, and FLgym do not involve outsourced auditing. Thus, their update auditing and aggregation are executed on the TS. We observe that, on the TS, most schemes exhibit comparable peak memory footprints, except for AlignIns. This is because AlignIns compute and store an additional sign vector for each model update, thereby incurring a higher memory footprint. For FL-Auditor and FL-OA, the auditing procedure is executed on the OS. The results show that the memory footprint of FL-OA is higher than that of FL-Auditor on the OS. This is because FL-OA computes additional PII vectors for model updates to identify critical parameters, which introduces extra memory overhead for storing these vectors. To evaluate the efficiency of FL-OA, we measure its server- side runtime and device-side runtime , as shown in Fig. 12. The Server-side runtime includes the time required for update auditing, denoted as T aud and update aggregation, denoted as JOURNAL OF L A T E X CLASS FILES, VOL. 14, NO. 8, AUGUST 202114 TABLE XI ACTUAL PER-ROUND RUNTIME FOR DIFFERENT SCHEMES. /FedAvgKrumFLTrustFL-AuditorAlignInsFLgymFL-OA Time (min)4.074.134.097.424.114.097.54 T agg . Notably, the runtimes measured in the evaluation does not include data transmission delay. Specifically, on CIFAR10 dataset, we compare the server-side runtime of FL-OA with that of FedAvg [46] and other robust FL schemes under same hardware environments. In each round, 30 devices are selected to participate in training. As shown in Fig. 12(a), the aggregation time T agg of all robust schemes is comparable to that of FedAvg, since T agg mainly corresponds to the cost of update aggregation. For the auditing time, FL-OA achieves a lower time cost compared with Krum and AlignIns, indicating that the auditing time introduced by FL-OA is acceptable. In addition, we evaluate the device-side runtime of FL-OA. Fig. 12(b) presents the device-side runtime of FL-OA and FedAvg under different numbers of local iterations. The results show that FL-OA introduces additional computational overhead compared with FedAvg. This is mainly because the gradient ascent step and the correction term are incorporated into local optimization, enhancing the consistency of model updates among devices. It is worth noting that we measure the actual per-round runtime of all schemes under the same hardware and band- width settings. The per-round runtime consists ofthe device- side runtime, data transmission delay, and server-side runtime, as shown in Table XI. We observe that FL-Auditor and FL- OA, both of which adopt outsourced auditing, require longer per-round runtime than the other schemes. This is mainly caused by the additional data transmission delay between the task server and the outsourced server. Although FL-OA introduces extra runtime overhead, it significantly improves the robustness of the FL system against Byzantine attacks. VII. CONCLUSION To defend against Byzantine attacks, FL-OA introduces an outsourced server that possesses an additional root dataset to perform update auditing. This design enables the task server to conduct Byzantine-robust aggregation without directly ac- quiring the dataset, thereby alleviating the difficulty faced by existing defense methods in satisfying the additional dataset requirement. Furthermore, FL-OA introduces a gradient ascent step and a correction term into the local training to regulate the direction of model updates, which mitigates the divergence among benign updates caused by Non-IID data. To address the curse of dimensionality in auditing, FL-OA further designs a parameter importance indicator to extract critical parameters of model update for auditing. Experimental results demonstrate that FL-OA can effectively defend against Byzantine attacks under various data distributions. Notably, the effectiveness of FL-OA relies on the as- sumption that the OS is honest and uncompromised. This assumption is reasonable because the OS can be regarded as an independent third-party auditing entity, whose behavior is generally constrained by legal regulations and company reputation. Nevertheless, we acknowledge that collusion be- tween the OS and malicious intelligent devices may occur in practical scenarios, which could lead to the following security risks. (i) OS may intentionally assign higher trust scores to malicious devices, thereby increasing the impact of malicious updates on the global model. (i) OS may also reduce the trust scores of benign devices, causing useful benign updates to be weakened. To mitigate the above risks, a viable improvement is to extend FL-OA by introducing multiple independent outsourced servers. By averaging the trust scores provided by these servers, FL-OA can reduce its reliance on a single OS and enhance its robustness against potential collusion between a single OS and malicious devices. In addition to the collusion issue, FL-OA faces potential privacy leakage risks. Specifically, attackers may exploit the model updates submitted by devices to launch privacy attacks, such as membership inference attacks, attribute inference attacks, and data reconstruction attacks, thereby illegally infer- ring or recovering the devices’ raw training data. For example, data reconstruction attacks [54] [55] can recover original training samples from the model updates submitted by devices. More importantly, poisoning attacks and privacy threats often coexist in practical FL scenarios [56]. Given the resource- constrained nature of intelligent devices, it is imperative to integrate privacy-preserving mechanisms into Byzantine attack detection in FL. For example, it is worth investigating how to incorporate differential privacy into local training and control the privacy budget and noise magnitude, thereby achieving an effective trade-off between privacy protection and global model performance. Thus, in future work, we will investigate lightweight privacy-preserving techniques suitable for intelli- gent devices to mitigate the privacy leakage risks during model update auditing. REFERENCES [1] Z. Wang, J. Ma, X. Wang, J. Hu, Z. Qin, and K. Ren, “Threats to training: A survey of poisoning attacks and defenses on machine learning systems,” ACM Comput. Surv., vol. 55, no. 7, p. 1–36, 2022. [2] K. N. Kumar, C. K. Mohan, and L. R. Cenkeramaddi, “The impact of adversarial attacks on federated learning: A survey,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 46, no. 5, p. 2672–2691, 2024. [3] P. Blanchard, E. M. El Mhamdi, R. Guerraoui, and J. Stainer, “Machine learning with adversaries: Byzantine tolerant gradient descent,” in Proc. Adv. Neural Inf. Process. Syst., 2017, p. 119–129. [4] S. Huang, Y. Li, C. Chen, L. Shi, and Y. Gao, “Multi-metrics adaptively identifies backdoors in federated learning,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2023, p. 4652–4662. [5] T. Krauß and A. Dmitrienko, “Mesas: Poisoning defense for federated learning resilient against adaptive attackers,” in Proc. ACM Conf. Com- put. Commun. Secur., 2023, p. 1526–1540. [6] H. Zhang, Z. Yu, G. Wang, F. Xu, Y. Zhang, C. Hu, X. Wang, and J. Yu, “Toward model-contrastive federated learning with lightweight privacy preservation and poisoning attack detection,” IEEE Trans. Dependable Secure Comput., vol. 23, no. 2, p. 1830–1846, 2026. [7] D. Yin, Y. Chen, R. Kannan, and P. Bartlett, “Byzantine-robust dis- tributed learning: Towards optimal statistical rates,” in Proc. Int. Conf. Mach. Learn., 2018, p. 5650–5659. [8] X. Cao, M. Fang, J. Liu, and N. Gong, “Fltrust: Byzantine-robust federated learning via trust bootstrapping,” in Proc. NDSS Symposium, 2021. [9] Y. Miao, Z. Liu, H. Li, K. Choo, and R. Deng, “Privacy-preserving byzantine-robust federated learning via blockchain systems,” IEEE Trans. Inf. Forensics Secur., vol. 17, p. 2848–2861, 2022. JOURNAL OF L A T E X CLASS FILES, VOL. 14, NO. 8, AUGUST 202115 [10] Z. Zhang, L. Wu, D. He, J. Li, N. Lu, and X. Wei, “Using third- party auditor to help federated learning: An efficient byzantine-robust federated learning,” IEEE Trans. Sustainable Comput., vol. 9, no. 6, p. 848–861, 2024. [11] X. Zheng, Q. Dong, and A. Fu, “Wmdefense: Using watermark to defense byzantine attacks in federated learning,” in Proc. Conf. Comput. Commun. Workshops, 2022, p. 1–6. [12] Q. Dong et al., “Carefl: Contribution guided byzantine-robust federated learning,” IEEE Trans. Inf. Forensics Secur., vol. 19, p. 9714–9729, 2024. [13] Y. Xia, C. Hofmeister, M. Egger, and R. Bitar, “Byzantine-resilient secure aggregation for federated learning without privacy compromises,” arXiv preprint arXiv:2405.08698, 2024. [14] M. P. Uddin, Y. Xiang, M. Hasan, J. Bai, Y. Zhao, and L. Gao, “A systematic literature review of robust federated learning: Issues, solutions, and future research directions,” ACM Comput. Surv., vol. 57, no. 10, p. 1–62, 2025. [15] X. Mu et al., “Feddmc: Efficient and robust federated learning via detecting malicious clients,” IEEE Trans. Dependable Secure Comput., vol. 21, no. 6, p. 5259–5274, 2024. [16] Z. Lu, H. Pan, Y. Dai, X. Si, and Y. Zhang, “Federated learning with non-iid data: A survey,” IEEE Internet Things J., vol. 11, no. 11, p. 19 188–19 209, 2024. [17] W. Huang et al., “Federated learning for generalization, robustness, fairness: A survey and benchmark,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 46, no. 12, p. 9387–9406, 2024. [18] J. Pei, W. Liu, J. Li, L. Wang, and C. Liu, “A review of federated learning methods in heterogeneous scenarios,” IEEE Trans. Consum. Electron., vol. 70, no. 3, p. 5983–5999, 2024. [19] J. Bai, D. Wu, S. Zeng, Y. Zhao, Y. Qu, and S. Yu, “Non-iid free federated learning with fuzzy optimization for consumer electronics systems,” IEEE Trans. Consum. Electron., vol. 71, no. 2, p. 7032– 7044, 2025. [20] Y. Sun, L. Shen, T. Huang, L. Ding, and D. Tao, “Fedspeed: Larger local interval, less communication round, and higher generalization accuracy,” in Proc. Int. Conf. Learn. Represent., 2023. [21] J. Liu, Y. Wu, W. Du, R. Sun, G. Xu, L. Liu, and C. Wu, “Byzantine- robust hierarchical aggregation for cross-device federated learning in consumer iot,” IEEE Trans. Consum. Electron., vol. 71, no. 2, p. 6359– 6370, 2025. [22] W. Guo et al., “Byzantine-resilient federated learning with trust-aware task scheduling for heterogeneous uav swarms,” IEEE Trans. Consum. Electron., p. 1–1, 2026. [23] J. Pei, R. Xue, C. Liu, and L. Wang, “Toward byzantine-resilient secure ai: A federated learning communication framework for 6g consumer electronics,” IEEE Trans. Consum. Electron., vol. 70, no. 3, p. 5719– 5728, 2024. [24] M. Verleysen and D. Franc ̧ois, “The curse of dimensionality in data mining and time series prediction,” in Proc. Comput. Intell. Bioinspired Syst., 2005, p. 758–770. [25] K. Beyer, J. Goldstein, R. Ramakrishnan, and U. Shaft, “When is “nearest neighbor” meaningful?” in Proc. Int. Conf. Database Theory, 1999, p. 217–235. [26] Z. Li, F. Liu, W. Yang, S. Peng, and J. Zhou, “A survey of convolutional neural networks: Analysis, applications, and prospects,” IEEE Trans. Neural Networks Learn. Syst., vol. 33, no. 12, p. 6999–7019, 2022. [27] Q. Li, B. He, and D. Song, “Model-contrastive federated learning,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2021, p. 10 713–10 722. [28] T. Li, A. K. Sahu, M. Zaheer, M. Sanjabi, A. Talwalkar, and V. Smith, “Federated optimization in heterogeneous networks,” in Proc. Mach. Learn. Syst., vol. 2, 2020, p. 429–450. [29] M. Andriushchenko and N. Flammarion, “Towards understanding sharpness-aware minimization,” in Proc. Int. Conf. Mach. Learn., 2022, p. 639–668. [30] Z. Qu, X. Li, R. Duan, Y. Liu, B. Tang, and Z. Lu, “Generalized federated learning via sharpness aware minimization,” in Proc. Int. Conf. Mach. Learn., 2022, p. 18 250–18 280. [31] Y. Zhao, H. Zhang, and X. Hu, “Penalizing gradient norm for efficiently improving generalization in deep learning,” in Proc. Int. Conf. Mach. Learn., 2022, p. 26 982–26 992. [32] F. Hanzely and P. Richt ́ arik, “Federated learning of a mixture of global and local models,” arXiv preprint arXiv:2002.05516, 2020. [33] V. Shejwalkar and A. Houmansadr, “Manipulating the byzantine: Opti- mizing model poisoning attacks and defenses for federated learning,” in Proc. NDSS Symposium, 2021. [34] A. Ma ́ ckiewicz and W. Ratajczak, “Principal components analysis (pca),” Computers & Geosciences, vol. 19, no. 3, p. 303–342, 1993. [35] L. Van der Maaten and G. Hinton, “Visualizing data using t-sne.” Journal of machine learning research, vol. 9, no. 11, 2008. [36] Y. J. Cho, J. Wang, and G. Joshi, “Client selection in federated learning: Convergence analysis and power-of-choice selection strategies,” arXiv preprint arXiv:2010.01243, 2020. [37] J. Wang, Q. Liu, H. Liang, G. Joshi, and H. Poor, “Tackling the objective inconsistency problem in heterogeneous federated optimization,” in Proc. Adv. Neural Inf. Process. Syst., vol. 33, 2020, p. 7611–7623. [38] J. Xu, Z. Zhang, and R. Hu, “Detecting backdoor attacks in federated learning via direction alignment inspection,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2025, p. 20 654–20 664. [39] S. Targ, D. Almeida, and K. Lyman, “Resnet in resnet: Generalizing residual architectures,” arXiv preprint arXiv:1603.08029, 2016. [40] A. Krizhevsky, G. Hinton et al., “Learning multiple layers of features from tiny images,” 2009. [41] J. Lin, “On the dirichlet distribution,” Master’s thesis, Dept. Math. Statist., Queen’s Univ., vol. 40, 2016. [42] Y. Fraboni, R. Vidal, and M. Lorenzi, “Free-rider attacks on model aggregation in federated learning,” in Proc. Artif. Intell. Statist., 2021, p. 1846–1854. [43] Z. Zhang et al., “Neurotoxin: Durable backdoors in federated learning,” in Proc. Int. Conf. Mach. Learn., 2022, p. 26 429–26 446. [44] P. Fang and J. Chen, “On the vulnerability of backdoor defenses for federated learning,” in Proc. AAAI Conf. Artif. Intell., vol. 37, no. 10, 2023, p. 11 800–11 808. [45] K. Xiao, Q. Wang, and C. Anagnostopoulos, “Flgym: Toward robust and byzantine-resilient federated learning,” IEEE Trans. Inf. Forensics Secur., vol. 21, p. 404–416, 2026. [46] B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. Arcas, “Communication-efficient learning of deep networks from decentralized data,” in Proc. Artif. Intell. Statist., 2017, p. 1273–1282. [47] D. Acar, Y. Zhao, R. Navarro, M. Mattina, P. Whatmough, and V. Saligrama, “Federated learning based on dynamic regularization,” arXiv preprint arXiv:2111.04263, 2021. [48] T. Zhou, J. Zhang, and D. H. K. Tsang, “Fedfa: Federated learning with feature anchors to align features and classifiers for heterogeneous data,” IEEE Trans. Mob. Comput., vol. 23, no. 6, p. 6731–6742, 2024. [49] T. Yu, E. Bagdasaryan, and V. Shmatikov, “Salvaging federated learning by local adaptation,” arXiv preprint arXiv:2002.04758, 2020. [50] Y. Zhao, M. Li, L. Lai, N. Suda, D. Civin, and V. Chandra, “Federated learning with non-iid data,” arXiv preprint arXiv:1806.00582, 2018. [51] Y. Netzer et al., “Reading digits in natural images with unsupervised feature learning,” in NIPS Workshops., vol. 2011, no. 2, 2011, p. 4. [52] L. N. Darlow, E. J. Crowley, A. Antoniou, and A. J. Storkey, “Cinic-10 is not imagenet or cifar-10,” arXiv preprint arXiv:1810.03505, 2018. [53] Y. Le, X. Yang et al., “Tiny imagenet visual recognition challenge,” CS 231N, vol. 7, no. 7, p. 3, 2015. [54] L. Zhu, Z. Liu, and S. Han, “Deep leakage from gradients,” in Proc. NIPS, vol. 32, 2019. [55] E. Sotthiwat, L. Zhen, C. Zhang, Z. Li, and R. S. M. Goh, “Generative image reconstruction from gradients,” IEEE Trans. Neural Netw. Learn. Syst., vol. 36, no. 1, p. 21–31, 2025. [56] R.-Y. Huang, D. Samaraweera, and J. M. Chang, “Exploring threats, defenses, and privacy-preserving techniques in federated learning: A survey,” Computer, vol. 57, no. 4, p. 46–56, 2024.