Paper deep dive
Concept Drift Detection and Adaptive Retraining of Malware Classification Models
Christofer Washington Berruz Chungata, Martin Jurecek, Katerina Potika, William B. Andreopoulos, Mark Stamp
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 8/16/2026, 2:32:58 AM
Summary
This paper investigates automated concept drift detection in malware classification models to optimize retraining efficiency. It compares three detection techniques: One-Class Support Vector Machines (OCSVM), Minibatch K-Means (MK-Means), and Maximum Mean Discrepancy (MMD). Experiments using four classifiers (MLP, RF, SVM, XGB) across static, periodic, and drift-aware retraining scenarios demonstrate that drift-aware retraining, particularly using OCSVM, achieves accuracy comparable to periodic retraining while significantly reducing computational costs.
Entities (10)
Relation Signals (11)
OCSVM → usedfor → Concept Drift Detection
confidence 95% · we analyze two machine learning-based approaches to automated concept drift detection-a novel approach based on One-Class Support Vector Machines (OCSVM)
MK-Means → usedfor → Concept Drift Detection
confidence 95% · a previously-studied technique based on Minibatch K-Means (MK-Means)
MMD → usedfor → Concept Drift Detection
confidence 95% · we also consider Maximum Mean Discrepancy (MMD), a statistical technique for detecting changes
Drift-Aware Retraining → achievesaccuracycomparableto → Periodic Retraining
confidence 92% · all three concept drift detection techniques achieve classification accuracy comparable to periodic retraining
Drift-Aware Retraining → hashigherefficiencythan → Periodic Retraining
confidence 92% · offering substantially greater efficiency in terms of the number of models that must be retrained
MLP → evaluatedin → Malware Classification
confidence 90% · comparing the effectiveness of four learning models, namely, Multilayer Perceptron...
SVM → evaluatedin → Malware Classification
confidence 90% · comparing the effectiveness of four learning models, namely, ... Support Vector Machines...
XGB → evaluatedin →
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Concept drift refers to changes over time in the statistical properties of data, as compared to the data that was used to train a learning model. Machine learning models for malware detection or classification are particularly susceptible to performance degradation caused by concept drift, as attackers constantly modify existing malware. In this chapter, we analyze two machine learning-based approaches to automated concept drift detection-a novel approach based on One-Class Support Vector Machines (OCSVM) and a previously-studied technique based on Minibatch K-Means (MK-Means). For comparison we also consider Maximum Mean Discrepancy (MMD), a statistical technique for detecting changes in multidimensional data. We conduct an extensive series of experiments comparing the effectiveness of four learning models, namely, Multilayer Perceptron, Random Forest, Support Vector Machines, and eXtreme Gradient Boosting. For each of these models, we consider three distinct scenarios: A static scenario where no model retraining occurs, a periodic scenario where models are constantly retrained irrespective of concept drift, and a drift-aware scenario where models are only retrained when concept drift is detected. Under the drift-aware scenario, we analyze the tradeoff between accuracy and training efficiency using Pareto Front analysis. We find that all three concept drift detection techniques achieve classification accuracy comparable to periodic retraining, while offering substantially greater efficiency in terms of the number of models that must be retrained. In addition, drift-aware retraining based on our OCSVM technique generally outperforms the MK-Means and MMD approaches. Overall, these results provide strong evidence that we can accurately detect concept drift in malware classification models.
Tags
Links
- Source: https://arxiv.org/abs/2608.13465v1
- Canonical: https://arxiv.org/abs/2608.13465v1
Trouble viewing inline? Open PDF directly →
Full Text
103,490 characters extracted from source content.
Expand or collapse full text
Concept Drift Detection and Adaptive Retraining of Malware Classification Models Christofer Washington Berruz Chungata11 1 Department of Computer Science, San Jose State University Martin Jureček22 2 Faculty of Information Technology, Czech Technical University in Prague Affiliation: Katerina Potika11footnotemark: 1 William B. Andreopoulos11footnotemark: 1 Mark Stamp11footnotemark: 1 33 3 mark.stamp@@sjsu.edu Abstract Concept drift refers to changes over time in the statistical properties of data, as compared to the data that was used to train a learning model. Machine learning models for malware detection or classification are particularly susceptible to performance degradation caused by concept drift, as attackers constantly modify existing malware. In this chapter, we analyze two machine learning-based approaches to automated concept drift detection—a novel approach based on One-Class Support Vector Machines (OCSVM) and a previously-studied technique based on Minibatch K-Means (MK-Means). For comparison we also consider Maximum Mean Discrepancy (MMD), a statistical technique for detecting changes in multidimensional data. We conduct an extensive series of experiments comparing the effectiveness of four learning models, namely, Multilayer Perceptron (MLP), Random Forest (RF), Support Vector Machines (SVM), and eXtreme Gradient Boosting (XGB). For each of these models, we consider three distinct scenarios: A static scenario where no model retraining occurs, a periodic scenario where models are constantly retrained irrespective of concept drift, and a drift-aware scenario where models are only retrained when concept drift is detected. Under the drift-aware scenario, we analyze the tradeoff between accuracy and training efficiency using Pareto Front analysis. We find that all three concept drift detection techniques achieve classification accuracy comparable to periodic retraining, while offering substantially greater efficiency in terms of the number of models that must be retrained. In addition, drift-aware retraining based on our OCSVM technique generally outperforms the MK-Means and MMD approaches. Overall, these results provide strong evidence that we are able to accurately detect concept drift in malware classification models. Furthermore, our concept drift detection techniques are efficient and practical, and the process of updating learning models can easily be fully automated. Keywords: Concept Drift ⋅· Malware ⋅· One-Class Support Vector Machines ⋅· OCSVM ⋅· Maximum Mean Discrepancy ⋅· MMD ⋅· MK-Means ⋅· Pareto Front 1 Introduction Change over time in the statistical properties of data on which learning models have been trained is known as concept drift. Given that machine learning models learn from data, concept drift can degrade the performance of such models during inference. Concept drift is a particularly important problem in malware detection as malware is constantly evolving. According to [3, 50], attackers modify malware for two purposes—to add or modify functionality and to evade detection. As a result, developers and researchers using machine learning models for malware detection or classification must take concept drift into consideration when deploying these models in real-world systems. The number of cyberattacks due to malware, such as ransomware attacks, has increased significantly in recent years [6]. A single malware sample that is incorrectly classified as benign can wreak havoc on a system by bringing down machines or holding data hostage for ransom. One solution to mitigate the effects of concept drift is to constantly retrain the models used for detection. However, the resource demands of this approach may be substantial, in terms of computational resources and energy usage. In an era where significant research efforts focus on minimizing the resource consumption of complex machine learning models, such as Deep Neural Networks (DNN) and Large-Language Models (LLM) [15, 34], constant retraining would seem to be a step backwards. In this research, we conduct experiments involving two machine learning-based approaches for concept drift detection in malware classification models—one based on Minibatch K-Means (MK-Means) and another based on One-Class Support Vector Machines (OCSVM). For comparison, we also consider concept drift detection using the well-known statistical technique of Maximum Mean Discrepancy (MMD), which can be applied to detect changes in multidimensional data. To demonstrate the effectiveness of these concept drift detection approaches, we consider the following three scenarios, which are discussed in more detail in Section 3.2. 1. A classification model is trained on the initial temporal segment of the training data, and no retraining occurs. This represents a scenario where concept drift is not considered. We refer to this as the static scenario. 2. As in i), a classification model is trained on the initial temporal segment of the training data. Then the model is retrained at regular periodic intervals. This represents a scenario where concept drift is considered, but no effort is made to minimize the number of models that must be retrained. We refer to this as the periodic retraining scenario. 3. As in i), a classification model is trained on the initial temporal segment of the training data and concept drift is considered. However, instead of periodic retraining, we only retrain when concept drift is detected via one of the methods mentioned above. This represents a scenario where we account for concept drift, but we also attempt to minimize model retraining. We refer to this as the drift-aware retraining scenario. For each of these three scenarios, we consider four classification models, namely, Multilayer Perceptron (MLP), Random Forest (RF), eXtreme Gradient Boosting (XGB), and Support Vector Machines (SVM). Furthermore, for each of these model-scenario combinations, we consider 20 distinct combinations of malware families. Thus, we conduct 80 static experiments, 80 periodic retraining experiments, and 240 drift-aware retraining experiments (80 using each of the OCSVM, MK-Means, and MMD drift detectors), giving us a total of 400 distinct experiments. The remainder of this chapter is organized as follows. Section 2 provides relevant background information on concept drift, MK-Means, OCSVM, MMD, and the learning models used in our classification experiments. In Section 3, we describe the dataset used in this research, and the development environment for our experiments. In addition, this section describes the static, periodic retraining, and drift-aware retraining scenarios in detail. Section 4 presents results and analysis for all of our experiments. We demonstrate that using drift-aware retraining yields a large increase in accuracy, as compared to the static case, and only a slight decrease in accuracy, as compared to periodic retraining. We further analyze the drift-aware retraining case, and provide detailed information on the tradeoff between accuracy and efficiency. We conclude the chapter in Section 5, where we also provide suggestions for future work. Note that this chapter expands on and extends the research that appeared in the paper [12]. Specifically, the MMD technique presented in this chapter is analyzed in [12], which in the present chapter, merely serves as a baseline for comparison of the MK-Means and OCSVM techniques. 2 Background In this section, we first discuss concept drift and malware detection. Then we provide details on the OCSVM, MK-Means, and MMD techniques, which form the basis for our automated concept drift detection experiments. Finally, we briefly introduce the learning models that are used in our experiments. 2.1 Concept Drift Concept drift is defined as changes in the statistical properties of data over time, which can affect the performance of machine learning models [5]. Machine learning models deployed in real-world systems may be adversely affected by concept drift, as the data used for training the model can age out or become irrelevant. This is certainly the case for machine learning-based malware detection systems, as malware families generally tend to evolve over time. Concept drift can occur abruptly, gradually, or it can be recurring [5]. Abrupt concept drift is the easiest to detect, while gradual and recurring concept drift require more sophisticated methods that incorporate previous knowledge of the data and the model [11, 43, 49, 59]. According to [5], any concept drift detection technique can be classified into one of four categories, namely, data distribution-based, performance-based, hybrid, and context-based. Performance-based methods are considered more reliable than data distribution-based methods, as they focus on instances where the model is performing poorly [5]. However, such methods rely on the ability to measure well-defined performance metrics (e.g., accuracy, precision, recall, F1-score), and when using machine learning models to classify malware, such information is generally lacking. Given the limitations of performance-based methods in the malware domain, we focus our attention on data distribution-based methods. That is, we require drift detection techniques that can be performed by directly analyzing the data, without relying on measures of model performance. We validate that our drift detection approaches succeed by comparing the performance of classification models under three distinct scenarios—details on these scenarios are given in Section 3.2. 2.2 Malware Detection Malware detection is the process of identifying whether a given piece of software is malicious (malware) or benign. Malware detection is a vitally important area of research for at least three reasons. First, false negatives (malware misclassified as benign) can lead to significant damage to a system, such as data loss and system downtime. Second, false positives (benign misclassified as malware) can lead to significant inconvenience to users. Even a relatively small false positive rate can result in a “boy who cried wolf” syndrome, where users lose trust in the system and ignore warnings. Third, malware is constantly evolving, making it challenging to detect new malware variants [3, 50]. Over the past 20 years, malware research has shifted dramatically, to where the focus today is dominated by machine learning (ML) and deep learning (DL) techniques. From classical ML models, such as Support Vector Machines (SVM) and Random Forests (RF), to DL models, such as Convolutional Neural Networks (CNN) and Recurrent Neural Networks (RNN), learning models are now routinely applied to the malware detection and classification problems [4, 35, 42]. In recent years, the impact of concept drift on malware detection has gained some attention from researchers—although certainly not as much attention as the importance of the topic warrants. For example, the authors of [31] proposed a novel technique for detecting and classifying concept drift in malware by learning supposedly drift-invariant features from control flow graphs. Another example of such research can be found in [1], which introduced MORPH, a pseudo-label-based concept drift adaptation method for neural networks as used for malware detection. Additionally, the paper [25] considers DREAM, a semi-supervised system that enhances drift detection in deep learning-based malware classifiers. Note that the approaches considered in our research differ from [1, 25, 31], as we use unsupervised machine learning techniques that are faster, simpler, and more efficient to implement. Our methodology builds on and extends the work in [36]. We note in passing that research into malware evolution is related to concept drift, in the sense that such research deals with changes to malware over time. However, in concept drift research, the focus is on how these changes affect the performance of machine learning models, while malware evolution research deals with the changes themselves. Hidden Markov Models (HMM) [53], Word2Vec [39], SVMs [55], and hierarchical tree models [60] have all been used to study malware evolution. Research into concept drift detection for malware requires data that contains not only a significant number of malware samples over an extended period of time, but also include reliable timestamps. This is challenging because the time when a malware sample was originally created is not generally available. Therefore, modification or collection dates are used as proxies for the creation date. Prior to the KronoDroid dataset [22], other malware datasets used for research involving temporal aspects of malware analysis included the Malicia dataset [37] and the malware collection in [28]. 2.3 Concept Drift Detection Techniques In this section, we introduce the two unsupervised learning techniques that we employ for concept drift detection, namely, Minibatch K-Means (MK-Means) and One-Class Support Vector Machines (OCSVM). Furthermore, we introduce Maximum Mean Discrepancy (MMD), an established technique to detect data drift in multidimensional data. Additional details on precisely how we employ these techniques for concept drift detection are provided in Section 3.2. 2.3.1 Minibatch K-Means Minibatch K-Means (MK-Means) [2] is a more scalable variant of the well-known K-Means clustering algorithm [24]. While many K-means variants exist, such as K-Means++ [27], MK-Means has been optimized for large datasets by using stochastic optimization principles. The main difference between K-Means and MK-Means is that the latter uses small random subsets of the data (i.e., minibatches) to update the centroids, instead of using the entire dataset at each iteration. As with K-means, MK-Means is an unsupervised clustering algorithm that uses a hill climb approach to find improved centroids of clusters. Formally, the K-Means algorithm aims to minimize Within-Cluster Sum of Squares (WCSS), which is defined as argminS1,…,SK∑i=1K∑x∈Si∥x−μi∥2 _\S_1,…,S_K\ _i=1^K _x∈ S_i\|x- _i\|^2 where μi _i is the mean, or centroid, of the data points in cluster SiS_i, and the nonempty subsets S1,S2,…,SKS_1,S_2,…,S_K of S form a partition of the set S. Note that a centroid μi _i need not be an actual data point in its cluster SiS_i. Given that it is a hill climb algorithm, MK-Means is sensitive to the initial placement of the centroids. Hence, MK-Means can produce different clustering results, even for a fixed number of clusters K. Intuitively, a good clustering result should produce clusters that are well separated from each other, while any individual cluster should be relatively compact. The silhouette coefficient provides a single value that incorporates both of these desirable aspects of cluster quality. For a given clustering, of the n data points x1,x2,…,xn\x_1,x_2,…,x_n\, the silhouette coefficient of a specific data point xix_i is defined as s(i)=b(i)−a(i)max(a(i),b(i))s(i)= b(i)-a(i) (a(i),b(i) ) where a(i)a(i) is the average distance between xix_i and all other points in its same cluster, and b(i)b(i) is the minimum of the average distances from xix_i to points in each of the other clusters. For a reasonable clustering, we expect b(i)>a(i)b(i)>a(i), in which case s(i)=1−a(i)b(i)s(i)=1- a(i)b(i) In this form, we see that a silhouette coefficient value s(i)s(i) closer to 1 represents a better clustering result for xix_i than values closer to 0. Thus, the larger the average silhouette coefficient s=1n∑i=1ns(i)s= 1n _i=1^ns(i) the better the overall quality of a given clustering. For concept drift detection, we use MK-Means to cluster consecutive (overlapping) pairs of temporal batches of samples, and we compute the average silhouette coefficient for each such clustering. If the change in average silhouette coefficient from one pair of batches to the next exceeds a specified threshold, this implies that the features representing the underlying samples have changed, and hence concept drift has occurred. More details on MK-Means for concept drift detection are provided in Section 3.2.2. We note that our use of MK-Means for concept drift detection in malware closely follows the approach in [36]. 2.3.2 One-Class Support Vector Machines Support Vector Machines (SVM) are supervised machine learning models used for classification and regression tasks. One-Class SVMs (OCSVM) are a variant of SVMs that are trained in an unsupervised mode, and hence labeled data is not required. OCSVMs were originally introduced in [45]. OCSVMs are used widely for outlier detection [16, 33]. Outlier detection is a generic problem where, for a given dataset, there exists two regions defined as inliers and outliers. The outliers differ significantly from the inliers, while the inliers form the bulk of the data. Many techniques exist for outlier detection, including Robust Covariance [44], Isolation Forest [32], and Local Outlier Factor [9]; refer to [46] for a visual comparison of the results of these techniques on a synthetic dataset. OCSVM tries to find the region where the training samples are representative of the dominant distribution of the data. OCSVMs include a regularization parameter ν∈(0,1)ν∈(0,1), which sets an upper limit on the fraction of training data points that can be considered outliers. Larger values of ν can lead to a more complex, but looser, decision boundary, while smaller values of ν yield a simpler, but tighter, decision boundary. For a dataset with a high number of outliers, too small of values of ν can result in long training times and models that are overly sensitive to noise. For a given OCSVM model, let noutliersn_ be the number of outliers and ninliersn_ the number of inliers. We define r=noutliersninliersr= n_ n_ (1) If this ratio r changes over time, it is reasonable to assume that the underlying distribution of the data has changed. Therefore, we can use OCSVM to detect concept drift by monitoring the ratio of the number of outliers to the number of inliers. Additional details on how we employ OCSVM for concept drift detection are given in Section 3.2.2. As far as the authors are aware, this is the first time that OCSVM has been used for concept drift detection, although OCSVMs have previously been used for malware anomaly detection [47]. 2.3.3 Maximum Mean Discrepancy Concept drift can cause model performance degradation due to changes in the underlying data distribution. The authors of [41] provide an empirical study of various distribution shift detection techniques, with the goal of better understanding the relationship between distribution shift and model performance degradation. Given two sets of samples, x=x1,x2,…,xnx=\x_1,x_2,…,x_n\ and x′=x1′,x2′,…,xm′x =\x _1,x _2,…,x _m\, the goal in distribution shift detection is to determine whether the probability distributions p(x)p(x) and q(x′)q(x ) are similar. To detect whether two distributions are similar, we can use hypothesis testing with a two-sample test. Formally, we want to test the null hypothesis H0:p(x)=q(x′)H_0:p(x)=q(x ) against the alternative HA:p(x)≠q(x′)H_A:p(x)≠ q(x ) at a specified significance level α. We note in passing that distribution shift depends on the chosen representation of the samples, which we refer to as the latent space. Depending on the latent space, two-sample tests can give different results. Maximum Mean Discrepancy (MMD) [21] is a popular kernel-based technique for multivariate two sample testing. MMD can distinguish between two distributions p and q based on the mean embeddings μp _p and μq _q in a special latent space (i.e., Hilbert space) that we denote as ℋ . MMD uses a kernel k to map samples onto ℋ . The MMD is computed as MMD(ℋ,p,q)=‖μp−μq‖ℋ. ( ,p,q)=\| _p- _q\|_ . Given samples drawn from p and q we compute the unbiased estimator MMD2=1n2∑i≠jnk(xi,xj)−2nm∑i=1n∑j=1mk(xi,xj′)+1m2∑i≠jmk(xi′,xj′) ^2= 1n^2 _i≠ j^nk(x_i,x_j)- 2nm _i=1^n _j=1^mk(x_i,x _j)+ 1m^2 _i≠ j^mk(x _i,x _j) where k is the Gaussian exponential kernel, k(x,x~)=e−∥x−x~∥2/σk(x, x)=e^-\|x- x\|^2/σ, and σ is the median distance between all pair of points (x,x′)(x,x ) [21]. The authors of [41] note that we can compute a p-value using permutation tests on the resulting kernel matrix to perform hypothesis testing. Once a p-value is computed, we accept or reject the null hypothesis H0H_0 at a chosen significance level α depending on whether p-value>α $p$- value>α (accept H0H_0) or p-value≤α $p$- value≤α (reject H0H_0). The Python package Alibi Detect [54] contains implementations of all techniques described in [41]; we use these implementations for our experiments. 2.4 Learning Models In this section, we introduce the four learning models that are consider in our experiments. These are standard learning models that serve to test our concept drift detection techniques. 2.4.1 Multilayer Perceptron A MultiLayer Perceptron (MLP) is a classic type of feedforward Artificial Neural Network (ANN). An MLP consists of an input layer, an output layer, and one or more hidden layers. Each hidden layer is fully connected to the layers above and below. MLPs are usually trained using backpropagation. Training MLPs is done in batches and epochs—one epoch represents a forward and backward pass through the entire training set, which is usually divided into minibatches. The main hyperparameters of an MLP architecture are the number of hidden layers, the number of neurons in each layer, and the choice of activation functions. For classification tasks, cross-entropy is widely used, and it is available in scikit-learn [48]. For binary classification, Binary Cross Entropy (BCE), also known as Log Loss, is used [51]. Assuming a classifier is a probabilistic model, BCE measures the distance between two Bernoulli distributions—the true distribution and the predicted distribution given by the classifier [51]. Thus, for n samples, the BCE is computed as BCE=−1n∑i=1n(yilog(pi)+(1−yi)log(1−pi)) =- 1n _i=1^n (y_i (p_i)+(1-y_i) (1-p_i) ) where yiy_i is the actual label (0 or 1) of sample i, and pip_i is the predicted probability of the ithi sample being in class 1. Note that due to the log term, BCE assigns a higher penalty to more confident, but incorrect, predictions. 2.4.2 Support Vector Machine A Support Vector Machine (SVM) is a supervised machine learning model. The goal of an SVM is to find the optimal hyperplane that separates the data into classes. To achieve separability, SVMs can map the data into a higher dimensional space using a kernel function. The kernel function is selected such that computing the dot product in a higher dimensional space does not incur any significant computational cost in terms of the input data—this is known as the “kernel trick.” The most popular kernel functions are linear, polynomial, and radial basis function (rbf) [57]. For additional details on SVMs, see, for example [14]. 2.4.3 Random Forest Random Forest (RF) is an ensemble machine learning technique that combines decision trees to make a prediction. Random Forest uses “bagging” to create uncorrelated decision trees [26]. Random Forest is a classical machine learning model that in practice can often rival more complex neural network models. An important hyperparameter of RF is the depth of the trees. A deeper depth means that more decisions are included in each tree, which can possibly lead to a higher accuracy, at the cost of a higher training time, as well as an increased chance of overfitting. RF models tend to generalize well given the uncorrelated nature of the component decision trees. For more details on RFs, refer to [8]. 2.4.4 Extreme Gradient Boosting Similar to RF, eXtreme Gradient Boosting (XGB) is an ensemble machine learning technique based on decision trees. It is important to note that XGBoost is an optimized version of a general gradient boosting algorithm, with the goal of maximizing performance and efficiency [10]. XGBoost uses boosting instead of bagging, which distinguishes it from RF. In practice, this means that while RF builds independent decision trees by random bagging of the features and samples, XGBoost builds decision trees sequentially. In XGBoost, each new (possibly weak) classifiers aims to predict the errors of the previous classifier [30]. The intuition behind XGBoost is that each new classifier will generally be able to improve on the errors of the previous classifiers. 3 Implementation In this section, we first discuss the dataset that we use in our experiments. Then we present a detailed view of our experimental setup. 3.1 Dataset The study of concept drift in malware detection requires a dataset in which samples can be placed on a relative timeline. We use the KronoDroid dataset, which was introduced in [22]; we retrieved the dataset from [23]. The KronoDroid dataset contains 41,382 Android malware samples belonging to 240 distinct malware families. Note that the KronoDroid dataset contains both real and emulated samples—we only use real samples in this research. Each sample in the dataset includes 200 static features (permissions, intents, hardware/software requirements, etc.) and 289 dynamic features (e.g., system calls). According to the paper introducing KronoDroid [22], each sample contains four distinct timestamps: Earliest Modification, Last Modification, First Seen VT (on Virus Total) and First Seen ITW (in the wild). However, in the dataset available from Github [23], malware samples contain only two timestamps: EarliestModDate and HighestModDate. In this research, we use the HighestModDate timestamp as it is the best available timestamp in the dataset with the least number of invalid samples, according to [22]. 3.1.1 Malware Families Although there are 240 malware families in the KronoDroid dataset, we focus our attention on the five malware families with the largest number of samples. By doing so, we can study concept drift and evaluate model performance for a significant number of samples over meaningful periods of time. In this respect, our approach is analogous to that followed in [36]. The five malware families under consideration are the following. Airpush/StopSMS is adware that displays unwanted ads and may silently collect and forward user data [17]. Subsequently, we refer this family simply as Airpush. SMSReg is riskware that, for example, may include a fake “Battery Improve” application [18]. Malap is spyware that collects sensitive information from a device [22]. Boxer is a Trojan that pretends to be a legitimate installer but actually sends premium-rate SMS messages, unbeknownst to the user [19]. Agent is a Trojan that downloads and installs adware or malware onto a victim device [20]. 3.1.2 Data Preprocessing As mentioned above, each malware sample in our dataset includes 489 features, 200 of which are static and 289 of which are dynamic. However, some features are not numerical, but are instead text. As a preprocessing step, we remove all non-numerical features using the Polars library [40]. The dimension of each feature vector after this preprocessing is 470. Additionally, we discard samples whose HighestModDate timestamp seems to be incorrect. In concrete terms, only samples with a timestamp in the format “M/D/Y” and years in the range of 2000 to 2025 are considered valid. Table 1 summarizes the extracted samples from the KronoDroid dataset; we explain the meaning of the “batches” column in Section 3.2.1. Table 1: KronoDroid dataset after preprocessing Family Samples Batches Earliest date Latest date Agent 2,826 056 2008-02-28 2020-07-17 Airpush 7,760 155 2008-02-29 2018-06-15 Boxer 3,597 071 2005-01-01 2018-06-15 Malap 4,018 080 2008-02-29 2020-11-11 SMSreg 4,989 099 2008-02-29 2020-11-09 Total 23,190 461 2005 2020 3.2 Experiment Setup As discussed above, in this research we consider three scenarios to evaluate the benefits of employing concept drift detection in the malware domain. We refer to these three scenarios as static training, periodic retraining, and drift-aware retraining. In each scenario, we train each of the four machine learning classifiers discussed above (MLP, SVM, RF, and XGB) on each of 20 distinct classification tasks. Intuitively, we expect that periodic retraining will yield the highest accuracy while the static training accuracy will be the lowest. If we are able to accurately detect concept drift, then the drift-aware retraining accuracies should be comparable to the periodic retraining accuracies, while reducing the work required to retrain models. 3.2.1 Definitions Let ℛ=ocsvm,mk-means,mmd R=\ , , \ denote the set of drift detection techniques considered. We let ℒ=MLP,SVM,RF,XGB L=\ , , , \ be the set of learning models under consideration. Let F denote the samples belonging to a specific malware family. For each experiment, let FdF_d be the samples of the malware family for which we want to detect where concept drift has occurred, and let FcF_c be the “control” samples, where FcF_c is a different malware family than FdF_d. For our binary classification experiments, FdF_d and FcF_c are the classes that we train models to distinguish. We employ the notation ℱ=(Fd,Fc)|Fd≠Fc F=\(F_d,F_c)\,|\,F_d≠ F_c\, and since we consider five families, |ℱ|=20| F|=20. For each malware family F, we order the samples from oldest to most recent, based on the HighestModDate timestamp. Let n be the number of samples in F, and denote the ordered samples of F as (f0,f1,…,fn−1)(f_0,f_1,…,f_n-1), where fif_i precedes fjf_j in the temporal ordering whenever i<ji<j. For a malware family F, we partition the samples into consecutive temporal batches, truncating so that all temporal batches are of the same size. Note that the smaller the batch size, the sooner we can detect concept drift, but smaller batch sizes also tend to increase the variance, adding noise to the process. In all of our experiments, we define the batch size as b=50b=50. Small-scale tests showed that the results did not improve significantly for larger values of b, while adverse variance effects were observed for smaller batch sizes. Assuming that |F|=n|F|=n, the number of batches in F is ⌊n/50⌋ n/50 . Letting BiB_i represent the ithi batch, we have Bi=fi⋅b,fi⋅b+1,…,f(i+1)⋅b−1B_i=\f_i· b,f_i· b+1,…,f_(i+1)· b-1\ Thus, B0=f0,f1,…,f49B_0=\f_0,f_1,…,f_49\, B1=f50,f51,…,f99B_1=\f_50,f_51,…,f_99\, and so on. Let BitrainB_i be the training subset and BitestB_i be the testing subset of BiB_i. We define BitrainB_i to be the first t samples in BiB_i and BitestB_i to be the remaining b−tb-t samples in BiB_i. We use t=30t=30 samples for training and hence the number of samples used for testing per batch is b−t=50−30=20b-t=50-30=20. For example, for the first batch B0B_0, we have B0train=f0,f1,…,f29B_0 =\f_0,f_1,…,f_29\ and B0test=f30,f31,…,f49B_0 =\f_30,f_31,…,f_49\. The “batches” column in Table 1 provides the number of batches per family. 3.2.2 Concept Drift Detection As mentioned above, we consider two machine learning-based approaches for concept drift detection—one based on MK-Means, another based on OCSVM—as well as the statistical-based MMD technique. In this section, we describe these three approaches in more detail. MK-Means — We employ the silhouette coefficient, based on MK-Means clustering, to detect concept drift as follows. Using the notation above, we let ℬ0=B0,B1 B_0=\B_0,B_1\ and ℬ1=B1,B2 B_1=\B_1,B_2\. Then we perform MK-Means clustering on the samples in ℬ0 B_0 and separately perform MK-Means clustering on the samples in ℬ1 B_1. Next, we compute the average silhouette coefficient s0s_0 for the clustering of ℬ0 B_0 and compute the average silhouette coefficient s1s_1 for the clustering of ℬ1 B_1. Then if |s1−s0|>mk-means|s_1-s_0|>T_ , where mk-meansT_ is a specified threshold, we assume that concept drift has occurred at batch B1B_1. We repeat this process for subsequent consecutive pairs of batches. OCSVM — We employ OCSVM to detect concept drift as follows. We train an OCSVM on B0B_0 and compute the ratio of outliers to inliers—as per equation (1)—which we denote as r0r_0. The ratio r0r_0 represents characteristics of the data in B0B_0. We then reuse this same OCSVM model to compute r1r_1, the ratio of outliers to inliers over B1B_1. If |r1−r0|>ocsvm|r_1-r_0|>T_ , where ocsvmT_ is a specified threshold, then we assume that the samples in B1B_1 differ from those in B0B_0, and hence concept drift has occurred. If concept drift is detected, we then retrain the OCSVM on B1B_1 and recompute r1r_1 on B1B_1 using this updated OCSVM model at the next step. On the other hand, if concept drift is not detected in B1B_1, we do not retrain the OCSVM. In either case, we then move on to consider B2B_2, and the process repeats. MMD — We employ MMD to detect concept drift as follows. To identify whether concept drift has occurred at batch BiB_i, we let Bref=Bi−1B_ =B_i-1 and Bnew=BiB_ =B_i. We then perform a two-sample test using MMD with a Gaussian kernel and compute a p-value for hypothesis testing as described in Section 2.3.3. Given a confidence level α, we let mmd=αT_ =α to keep notation homogenous across all drift detectors. If p-value≤mmd $p$-value _ , we conclude that drift has occurred at BiB_i; otherwise, no drift is detected. 3.2.3 Training a model Consider a given pair of malware families which, as above, we denoted as (Fd,Fc)(F_d,F_c), with N batches BiB_i over FdF_d. Let XtrainX and XtestX be two disjoint subsets of FcF_c. Let Ditrain=(Bitrain∪Xtrain)D_i =(B_i ∪ X ) and Ditest=(Bitest∪Xtest)D_i =(B_i ∪ X ). This yields a labeled dataset, where the samples in BiB_i are one class, and the selected samples from FcF_c, namely, Xtrain∪XtestX ∪ X , are the other class. A classification model MiM_i is trained on DitrainD_i and evaluated on DitestD_i . From Section 3.2.1, we recall that |Bitrain|=t|B_i |=t and |Bitest|=b−t|B_i |=b-t. Ideally, we want a balanced dataset with |Xtrain|=t|X |=t and |Xtest|=b−t|X |=b-t. Therefore, in all experiments we let Xtrain=f0c,f1c,…,ft−1cX =\f^c_0,f^c_1,…,f^c_t-1\ and Xtest=ftc,ft+1c,…,fb−1cX =\f^c_t,f^c_t+1,…,f^c_b-1\ As mentioned above, in all of our experiments, we let t=30t=30 and b=50b=50, and hence b−t=20b-t=20. 3.2.4 Static Training For a given drift family FdF_d, control family FcF_c, and learning model L∈ℒL∈ L, we train the model on D0trainD_0 . We denote this trained model as M0M_0. Then the accuracy Astatic(Fd,Fc,L)A_ (F_d,F_c,L) under the static training scenario is computed as Astatic(Fd,Fc,L)=∑i=0N−1M0(Ditest)/∑i=0N−1|Ditest|A_ (F_d,F_c,L)= _i=0^N-1M_0(D_i ) / _i=0^N-1|D_i | (2) where M0(Ditest)M_0(D_i ) is the number of correct predictions made by model M0M_0 over DitestD_i , and the drift family consists of N batches numbered 0 through N−1N-1. This experiment represents the scenario where we train one model on the initial temporal batch, and use this model to classify the test samples in all subsequent batches. 3.2.5 Periodic Retraining For a given drift family FdF_d, control family FcF_c, and learning model L, we train a model MiM_i on DitrainD_i for each batch in the drift family FdF_d. The overall accuracy for this periodic retraining experiment is given by Aperiodic(Fd,Fc,L)=∑i=0N−1Mi(Ditest)/∑i=0N−1|Ditest|A_ (F_d,F_c,L)= _i=0^N-1M_i(D_i ) / _i=0^N-1|D_i | (3) where Mi(Ditest)M_i(D_i ) is the number of correct predictions made by MiM_i over DitestD_i , and the drift family consists of N batches numbered 0 through N−1N-1. The periodic retraining experiment simulates the scenario where we train a new instance of the same model type on every batch of samples. Note that it is not necessary that the hyperparameters of the model remain the same across batches, and hence we recompute the hyperparameters for each model MiM_i. 3.2.6 Drift-Aware Retraining The drift-aware retraining scenario is slightly more involved. For a given drift family FdF_d, control family FcF_c, and learning model L, suppose that we detect concept drift at batches Bi1B_i_1, Bi2B_i_2, …, BiℓB_i_ , where 0<i1<i2<⋯<iℓ<N−10<i_1<i_2<·s<i_ <N-1 with N being the number of batches in the drift family FdF_d. Then defining i0=−1i_0=-1 and iℓ+1=N−1i_ +1=N-1, we have Adrift(Fd,Fc,L,R)=∑j=0ℓ∑k=ij+ij+1Mij+1(Dktest)/∑i=0N−1|Ditest|A_ (F_d,F_c,L,R)= _j=0 \, _k=i_j+1\!\!\!\!^i_j+1\!M_i_j+1(D_k ) / _i=0^N-1|D_i | (4) 3.2.7 Efficiency The models trained in the drift-aware scenario are a subset of the models trained in the periodic retraining scenario. This follows since the batches where concept drift is detected are a subset of all of the batches in FdF_d, and in the periodic retraining scenario, we retrain a model at every batch of FdF_d. Consequently, one measure of efficiency for the drift-aware scenario is the percentage of batches where concept drift is detected—the more drift points detected, the more models that must be trained, resulting in a higher overall cost for training. In this sense, the periodic retraining scenario has the worst possible efficiency, and the static scenario has the best possible efficiency, since at least one model must be trained. For specified families (Fd,Fc)(F_d,F_c) and drift detection technique R, let ℓ be the number of drift points detected. Then we define the drift-aware training efficiency as ℰ(Fd,Fc,R)=1−ℓN−2 if R=mk-means1−ℓN−1 otherwise E(F_d,F_c,R)= \ array[]l1- N-2& if R= \\[6.45831pt] 1- N-1& otherwise array . (5) where N is the number of batches in FdF_d. Since our clustering approach uses pairs of batches, for the MK-Means drift detection technique we have N−2N-2 possible drift points, whereas for the other two drift detection techniques, we have N−1N-1 possible drift points. Note that by the efficiency measures in equation (5), the static scenario will always achieve the maximum efficiency of 1. However, in any case where concept drift occurs, we expect the static scenario to have the lowest accuracy of our three experimental scenarios. Hence, the tradeoff between efficiency and accuracy must be considered. We provide a detailed discussion of this inherent tradeoff in Section 3.4. 3.2.8 Workflow The experiment workflow consists of preprocessing the dataset and then conducting the experiments. The preprocessing step includes removing non-numerical features and discarding samples with obviously incorrect timestamps. Details on the workflow for the experiments is given in Algorithm 1. Algorithm 1 Experiment workflow 1: // Step 0: Preliminaries 2: Input: Select drift and control malware families (Fd,Fc)∈ℱ(F_d,F_c)∈ F 3: Input: Let N=number of batches in FdN= number of batches in F_d 4: Input: Select learning model L∈ℒ=MLP,SVM,RF,XGBL∈ L=\ , , , \ 5: Input: Select drift detector R∈ℛ=ocsvm,mk-means,mmdR∈ R=\ , , \ 6: Input: Specify drift detector threshold RT_R 7: Output: Accuracy for static, periodic, and drift-aware scenarios 8: Output: Efficiency for drift-aware scenario 9: // Step 1: Train models (including hyperparameter tuning) 10: for i=1,2,…,Ni=1,2,…,N do 11: Train model MiM_i (of type L) on batch BiB_i 12: // Step 2: Static scenario 13: Use trained M0M_0 to compute Astatic(Fd,Fc,L)A_ (F_d,F_c,L) // equation (2) 14: // Step 3: Periodic retraining 15: Use all models MiM_i to compute Aperiodic(Fd,Fc,L)A_ (F_d,F_c,L) // equation (3) 16: // Step 4: Drift-aware retraining 17: Detect drift points D using R and threshold RT_R // Section 3.2.2 18: Use D to compute accuracy Adrift(Fd,Fc,L,R)A_ (F_d,F_c,L,R) // equation (4) 19: Compute efficiency ℰ(Fd,Fc,R) E(F_d,F_c,R) // equation (5) 20: // Step 5: Return results 21: Return: Astatic(Fd,Fc,L)A_ (F_d,F_c,L), Aperiodic(Fd,Fc,L)A_ (F_d,F_c,L), Adrift(Fd,Fc,L,R)A_ (F_d,F_c,L,R), 22: Return: and ℰ(Fd,Fc,R) E(F_d,F_c,R) 3.3 Hyperparameters Before presenting our experimental results, we need to consider hyperparameters. In this section, we first discuss hyperparameter tuning of the learning models, which affect the classification accuracy of each model. After discussing hyperparameter tuning for our learning models, we consider the hyperparameters for our drift detection techniques, which affect the tradeoff between accuracy and efficiency under the drift-aware scenario. 3.3.1 Hyperparameter Tuning of Learning Models An important step in our classification experiments is to perform hyperparameter tuning to obtain optimized learning models. We use Optuna [38], which performs a Tree-Structured Parzen Estimator (TPE) [56] search for the best hyperparameters. TPE is a Bayesian optimization algorithm. Given an objective function fθf_θ where θ is a set of parameters, TPE performs multiple trials over θ, iteratively searching for optimal values. This approach is different from a grid search because TPE does not try every value defined in the space—it uses Bayesian estimates to find better regions. We use a total of 100 trials per model. The set of hyperparameters considered for each ML model are given in Table 2. Here, we employ the notation (a,b)(a,b) to indicate that the open interval from a to b was specified for Optuna, while the notation x,y\x,y\ denotes a discrete set of values. Table 2: Hyperparameters for learning models Model Hyperparameters Search space SVM c (regularization) (e−4,e4)(e^-4,e^4) rbf RF _ (10,500)(10,500) _ (5,100)(5,100) XGB _ (10,300)(10,300) _ (5,15)(5,15) _ 0.01,0.05,0.1,0.2,0.3\0.01,0.05,0.1,0.2,0.3\ MLP _ (8,256)(8,256) relu,tanh,logistic\ , , \ adam,sgd\ , \ Given (Fd,Fc,L)(F_d,F_c,L), where L is the selected type of learning model, the objective function of each model MiM_i is defined as fiθ=Miθ(Ditest)|Ditest|f^θ_i= M_i^θ(D_i )|D_i | (6) where MiθM_i^θ is initialized using parameters θ, with the model trained on DitrainD_i , and Miθ(Ditest)M^θ_i(D_i ) denotes the number of correct predictions of MiθM^θ_i on DitestD_i , that is, we are optimizing each model based on the test accuracy. All other parameters needed for a model MiM_i not mentioned in the table are set to their default values in the respective libraries. Since we tune the hyperparameters of one model MiM_i per (Fd,Fc,L)(F_d,F_c,L) combination and per temporal batch, the selected hyperparameters need not be the same. For all models, we use early stopping—if a set of hyperparameters gives an objective function with ideal separation, we stop searching the space. For each drift family, there are four (Fd,Fc)(F_d,F_c) pairs, we have 461 total temporal batches (accounting for all drift families), and there are four machine learning models. Therefore, we have 4⋅4⋅461=7,3764· 4· 461=7,376 combinations. Furthermore, for each of these combinations, we run 100 trials, which gives us a total number of (Fd,Fc,L)(F_d,F_c,L) combinations for which hyperparameters are determined of 100⋅7,376=737,600.100· 7,376=737,600. We note in passing that the standard approach would be to tune the hyperparameters using a separate validation set. However, because of the small number of samples, such a validation set is not available. Tuning on the training set would also not be ideal because the reported metrics are computed on the test set, and overfitting is likely, especially given the limited sample size. Thus, the use of test-set tuning is intentional and can be interpreted as an oracle-style upper bound on performance rather than as a realistic deployment procedure. The goal of our experimental design is to initially establish the two baseline scenarios (static and periodic retraining), and then compare these two to various concept drift detection strategies under the drift-aware scenario. Under all three of these scenarios, the hyperparameter tuning is conducted in a similar fashion, and hence any change in accuracy in the drift-aware scenario should be attributable to the drift detection mechanism. 3.3.2 Hyperparameters for Drift Detectors It is clear from our experimental design that if all batches are detected as drift points, then the drift-aware scenario is identical to the periodic retraining scenario. Furthermore, in equation (5) we define efficiency so that the larger the number of detected drift points, the lower the efficiency in the drift-aware scenario. In practice, we would want to balance accuracy and efficiency. Since the hyperparameters of the drift detectors determine their respective detection processes, the tradeoff between accuracy and efficiency can be analyzed as a function of these hyperparameters. We can model this problem as a multi-objective optimization of conflicting objectives. This means that an increase in one objective produces a decrease in the other objective. Given a drift detector R∈ℛR∈ R, we denote its hyperparameter space as ΩR _R. The drift-aware accuracy AdriftA_ and retraining efficiency ℰ E are the objective functions. Note that hyperparameter tuning of the models MiM_i is orthogonal to the hyperparameter tuning of the drift detector, and hence we do not need to retrain the MiM_i models. Therefore, we can simply rerun Step 4 (lines 16–19) of Algorithm 1 using a grid search over ΩR _R to obtain multiple values for AdriftA_ and ℰ E. Table 3 shows the hyperparameter space that we use in a grid search for our OCSVM, MK-Means, and MMD concept drift detection techniques. We test all learning models and family combinations, and consequently, we run Step 4 of Algorithm 1 a total of 4⋅20⋅(540+240+100)=70,4004· 20·(540+240+100)=70,400 times. Table 3: Hyperparameters for each drift detector Drift detector R ΩR _R HyperparametersNNNN^N^N Values tested Total values MK-Means clusters 2,4,6,8\2,4,6,8\ 4⋅60=2404· 60=240 mk-meansT_ 0.01,0.02,…,0.6\0.01,0.02,…,0.6\ OCSVM ν 0.1,0.2,…,0.9\0.1,0.2,…,0.9\ 9⋅60=5409· 60=540 ocsvmT_ 0.01,0.02,…,0.6\0.01,0.02,…,0.6\ MMD mmd=αT_ =α 0.001,0.002,…,0.1\0.001,0.002,…,0.1\ 100 3.4 Pareto Front According to Algorithm 1, in the drift-aware scenario, the accuracy AdriftA_ depends on FdF_d, FcF_c, L, and R, while the efficiency ℰ E depends only on FdF_d, FcF_c, and R. However, this is not entirely correct, as both accuracy and efficiency depend on the selected hyperparameters ω∈ΩRω∈ _R where, as above, ΩR _R is a set of hyperparameters corresponding to drift detector R. Furthermore, there is an inherent tradeoff between accuracy and efficiency. One technique for selecting the hyperparameters ω is to find Pareto efficient solutions, also known as a Pareto Front [7]. Pareto Fronts are widely used in engineering and economics where tradeoffs have to be made between two conflicting objectives [13], such as the tradeoff between fuel efficiency and engine power. By using a Pareto Front, it is possible to find a set of points where optimal tradeoffs can be made—in a sense that is made clear below. The Pareto Front is also referred to as the set of non-dominated solutions. Formally, we consider a compact parameter space ℝnR^n and an objective space ℝmR^m with a mapping function f such that f:X→ℝmf:X ^m where X⊆ℝnX ^n. Then, the objective values can be represented as a set of vectors Y=y∈ℝm∣y=f(x),x∈X.Y=\\,y ^m y=f(x),\ x∈ X\. Let ≻ be a preference operator that dictates whether we want to maximize or minimize the objective. A vector y′∈ℝmy ^m is said to strictly dominate another vector y′∈ℝmy ^m if y′≻y′.y y . Then the Pareto Front can be expressed as P(Y)=y′∈Y∣y′∈Y∣y′≻y′,y′≠y′=∅.P(Y)=\\,y ∈ Y \\,y ∈ Y y y ,\ y ≠ y \,\= \. Intuitively, the points belonging to the Pareto Front create a boundary that surrounds all other points of the objective. Points forming the boundary have the best possible tradeoff, in the sense that moving inside the boundary produces a negative effect on at least one of the objectives. For a drift detector R and its parameter space ΩR _R, we perform a Pareto Front analysis per (Fd,Fc,L)(F_d,F_c,L) combination, that is, for each combination of drift family, control family, and learning model. Suppressing the dependence on (Fd,Fc,L)(F_d,F_c,L), as the objectives, we use (Δdrift,ℰ)( _ , E), where Δdrift=Adrift−Astatic. _ =A_ -A_ . Note that we use Δdrift _ instead of AdriftA_ in the objective because it better conveys the desired optimization, which is to maximize the difference from the static baseline. Of course, given Δdrift _ and the static baseline AstaticA_ , we can trivially determine AdriftA_ . We use the Paretoset Python package, which implements the Skyline Query operator [29, 52] to determine the Pareto Front for a set of points in the objective space. We note in passing that finding the Pareto Front without a set of points in the objective space can be achieved through evolutionary methods [13, 58] which fall under multi-objective optimization algorithms. We do not require a multi-objective optimization algorithm since a grid search generates the set of points from which we can directly determine the Pareto Front. 3.5 Consolidating Drift-Aware Experiments We now discuss how to use Pareto Front analysis to obtain meaningful results for comparing our drift detection techniques to the static and periodic scenarios. Again, this additional analysis is required since it is necessary to tune various hyperparameters of each drift detection technique, and when doing so, there is an inherent tradeoff between accuracy and efficiency. Recall that L∈ℒL∈ L is the learning model, R∈ℛR∈ R is the drift detection technique, ΩR _R set of all hyperparameters tested for R, while ω represents a specific selection of hyperparameters from ΩR _R. Also, FdF_d is the drift family, and FcF_c is the control family, with (Fd,Fc)∈ℱ(F_d,F_c)∈ F. We require the additional notation ΩR∗ _R =set of hyperparameters tested corresponding to Pareto Front = set of hyperparameters tested corresponding to Pareto Front ω∗ ω =specific selection of hyperparameters from ΩR∗ = specific selection of hyperparameters from~$ _R$ Since we have five malware families, |ℱ|=20| F|=20, and the values of |ΩR|| _R| are given in Table 3. 3.5.1 Mean Static and Periodic Accuracies In the static scenario, the accuracy depends only on FdF_d, FcF_c, and L, and hence we denote each of these accuracy values as Astatic(Fd,Fc,L)A_ (F_d,F_c,L). Then the average accuracy per learning model L is given by static(L)=1|ℱ|∑(Fd,Fc)∈ℱAstatic(Fd,Fc,L) A_ (L)= 1| F| _(F_d,F_c)∈ FA_ (F_d,F_c,L) Similarly, in the periodic scenario, the average accuracy per learning model is periodic(L)=1|ℱ|∑(Fd,Fc)∈ℱAperiodic(Fd,Fc,L) A_ (L)= 1| F| _(F_d,F_c)∈ FA_ (F_d,F_c,L) 3.5.2 Mean Drift-Aware Accuracy The drift-aware scenario is more complex. As in the static and periodic scenarios, the accuracy depends on FdF_d, FcF_c, and L. However, in the drift-aware scenario, the accuracy also depends on the drift detection technique R and, more specifically, the selected hyperparameters ω∈ΩRω∈ _R. Hence, in this scenario, we denote the accuracy as Adrift(Fd,Fc,L,R,ω)A_ (F_d,F_c,L,R,ω). For a given ω∈ΩRω∈ _R, we can average over all (Fd,Fc)∈ℱ(F_d,F_c)∈ F to obtain Adriftℱ(L,R,ω)=1|ℱ|∑(Fd,Fc)∈ℱAdrift(Fd,Fc,L,R,ω)A_ F(L,R,ω)= 1| F| _(F_d,F_c)∈ FA_ (F_d,F_c,L,R,ω) (7) On the other hand, for a given (Fd,Fc)∈ℱ(F_d,F_c)∈ F, we can average over all ω∈ΩRω∈ _R to obtain AdriftΩR(Fd,Fc,L,R)=1|ΩR|∑ω∈ΩRAdrift(Fd,Fc,L,R,ω)A_ _R(F_d,F_c,L,R)= 1| _R| _ω∈ _RA_ (F_d,F_c,L,R,ω) (8) From equation (7), the mean accuracy per (L,R)(L,R) pair can be computed as driftx(L,R)=1|ΩR|∑ω∈ΩRAdriftℱ(L,R,ω) A^x_ (L,R)= 1| _R| _ω∈ _RA_ F(L,R,ω) (9) Alternatively, based on equation (8), we can compute the mean accuracy per (L,R)(L,R) pair as drifty(L,R)=1|ℱ|∑(Fd,Fc)∈ℱAdriftΩR(Fd,Fc,L,R) A^y_ (L,R)= 1| F| _(F_d,F_c)∈ FA_ _R(F_d,F_c,L,R) (10) It is clear that driftx(L,R)=drifty(L,R) A^x_ (L,R)= A^y_ (L,R) and we denote this common value as drift(L,R) A_ (L,R). The calculations of drift(L,R) A_ (L,R), using either equation (9) or (10), is illustrated in Table 4. Table 4: Compute drift A_ as average of final row or final column (N=|ℱ|N=| F|) A(Fd0,Fc0,ω0)A(Fd0,Fc0,ω1)A(Fd0,Fc0,ω2)⋯A(Fd0,Fc0,ω|Ω|−1)AdriftΩ(Fd0,Fc0)A(Fd1,Fc1,ω0)A(Fd1,Fc1,ω1)A(Fd1,Fc1,ω2)⋯A(Fd1,Fc1,ω|Ω|−1)AdriftΩ(Fd1,Fc1)A(Fd2,Fc2,ω0)A(Fd2,Fc2,ω1)A(Fd2,Fc2,ω2)⋯A(Fd2,Fc2,ω|Ω|−1)AdriftΩ(Fd2,Fc2)⋮A(FdN−1,FcN−1,ω0)A(FdN−1,FcN−1,ω1)A(FdN−1,FcN−1,ω2)⋯A(FdN−1,FcN−1,ω|Ω|−1)AdriftΩ(FdN−1,FcN−1)Adriftℱ(ω0)Adriftℱ(ω1)Adriftℱ(ω2)⋯Adriftℱ(ω|Ω|−1)drift array[]c|c A(F_d_0,F_c_0, _0)&A(F_d_0,F_c_0, _1)&A(F_d_0,F_c_0, _2)&·s&A(F_d_0,F_c_0, _| |-1)&A_ (F_d_0,F_c_0)\\ A(F_d_1,F_c_1, _0)&A(F_d_1,F_c_1, _1)&A(F_d_1,F_c_1, _2)&·s&A(F_d_1,F_c_1, _| |-1)&A_ (F_d_1,F_c_1)\\ A(F_d_2,F_c_2, _0)&A(F_d_2,F_c_2, _1)&A(F_d_2,F_c_2, _2)&·s&A(F_d_2,F_c_2, _| |-1)&A_ (F_d_2,F_c_2)\\ & & & & & \\ A(F_d_N-1,F_c_N-1, _0)&A(F_d_N-1,F_c_N-1, _1)&A(F_d_N-1,F_c_N-1, _2)&·s&A(F_d_N-1,F_c_N-1, _| |-1)&A_ (F_d_N-1,F_c_N-1)\\ A_ F( _0)&A_ F( _1)&A_ F( _2)&·s&A_ F( _| |-1)& A_ \\ array 3.5.3 Mean Drift-Aware Accuracy over Pareto Front Computing of drift(L,R) A_ (L,R) using either equation (9) or (10) treats all hyperpameter combinations ω∈ΩRω∈ _R the same. This is clearly not ideal, as many hyperparameter values will yield suboptimal results. To mitigate this issue, we now consider an approach analogous to that in Section 3.5.2, but restricted to points on the relevant Pareto Front. For each ω∈ΩRω∈ _R, we first determine Adriftℱ(L,R,ω)A_ F(L,R,ω) using equation (7). Then we compute driftx∗(L,R)=1|ΩR∗|∑ω∗∈ΩR∗Adriftℱ(L,R,ω∗) A^x -1.0pt _ (L,R)= 1| _R| _ω ∈ _RA_ F(L,R,ω ) (11) where ΩR∗ _R is the set of hyperparameter values on the Pareto Front that correspond to Adriftℱ(L,R,ω)A_ F(L,R,ω). Note that this computation of driftx∗(L,R) A^x -1.0pt _ (L,R) is analogous to that of driftx(L,R) A^x_ (L,R) in equation (9), but restricted to the Pareto Front of Adriftℱ(L,R,ω)A_ F(L,R,ω). The calculation of driftx∗(L,R) A^x -1.0pt _ (L,R) is illustrated in Table 5. Table 5: Compute driftx∗ A^x -1.0pt _ as average of Adriftℱ(ωi∗)A_ F(ω _i) A(Fd0,Fc0,ω0)A(Fd0,Fc0,ω1)A(Fd0,Fc0,ω2)⋯A(Fd0,Fc0,ω|Ω|−1)A(Fd1,Fc1,ω0)A(Fd1,Fc1,ω1)A(Fd1,Fc1,ω2)⋯A(Fd1,Fc1,ω|Ω|−1)A(Fd2,Fc2,ω0)A(Fd2,Fc2,ω1)A(Fd2,Fc2,ω2)⋯A(Fd2,Fc2,ω|Ω|−1)⋮A(FdN−1,FcN−1,ω0)A(FdN−1,FcN−1,ω1)A(FdN−1,FcN−1,ω2)⋯A(FdN−1,FcN−1,ω|Ω|−1)Adriftℱ(ω0)Adriftℱ(ω1)Adriftℱ(ω2)⋯Adriftℱ(ω|Ω|−1)⏟driftx∗Determine Pareto Front Ω∗ from Adriftℱ(ωi)i=0|Ω|−1 array[]c|c A(F_d_0,F_c_0, _0)&A(F_d_0,F_c_0, _1)&A(F_d_0,F_c_0, _2)&·s&A(F_d_0,F_c_0, _| |-1)\\ A(F_d_1,F_c_1, _0)&A(F_d_1,F_c_1, _1)&A(F_d_1,F_c_1, _2)&·s&A(F_d_1,F_c_1, _| |-1)\\ A(F_d_2,F_c_2, _0)&A(F_d_2,F_c_2, _1)&A(F_d_2,F_c_2, _2)&·s&A(F_d_2,F_c_2, _| |-1)\\ & & & & \\ A(F_d_N-1,F_c_N-1, _0)&A(F_d_N-1,F_c_N-1, _1)&A(F_d_N-1,F_c_N-1, _2)&·s&A(F_d_N-1,F_c_N-1, _| |-1)\\ A_ F( _0)&A_ F( _1)&A_ F( _2)&·s&A_ F( _| |-1)\\ @intercol 6.0pt[0.0pt][0.0pt]$ 358.78723pt$ @intercol @intercol& A^x -1.0pt _ \\ @intercol Determine Pareto Front from \A_ F( _i)\_i=0^| |-1 @intercol @intercol\\ array Alternatively, we can first sum over the Pareto Front points for each combination of (Fd,Fc,L,R)(F_d,F_c,L,R). Analogous to equation (8), in this case we compute AdriftΩR∗(Fd,Fc,L,R)=1|ΩR∗|∑ω∗∈ΩR∗Adrift(Fd,Fc,L,R,ω∗)A_ _R(F_d,F_c,L,R)= 1| _R| _ω ∈ _RA_ (F_d,F_c,L,R,ω ) where ΩR∗ _R is the set of hyperameter values that correspond to Pareto Front points of (Fd,Fc,L,R)(F_d,F_c,L,R). Then averaging AdriftΩR∗(Fd,Fc,L,R)A_ _R(F_d,F_c,L,R) over all (Fd,Fc)∈ℱ(F_d,F_c)∈ F, we obtain drifty∗(L,R)=1|ℱ|∑(Fd,Fc)∈ℱAdriftΩR∗(Fd,Fc,L,R) A^y -1.0pt _ (L,R)= 1| F| _(F_d,F_c)∈ FA_ _R(F_d,F_c,L,R) (12) Note that this computation of drifty∗(L,R) A^y -1.0pt _ (L,R) is analogous to that in equation (10), but restricted to the relevant Pareto Front values. The calculation of drifty∗(L,R) A^y -1.0pt _ (L,R) is illustrated in Table 6. Table 6: Compute drifty∗ A^y -1.0pt _ as average of final column (N=|ℱ|N=| F|) A(Fd0,Fc0,ω0∗)A(Fd0,Fc0,ω1∗)A(Fd0,Fc0,ω2∗)⋯A(Fd0,Fc0,ω|Ω∗|−1∗)AdriftΩ∗(Fd0,Fc0)A(Fd1,Fc1,ω0∗)A(Fd1,Fc1,ω1∗)A(Fd1,Fc1,ω2∗)⋯A(Fd1,Fc1,ω|Ω∗|−1∗)AdriftΩ∗(Fd1,Fc1)A(Fd2,Fc2,ω0∗)A(Fd2,Fc2,ω1∗)A(Fd2,Fc2,ω2∗)⋯A(Fd2,Fc2,ω|Ω∗|−1∗)AdriftΩ∗(Fd2,Fc2)⋮A(FdN−1,FcN−1,ω0∗)A(FdN−1,FcN−1,ω1∗)A(FdN−1,FcN−1,ω2∗)⋯A(FdN−1,FcN−1,ω|Ω∗|−1∗)AdriftΩ∗(FdN−1,FcN−1)drifty∗ array[]c|c A(F_d_0,F_c_0,ω _0)&A(F_d_0,F_c_0,ω _1)&A(F_d_0,F_c_0,ω _2)&·s&A(F_d_0,F_c_0,ω _| |-1)&A_ (F_d_0,F_c_0)\\ A(F_d_1,F_c_1,ω _0)&A(F_d_1,F_c_1,ω _1)&A(F_d_1,F_c_1,ω _2)&·s&A(F_d_1,F_c_1,ω _| |-1)&A_ (F_d_1,F_c_1)\\ A(F_d_2,F_c_2,ω _0)&A(F_d_2,F_c_2,ω _1)&A(F_d_2,F_c_2,ω _2)&·s&A(F_d_2,F_c_2,ω _| |-1)&A_ (F_d_2,F_c_2)\\ & & & & & \\ A(F_d_N-1,F_c_N-1,ω _0)&A(F_d_N-1,F_c_N-1,ω _1)&A(F_d_N-1,F_c_N-1,ω _2)&·s&A(F_d_N-1,F_c_N-1,ω _| |-1)&A_ (F_d_N-1,F_c_N-1)\\ &&&&& A^y -1.0pt _ \\ array When considering all of the hyperparameter combinations ω∈ΩRω∈ _R, equations (9) and (10) show that the order of summation can be interchanged. However, when restricting our attention to hyperparameter values of R corresponding to Pareto Front points, this is no longer the case—in general, there is no reason to expect that driftx∗(L,R) A^x -1.0pt _ (L,R) will yield the same result as drifty∗(L,R) A^y -1.0pt _ (L,R). 3.5.4 Representative Accuracy over Pareto Front In practice, we would need to select a specific value for the hyperparameters of the drift detection technique R, and we would almost certainly want to select such a value from the Pareto Front. In this section we define such a representative value that is, in a sense, near the “middle” and, therefore, we refer to it as the median. For a give pair of malware families (Fd,Fc)(F_d,F_c), learning model L, and drift detection technique R, let (xmin,ymin)(x_ ,y_ ) be the point on the Pareto Front that gives the minimum ΔAccuracy Accuracy and let (xmax,ymax)(x_ ,y_ ) be the point on the Pareto Front where the maximum ΔAccuracy Accuracy is attained. Then let xa=xmin+xmax2 and ya=ymin+ymax2x_a= x_ +x_ 2 \ \ and\ \ y_a= y_ +y_ 2 The equation of the line from the origin through (xa,ya)(x_a,y_a) is given by y=yaxaxy= y_ax_a\,x (13) Let ωm∗∈ΩR∗ω _m∈ _R to be the hyperparameter values corresponding to the point on the Pareto Front that is closest to the line in equation (13). Then we define the median accuracy for a given (L,R)(L,R) pair as driftm∗(L,R)=1|ℱ|∑(Fd,Fc)∈ℱAdrift(Fd,Fc,L,R,ωm∗) A^m -1.0pt _ (L,R)= 1| F| _(F_d,F_c)∈ FA_ (F_d,F_c,L,R,ω _m) For each (L,R)(L,R) pair, we now have four distinct measures of the accuracy, namely, drift(L,R),driftx∗(L,R),drifty∗(L,R), and driftm∗(L,R) A_ (L,R),\ A^x -1.0pt _ (L,R),\ A^y -1.0pt _ (L,R), \ and\ A^m -1.0pt _ (L,R) (14) Of course, for each of the accuracy measures in equation (14) there is a corresponding efficiency that we denote as ℰdrift(L,R),ℰdriftx∗(L,R),ℰdrifty∗(L,R), and ℰdriftm∗(L,R) E_ (L,R),\ E^x -1.0pt _ (L,R),\ E^y -1.0pt _ (L,R), \ and\ E^m -1.0pt _ (L,R) (15) respectively. 3.6 Software and Development Environment Table 7 summarizes the libraries and software used in this research. Note that we primarily use Python for the experiments. In addition, we use Docker to run a PostgreSQL container and a volume to store the hyperparameter tuning trial information generated by Optuna [38]. The best models MiM_i are stored on disk as .pkl files. We export hyperparameter tuning information and experiment results as .csv files for later analysis using Matplotlib. All experiments were run on a CPU machine; no GPUs were employed. To allow for reproducibility, any machine learning model, function, or library used in this research that requires a random seed as input has been set to a fixed value of 420. Table 7: Software summary Software Purpose Python3.10 Main programming language used Poetry Python dependency management and packaging tool Polars Loading and preprocess KronoDroid dataset Type inference and data manipulation Matplotlib Build and visualize graphs and plots NumPy Numerical computations and array manipulations Alibi-Detect Implementation for MMD Pytorch Backend needed to run MMD Paretoset Find the Pareto Front Optuna Automatic hyperparameter optimization framework scikit-learn OCSVM and MK-Means implementations Implementations of MLP, SVM, and RF classifiers XGBoost XGB implementation Docker Postgres container for storing hyperparameters Psycopg2 PostgreSQL database adapter for Python 4 Experimental Results and Analysis In this section, we present the results for the concept drift detection experiments discussed above. The primary goal of these experiments is to determine the accuracy and efficiency of each drift detector (MK-Means, OCSVM, and MMD), as compared to the static and periodic retraining scenarios. 4.1 Static and Periodic Results Recall that our static and periodic experiments are independent of any drift detection technique. In subsequent sections, we consider accuracy and efficiency in the drift-aware scenario based on each of the three drift detectors under consideration. Figures 14 through 17 in the Appendix provides bar graphs of the static accuracies AstaticA_ and periodic accuracies AperiodicA_ . These results are provided for all four learning models L and all 20 family combinations for (Fd,Fc)(F_d,F_c). From Figures 14 through 17 we observe that MLP, RF, and MLP all perform well, with XGB lagging. We also generally observe qualitative consistency as, for example, across all models the periodic scenario yields a large improvement in accuracy for (Malap, Agent) and (SMSreg, Malap), while (Airpush, Boxer) and (Boxer, SMSreg) only improve marginally. Furthermore, SVM achieves the highest accuracy in the static scenario for 40% of the family combinations (8 out of 20), followed by MLP and RF at 25% each, with XGB performing best in the remaining 10% of the combinations. However, in the periodic retraining scenario, MLP performs best for 65% of the family combinations, followed by SVM and RF at 25% and 10%, respectively, with XGB being best for only 1 such combination. In Figure 1 we provide the average AstaticA_ and AperiodicA_ for each learning model across all family combinations. This graph clearly demonstrates the value of accounting for concept drift, as the best average accuracy in the static scenario is 0.8416 (SVM model), while the worst average accuracy in the periodic retraining scenario is 0.9334 (XGB model). Furthermore, using the best model in the periodic scenario (MLP), we obtain an average accuracy of 0.9666. MLPRFSVMXGB0.000.000.200.200.400.400.600.600.800.801.001.000.82480.82480.81520.81520.84160.84160.75780.75780.96660.96660.95290.95290.95000.95000.93340.9334Average accuracyStaticPeriodic Figure 1: Average static and periodic accuracy per learning model Another aspect to consider is the effect of hyperparameter tunning of each learning model instance. If hyperparameter tunning was generally successful for learning model L, we expect the distribution of its periodic accuracies to be more compact and have a higher median, as compared to the distribution of the static accuracies. This is indeed the case, as illustrated in Figure 2, which validates our use of TPE for hyperparameter tuning. Figure 2: Box plot of static and periodic accuracy Overall, these static and periodic results confirm the success of our experimental design and hyperparameter tuning. It is also clear that retraining models is necessary to counter concept drift. However, given the computational expense of training learning models, we would like to achieve comparable results to the periodic scenario with less work. To this end, we now consider drift-aware retraining for each of the three drift detectors introduced in Section 2.3. Our primary goal is to determine whether we can achieve results comparable to periodic retraining, but at a significantly lower cost, in terms of the number of models trained. 4.2 MK-Means Drift-Aware Results Recall that MK-Means has two hyperparameters, (mk-means,clusters)(T_ , ). Figure 3 contains a plot of the Pareto Front for the specific case (Fd,Fc,L,R)=(Agent,Airpush,MLP,mk-means)(F_d,F_c,L,R)=( Agent, Airpush, , ) As previously noted, from the Pareto Front coordinates, it is trivial to determine the corresponding hyperparameters (mk-means,clusters)(T_ , ). −0.050-0.050−0.025-0.0250.0000.0000.0250.0250.0500.0500.0750.0750.1000.1000.1250.1250.00.00.20.20.40.40.60.60.80.81.01.0ΔAccuracy AccuracyEfficiency(Fd,Fc,L,R)=(Agent,Airpush,MLP,MK-Means)(F_d,F_c,L,R)=( Agent, Airpush, , MK-Means)Non-Pareto FrontPareto Front Figure 3: MK-Means Pareto Front example From Figure 3 we can see the advantage of choosing hyperparameter values based on the Pareto Front. For example, if we require an efficiency in the range of, say, 60% to 65%, choosing hyperparameter values from the Pareto Front could provide an improvement in accuracy of more than 3%, as compared to values that are not on the Pareto Front. In Figure 4, we provide a bar graph comparing the average accuracies for each learning model under the static, drift-aware, and periodic scenarios, where the drift detection is based on MK-Means. As discussed in Section 3.5, for the drift-aware scenario, we have four distinct methods for computing the average accuracy, which we denote as drift A_ , driftx∗ A^x -1.0pt _ , drifty∗ A^y -1.0pt _ , and driftm∗ A^m -1.0pt _ . Recall that drift A_ is the average accuracy over all (mk-means,clusters)(T_ , ) pairs tested, while the latter three averages are based on Pareto Front analysis. MLPRFSVMXGB0.000.000.200.200.400.400.600.600.800.801.001.000.82480.82480.81520.81520.84160.84160.75780.75780.88620.88620.86860.86860.88120.88120.81850.81850.91350.91350.89410.89410.90520.90520.85290.85290.93770.93770.92350.92350.92660.92660.88710.88710.94800.94800.93580.93580.93780.93780.90230.90230.96660.96660.95290.95290.95000.95000.93340.9334AccuracyStaticdrift A_ driftx∗ A^x -1.0pt _ drifty∗ A^y -1.0pt _ driftm∗ A^m -1.0pt _ Periodic Figure 4: MK-Means accuracies From Figure 4, we observe that for all learning models tested, using MK-Means for concept drift detection produces a gain in accuracy, as compared to the static scenario. Also, by most of the measures considered, drift-aware retraining based on MK-Means results in only a relatively small loss in accuracy as compared to the more costly periodic scenario. For example, considering the median driftm∗ A^m -1.0pt _ for the MLP model, we see that the drift-aware scenario improves on the static scenario by nearly 15%, since (0.9480−0.8248)/0.8248=0.1494(0.9480-0.8248)/0.8248=0.1494, while being within 2% of the accuracy achieved in the more costly periodic scenario, since (0.9666−0.9480)/0.9666=0.0192(0.9666-0.9480)/0.9666=0.0192. Figure 5 shows the analogous efficiency results. The improvement in efficiency for the drift-aware scenario averaged over all hyperparameters (mk-means,clusters)(T_ , ) tested, as indicated by ℰdrift E_ , is over 80%, although the improvement when restricted to the better choices of hyperparameters, as represented by the Pareto Front-based values of ℰdriftx∗ E^x -1.0pt _ , ℰdrifty∗ E^y -1.0pt _ , and ℰdriftm∗ E^m -1.0pt _ is not quite as impressive. Nevertheless, for the median ℰdriftm∗ E^m -1.0pt _ we achieve a savings—in terms of the number of models that must be trained—of more than 58%, as compared to the periodic retraining scenario. MLPRFSVMXGB0.000.000.200.200.400.400.600.600.800.801.001.000.82100.82100.82100.82100.82100.82100.82100.82100.70930.70930.74990.74990.69890.69890.70460.70460.59490.59490.70450.70450.67070.67070.66790.66790.58670.58670.74850.74850.68990.68990.69640.6964Efficiencyℰdrift E_ ℰdriftx∗ E^x -1.0pt _ ℰdrifty∗ E^y -1.0pt _ ℰdriftm∗ E^m -1.0pt _ Figure 5: MK-Means efficiencies 4.3 OCSVM Drift-Aware Results Our analysis of concept drift detection based on OCSVM is similar to that for MK-Means, above. In Figure 6, we give the Pareto Front for the same specific case that we considered for MK-Means, namely, (Fd,Fc,L,R)=(Agent,Airpush,MLP,ocsvm)(F_d,F_c,L,R)=( Agent, Airpush, , ) As in the MK-Means case, we again see the value of Pareto Front analysis for selecting hyperparameters to meet a desired balance between accuracy and efficiency. −0.050-0.050−0.025-0.0250.0000.0000.0250.0250.0500.0500.0750.0750.1000.1000.1250.1250.00.00.20.20.40.40.60.60.80.81.01.0ΔAccuracy AccuracyEfficiency(Fd,Fc,L,R)=(Agent,Airpush,MLP,OCSVM)(F_d,F_c,L,R)=( Agent, Airpush, , OCSVM)Non-Pareto FrontPareto Front Figure 6: OCSVM Pareto Front example In Figures 7 and 8, respecitvely, we provide bar graphs of our various accuracy and efficiency metrics. We note that, similar to MK-Means, OCSVM achieves a substantial accuracy gain over the static baseline at a significant lower retraining cost than the periodic retraining scenario. For example, if we consider the median driftm∗ A^m -1.0pt _ for the MLP model, the drift-aware scenario improves on the static scenario by 15% ((0.9486−0.8248)/0.8248=0.1501(0.9486-0.8248)/0.8248=0.1501), while being within 1.9% of the accuracy achieved in the more costly periodic scenario ((0.9666−0.9486)/0.9666=0.0186(0.9666-0.9486)/0.9666=0.0186), a marginal improvement over MK-Means. MLPRFSVMXGB0.000.000.200.200.400.400.600.600.800.801.001.000.82480.82480.81520.81520.84160.84160.75780.75780.89540.89540.88010.88010.88160.88160.83350.83350.93250.93250.91870.91870.91760.91760.87850.87850.93910.93910.93090.93090.92730.92730.88880.88880.94860.94860.94340.94340.93680.93680.90470.90470.96660.96660.95290.95290.95000.95000.93340.9334AccuracyStaticdrift A_ driftx∗ A^x -1.0pt _ drifty∗ A^y -1.0pt _ driftm∗ A^m -1.0pt _ Periodic Figure 7: OCSMV accuracies MLPRFSVMXGB0.000.000.200.200.400.400.600.600.800.801.001.000.77850.77850.77850.77850.77850.77850.77850.77850.58610.58610.58260.58260.60850.60850.58950.58950.64630.64630.68200.68200.68730.68730.72560.72560.65860.65860.69510.69510.72830.72830.70950.7095Efficiencyℰdrift E_ ℰdriftx∗ E^x -1.0pt _ ℰdrifty∗ E^y -1.0pt _ ℰdriftm∗ E^m -1.0pt _ Figure 8: OCSVM efficiencies With respect to efficiency, the improvement for the OCSVM-based drift-aware scenario averaged over all hyperparameters tested is more than 77%. The improvement in efficiency when restricted to the median Pareto Front-based value ℰdriftm∗ E^m -1.0pt _ and MLP model is more than 65%. Recall that the corresponding median value MK-Means is about 58%. Thus, by this measure, OCSVM offers improved efficiency, as compared to MK-Means. 4.4 MMD Drift-Aware Results Our MMD-based concept drift detection technique differs from MK-Means and OCSVM in that for MMD we only consider one hyperparameter. This hyperparameter, which we denote as mmdT_ , is a confidence value for hypothesis testing. In Figure 9, we give the Pareto Front for the same specific case that we considered for both MK-Means and OCSVM, that is, (Fd,Fc,L,R)=(Agent,Airpush,MLP,mmd)(F_d,F_c,L,R)=( Agent, Airpush, , ) Compared to MK-Means and OCSVM, we observe that for this MMD example, there is more consistency in the sense that points tend to be very close to the Pareto Front, modulo a small number of outliers. 0.000.000.010.010.020.020.030.030.750.750.800.800.850.85ΔAccuracy AccuracyEfficiency(Fdrift,Fcontrol,L,R)=(Agent,Airpush,MLP,MMD)(F_ ,F_ ,L,R)=( Agent, Airpush, , MMD)Non-Pareto FrontPareto Front Figure 9: MMD Pareto Front example In Figure 10 we provide bar graphs of the various accuracy metrics and in Figure 11 we give the corresponding efficiency results. As with both MK-Means and OCSVM, MMD achieves a substantial gain over the static baseline at a significant lower retraining cost, as compared with the periodic retraining scenario. For example, if we consider the median driftm∗ A^m -1.0pt _ for the MLP model, the drift-aware scenario improves on the static scenario by 13.72%, since (0.9380−0.8248)/0.8248=0.1372(0.9380-0.8248)/0.8248=0.1372, while being within 3% of the accuracy achieved in the much more costly periodic retraining scenario, since (0.9666−0.9380)/0.9666=0.0296(0.9666-0.9380)/0.9666=0.0296. These numbers are slightly worse than the results we obtained using MK-Means and OCSVM. MLPRFSVMXGB0.000.000.200.200.400.400.600.600.800.801.001.000.82480.82480.81520.81520.84160.84160.75780.75780.93480.93480.92130.92130.91540.91540.88010.88010.93350.93350.91990.91990.91480.91480.87960.87960.93640.93640.92300.92300.91780.91780.88310.88310.93800.93800.92400.92400.91660.91660.88430.88430.96660.96660.95290.95290.95000.95000.93340.9334AccuracyStaticdrift A_ driftx∗ A^x -1.0pt _ drifty∗ A^y -1.0pt _ driftm∗ A^m -1.0pt _ Periodic Figure 10: MMD accuracies MLPRFSVMXGB0.000.000.200.200.400.400.600.600.800.801.001.000.64340.64340.64370.64370.64340.64340.64370.64370.67380.67380.67140.67140.66840.66840.66170.66170.69480.69480.69410.69410.69010.69010.69960.69960.69120.69120.69900.69900.70170.70170.69850.6985Efficiencyℰdrift E_ ℰdriftx∗ E^x -1.0pt _ ℰdrifty∗ E^y -1.0pt _ ℰdriftm∗ E^m -1.0pt _ Figure 11: MMD efficiencies With respect to efficiency, the improvement over the periodic scenario, averaged over all hyperparameters tested, is more than 64%. On the other hand, the efficiency when restricted to Pareto Front is slightly better—for example, the value ℰdriftm∗ E^m -1.0pt _ for the MLP model is more than 69%. Recall that the corresponding median value MK-Means is about 58%, and for OCSVM it is about 65%. For MK-Means and OCSVM, the average efficiency declined significantly when we restricted the hyperparameters to the Pareto Front. This indicates that for MK-Means and OCSVM, many of the hyperparameter choices fail to adequately detect concept drift, while for the better hyperparameter choices (i.e., those on the Pareto Front), we obtain strong results. This is not the case for MMD, where hyperparameter choices yield more consistent efficiency results, regardless of whether we restrict to the Pareto Front. The violin graphs in Figures 18 and 19 serve to further emphasize these points. These results strongly suggest that we could reduce the hyperparameter search space for MK-Means and OCSVM without affecting the accuracy or training efficiency results. 4.5 Comparison of Drift Detection Techniques In this section we provide a comparative analysis of the three drift detection techniques discussed above, with the emphasis on aggregate results. We pay particular attention to driftm∗ A^m -1.0pt _ and ℰdriftm∗ E^m -1.0pt _ , since since these values correspond to typical selections of drift-detector hyperparameters In Figure 12(a), we give accuracy results, averaged over all (Fd,Fc,L)(F_d,F_c,L), that is, all selections of drift family, control family, and learning model. Figure 12(b) gives the corresponding efficiency averages. drift(L,R) A_ (L,R)driftx∗(L,R) A^x -1.0pt _ (L,R)drifty∗(L,R) A^y -1.0pt _ (L,R)driftm∗(L,R) A^m -1.0pt _ (L,R)0.500.500.600.600.700.700.800.800.900.901.001.000.86360.86360.89140.89140.91870.91870.93100.93100.87260.87260.91180.91180.92150.92150.93340.93340.91290.91290.91200.91200.91510.91510.91570.9157Accuracymk-means ocsvm mmd ℰdrift(L,R) E_ (L,R)ℰdriftx∗(L,R) E^x -1.0pt _ (L,R)ℰdrifty∗(L,R) E^y -1.0pt _ (L,R)ℰdriftm∗(L,R) E^m -1.0pt _ (L,R)0.500.500.600.600.700.700.800.800.900.900.82100.82100.71570.71570.65950.65950.68040.68040.77850.77850.59170.59170.68530.68530.69790.69790.64350.64350.66880.66880.69460.69460.69760.6976Efficiencymk-means ocsvm mmd (a) Accuracy (b) Efficiency Figure 12: Drift detector comparison averaged over all (Fd,Fc,L)(F_d,Fc,L) From Figure 12 we see that the best driftm∗ A^m -1.0pt _ is obtained with the OCSVM drift-detector, while MK-Means performs only marginally worse. With respect to ℰdriftm∗ E^m -1.0pt _ , OCSVM and MMD are virtually identical. Recall that we define efficiency in terms of the number of times that concept drift is detected, which necessitates the retraining of the learning models. Since the same learning models are trained for all three drift detectors, the time required to train any individual learning model has no bearing on this measure of efficiency. However, the time required to train each of our three drift detection techniques varies significantly, and this is relevant when selecting between these techniques. A comparison of the average time per (Fd,Fc)(F_d,F_c) pair that is required for each of the three drift-detection techniques is given in the form of a bar graph in Figure 13. By this measure, OCSVM requires the least time, by a wide margin, with MK-Means requiring the most time, also by a wide margin. The primary reason for these large discrepancies is that the OCSVM model is only retrained when concept drift is detected, whereas MK-Means and MMD need to be updated for each block, regardless of whether drift is detected. It is also worth noting that OCSVM has the largest number of hyperparameters to test and, as discussed in Section 4.4, this number could likely be significantly reduced, which would further increase its timing advantage. MK-MeansOCSVMMMD00224466881.071.070.070.070.620.627.717.710.460.461.851.853.173.170.150.151.061.061.341.340.070.070.390.39SecondsMinMaxMeanStandard deviation Figure 13: Time required to detect drift points per (Fd,Fc)(F_d,F_c) pair According to a majority of the metrics considered, our OCSVM concept drift detector outperforms the MK-Means and MMD approaches. Thus, we conclude that OCSVM is the optimal choice from among the three concept drift detection techniques analyzed in this chapter. 5 Conclusion and Future Work In this chapter, we considered the problem of concept drift detection when using machine learning models for malware classification. We analyzed three distinct concept drift detection approaches, two of which use machine learning techniques, namely, Minibatch K-Means (MK-Means) and One-Class Support Vector Machine (OCSVM), and one of which uses Maximum Mean Discrepancy (MMD), a statistical analysis technique. We compared drift-aware detection strategies based on each of these three techniques, and in each case, we compared the results to a static scenario (no model retraining occurs), and to periodic retraining (models are constantly retrained, regardless of whether concept drift has occurred). We also performed an extensive investigation of the tradeoff between accuracy and training efficiency, using Pareto Front analysis. In addition, we analyzed the time required to train each of the concept drift detectors. Our OCSVM-based concept drift detection technique generally performed best, with respect to accuracy and training time, and with respect to training efficiency, OCSVM was equivalent to MMD. In terms of accuracy, efficiency, and training time, MK-Means was generally the least effective of the three drift detection techniques. Over the set of hyperparameters tested, we found that MMD was the most consistent, but for the best selections of hyperparameters, OCSVM performed somewhat better than MMD. It is interesting—and perhaps somewhat surprising—that a machine learning based approach (OCSVM) outperformed a statistical based approach (MMD). One plausible explanation is that while MMD is more sensitive to changes in the underlying statistics, OCSVM is better at detecting only those changes that are likely to actually matter, from a machine learning model perspective. This is a topic that is worthy of further investigation. An additional avenue for future research is refining the window size used for concept drift detection. While we only considered a fixed window of size 50, the smallest practical window size would be preferred, since we want to retrain our models as soon as malware has evolved. Additionally, a deeper analysis of the parameters of MK-Means and OCSVM could yield a better understanding of the underlying data distribution and thereby improve the accuracy and efficiency of these drift detectors. Another promising topic for further exploration is the use of Long-Short Term Memory (LSTM) models as drift detectors. LSTM models can account for temporal dependencies, which might enable such models to dynamically learn the optimal window size as part of the training process. Finally, evaluating the performance of proposed drift detection techniques in real-time malware detection systems would provide valuable insights into their practical applicability and scalability. References [1] M. T. Alam, R. Fieblinger, A. Mahara, and N. Rastogi (2024) MORPH: towards automated concept drift adaptation for malware detection. In Proceedings 2024 Network and Distributed System Security Symposium, NDSS. Note: https://arxiv.org/abs/2401.12790 Cited by: §2.2. [2] J. B. Alonso (2013) K-means vs Mini Batch K-means: a comparison. Note: http://hdl.handle.net/2117/23414 Cited by: §2.3.1. [3] J. Aycock (2006) Computer viruses and malware. Springer. Cited by: §1, §2.2. [4] M. Azeem, D. Khan, S. Iftikhar, S. Bawazeer, and M. Alzahrani (2024) Analyzing and comparing the effectiveness of malware detection: a study of machine learning approaches. Heliyon 10 (1), p. e23574. Cited by: §2.2. [5] F. Bayram, B. S. Ahmed, and A. Kassler (2022) From concept drift to model degradation: an overview on performance-aware drift detectors. Knowledge-Based Systems 245, p. 108632. Cited by: §2.1, §2.1, §2.1. [6] A. Bertia, B. X. Simon, G. J. W. Kathrine, and G. M. Palmer (2022) A study about detecting ransomware by using different algorithms. In 2022 International Conference on Applied Artificial Intelligence and Computing, ICAAIC, p. 1293–1300. Cited by: §1. [7] A. Bogomolnaia, H. Moulin, and R. Stong (2005) Collective choice under dichotomous preferences. Journal of Economic Theory 122 (2), p. 165–184. Cited by: §3.4. [8] L. Breiman (2001) Random Forests. Machine Learning 45, p. 5–32. Cited by: §2.4.3. [9] M. M. Breunig, H. Kriegel, R. T. Ng, and J. Sander (2000) LOF: identifying density-based local outliers. In Proceedings of the 2000 ACM SIGMOD International Conference on Management of Data, SIGMOD ’00, p. 93–104. Cited by: §2.3.2. [10] T. Chen and C. Guestrin (2016) XGBoost: a scalable tree boosting system. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’16, p. 785–794. Cited by: §2.4.4. [11] R. T. M. Chikushi, R. S. M. de Barros, M. G. N. M. da Silva, and B. I. F. Maciel (2021) Using spectral entropy and Bernoulli map to handle concept drift. Expert Systems with Applications 167, p. 114114. Cited by: §2.1. [12] C. W. B. Chungata, M. Jureček, K. Potika, and M. Stamp (2026) Maximum mean discrepancy for concept drift detection in malware classification models. In Proceedings of the 12th IEEE International Conference on Big Data Computing Service and Machine Learning Applications, IEEE BigDataServices. Cited by: §1. [13] C. A. Coello Coello, B. G. Lamont, and D. A. Van Veldhuizen (2007) Evolutionary algorithms for solving multi-objective problems. 2nd edition, Springer. External Links: ISBN 978-0-387-33254-3 Cited by: §3.4, §3.4. [14] C. Cortes and V. Vapnik (1995) Support-vector networks. Machine Learning 20, p. 273–297. Cited by: §2.4.2. [15] R. Desislavov, F. Martínez-Plumed, and J. Hernández-Orallo (2023) Trends in ai inference energy consumption: beyond the performance-vs-parameter laws of deep learning. Sustainable Computing: Informatics and Systems 38, p. 100857. Cited by: §1. [16] S. Dreiseitl, M. Osl, C. Scheibböck, and M. Binder (2010) Outlier detection with One-Class SVMs: an application to melanoma prognosis. In AMIA Annual Symposium Proceedings, p. 172–176. Cited by: §2.3.2. [17] (2025) F-Secure Labs: Adware:Android/Airpush. Note: https://w.f-secure.com/sw-desc/adware-android-airpush.shtml Cited by: item Airpush/StopSMS. [18] (2025) F-Secure Labs: Riskware:Android/SmsReg. Note: https://w.f-secure.com/sw-desc/riskware-android-smsreg.shtml Cited by: item SMSReg. [19] (2025) F-Secure Labs: Trojan:Android/Boxer. Note: https://w.f-secure.com/v-descs/trojan-android-boxer.shtml Cited by: item Boxer. [20] (2025) F-Secure Labs: Trojan:W32/Agent. Note: https://w.f-secure.com/v-descs/agent.shtml Cited by: item Agent. [21] A. Gretton, K. M. Borgwardt, M. J. Rasch, B. Schölkopf, and A. Smola (2012) A kernel two-sample test. Journal of Machine Learning Research 13, p. 723–773. Cited by: §2.3.3, §2.3.3. [22] A. Guerra-Manzanares, H. Bahsi, and S. N~omm (2021) KronoDroid: time-based hybrid-featured dataset for effective android malware detection and characterization. Computers & Security 110, p. 102399. Cited by: §2.2, item Malap, §3.1, §3.1. [23] A. Guerra-Manzanares (2021) GitHub - aleguma/kronodroid: KronoDroid dataset. Note: https://github.com/aleguma/kronodroid Cited by: §3.1, §3.1. [24] J. A. Hartigan and M. A. Wong (1979) Algorithm AS 136: a K-means clustering algorithm. Journal of the Royal Statistical Society. Series C (Applied Statistics) 28 (1), p. 100–108. Cited by: §2.3.1. [25] Y. He, J. Lei, Z. Qin, K. Ren, and C. Chen (2025) Combating concept drift with explanatory detection and adaptation for android malware classification. In Proceedings of the 2025 ACM Conference on Computer and Communications Security, CCS, p. 1–15. Cited by: §2.2. [26] (2021) IBM: Random Forest. Note: https://w.ibm.com/think/topics/random-forest Cited by: §2.4.3. [27] A. M. Ikotun, A. E. Ezugwu, L. Abualigah, B. Abuhaija, and J. Heming (2023) K-means clustering algorithms: a comprehensive review, variants analysis, and advances in the era of big data. Information Sciences 622, p. 178–210. Cited by: §2.3.1. [28] S. Kim (2018) PE header analysis for malware detection. Master’s Thesis, San Jose State University, California. Note: https://scholarworks.sjsu.edu/etd_projects/624/ Cited by: §2.2. [29] H. T. Kung, F. Luccio, and F. P. Preparata (1975) On finding the maxima of a set of vectors. Journal of the ACM 22 (4), p. 469–476. Cited by: §3.4. [30] A. Lev (2022) XGBoost versus Random Forest. Note: https://w.qwak.com/post/xgboost-versus-random-forest Cited by: §2.4.4. [31] A. S. Li, A. Iyengar, A. Kundu, and E. Bertino (2025) Revisiting concept drift in windows malware detection: adaptation to real drifted malware with minimal samples. In Proceedings 2025 Network and Distributed System Security Symposium, NDSS. Note: https://arxiv.org/abs/2407.13918 Cited by: §2.2. [32] F. T. Liu, K. M. Ting, and Z. Zhou (2008) Isolation forest. In 2008 Eighth IEEE International Conference on Data Mining, p. 413–422. Cited by: §2.3.2. [33] L. M. Manevitz and M. Yousef (2002) One-Class SVMs for document classification. Journal of Machine Learning Research 2, p. 139–154. Cited by: §2.3.2. [34] L. Mei and M. Stamp (2025) Energy considerations for large pretrained neural networks. Note: https://arxiv.org/abs/2506.01311 Cited by: §1. [35] H. E. Merabet and A. Hajraoui (2019) A survey of malware detection techniques based on machine learning. International Journal of Advanced Computer Science and Applications 10 (1). Cited by: §2.2. [36] A. Mishra and M. Stamp (2025) Cluster analysis and concept drift detection in malware. Journal of Computer Virology and Hacking Techniques 21. Cited by: §2.2, §2.3.1, §3.1.1. [37] A. Nappa, M. Z. Rafique, and J. Caballero (2014) The Malicia dataset: identification and analysis of drive-by download operations. International Journal of Information Security 14 (1), p. 15–33. Cited by: §2.2. [38] (2025) Optuna: efficient optimization algorithms — Optuna 3.5.0 documentation. Note: https://optuna.readthedocs.io/en/stable/tutorial/10_key_features/003_efficient_optimization_algorithms.html Cited by: §3.3.1, §3.6. [39] S. Paul and M. Stamp (2021) Word embedding techniques for malware evolution detection. In Malware Analysis Using Artificial Intelligence and Deep Learning, M. Stamp, M. Alazab, and A. Shalaginov (Eds.), p. 321–343. Cited by: §2.2. [40] (2025) Polars: data types and structures. Note: https://docs.pola.rs/user-guide/concepts/data-types-and-structures/ Cited by: §3.1.2. [41] S. Rabanser, S. Günnemann, and Z. C. Lipton (2019) Failing loudly: an empirical study of methods for detecting dataset shift. In Proceedings of the 33rd International Conference on Neural Information Processing Systems, Cited by: §2.3.3, §2.3.3. [42] A. Redhu, P. Choudhary, K. Srinivasan, and T. K. Das (2024) Deep learning-powered malware detection in cyberspace: a contemporary review. Frontiers in Physics 12. Cited by: §2.2. [43] G. J. Ross, N. M. Adams, D. K. Tasoulis, and D. J. Hand (2012) Exponentially weighted moving average charts for detecting concept drift. Pattern Recognition Letters 33 (2), p. 191–198. Cited by: §2.1. [44] P. J. Rousseeuw and K. V. Driessen (1999) A fast algorithm for the minimum covariance determinant estimator. Technometrics 41 (3), p. 212–223. Cited by: §2.3.2. [45] B. Schölkopf, J. C. Platt, J. Shawe-Taylor, A. J. Smola, and R. C. Williamson (2001) Estimating the support of a high-dimensional distribution. Neural Computation 13 (7), p. 1443–1471. Cited by: §2.3.2. [46] Scikit-learn: 2.7. novelty and outlier detection. Note: https://scikit-learn.org/stable/modules/outlier_detection.html Cited by: §2.3.2. [47] T. Shi, R. A. McCann, Y. Huang, W. Wang, and J. Kong (2024) Malware detection for Internet of Things using one-class classification. Sensors 24 (13), p. 4122. Cited by: §2.3.2. [48] (2010) Sklearn.neural_network.mlpclassifier — scikit-learn 0.20.3 documentation. Note: https://scikit-learn.org/stable/modules/generated/sklearn.neural_network.MLPClassifier.html Cited by: §2.4.1. [49] Y. Song, G. Zhang, H. Lu, and J. Lu (2020) A fuzzy drift correlation matrix for multiple data stream regression. In 2020 IEEE International Conference on Fuzzy Systems, FUZZ-IEEE, p. 1–6. Cited by: §2.1. [50] M. Stamp (2022) Introduction to machine learning with applications in information security. second edition, Chapman and Hall/CRC, Boca Raton. Cited by: §1, §2.2. [51] J. Terven, D. Cordova-Esparza, J. Romero-González, A. Ramírez-Pedraza, and E. A. Chávez-Urbiola (2025) A comprehensive survey of loss functions and metrics in deep learning. Artificial Intelligence Review 58, p. 195. External Links: Document Cited by: §2.4.1. [52] E. Tiakas, A. N. Papadopoulos, and Y. Manolopoulos (2015) Skyline queries: an introduction. In 2015 6th International Conference on Information, Intelligence, Systems and Applications, IISA, p. 1–6. Cited by: §3.4. [53] L. S. Tupadha and M. Stamp (2022) Machine learning for malware evolution detection. In Artificial Intelligence for Cybersecurity, M. Stamp, C. Aaron Visaggio, F. Mercaldo, and F. Di Troia (Eds.), p. 183–213. Cited by: §2.2. [54] A. Van Looveren, J. Klaise, G. Vacanti, O. Cobb, A. Scillitoe, R. Samoilescu, and A. Athorne (2019) Alibi detect: algorithms for outlier, adversarial and drift detection. Cited by: §2.3.3. [55] M. Wadkar, F. D. Troia, and M. Stamp (2020) Detecting malware evolution using support vector machines. Expert Systems with Applications 143, p. 113022. Cited by: §2.2. [56] S. Watanabe (2023) Tree-structured parzen estimator: understanding its algorithm components and their roles for better empirical performance. Note: https://arxiv.org/abs/2304.11127 Cited by: §3.3.1. [57] K. Yasar and F. Tabsharani (2023) What is a support vector machine (SVM)?. Note: https://w.techtarget.com/whatis/definition/support-vector-machine-SVM Cited by: §2.4.2. [58] Q. Zhang and H. Li (2007) MOEA/D: a multiobjective evolutionary algorithm based on decomposition. IEEE Transactions on Evolutionary Computation 11 (6), p. 712–731. Cited by: §3.4. [59] S. Zhang, J. Liu, and X. Zuo (2021) Adaptive online incremental learning for evolving data streams. Applied Soft Computing 105, p. 107255. Cited by: §2.1. [60] Y. Zhang, Q. Liu, and Y. Shi (2022) MANAGE: a novel malware evolution model based on digital genes. In 2022 7th IEEE International Conference on Data Science in Cyberspace, DSC, p. 64–70. Cited by: §2.2. Appendix In this Appendix, we provide additional relevant graphs. Figures 14 through 17 provide bar graphs for each learning model considered, where static and periodic accuracies are given for each combination of malware families (Fd,Fc)(F_d,F_c). Figure 18 contains violin plots related to accuracy and efficiency over all hyperparameters tested, while Figure 19 has the analogous plots restricted to the corresponding Pareto Front. 0.000.000.200.200.400.400.600.600.800.801.001.000.84420.84420.99060.99060.59640.59640.93750.93750.78000.78000.99190.99190.82230.82230.92480.92480.76410.76410.89330.89330.80140.80140.95950.95950.54940.54940.75310.75310.98410.98410.82370.82370.58890.58890.82470.82470.98280.98280.68310.68310.96290.96290.99820.99820.83350.83350.97900.97900.92760.92760.99230.99230.90680.90680.99150.99150.98270.98270.98770.98770.97640.97640.99720.99720.95340.95340.98060.98060.99470.99470.98750.98750.97370.97370.98860.98860.99600.99600.92150.9215AccuracyModel: MLPStaticPeriodic Figure 14: Static and periodic accuracies for MLP model 0.000.000.200.200.400.400.600.600.800.801.001.000.76610.76610.92410.92410.59380.59380.96430.96430.77420.77420.94630.94630.92580.92580.97480.97480.87500.87500.90740.90740.89230.89230.97080.97080.42750.42750.63750.63750.95590.95590.79560.79560.67100.67100.73130.73130.98130.98130.58990.58990.93790.93790.99600.99600.76830.76830.96920.96920.90020.90020.98440.98440.88320.88320.97550.97550.98030.98030.99190.99190.96940.96940.99790.99790.93470.93470.97440.97440.99160.99160.97120.97120.95560.95560.98610.98610.99390.99390.89550.8955AccuracyModel: RFStaticPeriodic Figure 15: Static and periodic accuracies for RF model 0.000.000.200.200.400.400.600.600.800.801.001.000.74690.74690.93350.93350.58350.58350.89200.89200.86680.86680.94500.94500.94020.94020.88600.88600.86830.86830.93450.93450.83840.83840.96060.96060.68940.68940.64780.64780.95470.95470.85780.85780.78510.78510.81110.81110.98810.98810.70150.70150.89420.89420.96210.96210.88480.88480.91920.91920.91740.91740.98350.98350.93310.93310.91550.91550.98560.98560.98350.98350.98350.98350.98520.98520.92810.92810.94780.94780.98910.98910.94870.94870.95400.95400.95580.95580.99600.99600.93260.9326AccuracyModel: SVMStaticPeriodic Figure 16: Static and periodic accuracies for SVM model (Agent, Airpush)(Agent, Boxer)(Agent, Malap)(Agent, SMSreg)(Airpush, Agent)(Airpush, Boxer)(Airpush, Malap)(Airpush, SMSreg)(Boxer, Agent)(Boxer, Airpush)(Boxer, Malap)(Boxer, SMSreg)(Malap, Agent)(Malap, Airpush)(Malap, Boxer)(Malap, SMSreg)(SMSreg, Agent)(SMSreg, Airpush)(SMSreg, Boxer)(SMSreg, Malap)0.000.000.200.200.400.400.600.600.800.801.001.000.73620.73620.97370.97370.57460.57460.82460.82460.64260.64260.99840.99840.68630.68630.82560.82560.65180.65180.70770.70770.80620.80620.94300.94300.54250.54250.51720.51720.83590.83590.85660.85660.63060.63060.89820.89820.81770.81770.63280.63280.93880.93880.98970.98970.78790.78790.94200.94200.79260.79260.97950.97950.96550.96550.96840.96840.87540.87540.96340.96340.92150.92150.97710.97710.90530.90530.96340.96340.98030.98030.96940.96940.91970.91970.97020.97020.98030.98030.87780.8778AccuracyModel: XGBStaticPeriodic Figure 17: Static and periodic accuracies for XGB model Figure 18: Violin plots for accuracy and efficiency over all hyperparameters Figure 19: Violin plots for accuracy and efficiency on Pareto Front