Paper deep dive
A TTP by TTP Approach: Precise Malware Detection via Malicious TTP Recognition
Yashovardhan Sharma
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 90%
Last extracted: 9/12/2026, 2:45:08 AM
Summary
This paper introduces a novel Deep Neural Network (DNN) architecture for precise malware detection in network traffic by leveraging the MITRE ATT&CK framework. Unlike purely data-driven approaches, the proposed system incorporates Tactics, Techniques, and Procedures (TTPs) as features, enabling multi-label classification to identify both the maliciousness of a sample and the specific TTPs utilized. The study demonstrates that this 'TTP by TTP' approach outperforms existing models, particularly in detecting rarely-used TTPs, handling limited training data, and resisting adversarial attacks.
Entities (7)
Relation Signals (5)
MITRE ATT&CK â provides â TTPs
confidence 95% ¡ We leverage the ontology of adversarial behaviour provided by the MITRE ATT&CK framework, comprising of tactics, techniques, and procedures (TTPs)
Bag of Flows â includes â TTP features
confidence 92% ¡ the Bag of Flows representation also includes the following information about the TTPs matched by the flows in the sample
STTP2STTP â outperforms â S2S
confidence 90% ¡ The STTP2STTP model has the best overall performance... consistently outperform the standard S2S model.
Deep Neural Network â utilizes â Bag of Flows
confidence 90% ¡ This Bag of Flows is given as input to our multi-label DNN architecture
YAF â generates â network flows
confidence 85% ¡ Each raw network traffic sample is represented as a set of network flows, using Yet Another Flowmeter (YAF)
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Machine learning methods, and especially neural networks, are now routinely used for malware detection in network traffic. Though very effective, systems based on such methods often (i) are purely data-driven, ignoring the substantial body of available knowledge about the tactics, techniques, and procedures (TTPs) possibly used, and, consequently (ii) are not precise, since they either cannot correlate malicious activity with TTP usage, or if they do, they are unable to explain which TTP has been maliciously used. In this paper we demonstrate that it is possible to precisely detect malware by (i) providing the neural network model with information about the TTPs used by any given sample, and (ii) teaching the neural network to detect not just the malicious activity as a whole, but which specific TTPs are maliciously used. We show that our approach consistently outperforms the three alternative models, which either do not exploit TTP information, or which are not taught to detect the malicious usage of TTPs, or both. Moreover, we show that our approach (i) is particularly beneficial in detecting malware that utilises rarely-used TTPs, a scenario which is particularly challenging for the other systems; (ii) allows for TTP by TTP tuning, further improving its ability to detect the malicious usage of TTPs; (iii) consistently outperforms other systems across a wide-range of scenarios, including when relying on limited training data or when subjected to adversarial attack.
Tags
Links
- Source: https://arxiv.org/abs/2609.06579v1
- Canonical: https://arxiv.org/abs/2609.06579v1
Trouble viewing inline? Open PDF directly â
Full Text
74,770 characters extracted from source content.
Expand or collapse full text
A TTP by TTP Approach: Precise Malware Detection via Malicious TTP Recognition Yashovardhan Sharma University of Oxford Oxford, United Kingdom yashovardhan.sharma@cs.ox.ac.uk AbstractâMachine learning methods, and especially neural networks, are now routinely used for malware detection in network traffic. Though very effective, systems based on such methods often(i)are purely data-driven, ignoring the substantial body of available knowledge about the tactics, techniques, and procedures (TTPs) possibly used, and, conse- quently(i)are not precise, since they either cannot correlate malicious activity with TTP usage, or if they do, they are unable to explain which TTP has been maliciously used. In this paper we demonstrate that it is possible to precisely detect malware by(i)providing the neural network model with information about the TTPs used by any given sample, and(i)teaching the neural network to detect not just the malicious activity as a whole, but which specific TTPs are maliciously used. We show that our approach consistently outperforms the three alternative models, which either do not exploit TTP information, or which are not taught to detect the malicious usage of TTPs, or both. Moreover, we show that our approach(i)is particularly beneficial in detecting malware that utilises rarely-used TTPs, a scenario which is particularly challenging for the other systems;(i)allows for TTP by TTP tuning, further improving its ability to detect the malicious usage of TTPs;(i)consistently outperforms other systems across a wide-range of scenarios, including when relying on limited training data or when subjected to adversarial attack. 1. Introduction Machine learning methods and especially neural net- works are now routinely used for malware detection in network traffic (see, e.g., [18], [20], [32]). Just within the research literature, Scopus indicates that there has been a 508% increase in papers referencing machine learning (ML) or deep learning (DL) within the context of cyber security over the last 5 years [36]. The rapid increase in the usage of such techniques within the domain of malware detection has been fuelled by the challenge of keeping up with constantly-evolving malware and the ability of ML models to handle large volumes of data. Approaches using neural networks in particular have been favoured since they do not require an expertâs domain knowledge to define discriminative feature used in mal- ware detection system [9]. Though very effective, such methods are often:(i)purely data-driven, ignoring the substantial body of available knowledge about the tactics, techniques, and procedures (TTPs) possibly used, and(i) not precise, since they either cannot correlate malicious activity with TTP usage or if they do, they give little to no explanation about which TTPs have been maliciously used. Furthermore, ML methods typically require large amounts of quality training data, and are often not robust to adversarial attacks [9], [32]. In this paper, we leverage the ontology of adversarial behaviour provided by the MITRE ATT&CK framework, comprising of tactics, techniques, and procedures (TTPs), to show that it is possible to precisely detect malware from network captures by(i)formulating the problem as a multi-label classification problem (see, e.g., [43]) with one set of labels to classify the TTPs, and another label to classify the maliciousness of the entire network traffic sample (henceforth referred to assample), and(i) designing a Deep Neural Network (DNN) architecture for the multi-label classification problem, that can take into account information about the TTPs utilised by the sample in the form of TTP features. In particular, we propose a methodology in which(i)each sample is analysed in order to automatically extract the TTPs utilised by it,(i)the network traffic features of the sample, together with the features about the matched TTPs, are then given as inputs to the DNN architecture, which is not only responsible for detecting the maliciousness of the sample, but also which TTPs are maliciously used. To the best of our knowledge, ours is the first system which is able to detect malicious network traffic and precisely characterise which TTPs have been maliciously used. This is a rather useful property since it enables our system to have different fine-grained outputs, as advocated in [3]. Compared to the existing purely data-driven approaches for malware detection (see, e.g., [18], [20]), we expect that by(i)pro- viding the information about the TTPs used by a sample, and(i)training the neural network to detect both the malicious activity as a whole and the specific TTPs that are maliciously used, our system is particularly well-suited for detecting malware that utilises uncommon or rarely- used TTPsâa scenario which is particularly challenging for any data-driven approach. We evaluate our system by conducting an extensive experimental analysis involving more than 1.5 million ma- licious and benign samples, curated from 5 different pub- licly available datasets, thus allowing us to test our system on a broad range of real-world samples. We consider 5 micro-average metrics usually used to evaluate systems for multi-label classification problems (F 1 -score, Subset Ac- curacy, Precision, Recall and False Positive Rate), together with the macro-average metrics corresponding to theF 1 - score, Precision and Recall. These metrics are widely used 1 (see, e.g., [8], [24], [33]), and each highlights a different aspect of our systemâs performance. For instance, we utilise the macro-average metrics to highlight the perfor- mance of our system at TTP classification since individual TTP classes are unbalanced in our data, and these metrics are suited for multi-label classification with unbalanced classes [30], [42]. On the other hand, we use micro- average metrics to highlight the overall performance of our system, specifically for sample classification, which is evaluated on balanced malicious and benign datasets. We further evaluate our approach across four possible configurations: 1)STTP2STTP: the system is given the sample and TTP features as input, and it has to detect both if the sample is malicious, and which TTPs are maliciously used by it. 2)STTP2S: the system is given the sample and TTP features as input, and it has to determine whether the sample is malicious or not. 3)S2STTP: the system is given just the sample as input, and it has to detect both if the sample is malicious, and which TTPs are maliciously used by it. 4)S2S: the system is given just the sample as input, and it has to determine whether the sample is malicious or not. We test our approach in all configurations, across all of our datasets, to thoroughly evaluate its performance. We conduct further experimental analysis to test our approach in various challenging conditions, such as when faced with limited training data or dealing with adversarial attack. Our experimental analysis confirms that: â˘TheSTTP2STTP,STTP2SandS2STTPmodels con- sistently outperform the standardS2Smodel. â˘TheSTTP2STTPmodel has the best overall per- formance, and significantly better average per- formance in detecting rarely-occurring malicious TTPs, as witnessed by an average improvement in the MacroF 1 -score, Macro Precision and Macro Recall of 37.55%, 11.82% and 46.05% respec- tively. â˘It is possible to further improve the performance of STTP2STTPby finely tuning the decision threshold for each individual TTP. In our datasets, we mea- sure average performance gains of 12.5%, 12.0%, and 47.6%, for the MacroF 1 -score, Macro Preci- sion, and Macro Recall respectively. â˘STTP2STTPmaintains its ability to better detect malicious TTPs thanS2STTPeven when con- sidering challenging scenarios where models are trained with a limited number of samples, or when tested with noisy malicious samples flooded with benign traffic by adversaries attempting to camou- flage their activities. The better performance ofSTTP2STTPwhen compared to any other model (be itS2STTP,STTP2SorS2S) corre- sponds to the possibility of trainingSTTP2STTPwith less data and still getting the same performance of the other model, thus at least partially overcoming the well-known data-greediness problem of ML models (see, e.g., [9]). For instance,STTP2STTPutilising 25% of the training data has a higher averageF 1 -score thanS2STTPutilising 100% of the training data, andSTTP2STTPwhen trained with 10% of the data has on average still a higher Recall than S2STTPtrained with 100% of the data. To recap, the main contributions of the paper can be summarised as follows: â˘We design a novel multi-label DNN architec- ture that can precisely detect network-based mal- ware along with the specific TTPs utilised by it, by exploiting TTP information in the form of automatically-generated features. â˘We outperform the state-of-the-art models, regard- less of whether or not they exploit the same TTP features used by our system. Our approach is particularly beneficial in detecting the malicious usage of uncommon or rarely-used TTPs. â˘Our system is highly configurable, allowing for TTP by TTP tuning, thus further increasing its performance. â˘We test our approach on a large dataset of real- world malicious and benign samples. We utilise 5 balanced datasets to fairly test our system, con- sisting of over 1.5 million samples. â˘We evaluate our system under challenging scenar- ios, such as when it has access to limited training data or when it is subjected to adversarial attack, and demonstrate that it consistently outperforms other systems even under these conditions. The rest of the paper is organised as follows: Section 2 gives a brief overview of the MITRE ATT&CK framework and discusses closely related work. Section 3 discusses the overall architecture of the system and the functioning of its individual components. The four modelsSTTP2STTP, S2STTP,STTP2SandS2Sare then comparatively evaluated in Section 4. Section 5 contains further experimental anal- ysis of our system, by testing it under various challenging conditions and tuning it on a TTP-by-TTP basis to further improve performance. A thorough discussion about the results presented is in Section 6, followed by the overall conclusions of our work in Section 7. 2. Background and Related Work The MITRE ATT&CK framework [35] is a well- known publicly available knowledge base of adversarial tactics, techniques and procedures, based on real-world observations. Within the ATT&CK framework: â˘Atacticrepresents a tactical goal of the adversary, i.e., the reason why the adversary performs any action. â˘Atechniqueis the method by which the adversary achieves their tactical goal. Techniques can have sub-techniques, corresponding to a more specific action within this method. â˘Aprocedureis a specific implementation of the method the adversary uses to fulfil their tactical goal. As an example, an adversary may dump OS cre- dentials (techniqueT1003) to achieve credential access (tactic). This may be done by dumping the contents of/etc/passwdor/etc/shadow(sub-technique T1003.008), using a tool such as LaZagne 1 (procedure). The MITRE ATT&CK framework is frequently updated to keep it up-to-date with the evolution of malware and novel adversarial approaches. The current version of the ATT&CK framework for Enterprise features 14 tactics, 201 techniques, and 424 sub-techniques. This paper will focus specifically on the techniques (and encompassing tactics) that utilise the network to achieve their goals, which is a smaller subset of the overall techniques present in the ATT&CK framework. The ATT&CK framework and the TTPs described within it have been used extensively to tackle various challenges within the context of cyber security. The broad idea is to exploit existing human domain knowledge in an automated manner, in order to assist solving problems that rely on the analysis of adversarial behaviour. TTPs from the ATT&CK framework have been utilised to track the evolution of malware and describe emerging trends [4], [23], locate TTPs within the control flow graph of malware executables [7], automatically extract TTPs from Cyber Threat Intelligence (CTI) reports [14], [19], [22], discover existing inter-dependencies in a TTP chain [1], detect Living-Off-the-Land (LotL) malware techniques and Ad- vanced Persistent Threat (APT) attack campaigns [12], [17], and build malware detection systems that exploit TTP information [29]â[31]. If we focus specifically on how TTPs have been used for the automatic detection of malicious activity, to the best of our knowledge, the only research literature that has explored this direction are the recent works by Sharma et al.[28]â[31]. In [30] and [29], the authors create an explainable and extensible IDS that exploits TTPs in order to detect malware. In this system, a different dataset is created for each TTP, where the label is considered positive if the sample is malicious and the TTP is present. For each dataset, a different decision tree [41] is trained and then used together with the others as an ensemble to decide on the maliciousness of the sample, thus resulting in an extensible and explainable system. However, the limitation of such an approach is that its modularity is achieved by having a separate ML engine for each TTP, thus sacrificing the possibility of exploiting the existing correlations between different TTPs to detect malicious activity (correlations among labels are often exploited to achieve better performance, see, e.g., [25], [34]). Addi- tionally, this system conducts its analysis on a per-flow basis, which while effective, is unable to consider all the flows in a sample together and thus cannot evaluate the behaviour of a malware sample as a whole. On the other hand, in [31] the authors create binary classification models, called TTPxML models, which incorporate TTP information as features into these models in order to detect network-based malware. They test this approach on a wide-variety of ML models, such as Support Vector Machines [5], Random Forests [13], and DNNs [21], and find that TTP features can be exploited to improve the malware detection capabilities of ML-based systems. Here, TTPs are first automatically extracted from the network sample, then a vectorial representation is created of both the sample features and the TTP features, which is subsequently provided as input to a binary classifier 1. https://github.com/AlessandroZ/LaZagne which labels the entire sample as malicious or not. This system is the state-of-the-art in terms of exploiting TTP information in an automated manner to detect network- based malware. Thus, we closely model this system in ourSTTP2Smodel, in order to be able to compare and contrast the results of our multi-label classification-based approach with their binary classification-based system. 3. System Architecture At a high level, our system classifies a given sample in the following manner: 1)A raw network traffic sample is taken as input, and modelled into a vectorial representation of the network traffic data, known as Bag of Flows (BoF). 2)TTP information about the sample is automati- cally extracted, and incorporated into the Bag of Flows in the form of TTP features. 3)This Bag of Flows is given as input to our multi- label DNN architecture, which performs the mal- ware classification. 4)As per the model configuration, our system gives as output:(i)a malicious or benign classification for the overall sample, or(i)a multi-label clas- sification of each maliciously-used TTP by the sample, in addition to the malicious or benign classification for the overall sample. We broadly describe the specifics of each of these steps in the subsequent subsections. 3.1. Constructing the Bag of Flows The first two steps are largely based on the method- ology of the TTPxML system described in [31]. Here, to make the paper more self-contained, we briefly describe how the raw network traffic samples are converted into their Bag of Flow representation, as required by the next step in our malware detection system. See [31] for more details. Each raw network traffic sample is represented as a set of network flows, using Yet Another Flowmeter (YAF) [15], and annotated with the set of features in Table 1. YAF has several advantages, including the fact that it has been designed for high performance. Then, each flow is annotated with the 13 TTPs from the MITRE ATT&CK framework as shown in Table 2. This TTP matching is done at the flow level, and utilises the method- ology described in [30]. These 13 TTPs encompass 11 tactics, 9 techniques, and 6 sub-techniques, and thus rep- resent a sufficiently wide variety of adversarial behaviour that can be exhibited by network-based malware. Each flow is then converted into a vector in which(i)each non-numerical feature is converted into a numerical one, (i)the categorical features âEnd-reasonâ, âProtocolâ and the 4 features representing TCP flags, are represented as one-hot encoded vectors, while(i)âDestination IPâ is converted into a 32-bit integer maintaining the structure information of IP addresses. Any flow properties that are not useful for classification or would unintentionally bias it, are dropped. Finally, the remaining flow properties are TABLE 1: Flow properties. FieldProperty Flow HashThe hash associated with each unique flow. Sample HashHash of the sample corresponding to the flow. Unique IDA unique ID associated with each flow. DatasetThe dataset to which a flow belongs. Source PortThe port from which the flow originates. Destination PortThe port on which the flow connects to. Start time, end timeFlow start or end time. DurationFlow duration in seconds. ProtocolIP protocol identifier in decimal format. EntropyThe Shannon Entropy for the flow payload. ApplabelThe application label, as identified by YAF. Source IP, Dest. IPSource and destination IPv4/IPv6 address. Type, CodeICMP type or code in decimal format. Isn, RisnForward or reverse TCP sequence number. Flags4 properties representing various TCP flags. Tag, Rtag802.1q VLAN tag in forward/reverse direction. Pkt, RpktNo. of packets in forward/reverse direction. Oct, RoctNo. of bytes in forward/reverse direction. RTTRound-trip time estimate in milliseconds. End-reasonReason for termination of flow. aggregated in a Bag of Flows, which is a vectorial repre- sentation of the original sample. Each BoF is constructed by(i)min-max scaling each flow-level feature, and(i) transforming each flow-level feature into a normalised histogram recording the proportion of values that fall within a certain range. In practice, for each non categorical featureB= 5bins are defined, and for each bin, the proportion of flows in the sample having a value belonging to that bin is recorded. In theSTTP2SandSTTP2STTPmodels, the Bag of Flows representation also includes the following informa- tion about the TTPs matched by the flows in the sample: â˘For each TTP two aggregate features are created: one representing the proportion of flows in the sample that match the given TTP, and the other representing the total number of flows matching the given TTP. This last feature is normalised over all the samples. â˘Then, the following information about the matched TTPs is also provided:(i)the number of unique TTPs matched by the sample,(i)the total number of TTPs matched by the sample,(i)the number of flows matching at least one TTP in the sample, (iv)the proportion of flows matching at least one TTP in the sample,(v)the mean of the number of TTPs matched by the sample, and(vi)the standard deviation of the number of TTPs matched by the sample. All these features are normalised over all the samples. At this point, the sample has been successfully mod- elled as a Bag of Flows, and its corresponding TTP information has also been incorporated into this BoF. This vectorial representation of the given sample is then passed on to the classifier. 3.2. Multi-label DNN Classification The multi-label DNN architecture utilised by our sys- tem has an input layer of neurons, that has the same size as that of the vector representing the Bag of Flows, which was created in the previous step. The neural network is implemented as a fully-connected feed-forward neural TABLE 2: List of supported TTPs from the MITRE ATT&CK framework. TacticsTechniques ReconnaissanceT1590 - Gathering Victim Network Information Credential AccessT1557.001 - Man-in-the-Middle Discovery T1124 - System Time Discovery T1135 - Network Share Discovery Lateral Movement T1021.001 - Remote Services (RDP) T1021.004 - Remote Services (SSH) T1550.003 - Use Alternate Authentication Material T1563.001 - Remote Service Session Hijacking (RDP) T1563.002 - Remote Service Session Hijacking (SSH) T1570 - Lateral Tool Transfer Command and Control T1071 - Application Layer Protocol T1090 - Proxy T1105 - Ingress Tool Transfer T1571 - Non-Standard Port ExecutionT1053 - Scheduled Task/Job network with ReLU non-linearity and four hidden layers. The output layer of the DNN contains 14 neurons, 13 corresponding to the outcome for the 13 TTPs supported by our system, and the final neuron corresponding to the maliciousness of the entire sample. Thus, this ar- chitecture allows us to operate in both multi-label and binary settings, enabling us to test our system in various configurations. As previously stated, our system allows for 4 different configurations depending on whether: 1)TTP features are given as input along with the sampleâs BoF or not, to the classification system. 2)The output of binary classification of the sam- ple, should also include information about which TTPs are maliciously used by the sample or not. These 4 configurations are:STTP2STTP(TTP informa- tion in both input and output),S2STTP(just the sample information in input and TTP information as output), STTP2S(TTP information in input and just the sam- ple maliciousness as output), andS2S(just the sample information in input and sample maliciousness as out- put). All the four classification tasks can be modelled as multi-label classification problems, and transformed into a binary classification problems, when dealing with configurations where the system only has to determine whether the overall sample is malicious or benign. In a multi-label classification model, an input vectorxof values is mapped to an output binary vector in which each element represents a property of the input sample x. In contrast to binary classification, an input samplex can have multiple elements in the output vector mapped to 1, thus corresponding toxsatisfying the properties represented by such output values. Formally, amulti-label classificationproblem is defined as a pair(A,X), where Ais a set of labels, whileXis a finite set of pairs(x,y) wherexâR N (NâĽ1) is a data point/sample, andyâA is the ground truth ofx, i.e., the set of labels associated withx. AmodelmforPis a functionm(¡,¡)which maps every labelAand data pointxâR N to[0,1]. For every labelAthe functionm A :R N 7â[0,1]is defined byx7âm(A,x), for each data pointxâR N . Finally, a data pointxâR N is predicted bymto have labelA wheneverm A (x)is greater than or equal to a user-defined thresholdθ A â[0,1]. Ideally, we would like to have one label per TTP that we are interested in detecting. In our case, this would correspond to having the set of labelsAequal to the set of TTPs in Table 2; the setXto be the set of Bag of Flows corresponding to the samples, in which each Bag of Flowsxis paired with the labelsycorresponding to the TTPs in Table 2 which are maliciously used by the sample. Such a picture has two difficulties: 1)All available datasets of malware network traffic only have information regarding the malicious- ness of the sample, and do not contain any infor- mation about which are the TTPs being used. 2)The list of TTPs in Table 2 is by no means exhaustive, meaning that there can certainly be malicious network traffic samples that do not match any of the TTPs supported by our system. We overcome the first difficulty by(i)automatically detecting the TTPs in each sample as described in the previous subsection, and(i)associating the set of labels corresponding to the detected TTPs, to the sample. To overcome the second difficulty, we introduce an additional label called Mal-Sample whose ground truth is assumed to be 1 when the sample is malicious, even if it does not match any of the TTPs our system supports. Notice that even assuming that our system would support all known TTPs, such an additional label is still useful given the possibility of either an attack using a never seen before TTP, or the TTP detection component failing to recognise a TTP (be it malicious or benign) in the network traffic sample. However, the presence of the label Mal-Sample indicating sample maliciousness and of the other labels indicating malicious usage of TTPs raises the following possibilities: 1)The TTP outputs and the sample output are com- patible with each other, meaning that either all the TTPs in the sample and the sample itself are classified as benign, or that some of the TTPs in the sample and the sample itself are classified as malicious. In this case, no issue arises and all the TTPs and the sample are considered according to their predictions. 2)The TTP outputs and the sample output are dif- ferent because all the TTPs in the sample are labelled as benign and the sample as malicious. Such a case reflects the possibility that the model has detected a malicious sample even though it does not match any of the supported TTPs. Given this, all the TTPs and the sample are considered according to their predictions, as in the first case. 3)The TTP outputs and the sample output are dif- ferent, implying that some TTP is labelled as malicious while the sample is labelled as benign. Such a situation does not reflect any real scenario, and it is possible given the inherent fact that machine learning models in general, and neural networks in particular, may fail to learn existing relationships between the labels (see, e.g., [10], for a discussion on this). In such a case, taking a prudential approach, we also consider the sam- ple as malicious and count it accordingly in the metrics. TABLE 3: Statistics about the original datasets. DatasetSamplesFlows Ember Malicious [2]435,74126,567,527 MalRec [27]37,76312,273,502 MalShare [37]1,268,92332,310,907 VirusShare [38]595,09812,217,493 Ember Benign [2]155,4321,423,023 Total (unique)2,131,83283,371,480 Indeed, from a technical point of view, the problem we are considering is the special case of a multi-label clas- sification problem with hierarchical constraints between the labels. Special techniques have been developed for such problems in order to guarantee coherency between hierarchical labels, (see, e.g., [11], [26], [40]). We did not consider such techniques given the relative simplicity of our hierarchy with only one level, in which, as we said, co- herency is guaranteed by simply flipping the Mal-Sample label when necessary. In practice, such incoherence shows up very rarely: in our experiments there are no differences (up to the third decimal place) in theF 1 -score, Precision, and Recall of label Mal-Sample when measured before and after Mal-Sample is turned to 1, in order to solve the incoherence between label Mal-Sample and the TTP labels. 4. Evaluation The goal of our experimental evaluation is to assess the benefits of(i)providing TTP information as additional features to the neural network; and(i)teaching the neural network to detect the malicious usage of TTPs, when classifying a given sample. Specifically, we pose the following research questions: 1)Is it possible to effectively detect the malicious usage of each TTP? 2)In the previous case, do the TTP features help? 3)Does detecting the malicious usage of TTPs, also help solve the simpler problem of determining the maliciousness of the overall sample? 4)In the previous case, do the TTP features help? Towards this end, we comparatively evaluate our dif- ferent modelsSTTP2STTP,S2STTP,STTP2SandS2Son the same 5 datasets used in [31]. In the following sub- sections, we first provide a detailed description of the datasets, highlighting some of their characteristics. Then, we present the metrics used in our evaluation. Finally, we present the results of our experimental evaluation, addressing the first two questions in Subsection 4.3, and the third and fourth questions in Subsection 4.4. 4.1. Datasets The datasets consist of a varied population of mal- ware and benign samples from the five publicly available datasets represented in Table 3. Of these five datasets, only one, namely âEmber Benignâ, contains benign samples. Starting from the datasets in Table 3, five new datasets have been constructed by randomly sampling the five TABLE 4: Statistics about the newly-derived datasets. Average number and percentage of matched samples per TTP, across the train/validation/test sets. Column #s is the number of matched samples. The Total number of TTPs does not include Tnull, since Tnull represents the samples that do not match any TTPs. Train(tot. samples = 186183)Validation(tot. samples = 62060)Test(tot. samples = 62061) BenignMaliciousBenignMaliciousBenignMalicious #s%#s%#s%#s%#s%#s% T15570.00.00%0.40.00%0.00.00%0.00.00%0.00.00%0.20.00% T113524.00.03%362.20.39%8.00.03%114.20.37%8.00.03%119.60.39% T112461962.666.56%71892.677.23%20654.266.56%23935.877.14%20654.266.56%23976.277.27% T10719.00.01%46.80.05%3.00.01%14.20.05%3.00.01%14.80.05% T1590 2.40.00%0.20.00%0.80.00%0.00.00%0.80.00%0.00.00% T15630.00.00%0.00.00%0.00.00%0.00.00%0.00.00%0.00.00% T110597.20.10%42.40.05%32.40.10%12.40.04%32.40.10%11.40.04% T10901.20.00%44.60.05%0.40.00%10.40.03%0.40.00%15.00.05% T15500.60.00%10.80.01%0.20.00%3.60.01%0.20.00%3.60.01% T15700.00.00%0.00.00%0.00.00%0.00.00%0.00.00%0.00.00% T157171584.276.90%86088.892.48%23861.476.90%28652.692.34%23861.476.90%28683.892.44% T10533.00.00%0.80.00%1.00.00%0.20.00%1.00.00%0.00.00% T102111.40.01%28.60.03%3.80.01%8.40.03%3.80.01%8.60.03% Tnull21548.023.15%6986.07.50%7045.022.70%2416.07.79%7178.023.13%2299.07.41% Total133695.6158518.244565.252751.844565.252833.2 original ones. Each of the five new datasets has then been split into training (60% of the samples), validation (20% of the samples), and test (20% of the samples) sets. These new datasets share the following properties:(i)they are balanced at the sample level, meaning that they contain a balanced number of malicious and benign samples,(i) they are split ensuring that they have pairwise disjoint test sets,(i)they are as large as possible given the previous two requirements. Though these new datasets should share the same distributions of samples, by considering five (and not just a single dataset) we are able to compute the mean and standard deviation of our results, thus minimising the risk of skewed results due to particular distributions in the training, validation, or test sets. Table 4 describes the mean statistics of our data across all five of our training, validation, and test sets. It also shows for each individual TTP: â˘The number of malicious and benign samples that contain the TTP. â˘The percentage of malicious and benign samples that contain the TTP. When we look at the last row of Table 4 which shows the total number of samples that match a TTP (excluding samples that do not match any TTP), we observe that there isnât a significant difference between the number of benign and malicious samples with TTPs (âź15% differ- ence). Further, notice that the average number of TTPs matched by benign samples is 1.4, while it is 1.7 for malicious ones. The second observation is that the dataset is highly unbalanced at the TTP level: T1124 and T1571 occur in most of the malicious and benign samples, while most of the others occur rarely in the datasets, with T1557, T1590, T1563, T1570, T1053 not occurring at least once on average in our train/validation/test sets. This implies that our models cannot always be prop- erly trained/validated/tested on the labels corresponding to such TTPs as there are no samples with such labels in the corresponding train/validation/test sets. For this reason, we do not consider the labels associated with T1557, T1590, T1563, T1570, T1053, and only consider the 9 labels T1135, T1124, T1071, T1105, T1090, T1550, T1571, T1021, each associated to the corresponding TTP, plus the label Mal-Sample corresponding to the entire sample. Finally, while the most frequent TTPs often occur in malicious samples, the opposite happens for some other TTPs, like T1105 and T1053. 4.2. Metrics Given theith labelA i (representing either a TTP or the label Mal-Sample associated to the sample), we assume 1)P i andN i denote the sets of malicious and benign samples with labelA i (and thus|P i |+|N i |=n wherenis the number of samples), and 2)P i andPN i denote the sets of predicted mali- cious and predicted benign samples respectively, with labelA i (and thus also|P i |+|PN i |=n). Then, all the metrics are defined on the basis of the confusion matrixrepresented by the different values for P i ,N i ,P i andNN i of the different labels. Considering theith labelA i , 1)theTrue Positive setTP i is the set of samples which are predicted to be positive (malicious) and are actually positive (malicious): TP i =P i âŠP i . 2)theFalse Positive setFP i is the set of samples which are predicted to be positive (malicious) and are actually negative (benign): FP i =N i âŠP i . 3)theTrue Negative setTN i is the set of samples which are predicted to be negative (benign) and are actually negative (benign): TN i =N i âŠPN i . TABLE 5: Average and standard deviation of the results obtained by runningSTTP2STTPandS2STTPon the five datasets, detecting the malicious TTPs. For each metric, theâ(resp.â) indicates that the higher (resp. lower) the result, the better the performance. For each pair of results, the best is in bold. STTP2STTPS2STTP F 1 -score (â)0.835Âą0.0030.826Âą0.003 Accuracy (â)0.972Âą0.0000.970Âą0.000 Subset Accuracy (â)0.841Âą0.0020.831Âą0.002 Precision (â)0.953Âą0.0020.951Âą0.002 Recall (â)0.744Âą0.0050.730Âą0.005 FPR (â)0.003Âą0.0020.004Âą0.000 4)theFalse Negative setFN i is the set of samples which are predicted to be negative (benign) and are actually positive (malicious): FN i =P i âŠPN i . On the basis of the confusion matrix of each label, many metrics have been defined and used to evaluate model performance in multi-label and binary classification prob- lems. A standard way to present the different metrics is to distinguish between âmicroâ and âmacroâ average scores, the latter usually distinguished by appending the word âMacroâ in front or after the name of the metric. In the micro case, each metric groups the classifi- cations by sample across all labels, and then calculates the overall metric, thus giving each prediction equal im- portance. Hence favouring the labels with highsupport, defined as the number of samples having a particular label. In contrast, each macro metric calculates the average of the metric across all classes, thus giving each label equal importance, irrespective of their support. In general, macro metrics are better suited in highlighting differences in per- formance when considering highly unbalanced datasets, as is our case when looking at the data at the TTP-level. We use the following micro and macro scores to present our results. See, e.g., [24] for a more in depth presentation of such metrics, including a study about the correlations existing among them. 4.2.1. Micro-average metrics. If we defineTP= P i |TP i |,FP= P i |FP i |,TN= P i |TN i |,FN= P i |FN i |,P= P i |P i |,N= P i |N i | andPP= P i |P i |the following (micro-average) met- rics are among the most commonly used: 1)F 1 -score (F 1 ), it is the harmonic mean of Preci- sion and Recall, defined as: F 1 = 2PrRe Pr+Re = 2TP 2TP+FP+FN = 2TP P+P . 2)Accuracy (Acc), it is defined as the ratio between the number of correctly predicted labels to the total number of predictions: Acc= TP+TN TP+TN+FP+FN = TP+TN P+N . 3)Precision (Pr), it is the proportion of the correctly predicted positive labels, i.e., the ratio between the correctly predicted labels and the total num- ber of predicted labels: Pr= TP TP+FP = TP P . 4)Recall (Re), it is the ratio between the correctly predicted labels and the total number of actual labels: Re= TP TP+FN = TP P . 5)False Positive Rate (FPR), it is the proportion of the positive cases that were incorrectly identified: FPR= FP FP+TN = FP N . Another standard metric used in multi-label classification, defined directly on the basis of the ground truthy i and the predicted labelsËy i of each samplex i , is Subset Accuracy. Subset Accuracy (SA)is defined as the percentage of samples that have all their labels classified correctly: SA= 1 n n X i=1 1(y i = Ëy i ) wherenis the total number of samples and1(.)is the in- dicator function returning 1 if the condition in parenthesis is true, and 0 otherwise. 4.2.2. Macro-average metrics. In multi-label classification problems, macro-average met- rics are computed by averaging the corresponding micro- average metric as defined in Section 4.2.1. So, ifNis the number of labels,MacroF 1 -scoreis defined as: MacroF 1 -score= 1 N X i 2|TP i | 2|P i |+|P i | , Macro Precisionas: Macro Precision= 1 N X i |TP i | |P i | , andMacro Recallas: Macro Recall= 1 N X i |TP i | |P i | . In our caseN= 9, corresponding to the 8 TTPs appearing on average at least once in our train/validation/test sets, plus the label Mal-Sample associated to the entire sample. 4.3. Evaluating Malicious Usage of TTPs Our first experiment tests the effectiveness of detecting malicious usage of TTPs and assesses the extent to which providing TTP features is helpful. We thus consider our STTP2STTPandS2STTPmodels with the following hyper- parameters: 1250 hidden dimensions, dropout rate of 0.4, learning rate of1.6Ă10 â4 , and weight decay of3.05Ă 10 â6 . All the models are trained using mini-batching (with a batch size of 750) and Adam optimizer [16]. The results for the different metrics are in Table 5, where the best results are highlighted in bold. Looking at the table, we observe that, as expected, pro- viding TTP features consistently helps across all metrics. TABLE 6: Average of the scores corresponding to each label. For each metric, theâ(resp.â) indicates that the higher (resp. lower) the result, the better the performance. For each pair of results, the best is in bold. For each label, column Support reports the number of (malicious) samples with that label. STTP2STTPS2STTP F 1 (â)Precision (â)Recall (â)F 1 (â)Precision (â)Recall (â)Support T11350.9620.9860.9400.8720.8840.866119.6 T11240.8500.9520.7680.8380.9500.75223976.2 T10710.6560.7420.5900.5660.8060.45814.8 T11050.6860.6600.7260.0000.0000.00011.4 T10900.9460.9720.9220.0780.6000.04415.0 T15500.1600.2000.1340.0000.0000.0003.6 T15710.8400.9520.7520.8300.9540.73628683.8 T10210.0000.0000.0000.3100.5940.2368.6 Mal-Sample0.8200.9520.7180.8100.9500.70831031.0 Macro avg0.6580.7130.6170.4780.6380.4229318.2 TABLE 7: Average and standard deviation of the results obtained by running each of the four models on the five datasets, to detect malicious samples. For each metric, theâ(resp.â) indicates that the higher (resp. lower) the result, the better the performance. For each metric, the best result is in bold, and the second best is underlined. The last row reports the average ranking. STTP2STTPS2STTPSTTP2S2S F 1 -score (â)0.820Âą0.0020.811Âą0.0030.811Âą0.0020.802Âą0.004 Accuracy (â)0.842Âą0.0020.835Âą0.0020.834Âą0.0010.827Âą0.004 Precision (â)0.952Âą0.0020.951Âą0.0020.939Âą0.0050.949Âą0.005 Recall (â)0.720Âą0.0040.708Âą0.0040.715Âą0.0040.700Âą0.002 FPR (â)0.037Âą0.0020.037Âą0.0020.047Âą0.0040.045Âą0.006 Avg ranking1.12.23.13.6 Secondly, the absolute value is very positive for each met- ric, and in particular, it is remarkably high for Accuracy (â0.97, meaning that more than 97% of the predictions are correct) and Precision (â0.95, meaning that more than 95% of the positive predictions are correct). We obtain the âlowestâ value for the Recall metric (â0.74, meaning that our models are able to detect roughly three out of four of the positive labels, each representing maliciousness). Next, we see that though the TTP information helps, the difference between the values associated with each metric does not appear to be remarkably high if we consider the (micro) statistics in Table 5. This fact underlines that, assuming we are interested in maximising such micro- average scores, it is possible to not provide the TTP features as input and let the model learn how to detect and properly classify each TTP by itself, with a relatively low decay in the micro-average performance. However, a careful analysis of the Precision, Recall, andF 1 -score of each label, reported in Table 6, reveals that: 1)TTP features are beneficial in all cases, except for T1021. 2)The contribution of the TTP features is relatively modest for the highly occurring TTPs (T1124 and T1571), and significantly higher for many of the others. For instance, if we consider T1105, theF 1 -score forSTTP2STTPis 0.686, and 0 for S2STTP. Such facts are reflected by the macro-average scores reported in the last line of Table 6, where we see huge improvements across all metrics: 1)MacroF 1 -score jumps from 0.478 to 0.658, with an increase of 37.55%. 2)Macro Precision goes from 0.638 to 0.713, with an increase of 11.82%. 3)Macro Recall jumps to 0.617 from 0.422, with the highest increase of 46.05%. In Section 5.1 we will see how such positive results can further be improved by fine-tuning the thresholdθ associated with each label to decide the maliciousness of the corresponding TTP and/or sample. As a final consideration, we remark that all the statis- tics reported in Tables 5 and Table 6 take into account the metrics associated with all the labels, including the label Mal-Sample associated with the entire sample. This is standard practice in both multi-label classification prob- lems and hierarchical multi-label classification problems (see, e.g., [39]). Further, if we do not count the outputs of Mal-Sample, the statistics in Table 5 and the macro- average statistics in Table 6 would only be moderately affected. Indeed, the Precision and Recall of the label Mal- Sample associated with the sample (in Table 6) are very close to the overall Precision and Recall in Table 5, and consequently the twoF 1 -scores of Mal-Sample computed bySTTP2STTPandS2STTP(equal toâ0.82andâ0.81, respectively) are also similar to the overallF 1 -score in Table 5 (equal toâ0.83). 4.4. Evaluating Sample Detection We now want to evaluate how the detection of TTPs and their classification as malicious has an impact on the 0.10.20.30.40.50.60.70.80.9 Threshold 0.0 0.2 0.4 0.6 0.8 1.0 F1-score 0.10.20.30.40.50.60.70.80.9 Threshold 0.0 0.2 0.4 0.6 0.8 1.0 Precision 0.10.20.30.40.50.60.70.80.9 Threshold 0.0 0.2 0.4 0.6 0.8 1.0 Recall Figure 1: Average and confidence intervals ofF 1 -score, Precision and Recall for TTP T1105 while varying the threshold θof the TTP label. TABLE 8: Precision, Recall andF 1 -score at different thresholds with theSTTP2STTPmodel.θ max is the threshold in between [0.1,0.9], step 0.1, for which we get the maximum resulting value. Forθ= 0.5we report the same values of Table 5 in order to make comparison easier. F 1 scorePrecisionRecall θvalueθ max valueinc.θvalueθ max valueinc.θvalueθ max valueinc. T1135:0.50.9630.40.9670.4%0.50.9880.80.9940.7%0.50.9400.10.9824.4% T1124:0.50.8500.40.8510.1%0.50.9520.90.9792.9%0.50.7680.10.99830.1% T1071:0.50.6560.30.7169.2%0.50.7410.30.89120.3%0.50.5910.10.82038.7% T1105:0.50.6870.40.7215.0%0.50.6600.60.6904.6%0.50.7260.10.89022.6% T1090:0.50.9470.60.9530.7%0.50.9730.60.9871.4%0.50.9220.10.9462.6% T1550:0.50.1600.10.652307.7%0.50.2000.10.610205.0%0.50.1330.10.771478.6% T1571:0.50.8410.40.8420.2% 0.50.9540.90.9802.8%0.50.7520.10.99832.8% T1021:0.50.0000.10.1410.50.0000.10.0780.50.0000.10.793 Mal-Sample0.50.8200.40.8210.2%0.50.9520.90.9802.9%0.50.7190.10.99838.7% Macro avg0.6580.74112.5%0.7130.79912.0%0.6170.91147.6% classification of the entire sample as either malicious or benign, and the impact of providing TTP features. We implementSTTP2SandS2Swith the same configuration and hyper-parameters asSTTP2STTPandS2STTP. The results obtained by running all the models (i.e., STTP2SandS2Sas well) are reported in Table 7, where the best results are highlighted in bold, while the second best are underlined. The last line reports the average ranking, obtained by ranking the performance of each system on a metric and then averaging the results. In the case of ties between models, average ranks are assigned to such models (see [6] for more details). Considering the table, the first observation is again thatSTTP2STTPhas the best performance across all the different metrics. Further,STTP2STTPalmost always has the most stable performance of the lot, as measured by the standard deviation: the only exception to the rule is for Recall, for whichSTTP2STTPhas a standard deviation of 0.004, equal to the standard deviation ofS2STTPand STTP2S, but higher than that ofS2S. Then, the second most performing model isS2STTP, which underlines the fact that the precise detection and classification of TTPs improves the overall performance of the corresponding binary classifiers. This result may look surprising given that(i)bothSTTP2STTP(resp.S2STTP) has been trained on the same dataset used for trainingSTTP2S(resp.S2S), and(i)STTP2STTPandS2STTPare actually solving a more complex problem than their corresponding models STTP2SandS2S, namely identifying malicious usage of TTPs and determining the maliciousness of the overall sample. However, by having a separate label for each TTP, the neural network has now the possibility to focus on the detection of the malicious samples containing rarely used TTPs, and classify them accordingly. The relatively good performance ofS2STTPeven when used as a binary classifier confirms that(i)it is possible to not provide TTP features as input,(i)have the model learn how to detect the malicious usage of TTPs, and(i)still not get a huge performance decay. Additionally, modelSTTP2Shas better performance than ofS2S, confirming the fact that providing TTP infor- mation improves the modelâs performance, as also high- lighted by the better performance ofSTTP2STTPcompared toS2STTP. Overall, the main results of this experimental analysis are that: 1)Teaching the neural network to detect the mali- cious usage of TTPs helps to improve the per- formance, as demonstrated by the good perfor- mances obtained bySTTP2STTPandS2STTP, 2)Providing TTP features improves performance, as demonstrated by the performance ofSTTP2STTP vsS2STTPandSTTP2SvsS2S, and 3)Teaching the neural network to detect the mali- cious usage of TTPs and providing TTP features, leads to the best overall results, as demonstrated bySTTP2STTPand its results across all metrics. 5. Experimental Analysis In this Section, we present the results of further ex- perimental analysis showing the results of(i)tuning the label decision thresholds inSTTP2STTP(Subsection 5.1), (i)trainingSTTP2STTPandS2STTPwith fewer samples (Subsection 5.2), and(i)testingSTTP2STTPandS2STTP with samples in which benign flows have been injected in the malicious samples in an attempt to camouflage their malicious activity (Subsection 5.3). 5.1. Threshold Variation per TTP The already positive performances ofSTTP2STTPre- ported in Section 4.3, can be further improved by tuning the decision thresholdθassociated to each label and above which the corresponding TTP is predicted to have been maliciously used. For instance, if we consider TTP T1105 and plot itsF 1 -score, Precision and Recall scores while varying its threshold, we obtain the graphs in Figure 1. As it can be observed, T1105F 1 -score peaks at some value in between(0,1)which is not necessarily equal to0.5, which is the standard and intuitive threshold we used in our experiments. Indeed, we can expectF 1 -scores to have a low value when the threshold is (extremely) low or (extremely) high. For instance, considering T1105, the averageF 1 -score across the five datasets is equal to 11.4/62061â2Ă10 â4 whenθ= 0(i.e., equal to the ratio between its average support, and the average number of samples in the test set), and equal to 0 whenθ= 1 (as in this case no sample is predicted to maliciously use T1105). At the standard settingθ= 0.5, theF 1 -score for T1105 is 0.686, which raises to 0.721 (5.1% improvement) ifθis set to 0.4. A similar behaviour can be observed for T1105 Precision, which peaks at0.6. Indeed, forθ= 0, every sample is positively predicted and thus the Precision is equal to the ratio between the support of the label and the number of samples in the test set, which for T1105 is equal to11.4/62061â2Ă10 â4 on average. Then, we can expect the Precision to increase withθunder the assumption that the value returned by the neural network reflects the probability of the TTP being maliciously used. However, as the figure shows, it may not be the case that Precision is ensured to be monotonically increasing as, in the limit case ofθ= 1, no labels are predicted and thus the Precision gets the undefined value0/0. On the contrary, Recall is bound to monotonically decrease from value 1 forθ= 0to value 0 forθ= 1, as it is indeed the case for the Recall of T1105. All such considerations are reflected by the plots in Figure 1, which also shows the confidence interval of each metric that, for Precision, gets relatively high asθapproaches the value 1. Table 8 reports the best result for Precision, Recall, andF 1 -score for the various labels while varying their respective thresholds. As it can be observed, substantial improvements can be obtained for either Precision or Recall orF 1 -score, which one depending on the particular metric we are interested in for each single TTP. The substantial improvements are highlighted by the macro- average metrics reported in the last line: 1)MacroF 1 -score goes from 0.658 to 0.741, with a 12.5% increase, 2)Macro Precision goes from 0.713 to 0.799, with a 12.0% improvement, and 3)Macro Recall jumps to 0.911 from 0.617, with the highest 47.6% improvement. Of course, such a fine-tuning at the TTP level is also possible forS2STTP, while it is not forSTTP2SandS2S. 20406080100 Percentage of Data Points Used 0.805 0.810 0.815 0.820 0.825 0.830 0.835 0.840 F1-score (Micro) Without TTP Features With TTP Features (a)F 1 -score 20406080100 Percentage of Data Points Used 0.810 0.815 0.820 0.825 0.830 0.835 0.840 Subset Accuracy Without TTP Features With TTP Features (b) Subset Accuracy 20406080100 Percentage of Data Points Used 0.92 0.93 0.94 0.95 Precision (Micro) Without TTP Features With TTP Features (c) Precision 20406080100 Percentage of Data Points Used 0.71 0.72 0.73 0.74 0.75 Recall / True Positive Rate (Micro) Without TTP Features With TTP Features (d) Recall 20406080100 Percentage of Data Points Used 0.0035 0.0040 0.0045 0.0050 0.0055 0.0060 0.0065 0.0070 0.0075 False Positive Rate Without TTP Features With TTP Features (e) FPR Figure 2: Performance of the models when varying the size of the training set. On thex-axis the percentage of randomly sampled datapoints from the training set can be found. 0246810 Noise Ratio 0.2 0.3 0.4 0.5 0.6 0.7 0.8 F1-score (Micro) Without TTP Features With TTP Features (a)F 1 -score 0246810 Noise Ratio 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 Subset Accuracy Without TTP Features With TTP Features (b) Subset Accuracy 0246810 Noise Ratio 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Precision (Micro) Without TTP Features With TTP Features (c) Precision 0246810 Noise Ratio 0.2 0.4 0.6 0.8 1.0 1.2 Recall / True Positive Rate (Micro) Without TTP Features With TTP Features (d) Recall 0246810 Noise Ratio 0.00 0.02 0.04 0.06 0.08 0.10 0.12 0.14 False Positive Rate Without TTP Features With TTP Features (e) FPR Figure 3: Performance ofSTTP2STTPandS2STTPwhen tested with varying percentages the noise ratio, defined as the ratio between the number of injected benign flows and the number of flows in the original sample. 5.2. Data Scarcity One of the known problems of machine learning models is that their performance is very sensitive to the amount of data used for training. However, we expect the modelSTTP2STTPto be less dependent on the amount of data used for training, thanks to the injection of the TTP information. Thus, the existing gap between the per- formance obtained withSTTP2STTPandS2STTPshould either remain unvaried or increase as we train with less data. To test our hypothesis, we consider each of the five datasets used for training our models and randomly sample it in order to obtain 5 new datasets, each containing either 5%, or 10%, or 25%, or 50% or 75% of its datapoints. Thus, for each percentage, we have 5 new sets of samples, that we use to train bothSTTP2STTPandS2STTP, which are then tested on the same test sets used in Section 4. The results obtained according to different metrics are shown in Figure 2, in which we also consider the performance of the models when trained with 100% of the samples. We also plot the 95% confidence interval for each result, which is represented by the shading around each line in the figure. Considering the figures, we see that: 1)as expected,STTP2STTPhas better performance thanS2STTPwhen consideringF 1 -score, Subset Accuracy and Recall, 2)even for such metrics the gap between the two models varies depending on the percentage of data points used for training, while 3)for Precision and False Positive Rate, for some percentages,S2STTPhas better performance than STTP2STTP, though this happens only for very few points. Such behaviours can be explained by considering again the average characteristics of our five datasets reported in Table 4. Indeed, the labels with low support in the original datasets will very likely have correspondingly even lower support in the new sets used for training, as the percentage used for sampling decreases. This likely corresponds to a corresponding degradation of the performance, which is (almost always) reflected by the plot in the figure, and which we expect to be particularly the case for the labels with low support. At the same time, the micro-average scores in the Figure, favouring the highly supported labels, will cause the neural network to reward the still good performance it can get (even when training on 5% of the datasets) for the labels T1124, T1571 and Mal-Sample. Indeed, when sampling with the 5% percentage, 1)T1124, T1571 and Mal-Sample are expected to have, on average, 3594.63, 4304.44 and 4654.65 as support in the dataset used for training, 2)all the other ones (except for T1135 which still should have a supportâ18.11) should have support smaller than 3, and 3)the micro-average scores of all the labels different from T1124, T1571 and Mal-Sample are in the vast majority of cases equal to 0, even when given the TTP information. Thus, in many cases, when considering the performance of STTP2STTPandS2STTPtrained on smaller datasets, they are mostly considering just the labels T1124, T1571, and Mal-Sample, each of them with relatively high support in the training set even when considering 5% of the samples, thus allowing alsoS2STTPto recognise the malicious usage of the TTPs and the sample maliciousness. 5.3. Adversarial Benign Flow Injection In this last experiment, we consider the scenario in which a malicious attacker tries to camouflage his ma- licious activity by doing various other benign activities within the same traffic sample. To test the robustness of bothSTTP2STTPandSTTP2Sto such scenario, 1)we consider benign traffic flows extracted from the Ember Benign dataset in Table 3, 2)to each malicious sample, we add a varying per- centage of benign flows, starting from 0% up to 1000% with step 100%, and 3)we testSTTP2STTPandS2STTPperformance without retraining such systems. We measure the activity of the attacker in camouflaging its malicious activity in terms of the Sample Noise Ratio, defined as the ratio of injected benign flows to malicious flows (thus, a sample noise ratio equal to 2 means that each malicious sample withfflows has been added with 2fbenign flows). Indeed, we expect that on such new test sets, bothSTTP2STTPandS2STTPshould have a decay in performance given that the newly generated samples have different characteristics from the ones of the samples used for training both systems. The research questions we want to investigate are(i)whether the TTP information still helpsSTTP2STTPin detecting the malicious usage of the TTPs,(i)to what extentSTTP2STTPandS2STTP performance decay, and(i)whether the performance gap betweenSTTP2STTPandS2STTPincreases or decreases with the sample noise ratio. Figure 3 shows the plots for theF 1 -score, Subset Accuracy, Precision, Recall, and False Positive Rate, in their micro-average form. On thex-axis there is the used sample noise ratio. As it can be observed from the figures, the results confirm our expectations: 1)STTP2STTPhas always better performance than S2STTP, 2)the performance gap betweenSTTP2STTPand S2STTPincreases with a rather steep gradient up to a certain point after which the difference becomes roughly constant or only marginally in- creasing with the sample noise ration, and 3)both models have a large standard deviation, usuallyS2STTPlarger thanSTTP2STTPwith the exception for Recall for which the opposite hap- pens. As we said, such behaviours were somehow expected. The only relatively surprising fact is that the scores tend to stabilise when the number of injected flows is 200%. However, this fact can be easily explained by looking at the plot for Subset Accuracy, which measures the number of samples that are exactly labelled by the models. As it can be seen, from noise ratio 2 on, bothSTTP2STTP andS2STTPhave an (almost) constant Subset Accuracy, implying that in the original testing set there are: 1)malicious samples whose traffic flow (even when represented as a bag of flow) still make the detection of the used TTP robust to the amount of noise added, even when not using the TTP information, and 2)the number of such samples becomes bigger when using TTP information. Such analysis is confirmed by computing the macro- average metrics which have exactly the same behaviour of the micro-average metrics in Figure 3. 6. Discussion Consistent Improvement over the State-of-the-Art.Our results highlight the consistency of our systemâs improve- ment over the state-of-the-art when it comes to detecting malware with the help of TTP features. Both versions of our system based on our multi-label DNN architecture (STTP2STTPandS2STTP) clearly outperform the binary DNN systems (STTP2SandS2S). This further highlights the power of our novel multi-label DNN architecture, since it enables us to outperform models likeSTTP2Sthat utilise TTP features, even when using a model likeS2STTP which does not utilise TTP features. Explainability and Precision in Detecting Malware. Our overall malware detection is not only better but also much more precise, as a result of our system being able to identify the malicious usage of each TTP individually. Instead of a binary output indicating the malicious or benign nature of a sample, our system can additionally identify which particular behaviours, as characterised by specific TTPs, were responsible for a malicious classi- fication. Thus, making the output not only precise, but also more explainable to the user. For instance, the mean Accuracy ofSTTP2STTPover all our datasets increases by 0.83%, 0.95%, and 1.81% when compared toS2STTP, STTP2S, andS2S. Recognition of rarely-occurring TTPs.ML-based mal- ware detection systems rely on ample amounts of repre- sentative training data in order to reliably detect similar or perhaps unseen malware. Similarly, ML-based systems that rely on TTPs also require training on a significant number of datapoints for each TTP in order to later identify them. As a result, unlike other ML-based sys- tems where TTPs with only a few datapoints might be overlooked during model training, our system can also accurately identify malicious behaviour using TTPs which only have a few datapoints present in the training data. As shown in our experiments, our system can identify malware using TTPs with as few as 15 datapoints with an averageF 1 -score of 0.946, as in the case of T1090. System Configurability.A key part of our system design is that it can detect malicious usage of individual TTPs, thus allowing a user to choose the specific TTPs most rel- evant to their threat model. Additionally, we demonstrate that we can tune the detection thresholds for each TTP in our system, to enable even greater granularity of detection based on specific requirements. For instance, the threshold for a rarely-occurring but high-risk TTP can be lowered in a given environment, while a commonly-utilised TTP which does not pose a significant risk can have its detec- tion threshold increased, thus increasing and reducing the number of positively detected samples respectively. Our results across our datasets highlight the benefits of this configurabilityâresulting in an average improvement of Macro F1-score by 12.5%, Macro Precision by 12%, and Macro Recall by 47.6%. Utility of TTPs in Malware Detection.Our experi- mental analysis highlights the utility of TTP informa- tion in improving malware detection capabilities. When utilising our system with TTPs (STTP2STTP), we see an averageF 1 -score improvement of 1.08% at the sample level and 37.65% at the TTP-level, as compared to our system without TTPs (S2STTP). This can further be seen when comparing results at just the sample level (i.e., bi- nary classification)âon averageSTTP2STTPoutperforms S2STTP,STTP2S, andS2Sacross every metric (F 1 -score, Accuracy, Precision, Recall, and FPR)âthus showing the utility of TTPs in improving standard malware detection capabilities. Data Scarcity.We find that even when utilising limited training data, the addition of TTP features in our system architecture helps improve overall performance. Across the 30 experiments we performed (5 datasets x 6 train- ing data percentages)STTP2STTPoutperformsS2STTP 96.67% of the time according to theF 1 -score. The ad- dition of TTPs significantly improves the capability of the system even when using much less training data. For instance,STTP2STTPutilising 25% of the training data has a higher averageF 1 -score thanS2STTPutilising 100% of the training data, andSTTP2STTPutilising 10% of the training data has a higher average Recall thanS2STTP utilising 100% of the training data. Resilience to Adversarial Attack.Our experimental analysis also evaluates the capability of our system to resist adversarial attack, specifically camouflaging attacks, where an adversary deliberately injects benign flows into a malicious sample in order to camouflage its malicious activity. We measure this in terms of the Sample Noise Ratio (ratio of injected benign flows to malicious flows), and find that while system performance decreases as expected when the Noise Ratio increases,STTP2STTPis consistently able to perform better under these conditions thanS2STTP. Indeed, from an average improvement of 1.21% inF 1 -score when the Noise Ratio is 0.0, we see an average improvement of up to 28.93% inF 1 -score when the Noise Ratio is 10.0. Similarly, we observe a consistent average improvement across every metric in the noisiest situationâ62.17% in Subset Accuracy, 48.71% in Precision, 14.50% in Recall, and 20.23% in terms of FPR. Minimal Additional Data.Our approach also highlights the minimal data requirements for achieving these sig- nificant performance gains. The TTPs we utilise in our system, as described in the MITRE ATT&CK framework, are open-source information, and our procedure to cre- ate TTP features using them does not require additional training data. Since these features can be created directly from existing data, utilising open-source domain knowl- edge regarding adversarial behaviour, it provides a vital performance benefit for the data-greedy machine learning algorithms. Different Levels of Granularity.Our approach allows us to report the maliciousness of each sample at two levels of granularity which, as explained in [3], allows for different fine-grained outputs that can be used under divergent levels of expertise. In our case, we have just two levels (sample and TTPs) but it is of course possible to(i)group the TTPs together according to the tactical goal of the attacker,(i)subdivide each technique into the various sub-techniques it may include, and(i)also include the procedures used to implement the technique or sub-technique. This would correspond to a five-level hierarchy reflecting the TTP ontology described in the MITRE ATT&CK framework, which, if incorporated into our model (thus having one hierarchical label per tactic, technique and procedure, plus the label for sample mali- ciousness) would give our system the ability to(i)report a more fine-grained output, and(i)recognise a malicious activity at different levels of granularity, each with its own level of confidence. For instance, our system would be able to recognise a sample as malicious, e.g., at the tactical level while failing to recognise the specific technique and sub-technique or procedure being used: though not optimal, this would still be better than just reporting that the sample is malicious, or incorrectly detect the malicious usage of a specific technique and/or sub-technique and/or procedure. 7. Conclusions In this paper, we have designed a novel multi-label DNN architecture that detects malicious behaviour at two different levels of granularityâfor the TTPs and the over- all sample. This unique ability allows us to improve the overall performance of malware detection, improve on the state-of-the-art when it comes to detecting malicious behaviour when exploiting TTPs, and identifying hard-to- detect malware which utilises uncommon or rarely-used TTPs. We back up our findings with an extensive com- parative analysis, using a dataset of over 1.5 million real- world malware and benign samples. Lastly, we test the ro- bustness of our system in various challenging conditions, such as when relying on limited training data or subjected to adversarial camouflage attacks, and demonstrate that it continues to outperform other comparable systems. As discussed, our system requires a minimal amount of additional data when compared to traditional models for detecting malicious activities and is able to provide precise outputs about which TTPs being used maliciously used. This opens up the possibility of future extensions in which the model will have a richer hierarchical representation of the TTPs, thus allowing for a finer level of granularity when it comes to the malware detection. Data Availability All the materials necessary to reproduce this work, including source code and data, will be made available once the paper is published. References [1]Rawan Al-Shaer, Jonathan M Spring, and Eliana Christou. Learn- ing the Associations of MITRE ATT&CK Adversarial Techniques. InProc. of CNS, 2020. [2]Hyrum Anderson and Phil Roth. EMBER: An Open Dataset for Training Static PE Malware ML Models, 2018. [3]D. Chang, K. Pang, Y. Zheng, Z. Ma, Y. Song, and J. Guo. Your âflamingoâ is my âbirdâ: Fine-grained, or not. In2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11471â11480, 2021. [4]Veronica Chierzi and Fernando Merc Ë es. Evolution of IoT Linux Malware: A MITRE ATT&CK TTP Based Approach. IneCrime Symposium, 2021. [5]Corinna Cortes and Vladimir Vapnik. Support-vector networks. Mach. learn., 1995. [6]Janez Demsar. Statistical comparisons of classifiers over multiple data sets.J. Mach. Learn. Res., 7:1â30, 2006. [7]Jeffrey Fairbanks, Andres Orbe, Christine Patterson, Janet Layne, Edoardo Serra, and Marion Scheepers.Identifying ATT&CK Tactics in Android Malware Control Flow Graph Through Graph Representation Learning and Interpretability. InIEEE Big Data, 2021. [8]Lei Feng, Bo An, and Shuo He.Collaboration based multi- label learning. InThe Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27 - February 1, 2019, pages 3550â3557. AAAI Press, 2019. [9]Daniel Gibert, Carles Mateu, and Jordi Planes. The rise of machine learning for detection and classification of malware: Research developments, trends and challenges.Journal of Network and Computer Applications, 2020. [10] Eleonora Giunchiglia, Fergus Imrie, Mihaela van der Schaar, and Thomas Lukasiewicz.Machine learning with requirements: a manifesto.CoRR, abs/2304.03674, 2023. [11] Eleonora Giunchiglia and Thomas Lukasiewicz. Coherent hier- archical multi-label classification networks. In Hugo Larochelle, MarcâAurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors,Advances in Neural Information Pro- cessing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. [12] Wajih Ul Hassan, Adam Bates, and Daniel Marino.Tactical provenance analysis for endpoint detection and response systems. InIEEE S&P, 2020. [13] Tin Ho. The random subspace method for constructing decision forests.IEEE Pattern Anal. Mach. Intell., 1998. [14] Ghaith Husari, Ehab Al-Shaer, Mohiuddin Ahmed, Bill Chu, and Xi Niu. TTPDrill: Automatic and Accurate Extraction of Threat Actions From Unstructured Text of CTI Sources.InProc. of ACSAC, 2017. [15] Christopher M Inacio and Brian Trammell. YAF: Yet Another Flowmeter. InLISA, 2010. [16] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017. [17] Aditya Kuppa, Slawomir Grzonkowski, Muhammad Asghar, and Nhien Le-Khac. Finding rats in cats: Detecting stealthy attacks using group anomaly detection. InTrustCom. IEEE, 2019. [18] Donghwoon Kwon, Hyunjoo Kim, Jinoh Kim, Sang C. Suh, Ikkyun Kim, and Kuinam J. Kim. A survey of deep learning-based network anomaly detection.Clust. Comput., 2019. [19] Valentine Legoy, Marco Caselli, Christin Seifert, and Andreas Peter. Automated Retrieval of ATT&CK Tactics and Techniques for Cyber Threat Reports.arXiv:2004.14322, 2020. [20] Ritesh K. Malaiya, Donghwoon Kwon, Sang C. Suh, Hyunjoo Kim, Ikkyun Kim, and Jinoh Kim. An empirical evaluation of deep learning for network anomaly detection.IEEE Access, 2019. [21] Warren McCulloch and Walter Pitts. A logical calculus of the ideas immanent in nervous activity.The bulletin of mathematical biophysics, 1943. [22] Otgonpurev Mendsaikhan, Hirokazu Hasegawa, Yukiko Yam- aguchi, and Hajime Shimada. Automatic Mapping of Vulnerability Information to Adversary Techniques. InSecureWare, 2020. [23] Kris Oosthoek and Christian Doerr. SoK: ATT&CK Techniques and Trends in Windows Malware. InSecurity and Privacy in Communication Networks, 2019. [24] Rafael B. Pereira, Alexandre Plastino, Bianca Zadrozny, and Luiz H.C. Merschmann.Correlation analysis of performance measures for multi-label classification.Information Processing & Management, 54(3):359 â 369, 2018. [25] Jesse Read, Bernhard Pfahringer, and Geoffrey Holmes. Multi-label classification using ensembles of pruned sets. InProceedings of IEEE ICDM, 2008. [26] Leander Schietgat, Celine Vens, Jan Struyf, Hendrik Blockeel, Dragi Kocev, and Saso Dzeroski. Predicting gene function using hierarchical multi-label decision tree ensembles.BMC Bioinfor- matics, 11:2, 2010. [27] Giorgio Severi, Tim Leek, and Brendan Dolan-Gavitt. Malrec: Compact Full-Trace Malware Recording for Retrospective Deep Analysis. InProc. of DIMVA, 2018. [28] Y Sharma.Exploiting Tactics, Techniques, and Procedures for Malware Detection. PhD thesis, University of Oxford, 2023. [29] Y Sharma, S Birnbach, and I Martinovic. Exploiting TTPs to design an Extensible and Explainable Malware Detection System. Journal of Universal Computer Science, 30(9):1140â1162, 2024. [30] Yashovardhan Sharma, Simon Birnbach, and Ivan Martinovic. RADAR: A TTP-based Extensible, Explainable, and Effective System for Network Traffic Analysis and Malware Detection. In Aleksandra Mileva, Steffen Wendzel, and Virginia N. L. Fran- queira, editors,Proceedings of the 2023 European Interdisciplinary Cybersecurity Conference, EICC 2023, Stavanger, Norway, June 14-15, 2023, pages 159â166. ACM, 2023. [31] Yashovardhan Sharma, Eleonora Giunchiglia, Simon Birnbach, and Ivan Martinovic. To TTP or not to TTP?: Exploiting TTPs to Improve ML-based Malware Detection.InIEEE International Conference on Cyber Security and Resilience, CSR 2023, Venice, Italy, July 31 - Aug. 2, 2023, pages 8â15. IEEE, 2023. [32] Kamran Shaukat, Suhuai Luo, Vijay Varadharajan, Ibrahim A. Hameed, and Min Xu. A survey on machine learning techniques for cyber security in the last decade.IEEE Access, 2020. [33] Adane Nega Tarekegn, Mario Giacobini, and Krzysztof Michalak. A review of methods for imbalanced multi-label classification. Pattern Recognition, 118:107965, 2021. [34] Grigorios Tsoumakas, I. Katakis, and I. Vlahavas. Effective and efficient multilabel classification in domains with large number of labels. InProceedings of ECML/PKDD â Workshop on Mining Multidimensional Data, 2008. [35] https://attack.mitre.org. ATT&CK. [36] https://bit.ly/scopus trend. Scopus. [37] https://malshare.com. MalShare. [38] https://virusshare.com. VirusShare. [39] Celine Vens, Jan Struyf, Leander Schietgat, Saso Dzeroski, and Hendrik Blockeel.Decision trees for hierarchical multi-label classification.Machine Learning, 73(2):185â214, 2008. [40] Jonatas Wehrmann, Ricardo Cerri, and Rodrigo C. Barros. Hierar- chical multi-label classification networks. InProceedings of ICML, pages 5225â5234, 2018. [41] Xindong Wu, Vipin Kumar, J Ross Quinlan, Joydeep Ghosh, Qiang Yang, Hiroshi Motoda, Geoffrey J McLachlan, Angus Ng, Bing Liu, S Yu Philip, et al.Top 10 algorithms in data mining. Knowledge and information systems, 14(1):1â37, 2008. [42] Ě Yanminsun, Andrew Wong, and Mohamed S. Kamel. Classification of imbalanced data: a review.International Journal of Pattern Recognition and Artificial Intelligence, 23, 11 2011. [43] Min-Ling Zhang and Zhi-Hua Zhou. A review on multi-label learning algorithms.IEEE Transactions on Knowledge and Data Engineering, 26(8):1819â1837, 2014.