Paper deep dive
Mean Masked Autoencoder with Flow-Mixing for Encrypted Traffic Classification
Xiao Liu, Xiaowei Fu, Fuxiang Huang, Lei Zhang
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 96%
Last extracted: 4/1/2026, 1:35:53 AM
Summary
The paper introduces Mean Masked Autoencoder (MMAE), a self-supervised pre-training paradigm for encrypted traffic classification. MMAE utilizes a teacher-student architecture with self-distillation and a novel dynamic Flow Mixing (FlowMix) strategy. FlowMix incorporates a Statistics-based Flow Matcher (SFM) and a Packet-importance aware Mask Predictor (PMP) to replace random masking, enabling the model to learn multi-granularity representations from byte, packet, and flow levels.
Entities (5)
Relation Signals (4)
MMAE → performs → Encrypted Traffic Classification
confidence 99% · Mean Masked Autoencoder with Flow-Mixing for Encrypted Traffic Classification
MMAE → uses → FlowMix
confidence 98% · MMAE employs a self-distillation mechanism... we introduce a dynamic Flow Mixing (FlowMix) strategy
FlowMix → incorporates → SFM
confidence 95% · We further design a Statistics-based Flow Matcher (SFM) and a Packet-importance aware Mask Predictor (PMP) to implement cross-flow interference in FlowMix.
FlowMix → incorporates → PMP
confidence 95% · We further design a Statistics-based Flow Matcher (SFM) and a Packet-importance aware Mask Predictor (PMP) to implement cross-flow interference in FlowMix.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Network traffic classification using self-supervised pre-training models based on Masked Autoencoders (MAE) has demonstrated a huge potential. However, existing methods are confined to isolated byte-level reconstruction of individual flows, lacking adequate perception of the multi-granularity contextual relationship in traffic. To address this limitation, we propose Mean MAE (MMAE), a teacher-student MAE paradigm with flow mixing strategy for building encrypted traffic pre-training model. MMAE employs a self-distillation mechanism for teacher-student interaction, where the teacher provides unmasked flow-level semantic supervision to advance the student from local byte reconstruction to multi-granularity comprehension. To break the information bottleneck in individual flows, we introduce a dynamic Flow Mixing (FlowMix) strategy to replace traditional random masking mechanism. By constructing challenging cross-flow mixed samples with interferences, it compels the model to learn discriminative representations from distorted tokens. Furthermore, we design a Packet-importance aware Mask Predictor (PMP) equipped with an attention bias mechanism that leverages packet-level side-channel statistics to dynamically mask tokens with high semantic density. Numerous experiments on a number of datasets covering encrypted applications, malware, and attack traffic demonstrate that MMAE achieves state-of-the-art performance. The code is available at this https URL
Tags
Links
- Source: https://arxiv.org/abs/2603.29537v1
- Canonical: https://arxiv.org/abs/2603.29537v1
Trouble viewing inline? Open PDF directly →
Full Text
74,771 characters extracted from source content.
Expand or collapse full text
Mean Masked Autoencoder with Flow-Mixing for Encrypted Traffic Classification Xiao Liu, Xiaowei Fu, Fuxiang Huang, and Lei Zhang This work was partially supported by National Natural Science Fund of China under Grants 92570110 and 62271090, Chongqing Natural Science Fund under Grant CSTB2024NSCQ-JQX0038, and National Youth Talent Project. (Corresponding author: Lei Zhang)X. Liu, X. Fu and L. Zhang are with the School of Microelectronics and Communication Engineering, Chongqing University, Chongqing 400044, China. (E-mail: liuxiao@stu.cqu.edu.cn, xwfu@cqu.edu.cn, leizhang@cqu.edu.cn,) Fuxiang Huang is with the School of Data Science, Lingnan University, Hong Kong, China. (E-mail: fxhuang1995@gmail.com) Manuscript received April 19, 2021; revised August 16, 2021. Abstract Network traffic classification using self-supervised pre-training models based on Masked Autoencoders (MAE) has demonstrated a huge potential. However, existing methods are confined to isolated byte-level reconstruction of individual flows, lacking adequate perception of the multi-granularity contextual relationship in traffic. To address this limitation, we propose Mean MAE (MMAE), a teacher-student MAE paradigm with flow mixing strategy for building encrypted traffic pre-training model. MMAE employs a self-distillation mechanism for teacher-student interaction, where the teacher provides unmasked flow-level semantic supervision to advance the student from local byte reconstruction to multi-granularity comprehension. To break the information bottleneck in individual flows, we introduce a dynamic Flow Mixing (FlowMix) strategy to replace traditional random masking mechanism. By constructing challenging cross-flow mixed samples with interferences, it compels the model to learn discriminative representations from distorted tokens. Furthermore, we design a Packet-importance aware Mask Predictor (PMP) equipped with an attention bias mechanism that leverages packet-level side-channel statistics to dynamically mask tokens with high semantic density. Numerous experiments on a number of datasets covering encrypted applications, malware, and attack traffic demonstrate that MMAE achieves state-of-the-art performance. The code is available at https://github.com/lx6c78/MMAE Index Terms: Encrypted Traffic Classification, Masked Autoencoder, Pre-training, Self-Distillation. I Introduction With the rapid development of the Internet, network traffic classification has become a key challenge. Traffic data streams provide critical insights into network behavior through payloads and metadata. Effective traffic monitoring and analysis are crucial for network management and security, ensuring Quality of Service (QoS) and detecting malicious activities [27], [14]. Consequently, network traffic classification has emerged as a vital research area. The main objective is to identify potential threats, various applications and services. However, the encryption protocols (e.g., TLS) and anonymity networks (e.g., VPNs, Tor) imposes further challenge. Figure 1: Comparison of reconstruction loss during pre-training between MAE and its variant with Self-Distillation strategy (i.e., MAE+SD). By introducing flow-level semantics via SD, the loss is consistently lower. Recently, pre-training methods [21], [45], [39], [46], [23] towards network traffic classification have shown immense promise and superior performance. Based on learned generalizable representations from massive unlabeled data, these models can be effectively fine-tuned on limited labeled datasets. These pre-training methods typically employ a strategy of mask-then-reconstruct. For instance, ET-BERT [21], FlowletFormer [23] and Trafficformer [46] adopt masking strategies derived from language modeling. YaTC [45] treats each traffic flow as an image to perform masked flow reconstruction, whereas NetMamba [39] employs a 1D sequence-specific masking strategy for pre-training objectives. Despite the above progress, such Masked Autoencoder (MAE) pre-training paradigms for network traffic classification still face critical challenges. These limitations can be summarized in two aspects. 1) Current MAE methods are restricted to isolated byte-level reconstruction of individual flows, failing to capture the inherent multi-granularity structure of network traffic from packet or flow. 2) The random masking strategy in MAE focuses only on byte-level but ignores traffic-specific attributes (e.g., packet-level and flow-level), struggling to construct sufficiently challenging pretext tasks and hindering the model from learning discriminative semantics. The rational behind the first limitation is that accurate classification requires moving beyond the perception of local regions within a single flow [44], [34]. For example, different applications often contain similar local traffic patterns, which means that relying solely on local byte-level information can only provide limited application-specific semantics and low-confidence classification results [8], [38]. The rational behind the second point is that since not all tokens possess equal information density, random masking often leads to targeting less informative tokens for reconstruction. Such trivial pretext tasks hinder learning useful representations. Therefore, exploring, harnessing and improving MAE is challenging but critical for encrypted traffic classification. Figure 2: Comparison of pre-training paradigms. (a) Standard MAE with a random masking strategy in an isolated single flow. (b) MMAE (ours): A novel teacher-student pre-training architecture with a flow-mixing strategy, where the teacher is a copy of student and interacted via self-distillation strategy under the unmasked flow-level semantic supervision from the teacher. To validate the above suspicions that the conventional MAE inadequately captures traffic information, we conduct a pilot experiment. Specifically, we deploy a self-distillation (SD) strategy into the MAE framework. In this setup, a copy of MAE without masking is deployed as a teacher to extract flow-level global semantics to provide richer supervision, while the student model still performs standard masked token reconstruction but under an extra supervision from the teacher. As shown in Fig. 1, we find that the reconstruction loss of the student model continues to decrease by deploying the SD strategy. This indicates that standard MAE pre-training strategy fails to fully explore useful information of traffic. To better represent traffic in multi-granularity, we rethink the pre-training paradigm and propose a Mean Masked Autoencoder (MMAE) with flow mixing mechanism. This framework extracts rich representations from unlabeled traffic data, achieving a comprehensive understanding of multi-granularity semantic features. Specifically, MMAE consists of a MAE (i.e., student) and its copy (i.e., teacher), which interacts via a self-distillation strategy based on EMA (Exponential Moving Average). This is the basic connotation of Mean MAE. Fig. 2 illustrates the key distinctions between MAE and our MMAE. Conventional MAE is limited to single-flow reconstruction with random masking mechanism, which tends to local byte guessing, failing to capture inter-flow class boundaries and discriminative semantics. To break the single-flow information bottleneck, MMAE allows the teacher (a copy of MAE without masking) to extract complete semantics from the unmasked input and provide flow-level supervision. Distinct encrypted traffic often exhibit similarities when viewed as isolated byte fragments. But in fact, their differences are reflected through multi-granularity contexts. Consequently, conventional random masking strategy of MAE in only byte-level is passive and trivial. To construct more challenging traffic-aware pretext tasks beneficial to discriminative semantic learning, we propose dynamic flow mixing (FlowMix). The core idea is to inject cross-flow interference and replace the conventional random masking strategy. To ensure this interference is challenging rather than trivially distinguishable, in FlowMix, we introduce a Statistics-based Flow Matcher (SFM) leveraging statistical priors to pair physically similar flows. After selecting the paired flows through SFM, we deploy a Packet-importance aware Mask Predictor (PMP) to determine where to inject interference. Specifically, PMP dynamically identifies regions of high semantic density, and then perform Dynamic Mixed Masking (DMM) to replace these critical regions with the matched distorted tokens. Thus the student is compelling to learn highly discriminative representations. This above design inherently guarantees multi-granularity perception. 1) Byte-level details are captured by reconstructing from the challenging flow mixed tokens. 2) Packet-level dependencies are learned through a Packet-importance aware Mask Predictor (PMP), which is guided by packet-level physical priors and subsequently fed back to dynamically construct mixed inputs for the next training iteration. 3) Flow-level global semantics are established by aligning the student’s representation with the teacher. Fig. 3 indicates the key differences in hierarchical representational capacity, in which MAE only captures byte-level information, while MMAE covers byte-level, packet-level as well as flow-level details. The main contributions are summarized as follows: • We propose Mean Masked Autoencoder (MMAE), a novel self-supervised pre-training paradigm for encrypted traffic classification. The framework leverages a twin architecture (teacher-student) of MAE, interacted via self-distillation under the supervision of unmasked flow-level semantics from teacher to enable the student learning multi-granular representations. • We propose a dynamic Flow Mixing (FlowMix) strategy beyond the conventional random masking in MAE to fully explore byte-level, packet-level and flow-level information. The strategy actively injects cross-flow interference to construct highly challenging pretext tasks, enabling the model to effectively delineate true traffic boundaries through multi-granularity contexts and learn discriminative semantics. • We further design a Statistics-based Flow Matcher (SFM) and a Packet-importance aware Mask Predictor (PMP) to implement cross-flow interference in FlowMix. SFM leverages statistical priors to pair physically similar flows. PMP identifies semantically dense regions that are subsequently replaced via Dynamic Mixed Masking (DMM). I Related Work I-A Traditional Methods Driven by the increasing complexity of network environments and the escalating demands of network management, traffic classification methods have undergone significant evolution over the past decade. Earlier approaches primarily relied on port numbers and simple rule matching [31], [47]. However, the widespread adoption of encryption and traffic obfuscation techniques has rendered these plaintext- and rule-based methods largely ineffective. Early machine learning approaches [25], [4], [26], [1], [35] explored classifiers such as decision trees, random forests, and SVMs for traffic classification. These techniques primarily utilized expert-designed, flow-level statistical summaries and protocol-specific features as model inputs. Although lightweight, fast, and interpretable, their performance heavily depends on time-consuming manual feature engineering, which limits their adaptability to new protocols. Consequently, such approaches struggle to flexibly adapt rapidly evolving network protocols. Despite these limitations, ML-based methods retain significant value in modern contexts. Handcrafted feature, such as traffic statistics, are relatively stable and frequently utilized to complement other methods in traffic classification [36], [20], [43]. Deep learning (DL) models (e.g., CNNs, RNNs, and GNNs) have significantly improved the performance by automatically extracting representations from raw packets, such as raw bytes or key packet attributes [34], [33], [32], [42], [22]. However, these conventional neural architectures exhibit limitations in capturing long-range sequential dependencies, which are prone to inductive biases and suffer from low computational efficiency. Transformer-based methods [41], [7], [24] offer powerful sequence modeling capabilities compared to CNNs by leveraging self-attention. However, their high performance typically relies on large-scale labeled datasets, which are costly to obtain in practical applications. Figure 3: Comparison of hierarchical semantic extraction capabilities between MAE and the proposed MMAE. MAE relies solely on a single flow and random masking for reconstruction. Our MMAE incorporates cross-flow mixing and packet-level side-channel statistical features. Within the self-distillation teacher-student architecture, flow-level reference features are used as supervision for reconstruction and mask prediction tasks. I-B Pre-training based Methods Pre-training methods significantly reduce the demand of labeled training data. This paradigm typically consists of two distinct phases: pre-training stage based on unlabeled traffic data via self-supervised learning and fine-tuning stage based on a few labeled downstream traffic data. With the success of large models such as BERT[11] and GPT [30], pre-training methodologies have rapidly emerged as the dominant approach for sequence modeling. Recent studies focus on Transformer-based pre-training paradigms for network traffic classification. For example, PERT [14] directly feeds raw packet sequences into a Transformer for feature extraction. Conversely, YaTC [45] transforms traffic into two-dimensional images and trains a visual masked autoencoder. Additionally, frameworks such as ET-BERT [21] and TrafficFormer [46] partition traffic into bursts by analyzing packet transmission directions. These bursts are subsequently serialized into hexadecimal strings, and the subword tokenization techniques are applied to construct a fixed-size vocabulary. Although these methods have achieved notable success, directly adopting local reconstruction strategies from natural language processing or computer vision often breaks the inherent properties of network traffic. Consequently, such designs fail to adequately capture the unique semantic characteristics of network traffic. I-C Knowledge Distillation Knowledge Distillation (KD) is a classic knowledge transfer and model compression technique [15]. The core objective is to train a compact student model to approximate the targets generated by a complex teacher model, which enables the model to inherit the strong generalization capabilities. Self-distillation deploys the same underlying architecture for teacher and student, utilizing its own outputs rather than relying on posterior distributions. As a result, it is often considered a discriminative self-supervised objective [2], [12], [16], [18]. Under this mechanism, the network learns from its historical iterations or differently perturbed views, progressively refining its high-dimensional representations. By eliminating the prerequisite to pre-train and freeze a massive external teacher network, self-distillation significantly reduces the memory and time overhead. More importantly, it typically enhances the model’s robustness [3], [6], [9], [19]. Motivated by the self-distillation, in the proposed MMAE framework, we seamlessly integrate the self-distillation mechanism between a twin of MAEs to achieve interactive pre-training. I The Proposed Mean Masked AutoEncoder An overview of our MMAE is illustrated in Fig. 4, which contains a traffic preprocessing unit (d), a FlowMix unit (a), a student MAE (b), a teacher MAE (c), and a Statistics-based Flow Matcher (e) for constructing flow pairs. I-A Traffic Preprocessing To transform the heterogeneous and variable-length raw network packets into representations suitable for models, Fig. 4 (d) illustrates the complete preprocessing workflow. First, we use the SplitCap tool to segment the PCAP files captured from network interfaces into independent session flows based on the five-tuple rule. To eliminate potential model bias towards specific hosts, the IP addresses and port details information within the flows are anonymized. Subsequently, we apply uniform truncation and padding to the packets, fixing each packet to 320 bytes (80 bytes of protocol header features and 240 bytes of payload). This specific allocation preserves critical initial information from both the control and application layers while avoiding interference from excessive zero-padding. We extract the first 5 packets from each standardized session. If a session contains fewer than 5 packets, appropriate zero-padding is applied. These extracted packets are then concatenated to form a one-dimensional integer array with a fixed length of L=1600L=1600 bytes. Finally, to improve the model’s convergence and numerical stability, each byte value is normalized from the original [0,255][0,255] to [0,1][0,1] interval. Figure 4: Flowchar of the proposed MMAE, which mainly includes FlowMix and teacher-student based twin MAEs. MMAE incorporates a traffic pre-processing unit, a statistics-based flow matcher and a cross-flow mixing unit (i.e., FlowMix). Within the self-distillation teacher-student twin MAE architecture, flow-level reference features are exploited as supervision for reconstruction and mask prediction tasks. TABLE I: Summary of Extracted Side-Channel Statistical Features Category Feature Name Description Time (6 dims) F-Duration Total time of a flow from start to end F-Time Timestamp of the first packet in a flow P-inter-min Shortest time interval between packets P-inter-max longest time interval between packets P-inter-avg Average deviation of packet intervals P-inter-std Standard deviation of packet intervals Size (11 dims) P-total Total number of packets in a flow Downlink-Bytes Total bytes transmitted downlink Downlink-Count Total packets count transmitted downlink Payload-P-Count Number of packets with payload data Payload-P-min Smallest payload sizes Payload-P-max Largest payload sizes Payload-P-std Standard deviation of payload sizes P-Length-min Smallest and largest packet lengths P-Length-max Smallest and largest packet lengths P-Length-avg Average deviation of packet lengths P-Length-std Standard deviation of packet lengths Flag (10 dims) TCP Count Number of TCP packets in a flow UDP Count Number of UDP packets in a flow DNS Count Number of DNS packets in a flow ICMP Count Number of ICMP packets in a flow SYN Count Number of SYN flags in a flow FIN Count Number of FIN flags in a flow ACK Count Number of ACK flags in a flow PSH Count Number of PSH flags in a flow URG Count Number of URG flags in a flow RST Count Number of RST flags in a flow I-B FlowMix Mechanism To overcome the limited contextual information within a single flow, we introduce a dynamic Flow Mixing strategy to enhance the model’s flow-level discriminative capabilities with perturbation tokens. However, random pairwise mixing of traffic flows have drastically different behavioral patterns. In such cases, the resulting reconstruction task becomes simple, as the model can easily distinguish them, failing to learn deep semantics. Therefore, effective cross-flow mixing requires identifying traffic pairs with similar patterns. Side-channel statistical features are inherent physical attributes of network traffic, which remain unaffected by encryption and can therefore intuitively reflect flow-level patterns, packet-level dynamics, and the overall communication environment. Statistics-based Flow Matcher (SFM) leverages these statistical priors as anchors to reliably estimate the physical similarity between flows, which establishes a principled foundation for the mixing process. I-B1 Statistics-based Flow Matcher As illustrated in Fig. 4 (e), SFM extracts 27 highly expressive side-channel statistical features from flow-level PCAP files. These features encompass packet-level behaviors, temporal patterns, and protocol interaction modes. Based on their primary physical implications, we categorize these attributes into three principal groups: Time Features, Size Features, and Flag & Protocol Features. • Time Features: These features describe the temporal patterns of packet arrivals and session persistence. For example, real-time communication applications typically exhibit stable and frequent inter-arrival times, while non-real-time applications show bursty time intervals. • Size Features: These attributes capture the distributional patterns of data volume transmission across the network flow, including the statistical regularity of overall packet length, payload size, and downlink traffic distribution. • Flag & Protocol Features: These features record the trigger frequency of control flags (e.g., SYN, FIN, PSH), which provide intuitive insights into the connection management strategies. For example, applications that frequently establish connections generate a large number of SYN/FIN flags, while persistent connection applications rarely generate such flags. The specific names and physical meanings of the 27 side-channel statistical features are shown in Table I. Subsequently, an encoder maps the raw features to a higher-order abstract representation. We denote the input raw statistical feature matrix as Xstat∈ℝ1×27X_stat ^1× 27. The entire feature encoding process can be represented as follows: Xnorm X_norm =BatchNorm(Xstat) =BatchNorm(X_stat) (1) Fcnn F_cnn =MaxPool(CNN1D(Xnorm)) =MaxPool (CNN1D(X_norm) ) Fdeep F_deep =MLP(Fcnn) =MLP(F_cnn) Fout F_out =Concat(Xnorm,Fdeep) =Concat(X_norm,F_deep) where Xnorm∈ℝ1×27X_norm ^1× 27 denotes the features processed via 1D Batch Normalization. CNN1D(⋅)CNN1D(·) represents a deep network comprising two 1D convolutional layers. Subsequently, an adaptive max-pooling layer MaxPool(⋅)MaxPool(·) is used for feature compression to extract the most salient sequence features FcnnF_cnn. Finally, the pooled features are mapped to a high-order deep representation Fdeep∈ℝ1×64F_deep ^1× 64. To avoid losing the explicit semantics of the original physical properties, the encoder employs a residual concatenation operation, resulting in a 9191-dimensional fused feature FoutF_out. Next, we must precisely pair each main flow with a corresponding support flow to construct the mixing input. To this end, we construct symmetric pairs using a greedy matching [29] strategy within a mini-batch, which is detailed in Algorithm 1. The final output of this algorithm is an index vector P, which encapsulates the symmetric pairing assignments for the entire batch. I-B2 Dynamic Mixed Masking After selecting the paired flows through SFM, executing an effective mixing strategy is crucial. To avoid the triviality of blindly swapping random tokens with varying information densities, we design the Dynamic Mixed Masking (DMM) module. Instead of the conventional random masking in standard MAE, we guide the selection process using the prediction loss LpredL_pred from the student model, which is elaborated as the Packet-importance aware Mask Predictor (PMP) in Section I-D. Regions with higher prediction errors indicate dense semantics that the model struggles to learn. Therefore, we prioritize masking and replacing these challenging regions with interfering features from paired flows indexed by P from SFM module. Specifically, we partition the input traffic sequence into Nx=400N_x=400 patches with a step size of 4, and linearly project them into D=256D=256 dimensions. This yields the main flow’s token sequence, denoted as Xmain∈ℝNx×DX_main ^N_x× D. Given masking ratio rmaskr_mask, the number of tokens to be replaced by support flow features XsuppX_supp corresponding to paired flows chosen by SFM is Nmask=⌊Nx⋅rmask⌋N_mask= N_x· r_mask , leaving Nkeep=Nx−NmaskN_keep=N_x-N_mask tokens retained, where ⌊⋅⌋ · represents the operation of rounding down. To progressively increase the difficulty of the pretext task, the total masked tokens NmaskN_mask are divided into replaced parts with the hard mask ratio of rhardr_hard and random mask parts with a ratio of randr_rand, respectively. Based on the predicted loss ℒpred∈ℝNxL_pred ^N_x from PMP, we identify the indices Ωhard _hard of the KhardK_hard most challenging patches via the TopK_IndicesTopK\_Indices operation: Ωhard=TopK_Indices(ℒpred,Khard) _hard=TopK\_Indices(L_pred,K_hard) (2) To efficiently execute both random and targeted masking simultaneously, we first generate uniform noise ∼U(0,1)NxN U(0,1)^N_x, and reformulate each value i,jN_i,j as: ^=i,j+η,if j∈Ωhardi,j,otherwise N= (3) given that η≫1η 1, the values of selected challenging patches in N become significantly larger than the uniform noise. By sorting N in ascending order along the sequence dimension, we construct the binary mask matrix M∈0,1NxM∈\0,1\^N_x: Mj=0,if j<Nkeep(Keep)1,if j≥Nkeep(Replace) M_j= (4) where j is the index in the sorted sequence, and NkeepN_keep is the target number of retained tokens. However, during the early stages of training, the model has not yet extracted meaningful feature representations, and LpredL_pred is highly uncertain. Therefore, we formulate the ratio of ℒpredL_pred-guided hard patches as a dynamic scaling function that evolves with the training epoch. This design fundamentally incorporates the paradigm of Curriculum Learning. For the current training epoch t, the definition is as follows: rhard(t)=Rmax⋅(tTtotal) r_hard(t)=R_max· ( tT_total ) (5) as t increases from 0 to TtotalT_total, the ratio rhard(t)r_hard(t) scales linearly up to its maximum threshold RmaxR_max. Algorithm 1 Symmetric Pairing Construction 1:Batch side-channel statistical feature matrix Fout∈ℝB×DF_out ^B× D (Batch size B, Feature dimension D=91D=91) 2:Symmetric pairing index vector P∈ℤBP ^B 3:Fnorm←Fout‖Fout‖2F_norm← F_out\|F_out\|_2 ⊳ Row-wise L2 normalization 4:←Fnorm⋅Fnorm⊤S← F_norm· F_norm 5:for i=1i=1 to B do 6: [i,i]←−∞S[i,i]←-∞ ⊳ Prevent self-pairing 7:end for 8:P←[1,2,…,B]P←[1,2,…,B] ⊳ Initialize indices 9:current←S_current 10:for step=1step=1 to ⌊B/2⌋ B/2 do 11: row,col←argmaxi,j(current)row,col← argmax_i,j(S_current) ⊳ Maximum coordinates 12: if current[row,col]=−∞S_current[row,col]=-∞ then 13: break ⊳ Terminate if all elements are masked 14: end if 15: P[row]←colP[row]← col 16: P[col]←rowP[col]← row ⊳ Symmetric bi-directional mapping 17: ⊳ Exclude paired rows and columns 18: current[row,:]←−∞S_current[row,:]←-∞; current[:,row]←−∞S_current[:,row]←-∞ 19: current[col,:]←−∞S_current[col,:]←-∞; current[:,col]←−∞S_current[:,col]←-∞ 20:end for 21:return P Finally, utilizing the pairing index vector P generated by Algorithm 1, we extract the corresponding support flow token sequences XsuppX_supp within the batch. The final flow mixing sequence XmixX_mix is: Xmix=Xmain⊙(1−M)+Xsupp⊙M X_mix=X_main (1-M)+X_supp M (6) where ⊙ denotes the Hadamard product. When a value in M is 0, the token retains the main flow feature XmainX_main. Otherwise, it is replaced by a feature from the support flow XsuppX_supp. Notably, the support flow serves only as a repository of distorted features. It is not fed into the network as an independent sequence as indicated by the “no input” state in Fig. 4 (a). Thus, the core entity of the pre-training task is always based on the main flow. To preserve the positional relationships of packets and bytes in traffic, we add positional encoding Epos∈ℝN×DE_pos ^N× D to the sequence, where N=Nx+1N=N_x+1 is the total sequence length. Following the standard Transformer architecture, we also prepend a learnable class token xcls∈ℝ1×Dx_cls ^1× D. Finally, the mixed feature embedding fed into the student model is formulated as: Xstudent=[xcls;Xmix]+Epos X^student=[x_cls;X_mix]+E_pos (7) where Xmix∈ℝNx×DX_mix ^N_x× D represents the cross-flow mixed feature and [;][;] represents the concatenation operation. The unmasked raw feature fed into teacher model is: Xteacher=[xcls;Xmain]+Epos X^teacher=[x_cls;X_main]+E_pos (8) I-C Student Branch The core objective of the student branch is to extract representative features from the cross-flow mixed sequence XstudentX^student by reconstructing the main flow feature. First, the student encoder ℰSE_S processes the mixed token sequence XstudentX^student. Through multi-layer self-attention, the features of the main and support flows are deeply integrated: Zstudent=ℰS(Xstudent) Z^student=E_S(X^student) (9) were Zstudent∈ℝN×DZ^student ^N× D represents the hidden-layer feature. This representation is then fed into its decoder and another parallel branch, i.e., PMP. In the following, we first detail the decoder and its reconstruction mechanism. I-C1 Dual-view Construction and Decoding To separately recover the semantics of the main and support flows from the mixed tokens ZstudentZ^student, we construct dual views before the decoder. Using the binary mask M∈0,1NxM∈\0,1\^N_x generated in the DMM, we obtain the main flow view VmainV_main and its symmetric support flow view VsuppV_supp. Specifically, VmainV_main retains the main flow tokens and replaces the positions of the support flow with learnable mask token E[MASK]E_[MASK]: Vmain=Zstudent⊙(1−M)+E[MASK]⊙M V_main=Z^student (1-M)+E_[MASK] M (10) Conversely, VsuppV_supp is represented as: Vsupp=Zstudent⊙M+E[MASK]⊙(1−M) V_supp=Z^student M+E_[MASK] (1-M) (11) These two views are concatenated along the batch dimension and fed into the decoder SD_S for reconstruction: Hmain H_main =S(Vmain+Eposdec) =D_S(V_main+E_pos^dec) (12) Hsupp H_supp =S(Vsupp+Eposdec) =D_S(V_supp+E_pos^dec) where Hmain∈ℝNx×DH_main ^N_x× D and Hsupp∈ℝNx×DH_supp ^N_x× D represent the reconstructed main and support view tokens, respectively. For brevity, the class token in the output sequence is omitted here. Figure 5: Architecture of the Packet-importance aware Mask Predictor (PMP). PMP leverages packet-level side-channel priors to generate a low-rank attention bias via a token-aware gating mechanism. This bias dynamically modulates self-attention to identify and mask challenging, semantically dense regions. I-C2 Unmixing and Reconstruction HmainH_main and HsuppH_supp contain the predictions for the masked regions. To recover the complete semantics of the main flow, we perform unmixing and reconstruction. First, we remap the index of support view token HsuppH_supp back to their original batch positions according to the index vector P from SFM in Algorithm 1, obtaining the aligned token: Haligned=Hsupp[P] H_aligned=H_supp[P] (13) The unmixing process is then formulated as: Hunmix=Hmain⊙M+Haligned⊙(1−M) H_unmix=H_main M+H_aligned (1-M) (14) Notably, HunmixH_unmix restores the original structure of the main flow, utilized for global semantic alignment with the reconstructed full tokens by the teacher network. Following unmixing, the model performs fine-grained byte-level reconstruction and computes the reconstruction loss: ℒrec=MSE(Hunmix,Xmain) _rec=MSE(H_unmix,X_main) (15) where MSE(⋅)MSE(·) indicates Mean Squared Error. Since the model predicts masks from dual views simultaneously in a single forward pass, the MSE can be directly calculated across all tokens to fully optimize the network. I-D Packet-importance aware Mask Predictor To effectively identify the most informative patches, we introduce a Packet-importance aware Mask Predictor (PMP). Packet-level side-channel features are natural indicators of reconstruction difficulty. Inter-arrival time (StimeS_time) reflects macroscopic burstiness, while payload length (SlenS_len) determines microscopic information capacity. Leveraging these multi-granularity priors, PMP dynamically allocates attention to accurately predict reconstruction difficulty. As shown in Fig. 5, this is achieved by generating an attention bias. We first broadcast the packet-level statistical signal StimeS_time and SlenS_len along the byte dimension to match the original sequence length. Then, respective 1D convolutions are applied to map them into the token latent space. The resulting embeddings EtE_t and ElE_l, are concatenated and fused to form the statistical context feature Zstat∈ℝNx×DstatZ_stat ^N_x× D_stat: Et E_t =Conv1D(Broadcast(Stime))+PEt =Conv1D(Broadcast(S_time))+PE_t (16) El E_l =Conv1D(Broadcast(Slen))+PEl =Conv1D(Broadcast(S_len))+PE_l Zstat Z_stat =LayerNorm(Wfuse[Et;El]) =LayerNorm(W_fuse[E_t;E_l]) where PEtPE_t and PElPE_l denote the respective positional encodings, and WfuseW_fuse is a learnable weight matrix used for feature fusion. To modulate the attention distribution, PMP generates suppressive and enhancing matrices, defined by a policy indicator k∈sup,enhk∈\sup,enh\. For each policy, the statistical feature ZstatZ_stat is projected into a rank-r space via learnable matrices WUkW_U^k and WYk∈ℝDstat×rW_Y^k ^D_stat× r, producing low-rank factors. k=ZstatWUk,k=ZstatWYk ^k=Z_statW_U^k, ^k=Z_statW_Y^k (17) The dense bias matrix Zk∈ℝNx×NxZ^k ^N_x× N_x is then computed via inner product: Zk=1r(k)(k)⊤ Z^k= 1 r(U^k)(Y^k) (18) Since different tokens play varying roles in traffic contexts, a gating mechanism dynamically decides whether to suppress or enhance each position. We extract three metrics for the i-th token: a similarity feature FsimiF^i_sim, a saliency feature FsaliF^i_sal, and a statistical context feature FctxiF^i_ctx: Fsimi F^i_sim =1Nx∑jReLU(cos(vi,vj)) = 1N_x _jReLU( (v_i,v_j)) (19) Fsali F^i_sal =Linear(‖vi‖2) =Linear(\|v_i\|_2) Fctxi F^i_ctx =Linear(Zstat,i) =Linear(Z_stat,i) where vi∈ℝDv_i ^D denotes the i-th token in the main flow view VmainV_main. These metrics are concatenated to form the comprehensive gated features Fgatei=[Fsimi;Fsali;Fctxi]F^i_gate=[F^i_sim;F^i_sal;F^i_ctx]. Subsequently, gating logits gateL_gate are generated using a similar matrix low-rank projection parameterized by learnable weight matrices aW_a and bW_b. These logits are then used to calculate the suppression probability psup^sup and enhancement probability penhp^enh for each position: gate _gate =(Fgatea)(Fgateb)⊤ =(F_gateW_a)(F_gateW_b) (20) psup p^sup =Sigmoid(gate) =Sigmoid(L_gate) penh p^enh =1−psup =1-p^sup Finally, the suppressive and enhancing bias matrices are weighted by the calculated gating probabilities. To ensure numerical stability, the final attention bias matrix attn B_attn is clamped to the interval [−5,5][-5,5]: attn=Clamp(psup⊙(−Softplus(γsup)⋅Zsup)+penh⊙(Softplus(γenh)⋅Zenh),−5,5) split B_attn=Clamp (p^sup (-Softplus(γ^sup)· Z^sup)\\ +p^enh (Softplus(γ^enh)· Z^enh),-5,5 ) split (21) where γsupγ^sup and γenhγ^enh are learnable scalars controlling the bias magnitudes. In the implementation, PMP processes ZstudentZ^student through a dedicated prediction network ℱPMPF_PMP to output the patch-wise reconstruction difficulty prediction: Mpred M_pred =ℱPMP(Zstudent) =F_PMP(Z^student) (22) where Mpred∈ℝNx×DM_pred ^N_x× D, and ℱPMPF_PMP is implemented in the structure of a 2-layer Transformer block. Within ℱPMPF_PMP, the generated bias matrix attn B_attn is injected directly into the attention score computation: Attention(Q,K,V) (Q,K,V) =softmax(QK⊤dk+attn)V =softmax ( QK d_k+ B_attn )V (23) The prediction MpredM_pred is subsequently fed back to the module described in Section I-B to guide the dynamic mask generation of cross-flow samples for the next training iteration. PMP is supervised by the reconstruction loss ℒrecL_rec. However, as pre-training progresses, the absolute reconstruction loss undergoes drastic decay and becomes noisy. To circumvent this instability, we adopt a pairwise ranking loss inspired by RankNet [5]. For any two patches i and j, we construct binary ranking labels based on ℒrecL_rec: i,j+ _i,j^+ =1,if ℒrec(i)>ℒrec(j)0,otherwise = (24) i,j− _i,j^- =1,if ℒrec(i)<ℒrec(j)0,otherwise = We then compute the prediction difference Δi,j=Mpred(i)−Mpred(j) _i,j=M_pred^(i)-M_pred^(j) and optimize this relative ranking using binary cross-entropy (BCE): ℒpred=BCE(σ(Δi,j),i,j+)+BCE(1−σ(Δi,j),i,j−)L_pred=BCE (σ( _i,j),I_i,j^+ )+BCE (1-σ( _i,j),I_i,j^- ) (25) where σ(⋅)σ(·) is the Sigmoid activation function. Algorithm 2 Training Pipeline of MMAE 1:Unlabeled traffic data unlabelD_unlabel, labeled traffic data labelD_label, total pre-training epochs TtotalT_total, fine-tuning epochs EfineE_fine, initial student parameters θS _S, EMA momentum m. 2:Initialize Teacher parameters θT _T using θS _S 3:Phase 1: Pre-training 4:for t=1t=1 to TtotalT_total do 5: for each mini-batch in unlabelD_unlabel do 6: # Flow Mixing 7: Extract and encode side-channel statistical features (Eq. 1) 8: Construct symmetric pairing index P (Algorithm 1) 9: Generate dynamic mixed mask M (Eq. 2-5) 10: Construct cross-flow mixed input XstudentX_student (Eq. 6, 7) 11: Construct unmasked input XteacherX_teacher (Eq. 8) 12: # Student Branch 13: Construct and decode dual views Vmain,VsuppV_main,V_supp (Eq. 9-12) 14: Unmix features to obtain HunmixH_unmix (Eq. 13, 14) 15: Compute reconstruction loss ℒrecL_rec (Eq. 15) 16: # Packet-importance aware Mask Predictor 17: Generate attention bias attn B_attn (Eq. 16-23) 18: Compute mask prediction loss ℒpredL_pred (Eq.24, 25) 19: # Teacher Branch with Self-Distillation 20: Extract global contextual features HteacherH_teacher (Eq. 26) 21: Compute semantic alignment loss ℒalignL_align (Eq. 27) 22: # Model Optimization 23: Compute total pre-training loss ℒpreL_pre (Eq. 30) 24: Update Student parameters θS←θS−η∇θSℒpre _S← _S-η _ _SL_pre 25: Update Teacher parameters θT _T via EMA (Eq. 28) 26: end for 27:end for 28:Phase 2: Fine-tuning 29:# Downstream Initialization 30:Retain pre-trained student encoder θS _S and initialize downstream classification head 31:for epoch =1=1 to EfineE_fine do 32: for each batch (X,y)(X,y) in labelD_label do 33: # Supervised Optimization 34: Extract features using student encoder and predict probability distribution y y 35: Compute fine-tuning loss ℒCEL_CE (Eq. 31) 36: Update student encoder and classification head parameters by minimizing ℒCEL_CE 37: end for 38:end for 39:return Fine-tuned model parameters I-E Teacher Branch with Self-Distillation During the reconstruction phase, the student model inputs only severely disrupted mixed fragments. In contrast, the teacher model processes the complete global flow, extracting high-level semantic representations as supervision signals. The teacher branch takes the original feature embedding XteacherX^teacher constructed in Section I-B as input to extract the full contextual feature HteacherH_teacher: Hteacher=T(ℰT(Xteacher)) H_teacher=D_T (E_T(X^teacher) ) (26) where ℰTE_T and TD_T denote the encoder and decoder of the teacher, respectively. To guide the student in learning global traffic semantics within the latent space, we introduce a feature alignment mechanism. Specifically, we align the unmixed feature HunmixH_unmix from the student with the HteacherH_teacher from the teacher using the alignment loss ℒalignL_align: ℒalign=1−sim(Hunmix,Hteacher) _align=1-sim(H_unmix,H_teacher) (27) where sim(⋅)sim(·) denotes the cosine similarity. By minimizing ℒalignL_align, the student network is subjected to a strict constraint: even when the student’s input is heavily contaminated by other flow, its unmixed representation must align with the teacher’s view of the uncorrupted context. This semantic distillation improves model’s robustness against traffic obfuscation. Notably, the teacher’s weights are updated using an Exponential Moving Average (EMA) strategy: θT←m⋅θT+(1−m)⋅θS _T← m· _T+(1-m)· _S (28) where θT _T and θS _S denote the parameters of the teacher and student models, respectively. m∈[0,1]m∈[0,1] is the momentum coefficient controlling the update rate. In this paper, the momentum m is scheduled via cosine annealing: mt=mfinal+12(mbase−mfinal)(1+cos(πtT))m_t=m_final+ 12(m_base-m_final) (1+ ( π tT ) ) (29) where t and T denote the current and total training steps, respectively. m is smoothly increased from the base value mbasem_base to the final value mfinalm_final, ensuring model stability. Since the teacher and student are a twin of MAE, and the teacher is updated through EMA strategy, we therefore call the model Mean MAE. I-F Model Optimization Pre-training phase. The overall pre-training objective is formulated as: ℒpre=ℒrec+λ1ℒpred+λ2ℒalign _pre=L_rec+ _1L_pred+ _2L_align (30) where λ1 _1 and λ2 _2 are balancing hyperparameters. Fine-tuning phase. For downstream tasks, we fine-tune the full model by retaining only the pre-trained student encoder as the backbone. The extracted features are fed into a classification head to predict the probability distribution y^∈ℝc y ^c, where c is the number of traffic categories. The model is optimized using the Cross-Entropy loss between the ground-truth labels y and predictions y y: ℒCE=CrossEntropy(y,y^) _CE=CrossEntropy(y, y) (31) To provide a comprehensive overview, the training pipeline of the proposed MMAE is detailed in Algorithm 2. IV Experiments IV-A Experimental Setup IV-A1 Datasets and Preprocessing For the encrypted traffic classification task, the model is pre-trained using six public real-world datasets: ISCXVPN 2016 [13], ISCXTor 2016 [13], CrossPlatform (Android) [37], CrossPlatform (iOS) [37], USTC-TFC [40], and CICIoT2022 [10]. During the fine-tuning phase, we evaluate the model on these six datasets and the CSTNET-TLS 1.3 dataset [21]. Each dataset is split into training, validation, and test sets at an 8:1:1 ratio. • ISCXVPN2016: This dataset comprises communication application traffic captured by the Canadian Institute for Cybersecurity, which contains VPN and non-VPN traffic from 16 applications categorized into 7 types, yielding 16,048 processed samples. • ISCXTor2016: This dataset contains application traffic using Tor for encrypted communication, which includes traffic data from 8 communication categories and totally 14,569 samples. • USTC-TFC2016: This dataset consists of encrypted traffic from 20 application types, evenly split between 10 benign and 10 malicious categories, comprising 50,677 processed samples. • CrossPlatform (Android & iOS): These datasets include encrypted traffic from the top 100 Android and iOS apps in the US, China, and India. To prevent long-tail classes from degrading model performance, we discarded categories with fewer than 50 samples, resulting in 181 and 124 categories, which comprise 54,011 and 48,787 processed samples, respectively. • CICIoT2022: This dataset is collected from a laboratory network, designed for IoT vulnerability testing, encompassing six mainstream IoT malicious attack categories, yielding 22,634 samples. • CSTNET-TLS 1.3: This dataset contains encrypted traffic data over the encryption protocol TLS 1.3, which is one of the most cutting-edge, widely used, and thoroughly encrypted transport protocol standards on the internet. This data was captured from the China Science and Technology Network backbone across 120 applications, closely reflecting actual ISP traffic distributions, contributing 46,356 processed samples. TABLE I: Comparisons with previous SoTA methods on CrossPlatform(Android), CrossPlatform(iOS), and CICIoT2022 datasets Method CrossPlatform(Android) CrossPlatform(iOS) CICIoT2022 AC PR RC F1 AC PR RC F1 AC PR RC F1 AppScanner 0.1626 0.1646 0.1456 0.1413 0.1718 0.1400 0.1440 0.1283 0.7556 0.8093 0.7244 0.6938 FlowPrint 0.8739 0.8941 0.8739 0.8700 0.8712 0.8687 0.8712 0.8603 0.5820 0.4164 0.5820 0.4643 FS-Net 0.0147 0.0023 0.0147 0.0034 0.0293 0.0014 0.0293 0.0025 0.5747 0.3800 0.5747 0.4216 ET-BERT 0.9386 0.9451 0.9386 0.9401 0.9105 0.8809 0.9105 0.8850 0.9937 0.9938 0.9937 0.9937 YaTC 0.9042 0.9081 0.9042 0.9042 0.9310 0.9307 0.9310 0.9295 0.9959 0.9959 0.9959 0.9959 TrafficFormer 0.7664 0.6435 0.6204 0.6167 0.5679 0.4966 0.4697 0.4689 0.8725 0.8487 0.8343 0.8288 FlowletFormer - - - - - - - - 0.9109 0.8905 0.8866 0.8859 NetMamba 0.9869 0.9871 0.9869 0.9864 0.9881 0.9885 0.9881 0.9881 0.9985 0.9985 0.9985 0.9985 MMAE 0.9897 0.9898 0.9897 0.9893 0.9900 0.9872 0.9900 0.9881 0.9990 0.9990 0.9990 0.9990 IV-A2 Implementation Details The teacher model and the student model have the same network architecture, which comprises an encoder consisting of 7 Transformer blocks and a decoder consisting of 2 Transformer blocks. During the pre-training phase, we train the model for 150,000 iterations with a batch size of 128 using the AdamW optimizer. A linear learning rate scaling rule is applied with a base learning rate of 1×10−31× 10^-3. For the teacher’s EMA momentum parameters, mbasem_base and mfinalm_final are set to 0.96 and 0.99, respectively. For the objective function, λ1 _1 and λ2 _2 are set to 1.0 and 0.1, respectively. Furthermore, the random masking ratio randr_rand is set to 0.7, supplemented by a dynamic hard masking ratio rhardr_hard of 0.2. During the fine-tuning phase, the model is optimized over 120 epochs using the AdamW, with the base learning rate adjusted to 2×10−32× 10^-3 and the batch size set to 64. Our framework is implemented in PyTorch 2.0.0 and trained on a single NVIDIA GeForce RTX 4080 GPU. IV-A3 Evaluation Metrics To evaluate the classification performance of our model, we employ four standard metrics: Accuracy (AC), Precision (PR), Recall (RC), and Weighted F1-Score (F1), formulated as: Weighted F1=∑i=1K(niNtotal×2×Precisioni×RecalliPrecisioni+Recalli)Weighted F1= _i=1^K ( n_iN_total× 2×Precision_i×Recall_iPrecision_i+Recall_i ) (32) where nin_i is the number of samples in the i-th class, NtotalN_total is the total number of samples, and K is class number. TABLE I: Comparison with previous SoTA methods on ISCXTor2016, ISCXVPN2016, and USTC-TFC2016 datasets Method ISCXTor2016 ISCXVPN2016 USTC-TFC2016 AC PR RC F1 AC PR RC F1 AC PR RC F1 AppScanner 0.4034 0.2850 0.2149 0.2113 0.7643 0.8047 0.7045 0.7256 0.6998 0.8591 0.6062 0.6633 FlowPrint 0.1316 0.0173 0.1316 0.0306 0.9666 0.9733 0.9666 0.9681 0.7992 0.7745 0.7992 0.7755 FS-Net 0.7020 0.7010 0.7020 0.6999 0.7023 0.7487 0.7023 0.6660 0.4381 0.2011 0.4381 0.2672 ET-BERT 0.9980 0.9981 0.9980 0.9980 0.9566 0.9566 0.9566 0.9565 0.9910 0.9911 0.9910 0.9910 YaTC 0.9959 0.9959 0.9959 0.9959 0.9819 0.9820 0.9819 0.9819 0.9947 0.9749 0.9747 0.9734 TrafficFormer 0.8669 0.7545 0.7460 0.7472 0.8533 0.8445 0.8348 0.8279 0.9750 0.9789 0.9750 0.9746 FlowletFormer 0.9215 0.9263 0.9043 0.9116 0.9400 0.9471 0.9277 0.9364 0.9650 0.9689 0.9650 0.9648 MLETC 0.9948 0.9778 0.9954 0.9865 0.9875 0.9847 0.9847 0.9880 0.9902 0.9935 0.9918 0.9926 NetMamba 0.9993 0.9993 0.9993 0.9993 0.9899 0.9899 0.9899 0.9899 0.9990 0.9991 0.9990 0.9990 MMAE 1.0000 1.0000 1.0000 1.0000 0.9957 0.9957 0.9957 0.9957 0.9995 0.9995 0.9995 0.9995 IV-A4 Comparison Methods We comprehensively evaluate and compare MMAE against diverse state-of-the-art methods, specifically including the following paradigms: • Traditional Machine Learning: AppScanner [35] and FlowPrint [37], which rely on hand-crafted statistical features for classification. • CNN-based Models: FS-Net [22], which performs supervised learning directly on the raw network packets. • Transformer-based Models: ET-BERT [21], YaTC [45], TrafficFormer [46], FlowletFormer [23], and MLETC [28]. These architectures capture latent representations by treating traffic as either 2D visual images or 1D language sequences before downstream fine-tuning. • Mamba-based Models: NetMamba [39], a pure Mamba architecture that processes network traffic as block-wise 1D sequences. Aside from the supervised baselines (AppScanner, FlowPrint, and FS-Net), all methods utilize a pre-training paradigm. Regarding the pre-training corpora, TrafficFormer and FlowletFormer are pre-trained on the ISCX-VPN2016, CICIDS2017, and WIDE datasets [17], whereas MLETC employs a proprietary dataset encompassing diverse real-world protocols. TABLE IV: Comparison with SoTA methods on the CSTNET-TLS dataset Method CSTNET-TLS AC PR RC F1 AppScanner 0.7441 0.7232 0.6963 0.7023 FS-Net 0.7814 0.7670 0.7316 0.7311 ET-BERT 0.7993 0.7832 0.7689 0.7700 YaTC 0.8391 0.8364 0.8101 0.8140 TrafficFormer 0.7982 0.7883 0.7736 0.7704 FlowletFormer 0.8605 0.8578 0.8445 0.8473 NetMamba 0.9258 0.9271 0.9258 0.9249 MMAE 0.9435 0.9459 0.9435 0.9434 TABLE V: Ablation experiments of each component in MMAE Method CrossPlatform(Android) CrossPlatform(iOS) CSTNET-TLS AC PR RC F1 AC PR RC F1 AC PR RC F1 Baseline 0.9852 0.9855 0.9852 0.9850 0.9838 0.9836 0.9836 0.9829 0.9271 0.9284 0.9271 0.9267 + SD 0.9856 0.9858 0.9856 0.9854 0.9843 0.9855 0.9848 0.9840 0.9284 0.9302 0.9284 0.9280 + DMM 0.9878 0.9876 0.9878 0.9873 0.9881 0.9872 0.9881 0.9866 0.9336 0.9356 0.9336 0.9332 + SFM 0.9882 0.9882 0.9882 0.9880 0.9883 0.9857 0.9883 0.9866 0.9353 0.9361 0.9353 0.9346 + PMP 0.9897 0.9898 0.9897 0.9893 0.9900 0.9872 0.9900 0.9881 0.9435 0.9459 0.9435 0.9434 IV-B Main Results The comprehensive results are summarized in Tables I, I, and IV. MMAE consistently achieves state-of-the-art performance across all seven datasets, delivering F1-scores ranging from 0.9434 to 1.0000. While traditional machine learning and early deep learning methods capture primary statistical features, they exhibit pronounced limitations in complex scenarios. In contrast, MMAE sustains remarkable accuracy across all tasks. Although large Transformer-based architectures (e.g., ET-BERT, TrafficFormer, FlowletFormer, and MLETC) demonstrate baseline applicability, they suffer from adaptability and stability issues in traffic classification. For instance, TrafficFormer experiences severe performance degradation on cross-platform datasets (yielding F1-scores of merely 0.6167 and 0.4689) in Table I. Similarly, image-based YaTC introduces unnecessary spatial interference. Standing apart from these NLP- or CV-inspired approaches, MMAE aligns intrinsically with the essence of network traffic. Performance on cross-platform and IoT traffic datasets. MMAE achieves substantial improvements on cross-platform datasets. As shown in Table I, MMAE reaches an accuracy of 0.9897 and an F1-score of 0.9893, delivering a significant 5.11% accuracy improvement over ET-BERT on the CrossPlatform (Android) dataset. On the CrossPlatform (iOS) dataset, MMAE surpasses all baselines with an accuracy of 0.9900, outperforming YaTC by 5.90%. Furthermore, on the IoT traffic dataset CICIoT2022, MMAE attains an accuracy of 0.9990. Performance on highly encrypted and malicious traffic datasets. As shown in Table I, MMAE demonstrates exceptional analytical efficacy. On the ISCXTor2016 dataset, all evaluation metrics achieve a perfect score of 1.0000. On the ISCXVPN2016 and USTC-TFC datasets, the accuracies reach 0.9957 and 0.9995, respectively. These results indicate that our designed architecture (a twin of MAE) and masking strategy (FlowMix) better capture complex dependencies within multi-layer encrypted packets, maintaining higher stability in encrypted environments. Performance under TLS 1.3 protocol. Table IV highlights the limitations of existing models under the highly secure TLS 1.3 protocol. As shown in the Table IV, most baseline models witness significant performance degradation. In such restrictive scenarios, FS-Net leverages the packet-size sequence feature. By contrast, MMAE explicitly utilizes these key discriminant features by employing a multi-granularity modeling paradigm. As a result, it achieves an optimal accuracy of 0.9435 and an F1-score of 0.9434, outperforming the second-best method (NetMamba) by 1.77% and 1.85%, respectively. The overall capacity of MMAE is validated. IV-C Ablation Study To show the effect of each component in MMAE, we conduct an ablation study on the proposed method using a standard masked autoencoder with a random masking strategy as a baseline. We incrementally integrate Self-Distillation (+SD), Dynamic Mixed Masking (+DMM), Statistics-based Flow Matcher (+SFM), and the Packet-importance aware Mask Predictor (+PMP). Table V details the results on the CrossPlatform (Android and iOS), which has the largest number of categories and the highly challenging CSTNET-TLS dataset. Notably, the performance gains are most pronounced on CSTNET-TLS dataset, highlighting the superiority of the teacher-student based twin architecture in decoding complex modern encryption protocols. Specifically, adding the unmasked teacher (+SD) yields consistent improvements under multiple metrics, and introducing DMM brings a further performance improvement. For instance, on CSTNET-TLS, DMM boosts F1-score from 0.9280 to 0.9332. This confirms that breaking single-flow boundaries to construct cross-flow mixed pretext tasks effectively forces the encoder to extract discriminative features under severe interference. Subsequently, the addition of SFM provides a modest but crucial enhancement by ensuring that the paired flows constitute meaningful hard samples. Finally, equipped with PMP yields a much positive impact. On the heavily encrypted CSTNET-TLS dataset, PMP increases accuracy from 0.9353 to 0.9435. By dynamically allocating masking probabilities based on semantic density, PMP resolves the flaw of random masking, thereby maximizing the efficacy of the multi-granularity pre-training paradigm. (a) CrossPlatform(Android) (b) CrossPlatform(iOS) (c) CSTNET-TLS 1.3 Figure 6: Comparison between MAE and MAE+SD on three datasets. The charts show 10 classes exhibiting prominent F1-score enhancements. IV-D Visualization Results F1-score visualization. To demonstrate the effectiveness of our pre-training paradigm, Fig. 6 compares the F1-scores of standard MAE against MAE with Self-Distillation (MAE+SD) across three datasets. We highlight 10 representative traffic classes with the most significant performance gains for each dataset. In the challenging CSTNET-TLS1.3 dataset, standard MAE struggles with many F1-scores falling below 0.80. Similarly, despite the overall better baseline performance, the two cross-platform datasets still suffer from severe bottleneck classes (e.g., Class 0 in Android and Class 25 in iOS). This highlights that relying solely on localized byte-level reconstruction is inadequate for capturing the discriminative semantics required for such complex traffic. By integrating self-distillation, the teacher branch provides flow-level global supervision, driving substantial enhancements across these difficult scenarios. For instance, the F1-score for Class 54 in CSTNET-TLS1.3 surges from 0.55 to 0.82. Meanwhile, the bottleneck classes in the cross-platform datasets have remarkable improvement. For instance, Class 25 in iOS is largely improved from 0.22 to over 0.70. These results comprehensively prove that multi-granularity semantic guidance effectively overcomes the limitations of standard MAE, significantly enhancing the model’s discriminative capability. Visualization of feature distribution. Fig. 7 visualizes the t-SNE projections of the hidden state representations on the CSTNET-TLS 1.3 dataset, where different colors denote distinct categories. As observed, features from the pre-trained MAE are heavily entangled, while MAE+SD exhibits slightly better clustering. After fine-tuning via supervised cross-entropy loss, the separability of MAE in the feature space improves significantly. However, a large number of classes overlap in the central region. In contrast, the proposed MMAE yields highly discriminative representations, achieving distinguishable boundaries in the feature space. (a) Pre-trained MAE (b) Pre-trained MAE+SD (c) Fine-tuned MAE (d) Fine-tuned MMAE (Ours) Figure 7: t-SNE visualization of hidden state representations across different training stages on the CSTNET-TLS 1.3 dataset. Distinct colors denote different traffic categories. MMAE effectively learns highly discriminative representations, exhibiting significantly clearer class boundaries compared to the baseline. (a) Throughput Comparison (b) Memory Comparison (c) Inference Efficiency Comparison Figure 8: Efficiency evaluation of MMAE and baseline models. (a) Inference throughput, (b) GPU memory consumption across varying batch sizes, and (c) Comprehensive cost-effectiveness comparison at a batch size of 64. IV-E Computational Efficiency To evaluate the efficiency of MMAE, we compare its inference throughput and GPU memory consumption across batch sizes ranging from 242^4 to 2102^10. Throughput comparison. As illustrated in Fig. 8 (a), MMAE sustains a stable throughput of approximately 5,000 samples per second. It significantly outpaces large Transformer-based architectures, operating over 10×10× faster than ET-BERT and nearly 1.6×1.6× faster than YaTC. Although the pure state-space model NetMamba achieves the highest speed, MMAE remains highly competitive. Memory comparison. Fig. 8 (b) delineates the GPU memory footprint. MMAE demonstrates controlled memory growth, avoiding the steep memory overhead exhibited by YaTC and ET-BERT at larger batch sizes. Specifically, at a batch size of 64 (262^6), MMAE consumes approximately 50%50\% of the memory required by ET-BERT. While utilizing slightly more resources than the lightweight FS-Net and NetMamba, MMAE consumes substantially less memory than standard Transformer models. Inference efficiency comparison. To illustrate the cost-effectiveness, Fig. 8 (c) presents a multi-dimensional comparison at a fixed batch size of 64. Ultimately, MMAE achieves the highest classification performance with moderate computational overhead. Efficiency and memory comparison of different phases. Table VI summarizes the computational overhead of MMAE across different phases. Despite its strong representational capabilities, the model maintains a lightweight parameter footprint and low GPU memory consumption. TABLE VI: Computational Efficiency Across Different Phases Phase GPUs Time Unit/Granularity Params GPU Memory Pre-training 1 24.66 h 5.918 s/100 steps 8.65 M 18GB Fine-tuning 1 4.6 h 137.87 s / epoch 7.72 M 3.6GB Inference 1 - 5343 samples/sec - 787MB IV-F Limitation Analysis PMP utilizes packet sizes and inter-arrival times to guide the masking process, which makes it susceptible to sophisticated adversarial traffic obfuscation. Advanced defense mechanisms may intentionally destroy these temporal and volumetric patterns, which may degrade PMP’s masking efficacy. Furthermore, the pre-training phase incurs considerable computational overhead, although this is unavoidable for pre-training based paradigms. The optimization of teacher-student branches and the reliance on large batch sizes for cross-flow mixing require substantial GPU memory. Consequently, while inference remains lightweight, online pre-training on resource-constrained edge devices is currently impractical. V Conclusion and Future Work In this paper, we propose MMAE, a teacher-student based twin MAE paradigm with flow mixing strategy (FlowMix), to overcome the limitations of standard MAE in isolated byte-level reconstruction for encrypted traffic classification. By coupling a FlowMix strategy with the PMP, our framework learns robust, multi-granularity semantic representations. Extensive evaluations confirm that MMAE establishes a new state-of-the-art pipeline across seven diverse datasets, demonstrating particular effectiveness in handling complex encryption protocols such as TLS 1.3. Despite these promising results, several issues are still remained for future work. First, it is challenging to explore more robust feature extraction strategies that minimize susceptibility to adversarial obfuscation. Second, it is valuable to develop memory-efficient training pipeline to reduce computational overhead during the pre-training phase, enabling seamless deployment and online learning on resource-constrained edge devices. Finally, we intend to extend MMAE beyond standard classification to broader network management tasks, such as QoS prediction and malicious traffic detection. References [1] K. Al-Naami, S. Chandra, A. Mustafa, L. Khan, Z. Lin, K. Hamlen, and B. Thuraisingham (2016) Adaptive encrypted traffic fingerprinting with bi-directional dependence. In Proceedings of the 32nd Annual Conference on Computer Security Applications, p. 177–188. Cited by: §I-A. [2] A. Andonian, S. Chen, and R. Hamid (2022) Robust cross-modal representation learning with progressive self-distillation. In CVPR, p. 16430–16441. Cited by: §I-C. [3] A. Baevski, W. Hsu, Q. Xu, A. Babu, J. Gu, and M. Auli (2022) Data2vec: a general framework for self-supervised learning in speech, vision and language. In ICML, p. 1298–1312. Cited by: §I-C. [4] L. Bernaille, R. Teixeira, I. Akodkenou, A. Soule, and K. Salamatian (2006) Traffic classification on the fly. ACM SIGCOMM Computer Communication Review 36 (2), p. 23–26. Cited by: §I-A. [5] C. Burges, T. Shaked, E. Renshaw, A. Lazier, M. Deeds, N. Hamilton, and G. Hullender (2005) Learning to rank using gradient descent. In ICML, p. 89–96. Cited by: §I-D. [6] M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin (2021) Emerging properties in self-supervised vision transformers. In ICCV, p. 9650–9660. Cited by: §I-C. [7] X. Chen, L. Han, D. Zhan, and H. Ye (2025) Miett: multi-instance encrypted traffic transformer for encrypted traffic classification. In AAAI, Vol. 39, p. 15922–15929. Cited by: §I-A. [8] Z. Chen, G. Cheng, Z. Wei, D. Niu, and N. Fu (2023) Classify traffic rather than flow: versatile multi-flow encrypted traffic classification with flow clustering. IEEE Transactions on network and service management 21 (2), p. 1446–1466. Cited by: §I. [9] R. Cheng, B. Wu, P. Zhang, P. Vajda, and J. E. Gonzalez (2021) Data-efficient language-supervised zero-shot learning with self-distillation. In CVPR, p. 3119–3124. Cited by: §I-C. [10] S. Dadkhah, H. Mahdikhani, P. K. Danso, A. Zohourian, K. A. Truong, and A. A. Ghorbani (2022) Towards the development of a realistic multidimensional iot profiling dataset. In 2022 19th Annual International Conference on Privacy, Security & Trust (PST), p. 1–11. Cited by: §IV-A1. [11] J. Devlin, M. Chang, K. Lee, and K. Toutanova (2019) Bert: pre-training of deep bidirectional transformers for language understanding. In ACL: human language technologies, volume 1, p. 4171–4186. Cited by: §I-B. [12] X. Dong, J. Bao, Y. Zheng, T. Zhang, D. Chen, H. Yang, M. Zeng, W. Zhang, L. Yuan, D. Chen, et al. (2023) Maskclip: masked self-distillation advances contrastive language-image pretraining. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 10995–11005. Cited by: §I-C. [13] G. Draper-Gil, A. H. Lashkari, M. S. I. Mamun, and A. A. Ghorbani (2016) Characterization of encrypted and vpn traffic using time-related. In ICISSP, p. 407–414. Cited by: §IV-A1. [14] H. Y. He, Z. G. Yang, and X. N. Chen (2020) PERT: payload encoding representation from transformer for encrypted traffic classification. In 2020 ITU Kaleidoscope: Industry-Driven Digital Transformation (ITU K), p. 1–8. Cited by: §I, §I-B. [15] G. Hinton, O. Vinyals, and J. Dean (2015) Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531. Cited by: §I-C. [16] M. Ji, S. Shin, S. Hwang, G. Park, and I. Moon (2021) Refine myself by teaching myself: feature refinement via self-knowledge distillation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, p. 10664–10673. Cited by: §I-C. [17] C. Kenjiro (2000) Traffic data repository at the wide project. In Proc. USENIX 2000 Annual Technical Conference: FREENIX Track, San Diego, CA, Cited by: §IV-A4. [18] K. Kim, B. Ji, D. Yoon, and S. Hwang (2021) Self-knowledge distillation with progressive refinement of targets. In Proceedings of the IEEE/CVF international conference on computer vision, p. 6567–6576. Cited by: §I-C. [19] J. Li, R. Selvaraju, A. Gotmare, S. Joty, C. Xiong, and S. C. H. Hoi (2021) Align before fuse: vision and language representation learning with momentum distillation. Advances in neural information processing systems 34, p. 9694–9705. Cited by: §I-C. [20] L. Li, Y. Zhao, D. Jiang, Y. Zhang, F. Wang, I. Gonzalez, E. Valentin, and H. Sahli (2013) Hybrid deep neural network–hidden markov model (dnn-hmm) based speech emotion recognition. In 2013 Humaine association conference on affective computing and intelligent interaction, p. 312–317. Cited by: §I-A. [21] X. Lin, G. Xiong, G. Gou, Z. Li, J. Shi, and J. Yu (2022) Et-bert: a contextualized datagram representation with pre-training transformers for encrypted traffic classification. In Proceedings of the ACM Web Conference 2022, p. 633–642. Cited by: §I, §I-B, 3rd item, §IV-A1. [22] C. Liu, L. He, G. Xiong, Z. Cao, and Z. Li (2019) Fs-net: a flow sequence network for encrypted traffic classification. In IEEE INFOCOM 2019-IEEE Conference On Computer Communications, p. 1171–1179. Cited by: §I-A, 2nd item. [23] L. Liu, R. Li, Q. Li, M. Hou, Y. Jiang, and M. Xu (2025) FlowletFormer: network behavioral semantic aware pre-training model for traffic classification. arXiv preprint arXiv:2508.19924. Cited by: §I, 3rd item. [24] Z. Liu, Y. Xie, Y. Luo, Y. Wang, and X. Ji (2025) TransECA-net: a transformer-based model for encrypted traffic classification. Applied Sciences 15 (6), p. 2977. Cited by: §I-A. [25] A. W. Moore and D. Zuev (2005) Internet traffic classification using bayesian analysis techniques. In Proceedings of the 2005 ACM SIGMETRICS international conference on Measurement and modeling of computer systems, p. 50–60. Cited by: §I-A. [26] A. Panchenko, F. Lanze, J. Pennekamp, T. Engel, A. Zinnen, M. Henze, and K. Wehrle (2016) Website fingerprinting at internet scale.. In NDSS, Vol. 1, p. 23477. Cited by: §I-A. [27] E. Papadogiannaki and S. Ioannidis (2021) A survey on encrypted network traffic analysis applications, techniques, and countermeasures. ACM Computing Surveys (CSUR) 54 (6), p. 1–35. Cited by: §I. [28] J. Park, Y. Choi, B. Cho, S. Kim, and M. Kim (2025) Multi-level pre-training for encrypted network traffic classification. IEEE Access. Cited by: 3rd item. [29] R. Preis (1999) Linear time 1/2-approximation algorithm for maximum weighted matching in general graphs. In Annual Symposium on Theoretical Aspects of Computer Science, p. 259–269. Cited by: §I-B1. [30] A. Radford, K. Narasimhan, T. Salimans, I. Sutskever, et al. (2018) Improving language understanding by generative pre-training. Cited by: §I-B. [31] M. Roesch et al. (1999) Snort: lightweight intrusion detection for networks.. In Lisa, Vol. 99, p. 229–238. Cited by: §I-A. [32] R. Schuster, V. Shmatikov, and E. Tromer (2017) Beauty and the burst: remote identification of encrypted video streams. In 26th USENIX Security Symposium (USENIX Security 17), p. 1357–1374. Cited by: §I-A. [33] M. Shen, J. Zhang, L. Zhu, K. Xu, and X. Du (2021) Accurate decentralized application identification via encrypted traffic analysis using graph neural networks. IEEE Transactions on Information Forensics and Security 16, p. 2367–2380. Cited by: §I-A. [34] P. Sirinam, M. Imani, M. Juarez, and M. Wright (2018) Deep fingerprinting: undermining website fingerprinting defenses with deep learning. In Proceedings of the 2018 ACM SIGSAC conference on computer and communications security, p. 1928–1943. Cited by: §I, §I-A. [35] V. F. Taylor, R. Spolaor, M. Conti, and I. Martinovic (2017) Robust smartphone app identification via encrypted network traffic analysis. IEEE Transactions on Information Forensics and Security 13 (1), p. 63–78. Cited by: §I-A, 1st item. [36] M. B. Umair, Z. Iqbal, M. Bilal, T. A. Almohamad, J. Nebhen, and R. M. Mehmood (2021) An efficient internet traffic classification system using deep learning for iot. arXiv preprint arXiv:2107.12193. Cited by: §I-A. [37] T. Van Ede, R. Bortolameotti, A. Continella, J. Ren, D. J. Dubois, M. Lindorfer, D. Choffnes, M. Van Steen, and A. Peter (2020) Flowprint: semi-supervised mobile-app fingerprinting on encrypted network traffic. In NDSS, Vol. 27. Cited by: 1st item, §IV-A1. [38] B. Wang, B. Wang, Z. Wei, S. Zhao, S. Chen, Z. Li, and M. Wang (2025) MFSI: multi-flow based service identification for encrypted network traffic. Computer Networks 265, p. 111283. Cited by: §I. [39] T. Wang, X. Xie, W. Wang, C. Wang, Y. Zhao, and Y. Cui (2024) Netmamba: efficient network traffic classification via pre-training unidirectional mamba. In ICNP, p. 1–11. Cited by: §I, 4th item. [40] W. Wang, M. Zhu, X. Zeng, X. Ye, and Y. Sheng (2017) Malware traffic classification using convolutional neural network for representation learning. In ICOIN, p. 712–717. Cited by: §IV-A1. [41] J. Zhang, H. Zhao, Y. Feng, Z. Cai, and L. Zhu (2025) NetST: network encrypted traffic classification based on swin transformer.. Computers, Materials & Continua 84 (3). Cited by: §I-A. [42] J. Zhang, F. Li, F. Ye, and H. Wu (2020) Autonomous unknown-application filtering and labeling for dl-based traffic classifier update. In IEEE INFOCOM, p. 397–405. Cited by: §I-A. [43] X. Zhang and J. Yin (2025) Optimized extreme learning machines with deep learning for high-performance network traffic classification. Scientific Reports 15 (1), p. 33199. Cited by: §I-A. [44] R. Zhao, M. Zhan, X. Deng, F. Li, Y. Wang, Y. Wang, G. Gui, and Z. Xue (2024) A novel self-supervised framework based on masked autoencoder for traffic classification. IEEE/ACM Transactions on Networking 32 (3), p. 2012–2025. Cited by: §I. [45] R. Zhao, M. Zhan, X. Deng, Y. Wang, Y. Wang, G. Gui, and Z. Xue (2023) Yet another traffic classifier: a masked autoencoder based traffic transformer with multi-level flow representation. In AAAI, Vol. 37, p. 5420–5427. Cited by: §I, §I-B, 3rd item. [46] G. Zhou, X. Guo, Z. Liu, T. Li, Q. Li, and K. Xu (2025) Trafficformer: an efficient pre-trained model for traffic data. In 2025 IEEE symposium on security and privacy (SP), p. 1844–1860. Cited by: §I, §I-B, 3rd item. [47] D. Zuev and A. W. Moore (2005) Traffic classification using a statistical approach. In International workshop on passive and active network measurement, p. 321–324. Cited by: §I-A.