Paper deep dive
Mitigating The Effect of Class Imbalance in Data with Hierarchical and Dependable Structure
Bipin Chhetri, Deepika Giri, Avishek Kadel, Rabin Kumar Karki, Akbar Siami Namin
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 7/18/2026, 10:46:27 AM
Summary
This paper addresses the challenge of class imbalance in hierarchical cybersecurity vulnerability classification using the Common Weakness Enumeration (CWE) taxonomy. It demonstrates that traditional oversampling techniques like SMOTE and ADASYN degrade the performance of deep learning models by violating hierarchical parent-child constraints. The authors propose a Hierarchy-Aware RoBERTa framework that incorporates learnable parent-class embeddings to preserve taxonomic consistency, achieving a weighted F1-score of 0.76, which outperforms baselines such as BERT, CNN, and BiGRU, particularly on minority classes.
Entities (13)
Relation Signals (9)
Hierarchy-Aware RoBERTa → mitigates → Class Imbalance
confidence 95% · This paper proposes a Hierarchy-Aware RoBERTa framework that explicitly incorporates CWE structural information... to mitigate class imbalance
Hierarchy-Aware RoBERTa → outperforms → BERT
confidence 95% · Our proposed model achieves a weighted F1-score of 0.76... outperforming all baselines with notable gains on minority classes, including the Class category whose F1-score improved from 0.49 to 0.60 over the BERT baseline.
CWE → hasstructure → Hierarchical Dependencies
confidence 90% · strong hierarchical dependencies among weakness categories... parent-child constraints of the CWE hierarchy
Hierarchy-Aware RoBERTa → uses → SecureBERT+
confidence 90% · A pretrained SecureBERT encoder[1] produces a 768-dimensional [CLS] representation from the input text.
SMOTE → violates → Hierarchical Constraints
confidence 90% · synthetic interpolation in high-dimensional embedding spaces violates the inherent parent-child constraints of the CWE hierarchy
ADASYN → degrades → BiGRU
confidence 85% · BiGRU fell from 0.70 to 0.44 with ADASYN
ADASYN → degrades → CNN
confidence 85% · vanilla CNN dropped from 0.71 to 0.51 with ADASYN
SMOTE → →
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Classifying cybersecurity vulnerabilities using the Common Weakness Enumeration (CWE) taxonomy is challenging due to extreme class imbalance and strong hierarchical dependencies among weakness categories. Although oversampling techniques such as Synthetic Minority Oversampling Technique (SMOTE) and Adaptive Synthetic Sampling (ADASYN) are widely adopted to mitigate class imbalance, their effectiveness for hierarchical CWE text classification remains largely unexplored. This paper proposes a Hierarchy-Aware RoBERTa framework that explicitly incorporates CWE structural information through learnable parent-class embeddings, preserving taxonomic consistency. Our experiments demonstrate that synthetic interpolation in high-dimensional embedding spaces violates the inherent parent-child constraints of the CWE hierarchy, offering only marginal benefits for classical ML models while consistently degrading deep learning architectures. Evaluated on a CWE Research Concept dataset, the proposed model achieves a weighted F1-score of 0.76 without data augmentation, outperforming all baselines with notable gains on minority classes, including the Class category whose F1-score improved from 0.40 to 0.60 over the BERT baseline. Our results suggest that hierarchy-aware representation learning is a more principled alternative to oversampling for structured vulnerability classification.
Tags
Links
- Source: https://arxiv.org/abs/2607.11994v1
- Canonical: https://arxiv.org/abs/2607.11994v1
PDF not stored locally. Use the link above to view on the source site.
Full Text
30,074 characters extracted from source content.
Expand or collapse full text
Mitigating The Effect of Class Imbalance in Data with Hierarchical and Dependable Structure Bipin Chhetri1 Deepika Giri2 Avishek Kadel3 Rabin Kumar Karki4 Akbar Siami Namin1 Department of Computer Science1 Texas Tech University1 Cumberland University2 Yeshiva University3 University of Cumberlands4 bipin.chhetri, akbar.namin2@ttu.edu dgiri25@students.cumberland.edu akadel@mail.yu.edu rkarki34351@ucumberlands.edu Abstract Classifying cybersecurity vulnerabilities using the Common Weakness Enumeration (CWE) taxonomy is challenging due to extreme class imbalance and strong hierarchical dependencies among weakness categories. Although oversampling techniques such as Synthetic Minority Oversampling Technique (SMOTE) and Adaptive Synthetic Sampling (ADASYN) are widely adopted to mitigate class imbalance, their effectiveness for hierarchical CWE text classification remains largely unexplored. This paper proposes a Hierarchy-Aware RoBERTa framework that explicitly incorporates CWE structural information through learnable parent-class embeddings, preserving taxonomic consistency. Our experiments demonstrate that synthetic interpolation in high-dimensional embedding spaces violates the inherent parent-child constraints of the CWE hierarchy, offering only marginal benefits for classical ML models while consistently degrading deep learning architectures. Evaluated on a CWE Research Concept dataset, the proposed model achieves a weighted F1-score of 0.760.76 without data augmentation, outperforming all baselines with notable gains on minority classes, including the Class category whose F1-score improves from 0.490.49 to 0.600.60 over the BERT baseline. Our results suggest that hierarchy-aware representation learning is a more principled alternative to oversampling for structured vulnerability classification. I Introduction Mitigating weaknesses has become increasingly necessary as software systems grow in complexity and adversaries adopt sophisticated attack techniques. Software and hardware weaknesses cataloged in the CWE[15] enable systematic vulnerability assessment, mitigation planning, and security decision making[4]. The CWE taxonomy organizes weaknesses into abstraction levels such as Base, Class, and Pillar, where upper-level nodes capture broad concepts, and lower-level nodes represent specific details. In practice, a small number of weakness types dominate the corpus, while many rare but high-impact categories remain significantly underrepresented. Transformer-based encoders such as BERT [8] and RoBERTa [14] have become strong baselines for CWE text classification, capturing bidirectional contextual dependencies that outperform earlier Convolutional Neural Network (CNN) [17] and Recurrent Neural Network (RNN) architectures [5]. However, class imbalance remains a fundamental challenge that degrades performance on rare CWE categories, and CWE labels carry explicit parent-child dependencies [11]. For instance, a Variant is always a child of a Base, meaning misclassification at higher abstraction levels cascades downward to finer-grained classes. Research on hierarchical classification emphasizes that maintaining parent-child relationships is crucial for building reliable prediction models [11]. Traditional approaches, such as random oversampling and undersampling [18], rebalance label distributions but risk overfitting on minority samples or losing valuable information from the majority class. More advanced methods, SMOTE [3] and ADASYN [9], generate synthetic minority samples in feature space. SMOTE interpolates between minority instances to reduce classifier bias toward majority labels, while ADASYN concentrates synthesis on harder boundary regions where misclassifications are likely. The specific contributions of our work are as follows: 1. We empirically evaluate ML (RF, SVM), deep learning (CNN, BiGRU), and transformer-based (BERT) models under SMOTE and ADASYN resampling on an imbalanced hierarchical CWE dataset. 2. We demonstrate that synthetic interpolation in high-dimensional embedding spaces violates CWE parent-child constraints, marginally helping classical ML models while consistently degrading deep learning models. 3. We introduce a Hierarchy-Aware RoBERTa model that injects CWE structural priors through learnable parent-class embeddings for end-to-end joint modeling of semantic content and hierarchical structure. 4. Our proposed model achieves a weighted F1-score of 0.760.76 without augmentation, outperforming all baselines with Class F1 improving from 0.490.49 to 0.600.60 over BERT. This paper is structured into the following sections. Section I presents the related work. Section I presents the technical background of the models. Sections IV and V contain the methodology and experimental setup, respectively. Section VI presents the results. Section VII presents the discussion and limitations. Section VIII concludes the paper with future work. I Related Work I-A CWE-Based Vulnerability Classification Prior work primarily treats CWE classification as a supervised multiclass problem in data-rich settings. Chhetri et al. [4] showed that BERT outperformed CNN, LSTM, and HAN architectures in predicting the consequences of cyber attacks from the descriptions of CWE, but performance degraded under sparse and imbalanced conditions. VulnBERTa [16], a hierarchical RoBERTa-based classifier trained on reports from the National Vulnerability Database (NVD), achieved strong accuracy on frequent CWE classes but significant degradation on rare labels. Contreras et al. [6] evaluated BI-LSTM and BiGRU models on the Software Assurance Reference Dataset (SARD) and the NVD with strong overall results but struggled to distinguish semantically overlapping CWE classes, highlighting the persistent barrier of long-tail label distributions. I-B Semantic and Hierarchical Classification Previous work [13] applied BERT-based cross-encoders with binary chaining to the CWE View-1003 hierarchy, improving accuracy on high-cardinality label sets but remaining limited in tail classes. V2W-BERT [7] demonstrated that label embeddings and textual CWE definitions improve representations for single-label CWE assignment. Classical ML approaches including Random Forests (RF) [2] and Support Vector Machines (SVM) [10] perform well on frequent weaknesses but consistently degrade on rare ones, motivating approaches that explicitly model label structure. I-C Class Imbalance in Datasets SMOTE [3] and ADASYN [9] have proven effective for classifiers based on classical ML and CNN, and recent work has incorporated oversampling directly into deep learning training loops [12]. However, the behavior of SMOTE and ADASYN on hierarchically structured CWE text embeddings, where parent-child constraints impose structural dependencies that linear interpolation cannot respect, remains largely unexplored, a gap this paper directly addresses. I Technical Background I-A Machine Learning Models I-A1 Random Forest (RF)[2] constructs an ensemble of decision trees on bootstrapped subsets of training data, aggregating predictions to reduce variance and overfitting. Each tree recursively selects splits that maximize impurity reduction, performing well on structured feature spaces, but struggling on high-dimensional sparse text representations. Here, the best split is chosen as the feature and threshold that leads to the largest decrease in impurity between the parent and its children. I-A2 Support Vector Machine (SVM) [10] learns a maximum-margin hyperplane that separates classes in a high-dimensional feature space. A linear kernel is well suited for sparse text representations, making SVM a strong baseline for short-text classification tasks such as CWE descriptions. I-B Deep Learning Models I-B1 Convolutional Neural Networks (CNN) [17] applies 1-D convolutional filters over token embeddings to capture local n-gram patterns, followed by max-pooling to retain the most salient activations. While effective for short range dependencies, CNNs are sensitive to perturbations in embedding space introduced by synthetic oversampling. I-B2 Bidirectional Gated Recurrent Unit (BiGRU)[5] processes sequences in both forward and backward directions, concatenating hidden states at each position to capture full contextual dependencies. The gated mechanism controls information flow across time steps, making BiGRU effective for modeling order-sensitive vulnerability descriptions. I-C Transformers Based Models I-C1 Bidirectional Encoder Representations from Transformers (BERT) [8] uses stacked encoder layers with multi-head self-attention, pretrained via masked language modeling. Robustly Optimized Bidirectional Encoder Representations from Transformers (RoBERTa) [14] improves on BERT by removing next sentence prediction, adopting dynamic masking, and training on substantially larger corpora, yielding stronger contextual representations across NLP benchmarks. SecureBERT [1] extends RoBERTa through continuous pretraining on a large cybersecurity corpus and serves as the encoder backbone for our proposed model. IV Methodology IV-A Oversampling The dataset exhibited severe class imbalance across five classes, i.e., Base (393), Variant (219), Class (83), Compound (8), and Pillar (5). Oversampling was applied exclusively to the training set to prevent data leakage. SMOTE equalized all classes to 393 samples using k-neighbors=2=2, while ADASYN produced near parity counts (Compound 395, Base 393, Pillar 392, Class 387, Variant 362) by concentrating synthesis on harder boundary regions. Figure 1 illustrates the resulting class distributions before and after resampling. Figure 1: Class distribution before and after SMOTE and ADASYN oversampling. IV-B Model Architectures Figure 2: Overview of the Hierarchy-Aware RoBERTa framework. For classical baselines, RF was set with 100 decision trees (n-estimators =100=100) and a fixed random seed (random-state =42=42). The SVM classifier was configured to use a linear kernel, which is appropriate in high-dimensional, sparse feature representations. For deep learning, CNN used 128 convolutional filters of kernel sizes [4,6,6] with ReLU activation and adaptive max pooling, while BiGRU used a hidden dimension of 128 with stacked recurrent layers to capture long-range contextual dependencies. Both models mapped tokens to 256-dimensional embeddings trained end-to-end, fusing a 16-dimensional categorical status embedding via a 128-dimensional fusion layer (ReLU, dropout 0.2). BERT-base (uncased) used the [CLS] pooled output fused with a 16-dimensional status embedding, followed by a 128-dimensional linear layer (ReLU, dropout 0.2). All models were trained with a batch size of 8 for 15 epochs. IV-C Hierarchy-Aware RoBERTa We propose a Hierarchy-Aware RoBERTa architecture (Algorithm 1) that explicitly incorporates label hierarchy information into Transformer-based text classification. A pretrained SecureBERT encoder[1] produces a 768-dimensional [CLS] representation from the input text. To inject hierarchical knowledge, we introduce a learnable parent-category embedding that captures coarse-grained structural relationships among labels. Both parent categories are projected to a 24-dimensional dense embedding, trained together with the rest of the network. The BERT pooled representation is then fused with the parent embedding, resulting in a unified feature representation. This represents both semantic content and hierarchical context. This fused representation is regularized using dropout with a rate of 0.3 before being passed to a linear classification head that outputs logits over the target classes. Algorithm 1 Hierarchy-Aware RoBERTa for Text Classification 1:Input tokens X=x1,…,xLX=\x_1,…,x_L\, attention mask M, parent category IDs P, pretrained RoBERTa encoder ℬB, parent embedding matrix pE_p 2:Class logits y y 3:Encode input sequence using RoBERTa: 4: ←ℬ(X,M)H (X,M) 5:Extract pooled representation: 6: cls←[CLS]h_cls _[CLS] 7:Obtain parent embedding: 8: p←p(P)h_p _p(P) 9:Fuse semantic and hierarchical representations: 10: f←[cls,p]h_f←[h_cls,h_p] 11:Apply dropout: 12: ~f←Dropout(f) h_f (h_f) 13:Compute class logits: 14: y^←~f+ y h_f+b 15:Optionally, obtain class probabilities and loss (training): 16: ←softmax(y^)p ( y) 17: ℒ←CrossEntropy(,y)L (p,y) 18:return y y Figure 2 illustrates the proposed Hierarchy-Aware RoBERTa architecture, which combines semantic representations from a pretrained RoBERTa encoder [1] with learned parent-label embeddings. The textual embedding and hierarchical context are fused via concatenation and passed through a dropout-regularized linear classifier to produce fine-grained child class predictions. This design enables hierarchy-informed classification by jointly leveraging contextual language understanding and structured label information. To ensure consistency across all models, deep learning models (CNN, BiGRU, BERT), and our proposed Hierarchy-Aware RoBERTa were all trained using the same batch size of 8 and 15 epochs. V Experimental Procedure V-A Dataset The dataset used in this study was sourced from an enhanced version of the MITRE Common Weakness Enumeration (CWE) dataset. The CWE repository is continuously updated and maintained to incorporate newly identified software vulnerabilities and attack patterns. The dataset111https://cwe.mitre.org/data/slices/1000.html contains 944 data points with 23 columns. For the experiments, we selected three columns (i.e., CWE-ID, Name, Weakness Abstraction, Description). The Weakness Abstraction has five unique values as presented in Figure 3. The dataset includes five classes in “WeaknessAbstractionWeaknessAbstraction” column, Base sample of 524, Variant sample of 292, Class sample of 111, Pillar sample of 10, and a Compound sample of 7. Figure 3: Frequency of Weakness Abstraction Labels. V-B Data Processing In this experiment, all five abstraction classes were retained for classification. The Name and Description fields were concatenated using a [SEP] token after removing noise patterns and trimming whitespace. Figure 4 illustrates a sample of the processed dataset. Labels from Weakness Abstraction were encoded as integers (0–4) and tokenized using BertTokenizer. Data were split 75:25 for training and testing, with all inputs padded or truncated to a maximum sequence length of 128 tokens. Figure 4: A sample of CWE dataset after data preprocessing Figure 5: Model Accuracy Comparison Across Models VI Results VI-A Model Accuracy Figure 5 compares the performance of various machine learning, deep learning, and transformer-based models in terms of their accuracy as the main metric. RF achieved a baseline accuracy of 0.65, improving its performance to 0.69 with both SMOTE and ADASYN. SVM performed better than RF with an accuracy of 0.72, while SMOTE and ADASYN demonstrated the same performance, i.e., 0.71 and 0.72, respectively. Such findings suggest that oversampling is weakly helpful with classical models and helps address class imbalance. Similarly, the vanilla CNN dropped from 0.71 to 0.55 with SMOTE and 0.51 with ADASYN, while BiGRU fell from 0.70 to 0.53 and 0.44, respectively, confirming that oversampling can introduce noise that adversely impacts sequence-based and convolutional frameworks. BERT achieved an accuracy of 0.74, while the proposed Hierarchy-Aware RoBERTa achieved the highest accuracy of 0.76, demonstrating the advantage of structural priors over synthetic data generation. VI-B Classification Performance TABLE I: Per-Class Performance of Models Across Data Augmentation Methods Category Model Class BASELINE SMOTE ADASYN Precision Recall F1 Precision Recall F1 Precision Recall F1 ML RF Base 0.63 0.95 0.76 0.68 0.89 0.77 0.70 0.86 0.77 Class 0.50 0.04 0.07 0.60 0.32 0.42 0.67 0.43 0.52 Compound 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 Pillar 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 Variant 0.77 0.37 0.50 0.76 0.51 0.61 0.70 0.53 0.60 Weighted Average 0.65 0.65 0.58 0.68 0.69 0.67 0.68 0.69 0.68 SVM Base 0.74 0.85 0.79 0.73 0.84 0.78 0.74 0.85 0.79 Class 0.50 0.46 0.48 0.50 0.46 0.48 0.50 0.46 0.48 Compound 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 Pillar 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 Variant 0.78 0.63 0.70 0.76 0.62 0.68 0.78 0.63 0.70 Weighted Average 0.71 0.72 0.71 0.70 0.71 0.70 0.71 0.72 0.71 DL CNN Base 0.70 0.92 0.79 0.70 0.70 0.70 0.70 0.66 0.68 Class 0.56 0.18 0.27 0.14 0.11 0.12 0.09 0.07 0.08 Compound 0.00 0.00 0.00 0.02 0.50 0.04 0.02 0.50 0.04 Pillar 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 Variant 0.79 0.58 0.67 0.90 0.48 0.62 0.79 0.45 0.57 Weighted Average 0.70 0.71 0.68 0.68 0.56 0.60 0.64 0.52 0.56 BI-GRU Base 0.74 0.80 0.77 0.81 0.56 0.67 0.77 0.41 0.54 Class 0.50 0.32 0.39 0.19 0.14 0.16 0.22 0.25 0.23 Compound 0.00 0.00 0.00 0.00 0.00 0.00 0.02 0.50 0.04 Pillar 0.00 0.00 0.00 0.11 1.00 0.20 0.06 0.50 0.11 Variant 0.70 0.73 0.71 0.61 0.63 0.62 0.66 0.58 0.61 Weighted Average 0.68 0.71 0.69 0.66 0.53 0.58 0.66 0.44 0.52 BERT Base 0.82 0.75 0.78 - - - - - - Class 0.57 0.43 0.49 - - - - - - Compound 0.00 0.00 0.00 - - - - - - Pillar 0.00 0.00 0.00 - - - - - - Variant 0.69 0.89 0.78 - - - - - - Weighted Average 0.74 0.74 0.73 - - - - - - Ours Hierarchy-Aware RoBERTa Base 0.78 0.85 0.81 - - - - - - Class 0.56 0.64 0.60 - - - - - - Compound 0.00 0.00 0.00 - - - - - - Pillar 0.00 0.00 0.00 - - - - - - Variant 0.84 0.76 0.76 - - - - - - Weighted Average 0.76 0.76 0.76 - - - - - - Table I reports the precision, recall, and F1-score per-class across all models and augmentation conditions. The improvement was primarily observed in minority classes. For example, the Class F1-score increased from 0.07 to 0.42 with SMOTE and 0.52 with ADASYN for RF. However, Compound and Pillar remained at 0.00 F1 across all settings. SVM showed stronger overall performance with a weighted F1 of 0.71, maintaining stable scores of 0.70-0.71 under both oversampling methods, indicating robustness on well-represented classes. For deep learning models, CNN achieved a baseline weighted F1 of 0.68, with strong performance on Base (F1=0.79=0.79) and Variant (F1=0.67=0.67). However, applying SMOTE and ADASYN reduced the performance to 0.60 and 0.56, respectively, with Class F1 dropping to as low as 0.08 under ADASYN. BiGRU achieved a baseline weighted F1 of 0.69, dropping to 0.58 with SMOTE and 0.52 with ADASYN, with substantial degradation in minority classes, highlighting the sensitivity of sequence-based models to oversampling-induced noise. Transformer-based BERT achieved a strong baseline weighted F1 of 0.73, driven by high performance on Base (F1=0.78=0.78) and Variant (F1=0.78=0.78). No augmentation was applied for BERT, as its contextual embeddings were sufficient without synthetic oversampling. Our Hierarchy-Aware RoBERTa further improved performance, achieving the highest weighted F1 of 0.76, consistently outperforming all models across Base (F1=0.81=0.81), Class (F1=0.60=0.60), and Variant (F1=0.76=0.76). In particular, Compound and Pillar remain at 0.00 F1 in all models, reflecting the persistent challenge of supervised learning under extreme data sparsity. VII Discussion Although data augmentation techniques such as SMOTE and ADASYN are widely used to address class imbalance in classical machine learning, they are not well suited for transformer-based models. BERT does not use fixed, low-dimensional feature vectors, but learns contextualized, high-dimensional semantic representations directly from raw text, meaning that synthetic interpolation produces samples that lack linguistic coherence and add noise rather than signal. For CNN and BiGRU models, small perturbations in embedding space correspond to large taxonomic shifts, causing models to overfit to synthetic artifacts and degrading generalization performance on minority and hierarchical classes. Furthermore, SMOTE and ADASYN do not conserve the hierarchical relationship that exists within the data. Classes such as Variant are semantically dependent on their parent classes (e.g., Base). Synthetic oversampling does not respect such constraints and can produce samples that are inconsistent with their underlying class structure and confuse the model during training. In the CWE taxonomy, each entry links to a Related Weakness via an explicit parent-child chain, meaning errors at higher abstraction levels cascade downward to finer-grained classes, a structural dependency that oversampling methods ignore entirely by treating each sample as an independent entity in a flat feature space. CWEs are not isolated texts but are organized as a tree structured graph where more abstract weaknesses act as semantic parents to narrower ones, and interpolating between embeddings from different parent nodes produces representations of non-existent or indistinct weaknesses, giving statistically balanced but structurally invalid embeddings. Our Hierarchy-Aware RoBERTa model explicitly incorporates structural knowledge by introducing a learnable parent embedding. In the CWE taxonomy, weakness abstractions are strictly hierarchical. For instance, a "Variant" class is semantically and structurally dependent on its corresponding "Base" parent. To model this dependency, we formulate the classification task not as an independent prediction P(Ychild|Xtext)P(Y_child|X_text), but as a conditional probability problem P(Ychild|Xtext,Yparent)P(Y_child|X_text,Y_parent). We utilize "Parent ID" as a structural prior to guide the model to learn that if the parent is a Base type, the child is likely a Variant. We define two input vectors: Text Vector (VtextV_text): The output from BERT (768 dimensions). Parent Vector (VgraphV_graph): A learnable embedding for the parent ID (24 dimensions). The model fuses them via concatenation:Vfinal=Concat(Vtext,Vgraph)V_final=Concat(V_text,V_graph), Vfinal∈ℝ768+24V_final ^768+24. The classifier then predicts based on this combined knowledge: ypred=Softmax(W⋅Vfinal+b)y_pred=Softmax(W· V_final+b). VII-A Limitations of the Hierarchy-Aware RoBERTa Model Despite its improved performance, the proposed Hierarchy-Aware RoBERTa model has several limitations. First, the architecture assumes parent-level metadata availability during inference, though this is consistent with coarse-to-fine vulnerability pipelines where high-level categories are readily identifiable via static analysis tools (SAST) or manual triage. Second, the hierarchy is treated as fixed, and the model may not generalize to new or rearranged nodes without retraining, as the CWE taxonomy is subject to continual updates. Third, only direct parent information is captured, disregarding multi-hop ancestors, sibling relationships, and tree depth, which may limit discrimination among deeply nested classes such as Compound and Pillar. When training data is virtually absent, hierarchy awareness alone is insufficient, suggesting that weak supervision or external knowledge may be necessary. Finally, the learnable parent embeddings add modest parameter overhead that could hinder scaling to large taxonomies, and the approach has been evaluated only within the CWE domain, leaving generalizability to other hierarchical tasks empirically unvalidated. VIII Conclusion This study examined the effectiveness of oversampling techniques and hierarchy-aware modeling to address class imbalance in CWE vulnerability classification. Through extensive experimentation, we demonstrated that commonly used oversampling methods, such as SMOTE and ADASYN, provide limited benefits for machine learning models (i.e., RF and SVM) and consistently degrade performance for deep learning models (i.e., CNN and BiGRU). These methods generate synthetic samples through linear interpolation in embedding space, an assumption that does not hold for contextualized text representations and often disrupts the semantic and hierarchical integrity of CWE labels. To overcome these limitations, we proposed a Hierarchy-Aware RoBERTa model that explicitly integrates parent–child relationships from the CWE taxonomy into the classification process. By incorporating learnable parent embeddings alongside contextual language representations, the model enforces structural consistency and improves discrimination among minority and semantically dependent classes. The proposed model achieves a weighted F1-score of 0.760.76 without data augmentation, outperforming all baselines, including BERT, which achieved 0.740.74, demonstrating that structural priors are more effective than synthetic data generation in hierarchical imbalanced cybersecurity datasets. Despite these gains, extremely sparse categories such as Compound and Pillar remain difficult to predict, highlighting the inherent limitations of supervised learning when training data is severely limited. Future research may extend the hierarchy-aware framework to incorporate multi-hop ancestry and graph neural networks over the full CWE hierarchy, explore weak supervision and label propagation for extremely sparse classes, and evaluate the proposed approach across other hierarchical cybersecurity datasets to establish its generalizability beyond the CWE domain. Acknowledgment This research is partially supported by the U.S. National Science Foundation (Award #: 2319802). References [1] E. Aghaei, X. Niu, W. Shadid, and E. Al-Shaer (2022) Securebert: a domain-specific language model for cybersecurity. In International Conference on Security and Privacy in Communication Systems, p. 39–56. Cited by: §I-C1, §IV-C, §IV-C. [2] L. Breiman (2001) Random forests. Machine learning 45 (1), p. 5–32. Cited by: §I-B, §I-A1. [3] N. V. Chawla, K. W. Bowyer, L. O. Hall, and W. P. Kegelmeyer (2002) SMOTE: synthetic minority over-sampling technique. Journal of Artificial Intelligence Research 16, p. 321–357. External Links: Link Cited by: §I, §I-C. [4] B. Chhetri and A. S. Namin (2025) The application of transformer-based models for predicting consequences of cyber attacks. In 2025 IEEE 49th Annual Computers, Software, and Applications Conference (COMPSAC), p. 523–532. Cited by: §I, §I-A. [5] J. Chung, C. Gulcehre, K. Cho, and Y. Bengio (2014) Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555. Cited by: §I, §I-B2. [6] C. Contreras, H. Dokic, Z. Huang, D. Stan Raicu, J. Furst, and R. Tchoua (2023) Multiclass classification of software vulnerabilities with deep learning. In Proceedings of the 2023 15th International Conference on Machine Learning and Computing, p. 134–140. Cited by: §I-A. [7] S. S. Das, E. Serra, M. Halappanavar, A. Pothen, and E. Al-Shaer (2021) V2w-bert: a framework for effective hierarchical multiclass classification of software vulnerabilities. In 2021 IEEE 8th International Conference on Data Science and Advanced Analytics (DSAA), p. 1–12. Cited by: §I-B. [8] J. Devlin, M. Chang, K. Lee, and K. Toutanova (2019) Bert: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), p. 4171–4186. Cited by: §I, §I-C1. [9] H. He, Y. Bai, E. A. Garcia, and S. Li (2008) ADASYN: adaptive synthetic sampling approach for imbalanced learning. In IEEE International Joint Conference on Neural Networks (IJCNN), p. 1322–1328. External Links: Link Cited by: §I, §I-C. [10] M. A. Hearst, S. T. Dumais, E. Osuna, J. Platt, and B. Scholkopf (1998) Support vector machines. IEEE Intelligent Systems and their applications 13 (4), p. 18–28. Cited by: §I-B, §I-A2. [11] R. Jiang, M. Liu, H. Liu, and Y. Yu (2025) Hierarchy-consistent learning and adaptive loss balancing for hierarchical multi-label classification. In Proceedings of the 34th ACM International Conference on Information and Knowledge Management, p. 1190–1199. Cited by: §I, §I. [12] S. Kishanthan and A. Hevapathige (2025) Deep learning meets oversampling: a learning framework to handle imbalanced classification. International Journal of Information Technology, p. 1–13. Cited by: §I-C. [13] K. Kota, A. Manjunatha, et al. (2024) CWE prediction using cve description-the semantic similarity approach. Procedia Computer Science 235, p. 1167–1178. External Links: Document Cited by: §I-B. [14] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov (2019) Roberta: a robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. Cited by: §I, §I-C1. [15] MITRE (2025) CWE- home. Note: https://cwe.mitre.org/about/indexAccessed: Dec. 14, 2025 Cited by: §I. [16] H. Turtiainen and A. Costin (2024) VulnBERTa: on automating CWE weakness assignment and improving the quality of cybersecurity CVE vulnerabilities through ml/nlp. In 2024 IEEE European Symposium on Security and Privacy Workshops (EuroS&PW), p. 618–625. External Links: Document Cited by: §I-A. [17] R. Yamashita, M. Nishio, R. K. G. Do, and K. Togashi (2018) Convolutional neural networks: an overview and application in radiology. Insights into imaging 9 (4), p. 611–629. Cited by: §I, §I-B1. [18] C. Yang, E. A. Fridgeirsson, J. A. Kors, J. M. Reps, and P. R. Rijnbeek (2024) Impact of random oversampling and random undersampling on the performance of prediction models developed using observational health data. Journal of Big Data 11 (1), p. 7. Cited by: §I.