Paper deep dive
Detecting Cybersecurity Threats by Integrating Explainable AI with SHAP Interpretability and Strategic Data Sampling
Norrakith Srisumrith, Sunantha Sodsee
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 95%
Last extracted: 7/20/2026, 8:41:03 PM
Summary
This paper presents an integrated Explainable AI (XAI) framework for cybersecurity threat detection using the CIC-IDS2017 dataset. The methodology addresses three key challenges: handling massive datasets via a Strategic Sampling Methodology that preserves class distributions, ensuring experimental rigor through Automated Data Leakage Prevention, and providing operational transparency using SHAP analysis. The framework achieves high detection accuracy (99.92%) while reducing computational overhead and delivering actionable explanations for security analysts, demonstrating that explainability, efficiency, and integrity can be simultaneously achieved in Security Operations Centers.
Entities (14)
Relation Signals (11)
Sunantha Sodsee → affiliatedwith → King Mongkut’s University of Technology North Bangkok
confidence 99% · Sunantha Sodsee Dept. of Digital Network and Information Security Management King Mongkut’s University of Technology North Bangkok
Norrakith Srisumrith → affiliatedwith → King Mongkut’s University of Technology North Bangkok
confidence 99% · Norrakith Srisumrith Dept. of Digital Network and Information Security Management King Mongkut’s University of Technology North Bangkok
SHAP → provides → model-agnostic interpretability
confidence 96% · Integrated XAI Implementation using SHAP analysis for model-agnostic interpretability across algorithms
Explainable AI → enables → operational transparency
confidence 95% · providing operational transparency through Integrated XAI Implementation
CIC-IDS2017 → usedby → Strategic Sampling Methodology
confidence 95% · Applied to the CIC-IDS2017 dataset, our approach maintains detection efficacy... handling massive datasets through Strategic Sampling Methodology
Strategic Sampling Methodology → preserves → class distributions
confidence 94% · Strategic Sampling Methodology that preserves class distributions while enabling efficient model development
Automated Data Leakage Prevention → removes → contaminated features
confidence 93% · Automated Data Leakage Prevention that systematically identifies and removes contaminated features
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:The critical need for transparent and trustworthy machine learning in cybersecurity operations drives the development of this integrated Explainable AI (XAI) framework. Our methodology addresses three fundamental challenges in deploying AI for threat detection: handling massive datasets through Strategic Sampling Methodology that preserves class distributions while enabling efficient model development; ensuring experimental rigor via Automated Data Leakage Prevention that systematically identifies and removes contaminated features; and providing operational transparency through Integrated XAI Implementation using SHAP analysis for model-agnostic interpretability across algorithms. Applied to the CIC-IDS2017 dataset, our approach maintains detection efficacy while reducing computational overhead and delivering actionable explanations for security analysts. The framework demonstrates that explainability, computational efficiency, and experimental integrity can be simultaneously achieved, providing a robust foundation for deploying trustworthy AI systems in security operations centers where decision transparency is paramount.
Tags
Links
- Source: https://arxiv.org/abs/2602.19087v1
- Canonical: https://arxiv.org/abs/2602.19087v1
Trouble viewing inline? Open PDF directly →
Full Text
52,808 characters extracted from source content.
Expand or collapse full text
Detecting Cybersecurity Threats by Integrating Explainable AI with SHAP Interpretability and Strategic Data Sampling Norrakith Srisumrith Dept. of Digital Network and Information Security Management King Mongkut’s University of Technology North Bangkok Bangkok, Thailand norrakith.s@email.kmutnb.ac.th Sunantha Sodsee Dept. of Digital Network and Information Security Management King Mongkut’s University of Technology North Bangkok Bangkok, Thailand sunantha.s@itd.kmutnb.ac.th Abstract—The critical need for transparent and trustworthy machine learning in cybersecurity operations drives the devel- opment of this integrated Explainable AI (XAI) framework. Our methodology addresses three fundamental challenges in deploying AI for threat detection: handling massive datasets through Strategic Sampling Methodology that preserves class distributions while enabling efficient model development; ensur- ing experimental rigor via Automated Data Leakage Prevention that systematically identifies and removes contaminated features; and providing operational transparency through Integrated XAI Implementation using SHAP analysis for model-agnostic in- terpretability across algorithms. Applied to the CIC-IDS2017 dataset, our approach maintains detection efficacy while reducing computational overhead and delivering actionable explanations for security analysts. The framework demonstrates that explain- ability, computational efficiency, and experimental integrity can be simultaneously achieved, providing a robust foundation for deploying trustworthy AI systems in security operations centers where decision transparency is paramount. Index Terms—Explainable AI, Cybersecurity, Multi-class Clas- sification, Feature Selection, SHAP Analysis, Network Intrusion Detection I. INTRODUCTION The exponential growth of network-based attacks and so- phisticated cyber threats has fundamentally transformed cy- bersecurity from a traditional perimeter defense challenge to a complex data-driven detection problem [1], [2]. With the increasing volume and sophistication of cyber attacks, machine learning approaches have demonstrated remarkable success in identifying malicious activities across diverse network envi- ronments [3], [4]. However, the practical deployment of these advanced detection systems faces significant hurdles due to the inherent ”black box” nature of complex models, particularly ensemble methods and deep learning architectures that achieve state-of-the-art performance [5], [6]. This interpretability gap becomes critically problematic in security operations centers (SOCs), where analysts require not only high detection accu- racy but also transparent, understandable decisions that they can validate, trust, and act upon during incident response This manuscript has been accepted for publication in ICTIS 2026 and will appear in Springer Lecture Notes in Networks and Systems (LNNS). The final authenticated version will be available via SpringerLink. scenarios [7], [8]. The challenge is further compounded by the massive scale of modern network datasets, such as CIC- IDS2017 with over 2.8 million records, which necessitates efficient sampling strategies without compromising detection efficacy or introducing statistical biases [9], [10]. Recent advancements in Explainable AI (XAI) have begun to address these transparency concerns, with SHAP (SHapley Additive exPlanations) emerging as a prominent framework for model interpretation due to its strong theoretical foun- dations and model-agnostic capabilities [11], [12]. However, current implementations often treat explainability as a post- hoc analysis layer rather than integrating it throughout the machine learning pipeline [13], [14]. Simultaneously, feature selection methodologies have evolved to not only improve computational efficiency but also enhance model interpretabil- ity by identifying the most discriminative characteristics while reducing dimensionality [15], [16]. The work of Zhou et al. [17] on adaptive ensemble learning and Ma et al. [10] on fea- ture selection with large language models represents important steps toward practical deployment, yet these approaches often prioritize performance optimization over operational trans- parency. Furthermore, critical challenges around data leakage prevention, as highlighted by Pendlebury et al. [18] and Arp et al. [7], underscore the importance of rigorous experimental methodologies that ensure model evaluations reflect real-world deployment scenarios. Our research addresses these challenges through an in- tegrated Explainable AI framework that combines strategic sampling, state-of-the-art performance, and comprehensive in- terpretability. The main contributions are: 1) Strategic Sampling Methodology: A novel stratified sampling approach that maintains class distribution while enabling efficient model development 2) Automated Data Leakage Prevention: Comprehensive detection and removal of 37% potentially leaky features. 3) Systematic Algorithm Evaluation: Multi-stage vali- dation across XGBoost, Random Forest, and Logistic Regression 4) Feature Selection Optimization: Comparative analysis of MRMR and Chi2 methods across multiple reduction arXiv:2602.19087v1 [cs.CR] 22 Feb 2026 percentages 5) Integrated XAI Implementation: SHAP analysis for model-agnostic interpretability across all algorithms I. RELATED WORK A. Machine Learning in Cybersecurity Machine learning (ML) methods for cybersecurity threat detection have developed considerably in the last decade from traditional signature-based approaches to advanced behavior based anomaly detection systems [3], [19]. Early techniques in the detection models targeted traditional algorithms such as Decision Trees, Naive Bayes, and SVMs which had reasonable detection capabilities but suffered from adaptive attack patterns or scalability to large network scale [1]. The emergence of comprehensive benchmark datasets like CIC-IDS2017 [20], [21] has enabled more rigorous evaluation of ML approaches in realistic network settings. Recent studies have demonstrated varying performance levels, with ensemble methods and deep learning architectures showing particular promise. Ferrag et al. [4] conducted an extensive comparative study showing that deep learning approaches could achieve up to 97.16% accuracy on CIC-IDS2017, while Liu and Lang [22] provided a comprehensive survey highlighting the trade- offs between different ML paradigms in security contexts. Recent advancements have focused on addressing the com- putational challenges of large-scale security datasets. DistilX- IDS [9] introduced an efficient, lightweight transformer-based language model for real-time network intrusion detection, achieving high performance while maintaining explainability. Similarly, Ma et al. [10] proposed an IoT intrusion detection framework based on feature selection and large language mod- els fine-tuning, achieving improved efficiency through strategic feature reduction. These approaches represent significant steps toward practical deployment of ML systems in operational security environments. B. Ensemble Methods and Hybrid Models for Intrusion De- tection Ensemble learning has emerged as a powerful paradigm for intrusion detection, leveraging multiple models to improve accuracy and robustness beyond what individual classifiers can achieve. Recent work has explored various ensemble strategies, including voting, bagging, boosting, and stack- ing approaches. Among these, hybrid ensemble models that combine multiple ensemble techniques have shown particular promise in handling the complexity and variability of modern network threats. Ahmed et al. [23] proposed a novel Hybrid Adaptive Ensem- ble for Intrusion Detection (HAEnID) that integrates three dis- tinct ensemble methods: Stacking Ensemble (SEM), Bayesian Model Averaging (BMA), and Conditional Ensemble Method (CEM). Their approach demonstrated exceptional performance on the CIC-IDS2017 dataset, with BMA achieving 98.79% accuracy when using 20 selected features. The HAEnID model incorporates an adaptive mechanism that allows ensemble components to evolve as network traffic patterns change, pro- viding resilience against emerging attack vectors. Furthermore, the model integrates Explainable AI techniques including SHAP and LIME to enhance interpretability, addressing the ”black box” problem common in complex ensemble systems. A key innovation in the HAEnID framework is its strategic use of SMOTE (Synthetic Minority Oversampling Technique) [24] to address class imbalance in multi-class classification tasks. By generating synthetic samples for minority attack classes, the model ensures balanced representation during training, significantly improving detection rates for rare attack types. This sampling strategy, combined with comprehensive feature selection methods including correlation analysis, in- formation gain, and PCA, enables the model to maintain high performance while reducing computational complexity. The model represents a significant advancement in ensemble-based intrusion detection, demonstrating that hybrid approaches can achieve both high accuracy and practical interpretability. C. Explainable AI in Security Applications The growing complexity of machine learning models has spurred substantial research interest in explainable AI tech- niques, particularly in security-critical applications where de- cision transparency is essential [5], [6]. The fundamental chal- lenge lies in balancing model performance with interpretabil- ity, as noted by Miller [8], who emphasized that explanations must be comprehensible to human operators while accurately representing model reasoning. SHAP (SHapley Additive exPlanations) has emerged as a prominent framework for model interpretation due to its strong theoretical foundations and model-agnostic capabilities [11], [12]. Lundberg et al. [12] demonstrated that TreeSHAP could provide consistent and accurate feature importance explana- tions for tree-based models, making it particularly suitable for cybersecurity applications where random forests and gradient boosting machines are commonly employed. Recent research has produced several notable XAI im- plementations for intrusion detection. Mahmoud et al. [25] proposed XI2S-IDS, a two-stage framework that separates binary anomaly detection from multi-class attack classifica- tion, achieving 99.81% accuracy on CICIDS2017 while using SHAP to provide global explanations for the binary classifier. Their work specifically addresses the challenge of detecting low-frequency attacks through targeted multi-class training on attack-only data. Similarly, in the domain of hybrid ensem- ble models, recent work has demonstrated that combining multiple algorithms with SHAP-based explanations can yield both high performance and interpretability. This approach typically achieves superior detection rates while maintaining transparency through feature importance analysis and decision rationalization. Gait ́ an-C ́ ardenas et al. [13] developed explainable AI-based intrusion detection systems for cloud and IoT environments, demonstrating how XAI can enhance trust in automated security decisions. Uddin et al. [14] applied explainable transformer-based models for phishing email detection using a large language model approach, achieving high accuracy while providing natural language explanations for security alerts. These approaches highlight the growing recognition of XAI as essential for operational security systems. D. Feature Selection and Sampling Strategies Feature selection has long been recognized as a critical component of effective machine learning systems, particularly in high-dimensional domains like cybersecurity [26], [16]. Traditional approaches include filter methods (such as Chi- squared and mutual information), wrapper methods, and em- bedded methods, each with distinct advantages and limitations. Peng et al. [15] introduced the MRMR (Maximum Rele- vance Minimum Redundancy) criterion, which selects features that have high mutual information with the target while having low mutual information with other features. This approach has proven particularly effective in cybersecurity contexts where feature interdependencies can obscure detection pat- terns. Forman [27] provided an extensive empirical study of feature selection metrics, demonstrating that the choice of selection method significantly impacts model performance and interpretability. Recent work has explored the intersection of feature selec- tion and interpretability. Ma et al. [10] developed an IoT in- trusion detection framework combining feature selection with large language model fine-tuning, showing that careful feature selection can improve both performance and explainability. Their approach maintained high accuracy while providing clearer decision boundaries for security analysis. Similarly, DistilXIDS [9] incorporated efficient feature processing within their transformer architecture, demonstrating that strategic feature handling can enhance real-time detection capabilities. Sampling strategies have gained increased attention as cy- bersecurity datasets continue to grow in scale. Ahmed et al. [23] employed SMOTE [24] to address class imbalance in their hybrid ensemble model, demonstrating that synthetic over- sampling of minority classes can significantly improve multi- class classification performance for rare attack types. The computational efficiency demonstrated in Chen et al.’s RIDE framework [28] highlights the importance of strategic data handling for real-time intrusion detection. Their hardware- accelerated approach preserved detection performance while enabling efficient inference, making real-time model deploy- ment feasible in resource-constrained environments. E. Data Leakage Prevention in Cybersecurity ML Data leakage represents a significant challenge in cybersecu- rity machine learning, where subtle temporal dependencies and feature-target relationships can artificially inflate performance metrics [7]. Pendlebury et al. [18] introduced TESSERACT, a framework for eliminating experimental bias in malware classification across space and time, highlighting the criti- cal importance of proper dataset partitioning and evaluation methodologies. Anderson and Roth [2] emphasized the need for careful dataset construction in their work on EMBER, an open dataset for training static PE malware machine learning models. Their methodology includes rigorous procedures for preventing data leakage through temporal splitting and feature analysis, setting important precedents for the cybersecurity ML community. Recent work by Arp et al. [7] provided comprehensive guidelines for machine learning in computer security, high- lighting common pitfalls in experimental design and eval- uation. Their ”dos and don’ts” framework has become an essential reference for researchers working in this domain, emphasizing the importance of realistic evaluation scenarios and proper handling of temporal aspects in security data. F. Integration Gaps and Research Opportunities While recent research has advanced individual components of IDS pipelines, important gaps remain in their holistic, large- scale, and evaluation-rigorous integration for enterprise-grade deployment. Existing frameworks often optimize a single aspect (e.g., architecture, imbalance handling, or XAI) or combine techniques without explicitly addressing the com- putational and operational constraints of modern Security Operations Centers (SOCs). For example, HAEnID inte- grates SMOTE-based class rebalancing, multi-step feature selection, and SHAP/LIME explanations within a hybrid en- semble, illustrating the value of combined methods [23]. Likewise, XI2S-IDS employs a two-stage architecture with SHAP-based explanations and achieves strong performance on CIC-IDS2017 [25]. However, these works primarily tar- get algorithmic accuracy and interpretability, leaving several pipeline-level challenges insufficiently addressed. First, there is a gap in handling the scale and form of modern network traffic. HAEnID and XI2S-IDS are evalu- ated on datasets of manageable size and do not explicitly confront computational scaling for multi-million-flow corpora or real-time streaming common in enterprise networks. Their reliance on SMOTE focuses on synthetic data generation for class balance rather than reducing the raw data volume for initial model development. In contrast, this work introduces a strategic stratified sub-sampling methodology that preserves original class distributions while reducing CIC-IDS2017 from 2.83M to a representative subset, enabling efficient itera- tion without sacrificing statistical integrity—a prerequisite for large-scale deployment. Second, there is a pronounced gap in evaluation rigor and experimental integrity. Seminal work has documented pervasive issues in cybersecurity ML, including data leakage, temporal bias, and inflated performance estimates [7], [18]. Although systems such as XI2S-IDS adopt standard train–test splits, they do not make systematic leakage detection and temporal validation a central pipeline component. The present framework does so by performing automated data leakage prevention—identifying and removing 29 leaky features (37% of the original set)—and by employing multi-configuration validation with temporally consistent splits to better approx- imate real-world deployment scenarios. This aligns with the methodological guidance of Arp et al. and Pendlebury et al. on realistic, unbiased evaluation. Thus, the novelty of this work does not lie in the iso- lated use of SHAP, feature selection, or ensembles, but in a leakage-aware, sampling-aware, and explainable pipeline engineered for end-to-end operational challenges. Concretely, the framework uniquely combines: (1) strategic stratified sub-sampling for large-scale data tractability, distinct from synthetic oversampling; (2) automated leakage detection and multi-configuration validation to support rigorous, unbiased evaluation; and (3) integrated SHAP-based XAI embedded across configurations to provide consistent operational trans- parency. Addressing computational scalability, experimental validity, and operational trust simultaneously helps close the integration gap that currently limits the real-world deployment of AI-driven IDS in SOCs. G. Comparative Analysis with State-of-the-Art Table I provides a comprehensive comparison of recent approaches in cybersecurity machine learning, highlighting the distinct contributions of our work in integrating sampling, fea- ture selection, and explainability within a unified framework. This analysis reveals that while existing approaches excel in specific areas, our framework uniquely combines all three critical components. TABLE I COMPARATIVE ANALYSIS OF RECENT CYBERSECURITY ML APPROACHES StudySampling Strategy Feature Selection ExplainabilityPerformanceKey Contribution DistilXIDS (2026) [9]✓99.61%Efficient, lightweight transformer-based language model for real-time NIDS Gait ́ an-C ́ ardenas et al. (2023) [13] ✓Reported*XAI-based IDS for cloud and IoT environments Zhou et al. (2025) [17]Reported*Adaptive ensemble learning for real-time attack detection Chen et al. (2023) [28]✓99.80%Hardware-accelerated real-time IDS with explainable ML Uddin et al. (2024) [14]✓Reported*Explainable transformer-based model for phishing detection Ma et al. (2025) [10]✓Reported*IoT IDS with feature selection and LLM fine-tuning XI2S-IDS (2025) [25]✓99.81%Two-stage IDS (binary then multi-class) with SHAP, focusing on low-frequency attacks Hybrid Ensemble Model (2024) [23] ✓98.79%Hybrid Adaptive Ensemble for Intrusion Detection (HAEnID) with SMOTE sampling, multi-feature selection, and SHAP/LIME explainability Our Work✓99.92%Integrated framework with strategic sampling, feature optimization, and comprehensive XAI * Performance metrics for cited works are reported in their respec- tive publications. This table focuses on methodological comparisons rather than direct performance comparisons across different datasets and evaluation metrics. As shown in Table I, the HAEnID framework [23] represents a comprehensive approach that integrates sampling (SMOTE), feature selection, and explainability (SHAP/LIME), achieving 98.79% ac- curacy on CIC-IDS2017 through its hybrid ensemble architecture. Similarly, the XI2S-IDS framework [25] achieves strong performance (99.81% accuracy on CICIDS2017) through its innovative two- stage architecture. Our work bridges remaining gaps by integrating strategic sampling to handle massive datasets efficiently, optimized feature selection to enhance model interpretability and performance, and comprehensive SHAP-based explainability to ensure operational transparency. This holistic approach addresses the practical deploy- ment challenges faced by security operations centers while maintain- ing state-of-the-art detection performance. Given the inherent challenges in directly comparing cybersecurity ML approaches due to divergent datasets, evaluation metrics, and experimental setups, we contextualize our framework’s performance of 99.92% accuracy on CIC-IDS2017 within our specific methodol- ogy of strategic sampling and multi-configuration validation. While prior works—such as HAEnID for integrated ensemble detection and XI2S-IDS for structured two-stage detection—demonstrate effec- tiveness in their respective domains, our contribution addresses the integration gap by delivering a unified framework that achieves state- of-the-art performance while simultaneously ensuring computational efficiency and operational transparency. H. Strategic Sampling with Quantitative Validation Our framework implements a novel stratified sampling methodol- ogy to address the computational challenges of the full CIC-IDS2017 dataset while maintaining statistical representativeness. The sampling process, implemented in sampling2.py, employs: S sample = C X c=1 min(N c ,⌊p× N c ⌋)(1) where S sample is the total sampled size, C is the number of classes, N c is the count of class c, and p is the sampling percentage (20% in our implementation). This approach ensures: • Class Distribution Preservation: Maintains original dataset proportions • Minority Class Protection: Guarantees minimum representa- tion for rare attack types • Computational Efficiency: Enables rapid iteration and model development The sampling process reduced the dataset from 2,830,743 to 470,269 samples while maintaining identical class distribution char- acteristics (Table I). TABLE I DATASET CLASS DISTRIBUTION AFTER STRATEGIC SAMPLING ClassOriginalSampledPercentageAttack Category BENIGN2,359,097388,80882.7%Normal Traffic DoS Hulk231,07334,6207.4%Denial of Service DDoS128,02725,5885.4%Distributed DoS PortScan158,93017,9793.8%Reconnaissance DoS GoldenEye10,2932,1050.4%DoS Variant FTP-Patator7,9381,1690.2%Brute Force To quantitatively validate that our strategic sampling preserves the essential characteristics of the full dataset, we conducted a comprehensive distributional and performance analysis: TABLE I QUANTITATIVE VALIDATION OF STRATEGIC SAMPLING ON CIC-IDS2017 Validation MetricFull Dataset20% SampleDifference Distribution Similarity Mean Packet Length (Benign)145.7144.90.55% Mean Flow Duration (Attack)2.31s2.29s0.87% Feature Correlation Matrix Distance–0.012 Performance Consistency XGBoost Accuracy (Binary)99.88%99.91%+0.03% XGBoost F1-Macro (Multi-class)99.65%99.72%+0.07% ROC-AUC Macro Average0.99980.9999+0.01% As shown in Table I, our strategic sampling approach preserves both statistical distributions and detection performance. The minimal differences in key metrics (¡1% for distributional properties, ¡0.1% for performance metrics) confirm that the 20% sample maintains the essential patterns and relationships of the full 2.8 million-record dataset. This validation addresses the critical concern raised by Arp et al. [7] regarding the representativeness of sampled data in security evaluations. I. Temporal Validation Framework and Leakage Prevention Following the rigorous evaluation principles established in cyber- security machine learning research [7], [18], [2], we implement a temporally-aware validation framework that reflects realistic deploy- ment scenarios. The CIC-IDS-2017 dataset is inherently temporal, with attacks occurring in specific sequences across five consecutive days. Unlike random splitting methods that can create unrealistic performance estimates [18], our approach respects the temporal ordering of network events. Temporal Split Design: We partition the dataset chronologically, using the first four days for training and the fifth day for testing. This design simulates a realistic deployment scenario where models trained on historical data must detect attacks occurring in the future. The validation set is created from the latter portion of the training period, maintaining temporal separation from both training and testing data. Automated Data Leakage Prevention: To ensure experimental integrity, we implement comprehensive data leakage detection and prevention measures: • Temporal Leakage Prevention: Strict chronological separation between training, validation, and test sets, following TESSER- ACT’s recommendations for time-series security data [18] • Feature Leakage Analysis: Systematic identification and re- moval of 29 features exhibiting near-perfect correlation with target classes or containing future information • Cross-split Sample Overlap Detection: Verification that no overlapping samples exist between training, validation, and test sets • Preprocessing Isolation: All feature scaling parameters are computed exclusively from training data and applied to vali- dation and test sets Alignment with Security ML Best Practices: Our methodology explicitly addresses the ”dos and don’ts” outlined by Arp et al. [7]: • Do respect temporal ordering: We maintain chronological splits that reflect realistic attack evolution • Do prevent feature leakage: We systematically identify and remove contaminating features • Don’t use future information: Our preprocessing pipeline ensures no test set information influences training • Do validate on realistic data: Our test set represents future, unseen attacks as would occur in operational deployment This rigorous approach ensures that our performance metrics reflect realistic detection capabilities rather than optimistic estimates from temporally contaminated evaluations. By grounding our method- ology in established security ML evaluation principles, we provide a trustworthy foundation for assessing the operational viability of our integrated XAI framework. J. Experimental Framework Our experimental framework (Figure ??) implements a compre- hensive ML pipeline with integrated XAI components and multi- configuration validation: The framework employs a robust multi-configuration validation strategy evaluating three distinct data partitioning schemes to assess model generalization across varying data availability scenarios: • Configuration A (40-10-50): 40% Training, 10% Validation, 50% Test - Simulating limited training data scenarios • Configuration B (60-10-30): 60% Training, 10% Validation, 30% Test - Balanced data distribution • Configuration C (80-10-10): 80% Training, 10% Validation, 10% Test - Abundant training data scenario K. Robust Algorithm Selection with Cross-Configuration Val- idation We implemented a comprehensive algorithm selection process that evaluates performance across all three data configurations to identify the most robust model: • Cross-Configuration Averaging: Algorithm performance met- rics (Accuracy, F1-Macro, ROC AUC) are averaged across all three split configurations to determine overall robustness • Configuration-Specific Analysis: Individual algorithm per- formance is analyzed per configuration to identify context- dependent strengths • Best Overall Algorithm Selection: The algorithm with the highest average F1-Macro score across all configurations is selected for final deployment Three diverse machine learning algorithms were evaluated under this multi-configuration framework: • XGBoost: Configured with ensemble methods and regulariza- tion parameters to balance complexity and generalization (We used 100 trees with a maximum depth of 8 and a learning rate of 0.1.) • Random Forest: Employed multiple decision trees with con- trolled depth to prevent overfitting. (The model consisted of 100 trees, each with a maximum depth of 15.) • Logistic Regression: Utilized linear decision boundaries with multi-class handling capabilities. All models employed 5-fold stratified cross-validation within each configuration for robust performance estimation. L. Data Leakage Prevention and Validation To maintain experimental integrity across all configurations, we implemented comprehensive data leakage detection and prevention measures: • Cross-split sample overlap detection: Ensured no overlapping samples between training, validation, and test sets across all three configurations • Feature leakage analysis: Identified and removed 29 features exhibiting near-perfect correlation with target classes • Temporal validation: For time-series network data, maintained temporal ordering within splits to prevent future information leakage • Automated leakage detection: Systematic identification of features showing constant values, statistical anomalies, and predictive leakage patterns This comprehensive process reduced the feature set from 78 to 49 dimensions while maintaining detection capability across all experimental configurations. All feature scaling was performed using training-set statistics only, with identical transformations applied to validation and test sets: X scaled = X− μ train σ train (2) where: • X represents the feature matrix (can be training, validation, or test data) • μ train is the mean vector computed from the training set only • σ train is the standard deviation vector computed from the training set only • X scaled is the standardized feature matrix This approach ensures that no information from validation/test sets contaminates the preprocessing parameters, the model is evaluated on data that mimics real-world deployment scenarios, and statistical properties are estimated exclusively from training samples. The validation and test sets are transformed using the same μ train and σ train values, preventing the common data leakage pitfall where scaling parameters are computed using the entire dataset. M. Statistical Significance Testing We employed stratified 5-fold cross-validation within each split configuration to obtain performance distributions and compute Strategic Sampling 20% stratified 470K samples, 78 features Data Leakage Detection Remove 29 leaky features 49 clean features Algorithm Selection 50/30/10/10 Split Train Three Algorithms XGBoostRandom ForestLogistic Regression Cross-Validation Select Best Algorithm Feature Selection MRMR vs Chi2 Comparison 30%, 50%, 70% reduction Select Best FS Method Final Evaluation Holdout Testing 10% Performance Metrics SHAP Analysis Model InterpretabilityFeature ImportanceDecision Explanations Security Operations Integration Dataset A A Fig. 1. Comprehensive Experimental Framework with Integrated XAI and Multi-Split Validation confidence intervals, ensuring statistical reliability of our multi- configuration analysis: CI95% = ̄x± t0.025,n− 1× s √ n (3) where ̄x is the mean cross-validation score, s is the standard devia- tion, and n = 5 is the number of folds. This approach provided robust performance estimation across all three data partitioning strategies. N. Computational Efficiency Assessment Beyond predictive performance, we evaluated computational char- acteristics across all configurations to identify practical deployment considerations: • Training time: Wall-clock time for model training across different data volumes • Inference speed: Predictions per second on validation data, critical for real-time cybersecurity applications • Memory footprint: Model size and inference memory require- ments across configurations • Scalability analysis: Performance trends as training data vol- ume increases across configurations O. Feature Selection Methods We compared MRMR and Chi-squared feature selection across multiple reduction percentages (30%, 50%, 70%) using the best- performing algorithm and configuration identified through our multi- split analysis and computational assessment. P. Explainable AI Implementation SHAP analysis was integrated throughout the multi-configuration framework providing comprehensive interpretability: • Cross-Configuration Feature Importance: Comparative anal- ysis of feature importance patterns across different data splits • Robust Interpretability: Model explanations validated across multiple experimental configurations • Decision Transparency: Individual prediction rationales with configuration-aware interpretations • Algorithm Consistency: Assessment of explanation stability across different training data volumes I. EXPERIMENTS AND RESULTS A. Multi-Configuration Algorithm Performance Our comprehensive evaluation across three distinct data partition- ing strategies revealed critical insights into model generalization and configuration optimization: TABLE IV MULTI-CONFIGURATION ALGORITHM PERFORMANCE ANALYSIS AlgorithmSplitAccuracyF1-MacroROC-AUCTraining Time (s)Prediction SpeedCV Stability XGBoost60-10-3099.91%99.72%1.000033.90324,9130.99986 ± 0.00023 XGBoost80-10-1099.92%99.70%1.000047.02338,9610.99994 ± 0.00005 XGBoost40-10-5099.90%99.60%0.999914.73683,0000.99994 ± 0.00004 Random Forest60-10-3099.87%99.63%0.999980.0416,0490.99975 ± 0.00026 Logistic Regression60-10-3096.34%74.50%0.9895415.513,423,9220.98241 ± 0.00154 Continuous Improvement Security Operations XAI Components Raw Network Data ML Detection Engine Alert Generation Yes No High Confidence? Auto-ResponseSHAP Analysis Feature Importance Ranking Decision ExplanationAttack Pattern Analysis Security Analyst Review Medium Confidence Confidence Assessment Incident ResponseFurther InvestigationModel Feedback Threat ContainmentAdditional Context Model Retraining High ConfidenceLow Confidence Fig. 2. Detailed XAI Integration Workflow in Security Operations TABLE V COMPREHENSIVE FEATURE SELECTION PERFORMANCE MethodFeaturesAccuracyF1-MacroROC-AUCTraining Time (s)ReductionPerformance Gain MRMR70%3499.92%99.77%0.999921.4730.6%+0.05% F1 Chi270%3499.91%99.72%1.000012.2930.6%+0.00% F1 Original4999.91%99.72%1.000033.900%Baseline MRMR50%2499.89%99.61%1.000016.2051.0%-0.11% F1 Chi250%2499.91%99.72%1.000012.2851.0%+0.00% F1 MRMR30%1499.72%98.66%0.999923.8971.4%-1.06% F1 Chi230%1499.91%99.72%1.000012.0371.4%+0.00% F1 The multi-configuration analysis (Table IV) demonstrates XG- Boost ́ s consistent superiority across all partitioning strategies. The 60- 10-30 configuration emerged as optimal, achieving the best balance between training data utilization and validation robustness. Notably, cross-validation stability remained exceptionally high across all con- figurations, indicating robust model generalization. B. Feature Selection Impact and Optimization The feature selection analysis (Table V) reveals that MRMR at 70% reduction (34 features) not only maintained performance but achieved a slight improvement in F1-macro score (+0.05%) over the original feature set. This represents a significant advancement in feature optimization, demonstrating that strategic dimensionality reduction can enhance model performance while improving compu- tational efficiency. C. Final Configuration Performance The optimal configuration (XGBoost with 60-10-30 split and MRMR 70% feature selection) achieved exceptional performance on the test dataset (Table VI). The near-perfect ROC-AUC (0.99997) and balanced precision-recall metrics demonstrate robust generalization capability across all attack categories. D. Confusion Matrix and Classification Analysis The confusion matrix analysis (Figure 3) reveals exceptional classification performance with strategic insights: Per-Class Performance Analysis: TABLE VI FINAL OPTIMAL CONFIGURATION PERFORMANCE MetricPerformance Accuracy99.92% F1-Macro99.77% F1-Weighted99.92% Precision-Macro99.70% Recall-Macro99.85% ROC-AUC99.997% Features Used34 Test Samples47,027 Training Time21.47s Fig. 3. Confusion Matrix Analysis for Optimal Configuration (XGBoost + MRMR70% + 60-10-30) TABLE VII DETAILED ROC-AUC PERFORMANCE BY CLASS Attack ClassROC-AUC FTP-Patator1.0000 DDoS0.9999 DoS Hulk0.9999 DoS GoldenEye0.9999 BENIGN0.9999 PortScan0.9999 Micro-Average0.9999 Macro-Average0.9999 • BENIGN Traffic: 99.94% accuracy (38,856/38,881 correct), minimal confusion with PortScan • DDoS Attacks: 99.98% accuracy (2,557/2,559 correct), near- perfect detection • DoS GoldenEye: 99.52% accuracy (209/210 correct), robust minority class performance • DoS Hulk: 99.86% accuracy (3,457/3,462 correct), high-volume attack precision • FTP-Patator: 100% accuracy (117/117 correct), perfect minor- ity class identification • PortScan: 99.83% accuracy (1,795/1,798 correct), minimal false positives The overall accuracy of 99.92% with balanced performance across both high-frequency and low-frequency attack classes demonstrates the framework’s effectiveness in addressing class imbalance chal- lenges. E. Multi-Class ROC Analysis The ROC curve analysis (Figure 4) demonstrates exceptional discriminative capability across all attack categories: The consistent near-perfect ROC-AUC scores across all classes (Table VII) confirm the model’s robust discriminative power, with particularly strong performance on minority classes (FTP-Patator, Fig. 4. Comprehensive ROC Analysis for Multi-class Classification DoS GoldenEye) that are often challenging in cybersecurity clas- sification tasks. F. Comparative Feature Importance Analysis To elucidate the predictive mechanisms underlying each model’s decision-making process, we conducted complementary feature im- portance analyses using two distinct methodologies. Figure 5 presents Gini-based importance from the Gradient Boosting model, while Figure 6 displays SHAP values from the Random Forest model, providing both global feature importance and class-specific contri- butions. Fig. 5. Gini-based feature importance from Gradient Boosting (XGBoost) with MRMR feature selection (70% retention) on 60-10-30 split. Features are ranked by their mean decrease in impurity, with packet length statistics showing dominant predictive power. 1) Cross-Methodological Insights: Comparative analysis re- veals both consistent patterns and methodological differences in feature importance assessment: Fig. 6.SHAP-based feature importance from Random Forest on 40-10- 50 split. Bars represent mean absolute SHAP values (global impact), while colored dots show feature effects across individual classes. Flow rate metrics emerge as most influential, with notable class-specific variations. Consistent Predictive Features: • Flow rate metrics appear in both analyses: Flow_Bytes/s and Flow_Packets/s rank highly in SHAP analysis and appear in XGBoost’s top 20 • Packet length statistics show strong predictive power in both models, particularly Fwd_Packet_Length_Std and Min_Packet_Length • Temporal characteristics including Flow_IAT_Min maintain importance across methodologies Methodological Divergences: • Gradient Boosting emphasizes Bwd_Packet_Length_Max and protocol flags (Fwd_URG_Flags, Bwd_PSH_Flags) as primary discriminators • Random Forest SHAP analysis prioritizes flow rate metrics and reveals class-specific feature effects, particularly for Classes 0 and 5 • The 40-10-50 split with Random Forest shows stronger emphasisonmeanvalues(Packet_Length_Mean, Flow_IAT_Mean), while the 60-10-30 split with XGBoost favors extreme values and standard deviations Class-Specific Patterns: The SHAP analysis (Figure 6) reveals important class-level distinctions: • Flow_Bytes/s shows differential impacts, with strongest positive effects on Class 5 and negative effects on Class 0 • Min_Packet_Length exhibits contrasting directions of in- fluence across classes, suggesting different traffic patterns • Protocol flags (Bwd_URG_Flags, Fwd_PSH_Flags) show class-specific relevance not apparent in global importance rank- ings 2) MethodologicalImplications:Theconvergenceof key features across different models and importance metrics (Flow_Bytes/s, Flow_IAT_Min, packet length statistics) suggests these represent robust, model-agnostic indicators of network traffic patterns. However, the methodological differences highlight that: 1) Importance metrics capture different aspects of feature contri- bution (Gini impurity reduction vs. marginal prediction impact) 2) Data split ratios influence which features emerge as most predictive 3) SHAP analysis provides valuable class-level insights missing from global importance metrics 4) Ensemble methods with different randomization strategies sur- face complementary feature subsets IV. DISCUSSION A. Multi-Configuration Validation Insights Our multi-configuration approach provides several critical insights for real-world deployment: Optimal Data Partitioning: The 60-10-30 split configuration demonstrated the best balance between training data utilization and validation robustness, suggesting that approximately 60% of avail- able data provides optimal training efficiency for this cybersecurity classification task. Configuration-Specific Advantages: • 40-10-50: Best for scenarios with limited training data, main- taining 99.90% accuracy • 60-10-30: Optimal balance achieving best overall performance (99.92% accuracy) • 80-10-10: Maximum training data utilization with slightly re- duced generalization B. Limitations and Generalizability Considerations While our framework demonstrates exceptional performance on the CIC-IDS2017 dataset with a deployment readiness score of 9.1/10 across critical operational factors (Table VIII), we acknowledge im- portant limitations regarding generalizability. Our evaluation, though rigorous within the CIC-IDS2017 environment, represents promis- ing evidence rather than definitive proof of universal deployment readiness. As noted in comprehensive surveys of intrusion detection research, the performance of ML-based systems can vary significantly across different network environments, attack patterns, and data characteristics [3], [19]. The CIC-IDS2017 dataset, while being a valuable benchmark for traditional network intrusion detection, represents a specific network topology and attack profile that may not fully capture the diversity of modern threat landscapes. Recent work has highlighted the distinct challenges present in specialized environments such as IoT networks [10] and cloud infrastructures [13], where traffic patterns, device constraints, and attack vectors differ substantially from the enterprise network setting simulated by CIC-IDS2017. Our framework’s perfor- mance in these alternative domains remains untested and represents an important direction for future validation. Furthermore, as discussed in methodological critiques of cyber- security ML [7], there exists a well-documented gap between the controlled conditions of benchmark datasets and the dynamic, noisy realities of Security Operations Center (SOC) environments. Factors such as network evolution, zero-day attacks, and adversarial adap- tation—which are challenging to capture in static datasets—present additional hurdles for real-world deployment that are not addressed by our current evaluation. C. Feature Selection Strategic Value The MRMR feature selection success demonstrates several opera- tional advantages: Computational Efficiency: 30.6% feature reduction translates to: • 36.7% faster training times (33.90s→ 21.47s) TABLE VIII DEPLOYMENT READINESS ASSESSMENT FOR CIC-IDS2017 ENVIRONMENT Deployment FactorScoreRationale Detection Accuracy10/1099.92% overall accuracy False Positive Rate9/100.08% misclassification rate Computational Efficiency9/10324K predictions/second Model Interpretability10/10Comprehensive SHAP analysis Feature Optimization10/1030.6% reduction maintained performance Generalization9/10Consistent cross-configuration performance • Reduced memory requirements for model deployment • Faster real-time inference in production environments Enhanced Model Transparency: Reduced feature dimensionality improves analyst comprehension and facilitates better security rule development based on the identified critical features. D. Operational Deployment Considerations The deployment readiness assessment (Table VIII) confirms the framework’s suitability for production environments, with exceptional scores across all critical operational factors. V. CONCLUSION AND FUTURE WORK This research establishes a comprehensive multi-configuration framework for explainable AI in cybersecurity, achieving exceptional results on the CIC-IDS2017 dataset through rigorous methodology and strategic optimization: A. Key Contributions 1) Multi-Configuration Validation: Demonstrated configuration- dependent performance patterns, identifying 60-10-30 as opti- mal for cybersecurity classification on CIC-IDS2017 2) Feature Optimization Breakthrough: Achieved 30.6% di- mensionality reduction with performance improvement using MRMR selection 3) State-of-the-Art Performance on CIC-IDS2017: 99.92% ac- curacy and 99.77% F1-macro on comprehensive multi-class evaluation 4) Operational Explainability: Integrated SHAP analysis pro- viding actionable intelligence for security operations 5) Computational Efficiency: 324K predictions/second with ro- bust real-time deployment capability for similarly structured network data B. Limitations and Future Research Directions Our work provides a robust methodological foundation and demon- strates compelling results on a major benchmark dataset. However, we recognize the inherent limitation of single-dataset evaluation in cybersecurity machine learning. Future research should pursue several critical directions to address this limitation and strengthen the framework’s claims to generalizability: • Cross-Dataset Validation: Systematic evaluation on diverse network environments including IoT frameworks [10], cloud infrastructures [13], and specialized domains like industrial control systems. This addresses the call in survey literature for validation across heterogeneous environments [3], [19]. • Real-World Deployment Testing: Transition from benchmark datasets to real-time streaming environments, addressing the gap between controlled evaluation and operational deployment noted in methodological critiques [7]. • Adaptation to Evolving Threats: Extension of the framework to handle concept drift and zero-day attacks through continuous learning mechanisms, addressing limitations of static dataset evaluation. • Domain-Specific Optimization: Investigation of how strategic sampling, feature selection, and explainability components re- quire adjustment for different network topologies and threat models. • Integration with Threat Intelligence Feeds: Enhancement of the framework with external contextual information to improve detection accuracy and relevance in operational settings. This framework represents a promising stepping-stone in the pur- suit of trust-worthy, explainable AI within cybersecurity by reconcil- ing strong detection performance on benchmark data with reasonable methodological choices. Combining strategic sampling, automatic leakage prevention and full explainability fills central flaws of current IDS research pipelines. We will need further validation across a variety of environments to verify broad operational readiness, but our method offers a tightly scrutinized path toward functional, explainable security AI systems. REFERENCES [1] R. Sommer and V. Paxson, “Outside the closed world: On using machine learning for network intrusion detection,” in Proc. IEEE Symp. S&P, 2010, p. 305–316. [2] H. S. Anderson and P. Roth, “EMBER: An open dataset for training static PE malware machine learning models,” 2018, arXiv:1804.04637. [3] A. L. Buczak and E. Guven, “A survey of data mining and machine learning methods for cyber security intrusion detection,” IEEE Commun. Surv. Tutor., vol. 18, no. 2, p. 1153–1176, 2016. [4] M. A. Ferrag, L. Maglaras, S. Moschoyiannis, and H. Janicke, “Deep learning for cyber security intrusion detection: Approaches, datasets, and comparative study,” J. Inf. Secur. Appl., vol. 50, p. 102419, 2020. [5] C. Rudin, “Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead,” Nat. Mach. Intell., vol. 1, no. 5, p. 206–215, 2019. [6] A. Adadi and M. Berrada, “Peeking inside the black-box: A survey on explainable artificial intelligence (xai),” IEEE Access, vol. 6, p. 52 138– 52 160, 2018. [7] D. Arp, E. Quiring, F. Pendlebury, A. Warnecke, F. Pierazzi, C. Wress- negger, L. Cavallaro, and K. Rieck, “Dos and don’ts of machine learning in computer security,” in Proc. USENIX Security Symp., 2020. [8] T. Miller, “Explanation in artificial intelligence: Insights from the social sciences,” Artif. Intell., vol. 267, p. 1–38, 2019. [9] A. Ajayan et al., “Distilxids: Efficient, lightweight and explainable transformer-based language model for real-time network intrusion de- tection,” Neurocomputing, vol. 668, p. 132398, 2026. [10] H. Ma et al., “An iot intrusion detection framework based on feature selection and large language models fine-tuning,” Sci. Rep., vol. 15, p. 21158, 2025. [11] S. M. Lundberg and S.-I. Lee, “A unified approach to interpreting model predictions,” in Adv. Neural Inf. Process. Syst. (NeurIPS), 2017, p. 4765–4774. [12] S. M. Lundberg, G. Erion, H. Chen, A. DeGrave, J. M. Prutkin, B. Nair, R. Katz, J. Himmelfarb, N. Bansal, and S.-I. Lee, “From local explanations to global understanding with explainable ai for trees,” Nat. Mach. Intell., vol. 2, no. 1, p. 56–67, 2020. [13] M. C. Gaitan-Cardenas et al., “Explainable ai-based intrusion detection systems for cloud and iot,” in Proc. IEEE IC2E, 2023, p. 1–8. [14] M. A. Uddin et al., “An explainable transformer-based model for phishing email detection: A large language model approach,” 2024, arXiv:2402.13871. [15] H. Peng, F. Long, and C. Ding, “Feature selection based on mu- tual information criteria of max-dependency, max-relevance, and min- redundancy,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 27, no. 8, p. 1226–1238, 2005. [16] J. Li, K. Cheng, S. Wang, F. Morstatter, R. P. Trevino, J. Tang, and H. Liu, “Feature selection: A data perspective,” ACM Comput. Surv., vol. 50, no. 6, p. 1–45, 2017. [17] Y. Zhou et al., “Adaptive ensemble learning for real-time detection of price manipulation attacks on smart meters,” Energy Rep., vol. 12, p. 100–113, 2025. [18] F. Pendlebury, F. Pierazzi, R. Jordaney, J. Kinder, and L. Cavallaro, “Tesseract: Eliminating experimental bias in malware classification across space and time,” in Proc. USENIX Security Symp., 2019, p. 729–746. [19] A. Khraisat, I. Gondal, P. Vamplew, and J. Kamruzzaman, “Survey of intrusion detection systems: Techniques, datasets and challenges,” Cybersecurity, vol. 2, p. 1–22, 2019. [20] I. Sharafaldin, A. H. Lashkari, and A. A. Ghorbani, “Toward generating a new intrusion detection dataset and intrusion traffic characterization,” in Proc. ICISSP, 2018, p. 108–116. [21] M. Ring, S. Wunderlich, D. Grudl, J. Landthaler, and A. Hotho, “A toolset for intrusion and insider threat detection,” in IFIP Int. Conf. Digit. Forensics, 2019, p. 3–16. [22] H. Liu and B. Lang, “Machine learning and deep learning methods for intrusion detection systems: A survey,” Appl. Sci., vol. 9, no. 20, p. 4396, 2019. [23] U. Ahmed, J. Zheng, A. Almogren, S. Khan, M. T. Sadiq, A. Altameem, and A. U. Rehman, “Explainable ai-based innovative hybrid ensemble model for intrusion detection,” J. Cloud Comput., vol. 13, p. 150, 2024. [24] N. V. Chawla, K. W. Bowyer, L. O. Hall, and W. P. Kegelmeyer, “Smote: Synthetic minority over-sampling technique,” J. Artif. Intell. Res., vol. 16, p. 321–357, 2002. [25] M. M. Mahmoud, Y. O. Youssef, and A. A. Abdel-Hamid, “Xi2s-ids: An explainable intelligent 2-stage intrusion detection system,” Future Internet, vol. 17, no. 1, p. 25, 2025. [26] I. Guyon and A. Elisseeff, “An introduction to variable and feature selection,” J. Mach. Learn. Res., vol. 3, p. 1157–1182, 2003. [27] G. Forman, “An extensive empirical study of feature selection metrics for text classification,” J. Mach. Learn. Res., vol. 3, p. 1289–1305, 2003. [28] J. Chen et al., “Ride: Real-time intrusion detection via explainable machine learning implemented in a memristor hardware architecture,” 2023, arXiv:2311.16018.