Paper deep dive
SMSI: System Model Security Inference: Automated Threat Modeling for Cyber-Physical Systems
RoĂah Radaideh, Ali Khreis
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 98%
Last extracted: 6/21/2026, 7:35:50 AM
Summary
SMSI (System Model Security Inference) is a hybrid neuro-symbolic pipeline designed to automate threat modeling for Cyber-Physical Systems (CPS). It transforms SysML architecture models into prioritized NIST 800-53 security controls through a five-stage process: SysML ingestion, CPE-to-CVE mapping via the NVD, CVE-to-MITRE ATT&CK technique mapping (using retrieval, supervised classification, or LLMs), ATT&CK-to-NIST control recommendation, and automated report generation. The research evaluates various models, finding that while unsupervised MiniLM performs well for retrieval, the Gemma-4 26B LLM achieves high hit rates in zero-shot settings, and SecureBERT+ provides strong supervised classification performance.
Entities (9)
Relation Signals (6)
CVE â mappedto â MITRE ATT&CK
confidence 100% ¡ linking vulnerabilities to MITRE ATT&CK techniques
MITRE ATT&CK â mappedto â NIST 800-53
confidence 100% ¡ ATT&CK-to-NIST stage... combines the Center for Threat-Informed Defense (CTID) ATT&CKâ800-53 crosswalk
SMSI â outputs â NIST 800-53
confidence 100% ¡ produces a prioritized list of NIST 800-53 security controls
SecureBERT+ â performstask â CVE-to-ATT&CK mapping
confidence 100% ¡ a supervised multi-label classifier using fine-tuned SecureBERT+
SMSI â uses â SysML
confidence 100% ¡ starts from a SysML architecture model
MedGateway â validatedby â SMSI
confidence 100% ¡ We validate the pipeline on a synthetic healthcare IoT gateway, MedGateway
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Threat modeling for cyber-physical systems (CPS) remains a largely manual exercise. This project presents SMSI (System Model Security Inference), a hybrid neuro-symbolic pipeline that starts from a SysML architecture model and produces a prioritized list of NIST 800-53 security controls. The prototype has three main stages: a deterministic parser mapping system components to vulnerabilities via the NVD; a family of retrieval and classification models linking vulnerabilities to MITRE ATT&CK techniques; and a control recommender. We explore three approaches for CVE-to-ATT&CK mapping: a supervised classifier using fine-tuned SecureBERT+, retrieval-based dense encoders, and a zero-shot LLM approach using Gemma-4 26B. We validate the pipeline on a healthcare IoT gateway with nine software components. For the ATT&CK-to-NIST stage, pretrained SecureBERT achieves the highest control retrieval scores, demonstrating that dense embeddings provide a strong basis for automated control recommendation.
Tags
Links
- Source: https://arxiv.org/abs/2604.23905v1
- Canonical: https://arxiv.org/abs/2604.23905v1
Trouble viewing inline? Open PDF directly â
Full Text
38,079 characters extracted from source content.
Expand or collapse full text
ELG5271/CSI5388 â AI FOR CYBERSECURITY APPLICATIONS, UNIVERSITY OF OTTAWA, WINTER 2025/20261 SMSI: System Model Security Inference: Automated Threat Modeling for Cyber-Physical Systems Ro Ě Yah Radaideh and Ali Khreis AbstractâThreat modeling for cyber-physical systems (CPS) remains a largely manual exercise. Security analysts receive a system architecture diagram, scan vulnerability databases, and then spend hours deciding which attacks are plausible and which controls to implement. This project asks whether that workflow can be automated. We present SMSI (System Model Security Inference), a hybrid neuro-symbolic pipeline that starts from a SysML architecture model and produces a prioritized list of NIST 800-53 security controls, with every recommendation traced back to the specific system component, vulnerability, and attacker technique that motivated it. The prototype has three main stages: a deterministic parser that maps system components to software vulnerabilities via the National Vulnerability Database (NVD); a family of retrieval and classification models that link those vulnerabilities to MITRE ATT&CK techniques; and a control recommender that combines the Center for Threat-Informed Defense (CTID) ATT&CKâ800-53 crosswalk with text similarity and CVSS-weighted prioritization. We explore three complementary approaches for CVE-to- ATT&CK mapping: (1) a supervised multi-label classifier using fine-tuned SecureBERT+, achieving Micro F1 of 0.757 on 105 parent technique classes; (2) retrieval-based dense encoders (SecureBERT, ATTACK-BERT, MiniLM) with KEV-supervised fine-tuning, where MiniLM achieves the best MRR of 0.252 and Hits@10 of 0.578 on a 45-query KEV test split; and (3) a zero- shot LLM approach using Gemma-4 26B via LM Studio, which achieves 51.8% Hit Rate@1 on 419 KEV CVEs-substantially outperforming all embedding baselines on hit-rate metrics. We validate the pipeline on a synthetic healthcare IoT gate- way, MedGateway, with nine software components yielding 199 CVEs. For the ATT&CK-to-NIST stage, pretrained Secure- BERT achieves the highest control retrieval scores (MRR 0.582, Hits@10 0.798), demonstrating that the curated CTID crosswalk combined with dense embeddings provides a strong basis for au- tomated control recommendation. Taken together, SMSI reduces what would otherwise require days of expert manual analysis to an automated, fully traceable report suitable for analyst review. Index TermsâThreat modeling, cyber-physical systems, vul- nerability analysis, CVE, MITRE ATT&CK, NIST 800-53, neuro- symbolic AI, SecureBERT, SysML, large language models. I. INTRODUCTION W HEN a security team inherits a new cyber-physical sys- tem (say, a hospital IoT gateway that connects infusion pumps to a cloud analytics platform), their job is to figure out what can go wrong and what to do about it. In practice, this means consulting the system architecture, searching the All authors are with the School of Electrical Engineering and Computer Sci- ence, University of Ottawa, Ottawa, ON, Canada. Course: ELG5271/CSI5388 â AI for Cybersecurity Applications. Instructor: Prof. Paula Branco. Winter 2025/2026. National Vulnerability Database (NVD) for known flaws in the deployed software, figuring out which adversary techniques those flaws enable, and then mapping those techniques to the relevant NIST 800-53 controls. Each step is intellectually non-trivial and time-consuming; the pace of new vulnerability disclosures means the analysis is never really done. Between 2020 and 2026, the NVD accumulated over 198,000 CVE records, with over 44,000 in 2025 alone; a pace that makes any purely manual process unsustainable. The core obstacle is a semantic gap. On one side sit struc- tured artifacts: SysML models that describe what software and hardware components exist, Common Platform Enumeration (CPE) identifiers that name those components precisely, and CVE records that catalog known flaws. On the other side sit natural-language artifacts: CVE descriptions written by human analysts, ATT&CK technique definitions written to describe adversary behavior, and NIST control narratives written to de- scribe safeguards. Connecting these two sides requires reading comprehension that rule-based systems cannot provide. Prior work has addressed individual pieces of this problem. Transformer-based classifiers such as SecRoBERTa can map CVE descriptions to ATT&CK techniques with reasonable accuracy [2]. Semantic methods such as SMET use structured role labeling to extract adversary intent from vulnerability text [4], [5]. Model-based engineering toolkits such as CEMT can represent threat traceability explicitly [12]. No existing system, however, integrates all three needs: grounding predic- tions in the actual system architecture, mapping vulnerabilities to techniques with contextual awareness, and translating those techniques to recommended controls. SMSI is designed to fill that gap. This paper makes four contributions. First, we present a complete end-to-end pipeline from SysML to NIST 800- 53 controls, which to our knowledge is the first system to automate this entire workflow starting from a SysML model of a CPS gateway. Second, we introduce a retrieval- based CVE-to-ATT&CK mapping stage that combines a TFâ IDF lexical baseline, several dense encoders (SecureBERT, ATTACK-BERT, MiniLM), and KEV-supervised fine-tuning, with diagnostics that quantify where the models agree or disagree.[1], [5], [13] Third, we evaluate a zero-shot LLM approach (Gemma-4 26B) against both the retrieval models and a supervised SecureBERT+ classifier, showing that LLMs achieve substantially higher hit rates on KEV and SMET benchmarks despite having no task-specific training. Fourth, we show how ATT&CK-to-NIST recommendations can be arXiv:2604.23905v1 [cs.CR] 26 Apr 2026 ELG5271/CSI5388 â AI FOR CYBERSECURITY APPLICATIONS, UNIVERSITY OF OTTAWA, WINTER 2025/20262 framed as a ranking problem driven primarily by the CTID ATT&CKâ800-53 crosswalk, with TFâIDF and dense models providing auxiliary scores and with CVSS severity used to prioritize the result set for a concrete system of interest. I. LITERATURE REVIEW The problem of automating threat modeling sits at the intersection of three research threads: NLP-based vulnerability mapping, neuro-symbolic AI in cybersecurity, and model- based systems engineering (MBSE). We review each in turn, identify their gaps, and explain where SMSI fits. Table I summarizes the key approaches. A. NLP-Based CVE-to-ATT&CK Mapping The foundational work in this area is CVE2ATT&CK by Grigorescu et al. [1], which framed the CVE-to-technique problem as multi-label text classification and fine-tuned SciB- ERT on 1,813 labeled pairs, achieving a weighted F1 of 47.84%. The main bottleneck was severe class imbalance: ATT&CK has over 200 techniques, but most labeled CVEs cluster around a handful of high-frequency ones. Branescu et al. [2] expanded the training set to nearly 10,000 entries and found that domain-adapted encoders matter; SecRoBERTa hit 78.88% F1 where generic BERT struggled. Li et al. [3] improved further by operating at sentence granularity rather than document level and by introducing a relation-based post- processor that enforces ATT&CKâs own hierarchy. All three approaches treat CVEs in isolation; there is no notion of what system the CVE applies to. Unsupervised and weakly supervised methods reduce the dependence on labeled data. SMET [4], [5] uses Semantic Role Labeling (SRL) to extract subject-verb-object triples from CVE descriptions and trains a Siamese network to match these triples to ATT&CK technique descriptions, achieving Recall@5 of 67.71% without any labeled CVE-technique pairs. Recent benchmarks show that hybrid LLM-based sys- tems such as TRIAGE can reach higher ranked-retrieval scores on the KEV mapping dataset by encoding MITREâs CVE Mapping Methodology into structured prompts and in-context examples, but still depend on expert-labeled KEV data and careful governance [13]. Rafiey and Namadchian [8] show that large reasoning models such as Gemini 2.5 Pro and OpenAI o1 can reach F1 scores around 60% in zero-shot settings; reliability remains a concern due to hallucination, especially when the mapping must be auditable. B. Neuro-Symbolic AI in Cybersecurity Purely statistical approaches cannot enforce the structural constraints of cybersecurity. A BERT model might predict a firmware corruption technique for a cloud-based microservice, which is physically impossible; but the model has no way to know that. MITREtrieval [9] fuses a SBERT-based deep learning model with a cybersecurity ontology (COMAT) using a level-voting algorithm, so that ontology-derived inferences can validate or override neural predictions. On sparse datasets, this fusion achieved an F 2 -score of 69%, substantially outper- forming either component alone. Graph-based approaches such as AttacKG [10] and BRON [11] link CVEs, CWEs, CAPECs, and ATT&CK entities into a traversable graph, but these rely on static databases and struggle with new vulnerability disclosures. C. Model-Based Systems Engineering The CEMT toolkit [12] extends SysML with cybersecurity concepts; misuse cases, threat vectors, and control mappings; and provides explicit traceability from system components to NIST 800-53 safeguards. All of CEMTâs threat data is entered manually, however. It provides the scaffolding for traceable threat modeling but no automation for the analytical steps. SMSI can be thought of as an attempt to automate the input side of CEMT by wiring a SysML model into a CVEâATT&CKâcontrols pipeline. D. Mapping to NIST 800-53 Beyond CVE-to-ATT&CK, several authors explore partial automation of control selection. Sahu and Speretta [14] apply TFâIDF and cosine similarity to the text of NIST 800-53 control families, showing that statistical word analysis can help semi-automatically suggest which families are most relevant for a given organizationâs security documentation. Their work confirms that text similarity can meaningfully support, but not replace, human control tailoring. CTIDâs ATT&CK-to- CVE impact mapping and ATT&CK-to-800-53 crosswalk [15] provide a curated reference that we treat as ground truth in our ATT&CK-to-control experiments. E. Research Gap The literature shows three mature but isolated islands: NLP models for CVE-to-ATT&CK mapping, knowledge graphs for structural reasoning, and MBSE frameworks for traceability. None of these islands are connected to one another, and none starts from a system architecture model or ends with ranked security controls. SMSI is designed to bridge all three, while explicitly treating all CVE-to-ATT&CK inferences as hypotheses that must be validated against architecture and mission context. I. CASE STUDY: MEDGATEWAY To validate the pipeline end-to-end we designed a synthetic but representative system called MedGateway: a healthcare IoT gateway architecture that connects bedside medical devices to a cloud analytics platform. Healthcare CPS was chosen because security failures here have direct patient safety con- sequences, regulatory implications, and a well-documented vulnerability history. MedGateway is not a real deployed system, but it is deliberately structured to reflect real healthcare IoT deployments. MedGateway consists of nine components organized across four architecture layers, as specified in the SysML model (SOI/MedGateway_ReferenceArchitecture.sysml.xml): ⢠Application layer: ClinicalDashboard WebApp (Spring Framework 5.3.18) ELG5271/CSI5388 â AI FOR CYBERSECURITY APPLICATIONS, UNIVERSITY OF OTTAWA, WINTER 2025/20263 TABLE I COMPARATIVE SUMMARY OF RELATED APPROACHES ApproachInputOutputSystem ContextControls CVE2ATT&CK / SecRoBERTa [1], [2]CVE textATT&CK techniquesNoneNo SMET [4], [5]CVE / CTI textRanked ATT&CKNoneNo MITREtrieval [9]CTI reportsATT&CK techniquesLowNo Generative LLMs [8], [13]CVE / promptsATT&CK + summariesLowâMediumGeneric CEMT [12]SysML modelRisk diagrams + controlsHigh (manual)Yes (manual) SMSI (this work)SysML + CVE feedsATT&CK â NIST 800-53HighAutomated (decision support) TABLE I NVD CVE INVENTORY (2020â2026) YearTotal CVEsUsable Descriptions 202021,01319,340 202123,35722,512 202227,46826,372 202331,16730,551 202439,04738,284 202544,01042,263 202612,72112,418 Total198,783191,740 ⢠Middleware: AuditLogService (Apache Log4j 2.14.1), DeviceTelemetryBroker (Eclipse Mosquitto 2.0.14) ⢠Data layer: PatientData Store (PostgreSQL 14.2) ⢠Infrastructure:APIGatewayReverseProxy (Nginx1.21.6),TLSCryptoEngine(OpenSSL 3.0.2),ClinicianAuthService(Keycloak18.0.0), ContainerRuntime(DockerEngine20.10.14), EdgeHostOS (Ubuntu 22.04) All components include explicit CPE 2.3 tags in the SysML model, enabling deterministic NVD matching. Three trust boundaries partition the architecture: IoT LAN, Edge Process- ing, and Cloud DMZ. Log4Shell (CVE-2021-44228, CVSS 10.0) serves as the running example throughout this paper. It is one of the most severe vulnerabilities ever recorded, it affects Log4j directly, and its ATT&CK technique mappings are well-established, making it an ideal end-to-end test case. IV. DATASETS SMSI draws on four publicly available datasets. No propri- etary or sensitive data was used at any stage. A. National Vulnerability Database The NVD is NISTâs repository of CVE records [16]. We downloaded bulk JSON feeds for CVEs published between January 2020 and February 2026. Our NVD snapshot contains 198,783 total vulnerability records across seven annual files, of which 191,740 have usable English-language descriptions (Table I). For MedGatewayâs nine components, CPE-based matching against these feeds returned 199 unique CVEs, distributed as shown in Table I. TABLE I MEDGATEWAY COMPONENT CVE COUNTS ComponentCVEs EdgeHostOS (Ubuntu 22.04)65 TLSCryptoEngine (OpenSSL 3.0.2)40 Clinician AuthService (Keycloak 18.0.0)27 PatientData Store (PostgreSQL 14.2)25 ContainerRuntime (Docker 20.10.14)17 ClinicalDashboardWebApp (Spring 5.3.18)10 DeviceTelemetryBroker (Mosquitto 2.0.14)6 AuditLogService (Log4j 2.14.1)5 APIGateway ReverseProxy (Nginx 1.21.6)4 Total unique CVEs199 B. CWE-CAPEC-ATT&CK Knowledge Graph MITRE publishes a chain of mappings linking Common Weakness Enumeration (CWE) identifiers to Common Attack Pattern Enumeration and Classification (CAPEC) entries, and from CAPEC entries to ATT&CK techniques. We used these transitive mappings in Phase 1 of the supervised classifier pipeline to generate weakly labeled training data for the SecureBERT+ multi-label model. C. CISA KEV and CTID Mappings The CISA Known Exploited Vulnerabilities (KEV) catalog, cross-referenced with the CTID ATT&CK mapping method- ology, provides expert-curated (CVE, ATT&CK technique) pairs. We use the KEV mapping with 806 mapping objects covering 419 unique CVEs (ATT&CK v15.1 alignment) as evaluation ground truth for both the retrieval models and the LLM approach. The CTID ATT&CK-to-NIST 800-53 crosswalk covers 470 ATT&CK techniques and serves as ground truth for the control recommendation stage. D. NIST 800-53 Rev. 5 Control Catalog The official NIST 800-53 Rev. 5 control catalog [15] con- tains approximately 1,000 controls organized into 20 control families (e.g., ACâAccess Control; SCâSystem and Com- munications Protection; IRâIncident Response). We use this catalog to retrieve full control names and descriptions for any controls identified by the lookup or similarity-based stages. ELG5271/CSI5388 â AI FOR CYBERSECURITY APPLICATIONS, UNIVERSITY OF OTTAWA, WINTER 2025/20264 V. PROPOSED APPROACH The SMSI pipeline has five sequential stages. We describe each stage here, continuing to use Log4Shell as the running example. A. Stage 1: SysML Ingestion and Component Extraction The input is a SysML model serialized as XML. The parser walks the XML tree and extracts each sysml:Block element representing a deployed software or hardware component. For each block it reads at- tributes including vendor, product, version, and an ex- plicit cpeHint tag containing a CPE 2.3 identifier (e.g., cpe:2.3:a:apache:log4j:2.14.1). When the CPE tag is absent, a heuristic normalizer constructs a CPE string from the product name and version. This stage is entirely deterministic; no machine learning is involved. The output is a component registry of nine SysML block names paired with CPE identifiers. B. Stage 2: CPE-to-CVE Lookup via NVD For each CPE in the registry, the system queries local NVD JSON feeds over a configurable time window (2020â 2026 in our experiments) and retrieves all matching CVE records using CPE version-range matching with a fallback to product/version mention in descriptions. Each record includes a CVE ID, a natural-language description, a CVSS base score, and optionally a CWE identifier. The result is a vulnerability set of 199 unique CVEs in which each CVE is linked to one or more SysML components, preserving full traceability from architecture to flaw. For Log4j 2.14.1, this stage retrieves CVE-2021-44228 (CVSS 10.0), CVE-2021-45046, CVE-2021-45105, and two additional entries. C. Stage 3: CVE-to-ATT&CK Technique Mapping This is the core AI stage and we explore three complemen- tary approaches. 1) Approach A: Retrieval-Based Dense Encoders: We treat CVE-to-ATT&CK as ranked retrieval rather than classification. A TFâIDF baseline fits a vectorizer on ATT&CK technique descriptions, a system-of-interest (SOI) profile, and a sample of NVD descriptions. For each CVE, cosine similarity against all techniques yields a ranked list. Wethenreplacesparsevectorswithdense embeddingsfromthreeencoders:SecureBERT (ehsanaghaei/SecureBERT)withHFmean- pooling,ATTACK-BERT(basel/ATTACK-BERT) via SentenceTransformer,andMiniLM (all-MiniLM-L6-v2)asanunsupervisedbaseline. A DeBERTa-v3 NLI cross-encoder reranks MiniLMâs top candidates by entailment probability. To inject supervision, we fine-tune SecureBERT and ATTACK-BERT on KEV (CVE, technique) pairs using MultipleNegativesRankingLoss with 80/20 CVE- ID-grouped train/test splits. 2) Approach B: Supervised Multi-Label Classification: Separately, we train a supervised SecureBERT+ classi- fier on a large weakly labeled dataset constructed via CWEâCAPECâATT&CK transitive mappings and KEV gold labels. Phase 1 assemblesâź191K labeled CVE-technique pairs; Phase 1.5 collapses sub-techniques to 105 parent classes; Phase 2 cleans text and applies a minimum support filter; Phase 4 fine-tunes ehsanaghaei/SecureBERT_Plus with BCEWithLogitsLoss for multi-label classification; and Phase 5 performs threshold optimization on the test set. 3) Approach C: Zero-Shot LLM Inference: We evaluate a local LLM (Gemma-4 26B, served via LM Studio) in a zero-shot setting. The LLM receives a structured prompt containing the CVE description and the full ATT&CK parent technique catalogue (214 techniques with names), and returns a JSON list of up to 5 ranked technique predictions. We test two modes: no hint (CVE description only) and DB hint (with CWEâCAPEC-derived technique suggestions appended to the prompt). Evaluation uses two benchmarks: the KEV dataset (419 CVEs) and the SMET academic dataset (302 CVEs). D. Stage 4: ATT&CK-to-NIST 800-53 Control Recommenda- tion Given the ATT&CK techniques predicted for a componentâs CVEs, this stage recommends NIST 800-53 Rev. 5 controls using three mechanisms. 1) Direct Lookup via CTID Crosswalk: The CTID mapping covers 470 ATT&CK techniques. For any technique in this set, the associated controls are retrieved directly; this is a deterministic lookup with precision 1.0 on crosswalk-covered techniques [15]. 2) TFâIDF and Dense Similarity: We reuse the TFâIDF vectorizer from Stage 3, now fit on ATT&CK technique texts, NIST control texts, and the SOI profile. Dense embeddings from SecureBERT, ATTACK-BERT, and MiniLM produce ad- ditional similarity scores. Fine-tuned variants of SecureBERT and ATTACK-BERT trained on CTID crosswalk pairs using MultipleNegativesRankingLoss are also evaluated. 3) Hybrid Control Ranking and CVSS Weighting: A hybrid control score combines the binary crosswalk indicator and TFâ IDF similarity: hybrid score = 0.72¡ crosswalkscore + 0.28¡ tfidfscore, where the crosswalk score is 1.0 for pairs appearing in the CTID mapping and 0 otherwise. CVSS scores from Stage 2 propagate into a final priority score: priority = hybridmappingsoiĂ maxcvss. E. Stage 5: Report Generation and Traceability For every ranked control, SMSI emits the full trace: (SysML component â CPE â CVE â ATT&CK technique â NIST control), along with intermediate similarity scores and CVSS values. ELG5271/CSI5388 â AI FOR CYBERSECURITY APPLICATIONS, UNIVERSITY OF OTTAWA, WINTER 2025/20265 TABLE IV CVE-TO-ATT&CK RETRIEVAL (KEV TEST,n=45) ModelMRRH@1H@5H@10 TFâIDF0.0820.0440.0670.156 SecureBERT (pretrained)0.0330.0000.0220.089 SecureBERT (fine-tuned)0.0870.0220.0890.222 ATTACK-BERT (pretrained)0.1360.0220.2670.356 ATTACK-BERT (fine-tuned)0.1170.0440.1560.267 MiniLM (unsupervised)0.2520.1110.4440.578 VI. EXPERIMENTAL METHODOLOGY A. CVE-to-ATT&CK Evaluation: Retrieval Models We evaluate retrieval models against a held-out set of 45 KEV CVEs (GroupShuffleSplit, 20% test). Each model is evaluated as a ranked retrieval system using MRR and Hits@K for K â 1, 3, 5, 10. We com- pare six approaches: TFâIDF, pretrained SecureBERT, pre- trained ATTACK-BERT, fine-tuned SecureBERT, fine-tuned ATTACK-BERT, and MiniLM (unsupervised). B. CVE-to-ATT&CK Evaluation: Supervised Classifier The SecureBERT+ multi-label classifier is evaluated on a held-out test split using threshold optimization (sweeping 0.10â0.95 in steps of 0.05), reporting Micro/Macro F1, pre- cision, recall, Hamming loss, and per-class F1 at the optimal threshold. Quantitative results are summarized in Section VII (Table V, Figures 3 and 4). C. CVE-to-ATT&CK Evaluation: LLM Zero-Shot The Gemma-4 26B LLM is evaluated on two complete benchmarks without sampling: 419 KEV CVEs and 302 SMET CVEs. We report Hit Rate@K (fraction of CVEs where at least one true technique appears in the top-K predictions), true Recall@K (fraction of all true labels recovered), and classification metrics (weighted F1, macro F1, LRAP). D. ATT&CK-to-NIST Control Evaluation The CTID crosswalk serves as ground truth. We split by technique ID into 80% train and 20% test (94 test queries) and evaluate the same model families using MRR and Hits@K. VII. RESULTS A. CVE-to-ATT&CK: Retrieval Models Table IV summarizes the retrieval evaluation on the 45- query KEV test split. MiniLM (unsupervised) achieves the highest scores across all metrics, with MRR of 0.252 and Hits@10 of 0.578. Pretrained ATTACK-BERT is the second- best model (MRR 0.136, Hits@10 0.356). Fine-tuning im- proves SecureBERT substantially (MRR from 0.033 to 0.087) but unexpectedly degrades ATTACK-BERT (MRR from 0.136 to 0.117). Pretrained SecureBERT performs worst, even below TFâIDF. Figure 1 visualizes these results. The correlation heatmap (Figure 2) reveals that models cap- ture fundamentally different signals. TFâIDF and pretrained Fig. 1. CVE-to-ATT&CK retrieval: all methods vs KEV ground truth (n=45 test CVEs). MiniLM dominates all metrics. Fig. 2. Pearson correlation between all CVE-to-ATT&CK model scores. TFâ IDF and dense models capture largely independent signals. TABLE V SECUREBERT+ MULTI-LABEL CLASSIFIER (PARENT TECHNIQUES, 105 CLASSES) MetricScore Micro Precision0.770 Micro Recall0.744 Micro F10.757 Macro F10.383 Hamming Loss0.032 Optimal Threshold0.45 ATTACK-BERT are uncorrelated (r â 0.00), while fine- tuned ATTACK-BERT and MiniLM are strongly correlated (r = 0.79). Fine-tuned SecureBERT correlates highly with its pretrained version (r = 0.87) but not with other dense models. B. CVE-to-ATT&CK: Supervised Classifier The SecureBERT+ multi-label classifier, trained on the full CWE/CAPEC/KEV dataset and evaluated on 105 parent technique classes, achieves the results shown in Table V. The optimal threshold is 0.45. ELG5271/CSI5388 â AI FOR CYBERSECURITY APPLICATIONS, UNIVERSITY OF OTTAWA, WINTER 2025/20266 Fig. 3. SecureBERT+ aggregate test metrics at optimal threshold 0.45 (105 parent techniques). Hamming loss is 0.032 (table). Fig. 4. SecureBERT+ per-class F1: best 10 vs worst 10 parent techniques. The long tail of zero-scoring classes drives the low Macro F1. TABLE VI GEMMA-4 26B ZERO-SHOT LLM EVALUATION Dataset / ModeHR@1HR@5R@5W-F1 KEV, no hint (n=419)51.878.335.40.282 KEV, +DB hint (n=419)51.873.3â SMET, no hint (n=302)37.874.262.50.411 SMET, +DB hint (n=302)37.873.8â While the Micro F1 of 0.757 is competitive with prior work, the low Macro F1 (0.383) reveals severe class imbalance: the top-performing techniques (e.g., T1574 at F1=0.858, T1027 at F1=0.837) have thousands of training samples, while 10+ techniques score F1=0.000 due to insufficient support (Fig- ure 4). C. CVE-to-ATT&CK: Zero-Shot LLM The Gemma-4 26B LLM achieves substantially higher hit rates than all retrieval models (Table VI), with Hit Rate@1 of 51.8% on KEV (419 CVEs) and 37.8% on SMET (302 CVEs). On the KEV dataset, this represents a âź4Ă improvement over the best embedding baseline (ATTACK-BERT, 13.1%). However, true recall metrics are more modest (R@5 of 35.4% on KEV, 62.5% on SMET), and the LLMâs weighted F1 scores (0.282 on KEV, 0.411 on SMET) fall below the supervised Se- cureBERT+ classifier. Adding CWE/CAPEC-derived database hints did not improve results, slightly degrading R@5 on both datasets. Figure 5 compares the LLM against the ATTACK- BERT baseline. Figure 6 provides a cross-paradigm comparison of the supervised classifier against the zero-shot LLM. The super- vised SecureBERT+ dominates on precision-oriented metrics (Micro F1 0.757 vs W-F1 0.411), while the LLM shows stronger performance on macro F1 for the SMET dataset Fig. 5. Hit Rate comparison: Gemma-4 26B vs ATTACK-BERT embedding baseline on KEV (419 CVEs) and SMET (302 CVEs). Fig. 6. Supervised SecureBERT+ fine-tuning vs zero-shot Gemma-4 26B. The supervised model leads on precision and F1; the LLM is more competitive on recall and rare-class coverage. TABLE VII ATT&CK-TO-NIST CONTROL RETRIEVAL (CROSSWALK TEST,n=94) ModelMRRH@1H@5H@10 TFâIDF0.1790.0320.3400.404 SecureBERT (pretrained)0.5820.4470.7550.798 SecureBERT (fine-tuned)0.2390.1280.3720.500 ATTACK-BERT (pretrained)0.1130.0430.1280.266 ATTACK-BERT (fine-tuned)0.1950.0850.2870.436 MiniLM (unsupervised)0.1240.0430.1810.277 (0.405 vs 0.383), suggesting it handles rare technique classes more uniformly. D. ATT&CK-to-NIST Control Recommendation Table VII shows the ATT&CK-to-controls retrieval results on 94 test technique queries. Pretrained SecureBERT achieves the highest scores by a wide margin (MRR 0.582, Hits@10 0.798), substantially outperforming all other models including its own fine-tuned variant (MRR 0.239). This reversal from the CVE-to-ATT&CK results suggests that SecureBERTâs pre- trained embeddings already encode strong alignment between security concept descriptions and control language, and that fine-tuning on the relatively small crosswalk dataset introduces overfitting. The CVSS-weighted priority distribution (Figure 8) is heavy-tailed: most technique-control pairs cluster near zero, while a small fraction dominate the upper tail. This is desir- ELG5271/CSI5388 â AI FOR CYBERSECURITY APPLICATIONS, UNIVERSITY OF OTTAWA, WINTER 2025/20267 Fig. 7. ATT&CK-to-controls retrieval: all methods vs CTID crosswalk ground truth (n=94). Pretrained SecureBERT dominates. Fig. 8.Priority distribution for technique-control pairs (priority = hybrid_mapping_soi Ă max_cvss); clipped at the 99th percentile. able for triage because it concentrates analyst attention on a manageable subset of high-impact recommendations. E. End-to-End Pipeline: MedGateway TracingLog4Shellthroughthefullpipeline illustrateshowthestagesconnect.TheSysML parseridentifiestheAuditLog Serviceblockwith cpe:2.3:a:apache:log4j:2.14.1.TheNVD lookup returns 5 CVEs including CVE-2021-44228 with CVSS 10.0. The CVE-to-ATT&CK stage assigns high scores to techniques such as T1190 (Exploit Public-Facing Application) and T1059 (Command and Scripting Interpreter) across multiple models. The CTID crosswalk links these to controls including AC-3 (Access Enforcement), SI-10 (Information Input Validation), SC-7 (Boundary Protection), and SI-3 (Malicious Code Protection). The CVSS-weighted hybrid score pushes Log4Shell-related pairs toward the head of the global priority list. VIII. DISCUSSION A. Why MiniLM Outperformed Domain-Specific Models The most surprising CVE-to-ATT&CK result is that MiniLM (a generic 22M-parameter sentence encoder with no security-specific pretraining) outperforms both frozen and fine-tuned domain-specific models. We hypothesize two con- tributing factors. First, MiniLM was trained with a contrastive objective on diverse sentence pairs, producing an embedding space well-suited for cross-domain semantic similarity. Secu- rity domain pretraining (SecureBERT) or ATT&CK-specific training (ATTACK-BERT) may actually narrow the embedding space in ways that hurt cross-document retrieval between CVE descriptions and technique definitions, which use fundamen- tally different vocabulary. Second, the KEV fine-tuning set is small (fewer than 200 training CVEs), making overfitting likely for the larger domain models. B. Supervised vs Zero-Shot: Complementary Strengths The three CVE-to-ATT&CK approaches occupy different points in the precision-coverage tradeoff. The supervised Se- cureBERT+ classifier achieves the highest Micro F1 (0.757) but only on techniques with sufficient training data; its Macro F1 of 0.383 shows it effectively ignores rare classes. The Gemma-4 LLM achieves lower overall F1 but distributes its predictions more uniformly across the technique space (SMET Macro F1 of 0.405), suggesting that LLMs are better at rare-technique coverage. The retrieval models (MiniLM) provide ranked lists rather than binary predictions, making them suitable for shortlisting candidates that analysts can review. A production system could ensemble all three: use the classifier for confident high-frequency predictions, the LLM for rare techniques, and retrieval scores for ranking. C. SecureBERTâs Surprising Control-Retrieval Dominance In the ATT&CK-to-controls stage, pretrained SecureBERT (MRR 0.582) dramatically outperforms all other models in- cluding MiniLM (MRR 0.124). This is the opposite of the CVE-to-ATT&CK pattern, where SecureBERT performed worst. The explanation lies in the nature of the text: ATT&CK technique descriptions and NIST control descriptions both use formal security terminology, which is exactly what Se- cureBERTâs pretraining corpus covers. CVE descriptions, by contrast, mix security language with product-specific jargon (e.g., JNDI lookups, Spring DataBinder), creating a vocabulary mismatch that SecureBERTâs pretraining does not resolve. D. Limitations The NVD is queried as a static snapshot; new vulnerabilities disclosed after the query date are not captured. The KEV fine-tuning split contains only 45 test CVEs, limiting statis- tical power. The LLM evaluation depends on a local model server, making exact reproducibility hardware-dependent. The CWE/CAPEC knowledge graph is used for training data generation but not yet for inference-time reasoning. All mod- els operate on English-language artifacts and the Enterprise ATT&CK profile; extending to ICS/OT ATT&CK remains future work. CPE matching can fail for vague SysML descrip- tions. IX. CONCLUSION SMSI demonstrates that the end-to-end threat modeling workflow, from system architecture to ranked security controls, can be meaningfully automated for a concrete SysML case study with nine components and 199 CVEs. The key insight is that the problem has two distinct sub-problems that benefit from different strategies: structured data lookups (SysML to CPE to CVE; ATT&CK to NIST for crosswalk-covered ELG5271/CSI5388 â AI FOR CYBERSECURITY APPLICATIONS, UNIVERSITY OF OTTAWA, WINTER 2025/20268 techniques) are best handled deterministically, while semantic reasoning over natural language (CVE to ATT&CK; ATT&CK to NIST for uncovered techniques) benefits from neural models grounded by symbolic constraints and curated mappings. Our experiments reveal that no single approach dominates CVE-to-ATT&CK mapping. The supervised SecureBERT+ classifier achieves the best Micro F1 (0.757) but suffers from severe class imbalance. The zero-shot Gemma-4 LLM achieves dramatically higher hit rates (51.8% HR@1 vs <15% for embedding models) and better rare-class coverage. The unsupervised MiniLM provides the best retrieval-based rank- ing (MRR 0.252), offering a strong baseline that requires no labeled data. For ATT&CK-to-controls, pretrained Se- cureBERT embeddings provide surprisingly effective retrieval (MRR 0.582), and the CTID crosswalk remains the most reliable backbone signal. From a practical standpoint, SMSI reduces what would otherwise require days of manual expert analysis to an auto- mated pipeline producing fully traceable recommendationsâ provided they are reviewed by human analysts rather than treated as an oracle. X. FUTURE WORK The most impactful near-term improvement would be mak- ing the pipeline explicitly architecture-conditioned: incor- porating network exposure, component criticality, and data sensitivity into the risk weighting formula so that a Log4j instance on the public internet triggers different prioritization than one on an isolated subnet. Real-time NVD integration would enable continuous monitoring. The CVE-to-ATT&CK stage could incorporate the CWE/CAPEC knowledge graph at inference time, combining deterministic graph traversal with neural retrieval in a principled neuro-symbolic fashion [4], [11]. An ensemble of the supervised classifier, LLM, and retrieval models could be calibrated to optimize precision- recall tradeoffs for specific deployment contexts. Finally, a user study with practicing security analysts would measure whether SMSIâs recommendations genuinely reduce manual effort. ACKNOWLEDGMENT The authors would like to thank Prof. Paula Branco for her guidance throughout the course, and the course TA for their feedback during the progress check meeting. RoâYah also gratefully acknowledges the mentorship and support of Roderick Fernandes, Senior Special Advisor / Cyber Security Engineer at DND, whose guidance over the past three years has shaped much of her understanding of this field. REFERENCES [1] O. Grigorescu, A. Nica, M. Dascalu, and R. Rughinis, âCVE2ATT&CK: BERT-based mapping of CVEs to MITRE ATT&CK techniques,â Algo- rithms, vol. 15, no. 9, Aug. 2022, doi: 10.3390/a15090314. [2] I. Branescu, O. Grigorescu, and M. Dascalu, âAutomated mapping of common vulnerabilities and exposures to MITRE ATT&CK tactics,â Information, vol. 15, no. 4, Apr. 2024, doi: 10.3390/info15040214. [3] L. Li, C. Huang, and J. Chen, âAutomated discovery and mapping ATT&CK tactics and techniques for unstructured cyber threat in- telligence,â Comput. Secur., vol. 140, p. 103815, May 2024, doi: 10.1016/j.cose.2024.103815. [4] B. Abdeen, E. Al-Shaer, A. Singhal, L. Khan, and K. Hamlen, âSMET: Semantic mapping of CVE to ATT&CK and its application to cy- bersecurity,â in Data and Applications Security and Privacy XXXVII, V. Atluri and A. L. Ferrara, Eds. Cham: Springer, 2023, p. 243â260, doi: 10.1007/978-3-031-37586-6 15. [5] B. Abdeen et al., âSMET: Semantic mapping of CTI reports and CVE to ATT&CK for advanced threat intelligence,â J. Comput. Secur., 2024, to appear. [Online]. Available: https://doi.org/10.3233/JCS-230218. [6] A. Kuppa, L. Aouad, and N.-A. Le-Khac, âLinking CVEâs to MITRE ATT&CK techniques,â in Proc. 16th Int. Conf. Availability, Reliability and Security (ARES), New York, NY, USA: ACM, Aug. 2021, p. 1â12, doi: 10.1145/3465481.3465758. [7] X. Liu, Y. Tan, Z. Xiao, J. Zhuge, and R. Zhou, âNot the end of story: An evaluation of ChatGPT-driven vulnerability description mappings,â in Findings of the Assoc. Comput. Linguist.: ACL 2023, Toronto, Canada: ACL, Jul. 2023, p. 3724â3731, doi: 10.18653/v1/2023.findings-acl.229. [8] P. Rafiey and A. Namadchian, âMapping vulnerability description to MITRE ATT&CK framework by LLM,â May 2024, Research Square, doi: 10.21203/rs.3.rs-4341401/v1. [9] Y.-T. Huang et al., âMITREtrieval: Retrieving MITRE techniques from unstructured threat reports by fusion of deep learning and ontology,â IEEE Trans. Netw. Service Manag., vol. 21, no. 4, p. 4871â4887, Aug. 2024, doi: 10.1109/TNSM.2024.3401200. [10] Z. Li, J. Zeng, Y. Chen, and Z. Liang, âAttacKG: Constructing technique knowledge graph from cyber threat intelligence reports,â arXiv:2111.07093, May 2022, doi: 10.48550/arXiv.2111.07093. [11] E. Hemberg et al., âLinking threat tactics, techniques, and patterns with defensive weaknesses, vulnerabilities and affected platform con- figurations for cyber hunting,â arXiv:2010.00533, Feb. 2021, doi: 10.48550/arXiv.2010.00533. [12] S. Fowler, K. Joiner, and S. Ma, âCyber Evaluation and Management Toolkit (CEMT): Face validity of model-based cybersecurity deci- sion making,â Systems, vol. 12, no. 7, Jun. 2024, doi: 10.3390/sys- tems12070238. [13] A. Høst, P. Lison, and L. Moonen, âA systematic approach to predict the impact of cybersecurity vulnerabilities using LLMs,â 2025. [Online]. Available: https://arxiv.org/abs/2508.18439. [14] R. Sahu and M. Speretta, âStatistical word analysis to support the semiautomatic implementation of the NIST 800-53 cybersecurity frame- work,â in Proc. ASEE Annu. Conf., 2024. [15] Joint Task Force, Security and Privacy Controls for Information Sys- tems and Organizations, NIST Special Publication 800-53 Rev. 5, National Institute of Standards and Technology, Dec. 2020, doi: 10.6028/NIST.SP.800-53r5. [16] National Institute of Standards and Technology, âNational Vulnerability Database,â [Online]. Available: https://nvd.nist.gov/. Accessed: Feb. 21, 2026. [17] MITRE Corporation, âMITRE ATT&CK,â [Online]. Available: https: //attack.mitre.org/. Accessed: Feb. 21, 2026.