Paper deep dive
CyberCertBench: Evaluating LLMs in Cybersecurity Certification Knowledge
Gustav Keppler, Ghada Elbez, Veit Hagenmeyer
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 96%
Last extracted: 4/26/2026, 5:44:08 PM
Summary
The paper introduces CyberCertBench, a new suite of Multiple Choice Question Answering (MCQA) benchmarks designed to evaluate the domain-specific cybersecurity knowledge of Large Language Models (LLMs). The benchmark covers a gradient from general IT networking (Cisco CCNx) to specialized Operational Technology (OT) and formal standards (ISA/IEC 62443). The research identifies a performance hierarchy where frontier models excel in general IT security but struggle significantly with vendor-specific procedural knowledge and formal industrial standards. The authors also propose a Proposer-Verifier framework for generating interpretable explanations and analyze model scaling trends, noting that recent advancements are driven more by training methodology and data curation than by parameter count alone.
Entities (10)
Relation Signals (5)
CyberCertBench → contains → Cisco CCNx
confidence 100% · Specifically, we selected certifications from Cisco, Fortinet, and ISA/IEC...
CyberCertBench → contains → Fortinet NSE
confidence 100% · The Fortinet NSE dataset focuses on procedural, vendor-specific configuration
CyberCertBench → contains → ISA/IEC 62443
confidence 100% · The ISA/IEC 62443 benchmark is the most specialized...
Gemini 2.5 Pro → evaluatedon → CyberCertBench
confidence 100% · Figure 1 presents the performance of a representative set of 11 models across six cybersecurity benchmarks.
Proposer-Verifier framework → explainsperformanceof → LLMs
confidence 90% · a methodology to generate interpretable, natural language explanations for model performance.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:The rapid evolution and use of Large Language Models (LLMs) in professional workflows require an evaluation of their domain-specific knowledge against industry standards. We introduceCyberCertBench, a new suite of Multiple Choice Question Answering (MCQA) benchmarks derived from industry recognized certifications. CyberCertBench evaluates LLM domain knowledgeagainst the professional standards of Information Technology cybersecurity and more specializedareas such as Operational Technology and related cybersecurity standards. Concurrently, we propose and validate a novel Proposer-Verifier framework, a methodology to generate interpretable,natural language explanations for model performance. Our evaluation shows that frontier modelsachieve human expert level in general networking and IT security knowledge. However, theiraccuracy declines in questions that require vendor-specific nuances or knowledge in formalstandards, like, e.g., IEC 62443. Analysis of model scaling trend and release date demonstratesremarkable gains in parameter efficiency, while recent larger models show diminishing this http URL and evaluation scripts are available at: this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2604.20389v1
- Canonical: https://arxiv.org/abs/2604.20389v1
Trouble viewing inline? Open PDF directly →
Full Text
67,990 characters extracted from source content.
Expand or collapse full text
CyberCertBench: Evaluating LLMs in Cybersecurity Certification Knowledge Gustav Keppler, Ghada Elbez, and Veit Hagenmeyer Institute for Automation and Applied Informatics (IAI), Karlsruhe Institute of Technology (KIT), Karlsruhe, Germany gustav.keppler,ghada.elbez,veit.hagenmeyer@kit.edu Abstract The rapid evolution and use of Large Language Models (LLMs) in professional workflows re- quire an evaluation of their domain-specific knowledge against industry standards. We introduce CyberCertBench, a new suite of Multiple Choice Question Answering (MCQA) benchmarks de- rived from industry-recognized certifications. CyberCertBench evaluates LLM domain knowledge against the professional standards of Information Technology cybersecurity and more specialized areas such as Operational Technology and related cybersecurity standards. Concurrently, we pro- pose and validate a novel Proposer-Verifier framework, a methodology to generate interpretable, natural-language explanations for model performance. Our evaluation shows that frontier models achieve human expert level in general networking and IT security knowledge. However, their accuracy declines in questions that require vendor-specific nuances or knowledge in formal standards, like, e.g., IEC 62443. Analysis of model scaling trend and release date demonstrates remarkable gains in parameter efficiency, while recent larger models show diminishing returns. Code and evaluation scripts are available at: https://github.com/GKeppler/CyberCertBench. Keywords: Large Language Models, Benchmarking, Cybersecurity, Operational Technology, MCQA 1 Introduction Large language models (LLMs) have shown potential for generative and knowledge-intensive tasks that require domain knowledge, leading to rapid integration into professional workflows. Recent empirical studies demonstrate that software engineers and IT professionals increasingly rely on conversational AI tools for fact checking, code generation, debugging, and documentation tasks [1–3]. Similarly, researchers have described the growing adoption of LLM-based assistants in cybersecurity workflows, where professionals use them for vulnerability assessment, incident reporting, and threat- intelligence summarization [4–6]. Comparable trends are evident across more general high-stakes technical domains such as healthcare, engineering, and infrastructure operations [7,8]. This increasing dependence underscores a broader shift toward AI-mediated decision-making in professional contexts. However, it also introduces risks of LLMs "hallucinating" [9] plausible-sounding but unfaithful or nonsensical information, which can be particularly hazardous in domains where reliability and accuracy are critical. A growing body of research has begun to evaluate the Information Technology (IT) security capabilities of LLMs using question-answering benchmarks [10,11] and assessing task-specific skills like cyber threat intelligence analysis and vulnerability detection [4–6]. 1 arXiv:2604.20389v1 [cs.CR] 22 Apr 2026 However, specialized knowledge—such as the procedural, vendor-specific details required to manage enterprise security hardware or the high-stakes domain of Operational Technology (OT) cybersecurity is unexplored. These systems form the backbone of enterprise environments and critical infrastructure, including energy grids, water treatment facilities, where security failures can have catastrophic physical consequences [12]. The knowledge required to secure these environments includes unique protocols, real-world physical process implications, and security frameworks like ISA/IEC 62443. These are more sparsely represented in the web-scale corpora used to train foundation models, compared to general IT knowledge. Therefore, LLMs that show high capability in IT security-related benchmarks may strongly ‘hallucinate’ plausible but incorrect guidance for OT systems, creating a false sense of security for operators and engineers [13]. The extent to which LLMs can reliably provide guidance in these specialized areas is largely unquantified. The present paper addresses this challenge by introducing CyberCertBench, a suite of multiple- choice question answering benchmarks derived from professional certification exams that serve as a proxy for the practical and real-world knowledge. By spanning the gradient from IT to vendor-specific IT and specialized OT, our evaluation provides the first analysis of this performance hierarchy. We quantify the knowledge gaps that emerge as the required expertise becomes more specialized, posing a direct risk to the safe adoption of LLMs. Our key contributions are: •A suite of novel benchmarks for IT and OT cybersecurity knowledge of LLMs, derived from industry-recognized certification standards. •A systematic analysis of a wide range of LLMs, revealing a performance gradient where accuracy is high on benchmarks testing conceptual knowledge, both in IT and OT, but degrades significantly on those requiring specialized knowledge of vendor-specific procedures and formal standards. •A novel Proposer-Verifier framework to generate interpretable descriptions of question difficul- ties, offering a method to understand the causes of these knowledge gaps. 2 Related Work A variety of specialized benchmarks are used for evaluating the cybersecurity capabilities of Large Language Models (LLMs). These benchmarks have evolved from broad knowledge assessments to more practical and domain-specific evaluations. Security Knowledge Benchmarks Multiple-choice question benchmarks offer measurability and scalability. Besides the MMLU Computer Security subtask [10], WMDP [14] offers questions on biosecurity, cybersecurity, and chemical security. CyberSecEval uses free-response questions evaluated by another LLM. Other benchmarks like SecEval [15], SecQA [16], and CyberMetric [11] use LLMs to generate questions from textbooks and other technical documents. While comprehensive, these benchmarks often suffer from saturation, where top models achieve near-perfect scores, limiting their ability to differentiate between frontier models. Specialized benchmarks such as CTIBench[4], which is designed specifically for Cyber Threat Intelligence tasks, and OCCULT[17], a multiple-choice benchmark focused on offensive cybersecurity tactics, provide targeted evaluations for distinct areas within the cybersecurity domain. Agent-based and Offensive Capability Evaluation Recent literature is evaluating the practical, and often offensive, capabilities of LLM-based agents. This is motivated by the need to understand 2 potential misuse and to develop more robust defenses [5,18]. Early work demonstrated that LLMs could assist Pentesting PentestGPT [19] and solving Capture the Flag Challenges [20], leading to the creation of dedicated benchmark datasets like the NYU CTF Dataset [21]. This also includes the development of sophisticated LLM agents such as AutoAttacker [22], and EnIGMA [23], and multi-agent frameworks like D-CIPHER [24] designed to autonomously solve these challenges. Multitask Frameworks To standardize these practical evaluations, several comprehensive frame- works have been proposed. The Catastrophic Cyber Capabilities Benchmark (3CB) [25] specifically focuses on assessing offensive capabilities like vulnerability exploitation and privilege escalation. CyberBench [26] evaluates multiple tasks, including text classification and summarization. Concur- rently, major AI labs have released their own evaluation suites, such as CYBERSECEVAL [27,28] by Meta or Google Deepmind [5], which measure both the beneficial capabilities and potential risks of frontier models. These agent-based evaluations test knowledge, tool use, and planning in dynamic environments. Evaluation Using Professional Certifications To ensure authenticity and quality while avoiding the synthetic nature of benchmarks, some studies have turned to professional certification exams. Tann et al. [29] first demonstrated that LLMs could pass Cisco certification exams. Earlier work [30] also evaluated models on publicly available Cisco CCNA and CCNP materials, establishing a reproducible baseline for Information Technology (IT) security knowledge. 3 Methodology The methodology is centered around a novel collection of benchmarks derived from professional certifications, and compared to established computer security benchmarks. This section details the creation and composition of these benchmarks, the selection of models for evaluation, and the experimental setup. 3.1 Benchmark Collection and Curation To comprehensively evaluate the cybersecurity knowledge of LLMs, we curated a suite of Multiple Choice Question Answering (MCQA) benchmarks. This collection includes established academic benchmarks [10,11] to establish a baseline, alongside several novel datasets derived from industry career certifications. This approach allows for an analysis of both general and specialized knowledge, with a particular focus on the under-evaluated domain of OT. Specifically, questions were collected from freely available content from websites such asitexams.com, examtopics.com, and free-braindumps.com. These platforms were chosen for their extensive and frequently updated question banks, which serve as a proxy for the material covered in the official certification exams. We selected certifications from Cisco, Fortinet, and ISA/IEC as they represent widely recognized industry standards for IT networking and OT systems. Despite this selection and expert validation of labels, the resulting datasets remain subject to sampling bias. The question pools are drawn from community-driven platforms where users tend to share items they find difficult, surprising, or exam-relevant, rather than a random sample of all possible certification questions or real-world issues. As a consequence, certain topics and question styles may be overrepresented, while scenario-based, free-form, or practical configuration tasks are underrepresented. The scraped multiple-choice questions were preprocessed: 3 CriterionCisco CCNxFortinet NSEFortinet ICS/SCADA ISA/IEC 62443 Benchmark Category IT NetworkingSpecialized IT Security Conceptual OT Security Formal OT Security Standard Knowledge Domain Cisco Networking Fortinet Security Ecosystem Foundational ICS/OT Concepts The ISA/IEC 62443 Standard Question Focus Foundational & General. IT networking and security principles, with vendor specifics canonical in the industry. Procedural & Vendor-specific. Configuration and management of a proprietary product ecosystem. Conceptual & Domain-general. Core OT security principles and technologies, often independent of a specific vendor. Standard-specific & Formal. Detailed terminology, structure, and prescribed methodologies of the standard. Table 1: Analysis of the CyberCertBench Suite across a gradient of knowledge specificity. 1. De-duplication: Exact and near-duplicate questions were identified and removed. 2.Formatting Normalization: Questions with non-standard formats, such as "drag-and-drop" or with reference images, were excluded. 3.Standardization: All questions, choices, and answers were converted into a unified MMLU- compatible format for seamless integration with evaluation harnesses. 4.Expert Label Verification: To mitigate label noise arising from the user-generated nature of the source platforms, we manually reviewed all questions and answer keys. Questions with ambiguous wording, inconsistent answers across sources, or suspected wrong solutions were either corrected, based on authoritative documentation, or removed from the dataset entirely. This manual curation step ensures that the final benchmark reflects a high-quality, expert-validated ground truth, although residual noise cannot be fully ruled out. 3.2 Baseline Academic Benchmarks To compare the results, we include two established cybersecurity benchmarks: MMLU Computer Security [10] is a subtask from the widely used Massive Multitask Language Understanding benchmark. It consists of multiple-choice questions designed to test general knowledge in computer security. We use it as a baseline to compare the performance of models on the specialized datasets. CyberMetric80 [11] is a more recent benchmark created using Retrieval-Augmented Generation (RAG) from authoritative sources like NIST standards and research papers. The quality of the content was approved by human experts. 3.3 Industry Certification Benchmarks The core of our evaluation is CyberCertBench, a novel suite of datasets derived from industry- recognized certifications. This approach provides high-quality questions that directly measure the practical knowledge required of professionals across a gradient of specificity. As detailed in Table 1, 4 the benchmarks test distinct types of cybersecurity knowledge. The Fortinet NSE dataset focuses on procedural, vendor-specific configuration, whereas the Fortinet ICS/SCADA dataset assesses broader, conceptual knowledge of the OT security domain. The ISA/IEC 62443 benchmark is the most specialized, testing formal methodology and terminology of that series of standards that address security for OT in automation and control systems. Gemini 2.5 Pro 0506 Claude 3.5 Sonnet Moonshot AI Kimi K2 OpenAI GPT 5 DeepSeek v3 0324 OpenAI GPT OSS 120B Qwen3 235B A22B Mistral Small 3.1 24B Llama3.1 405B Llama3.2 3B Gemma 3 4B 30 40 50 60 70 80 90 100 Accuracy (%) 98 95 9898 95 99 91 94 100 84 85 89 87 87 88 84 91 89 82 84 6868 92 8686 86 82 79 80 80 81 53 51 62 60 61 53 62 52 52 53 54 42 37 86 8484 88 79 88 79 88 81 74 53 74 81 77 70 77 62 74 66 63 38 40 Benchmark Categories Academic Baseline CyberMetric80 MMLU Computer Security IT Enterprise Networking Cisco CCNx Fortinet NSE OT/Industrial Security Fortinet ICS/SCADA ISA/IEC 62443 Passing Grade 80%) Figure 1: Overview of 5-shot accuracy for a selection of LLMs across six cybersecurity benchmarks ordered by average accuracy. Top proprietary models show similar performance compared to open- weight competitors, with specialized OT datasets proving more challenging for all models. 3.4 Model Selection Our model selection aims to cover the state-of-the-art across proprietary and open-weight models of various sizes, leveraging MMLU [31] leaderboards. Selection criteria include a range of models in terms of: • Modelsize: From small to the largest models • Region: Models from the US, Europe and Asia • Release Date: Models released since ChatGPT in 2022 • Availability: Proprietary and open-weight models The selection includes the latest proprietary frontier models from leading US-based labs, including OpenAI, Anthropic, and Google. To capture the full performance spectrum, we also include their smaller, efficiency-focused counterparts. The evaluation also features a wide array of open-weight models to analyze the broader ecosystem. This includes foundational model families from Western developers, such as Meta’s Llama series, Mistral AI’s models, and Google’s Gemma family. To provide a global perspective, we have incorporated leading models from Asia, including Alibaba’s Qwen series and Moonshot AI’s Kimi K2. The recent open-weight contributions from OpenAI (GPT-OSS series) are also included. 5 3.5 Experimental Setup and Metrics The evaluation is conducted as an MCQA task, where a model is given a questionqand a set of choicesC, from which it must select the correct option(s)a. We use a 5-shot prompting approach, providing five in-domain examples before the actual question to set the context. For all tests, the model temperature is set to zero to ensure deterministic outputs. To contextualize model performance, we establish a human expert baseline. The passing score for professional certifications like the Cisco exams is typically around 80%. This score represents the minimum competency expected of a certified professional and serves as a target benchmark. Performance is measured using exact accuracy. A response is considered correct only if the model identifies all of the correct option(s) and none of the incorrect ones. This strict metric is suitable for the MCQA format where partial credit is not applicable. Figure 2: Average accuracy across all certification datasets plotted against release date. The upward trend highlights progress, with smaller, newer models approaching the performance of larger, older models. 3.6 Legal Considerations The certification-style questions used in CyberCertBench were obtained via automated scraping from publicly accessible platforms. Our use of this material is limited to internal text-and-data processing for non-commercial scientific research, and based on access in the sense that no technical protection measures were circumvented. In line with the restrictions on handling IP-sensitive content, we do not publicly redistribute the full question text. The paper reports only aggregate statistics and derived analyses. Access to the underlying questions can be granted to qualified researchers on request. 4 Quantitative Evaluation This section presents a quantitative analysis of LLM performance across cybersecurity benchmarks. First, the overall model accuracies in comparison to human baseline is analyzed. Then the influence of model scale and release date factors on performance is investigated revealing trends in scaling laws and open-weight competition. 6 4.1 Performance on Cybersecurity Certifications Figure 1 presents the performance of a representative set of 11 models across six cybersecurity benchmarks. The results reveal a performance hierarchy, with large proprietary models such as Gemini 2.5 Pro, Claude 3.5 Sonnet, and OpenAI GPT 5 consistently achieving the highest scores. Notably, top-tier open-weight models, like Moonshot AI’s Kimi K2, demonstrate highly competitive performance. On general IT security benchmarks like CyberMetric80 and MMLU Computer Security, frontier models approach or achieve perfect scores, suggesting these benchmarks are becoming saturated. To contextualize these results, we compare model performance against a human expert baseline, represented by the 80% passing grade typical for professional certifications. On the IT-focused Cisco certification, top models consistently meet or exceed this professional standard. On the Fortinet ICS/SCADA benchmark, which tests more conceptual OT security knowledge, many top models still achieve a passing grade. This indicates that general concepts in OT security are somewhat accessible. However, on the most specialized benchmark, ISA/IEC 62443, which tests formal, standard- specific knowledge, performance collapses. Only a single model, Claude 3.5 Sonnet at 81%, achieves a passing score. Notably, Claude 4.5 Sonnet has a lower performance on this benchmark and a lower average performance. This shows that while LLMs may have a conceptual understanding at the surface level of OT, they lack the deep, formal knowledge required to be reliable in safety-critical industrial environments. The challenge is most pronounced on the Fortinet NSE benchmark, which focuses on vendor- specific procedural knowledge. No model comes close to the passing grade, with the highest score being 62.4%. This indicates that while LLMs trained on web-scale data have acquired a high conceptual IT and OT security knowledge, they lack an understanding of formal standards and vendor-specific operational details. This specialized knowledge remains a significant and measurable frontier. 4.2 Model Performance Over Time Figure 2 analyzes model performance as a function of release date to offer insight into the pace of progress. The results reveal a clear positive overall trend, indicating that the knowledge of LLMs in cybersecurity is advancing. However, this progress is not consistent across all model families, and model sizes. The Qwen series demonstrates a consistent improvement trajectory from Qwen2 through Qwen3, with newer models maintaining or outperforming older ones. In contrast, the Llama 4 series does not show an improvement over the Llama 3 series on the benchmarks, and the Gemma 3 series exhibits lower average performance than its predecessor, Gemma 2. This suggests that progress is highly dependent on the specific training data used for each model generation, which may not always align with improvements on these specialized tasks. The top of the performance leaderboard also shows Google’s Gemini 2.5 Pro (May 2025) surpassing the earlier Claude 3.5 Sonnet (July 2024) as the best-performing model overall for almost one year, while in the open-weight category, DeepSeek v3 and then Kimi K2 surpass Qwen2 72B. While performance improvements are less profound in the largest models, the increase among smaller, more efficient language models is to be noted. Models in the 14B to 32B parameter range, such as Qwen3 14B/32B and Mistral Small 3.1 24B, show a substantial leap in capability over time, achieving performance levels that are highly competitive with much larger models from previous generations. This trend indicates that progress is driven not by scaling up parameter counts but by 7 advances in training methodology, particularly in data curation. Figure 3: Open-weight model average accuracy across all certification datasets versus parameter size in billions, on a logarithmic scale. The plot shows a clear positive trend where larger models generally perform better, but also highlights significant differences in parameter efficiency among models of similar sizes. 4.3 Scaling Laws: Performance vs. Parameters To investigate the relationship between model size and cybersecurity knowledge, the average accuracy of each open-weight model is plotted against its parameter count, as shown in Figure 3. While this plot illustrates trends in the open-weight ecosystem, the exact parameter counts of the top-performing proprietary models are not public. The plot reveals a strong positive correlation between model size and average performance. The largest open-weight models, such as Moonshot AI’s Kimi K2 ( 1 trillion parameters) and DeepSeek v3 0324, occupy the top-right quadrant. However, the distribution also highlights significant variations in parameter efficiency. Models of similar size exhibit a wide performance spread. A clear example is the Llama2 70B model, which is substantially outperformed by more recent models in the same size class, such as Qwen2 72B and Llama3.1 70B, demonstrating generational progress. This trend is even more pronounced when comparing models across different size classes. For instance, models such as Qwen3 14B and Mistral Small 3.1 24B achieve average accuracies that are highly competitive with much larger models like Llama 3.1 405B. This demonstrates that improvements in parameter efficiency allow smaller models to achieve performance on par with counterparts an order of magnitude larger in these specialized domains. 4.4 Evaluating on a More Challenging "PRO" Benchmark As state-of-the-art models approach saturation on standard benchmarks, their utility for differentiat- ing between top performers is reduced. Inspired by the MMLU-Pro [32], we created a more difficult 8 "PRO" version of the benchmark suite through a data-driven curation process that filters out easy questions. A question is classified as easy if it is answered correctly by at least two out of three small models: Ministral 3B, Gemma 3 4B, and Llama3.2 3B. As shown in Table 2, this filtering process removed approximately 50% of the questions from the certification-based benchmarks. The removal rate varied from 39.0% for the Fortinet NSE dataset to 67.4% for the Fortinet ICS/SCADA dataset, indicating differences in baseline difficulty across the domains. Table 2: Statistics of the PRO benchmark showing the original number of questions, easy questions removed, and remaining in the PRO version. Dataset#Tot #Easy #PRO CCNx20010991 Fort. ICS/SCADA432914 ISA/IEC 62443532429 Fortinet NSE1415586 Gemini 2.5 Pro 0506 Kimi K2 Mistral 3.1 24B Qwen2.5 7B Ministral 3B 0 20 40 60 80 100 Average Accuracy 84 72 83 68 77 61 70 45 62 18 Original Benchmark Benchmark PRO Figure 4: Average model accuracy on the original benchmarks versus the “PRO” version on selected models. While performance drops universally, the magnitude of the drop is not uniform and correlates with model capability. Top-tier models like Gemini 2.5 Pro and Moonshot AI Kimi K2 show a low performance decrease. In contrast, smaller models experience a higher performance degradation. The accuracy of Qwen2.5 7B falls by 25 percentage points, while the small baseline model, Ministral 3B, falls by 44 points from 62% to just 18%, indicating that it still answers some questions correctly that the other two small models cannot. 5 Interpretable Difficulty Analysis 1. Question Difficulty Grouping 2. Concept Proposal 3. Concept Validation hard medium Questions Evaluation Models Proposer LLM Verifier LLM Test Questions True Labels Predicted Labels Validation Score (F1) easy train test Concept easy Concept medium Concept hard Figure 5: The Proposer-Verifier framework workflow. The process involves (1) grouping questions by difficulty, (2) using a Proposer LLM to generate difficulty descriptions from a training set, and (3) using a Verifier LLM to validate these descriptions by classifying questions from a held-out test set to produce the final validation statistics. 9 The quantitative analysis reveals a significant discrepancy between model performance on IT and OT benchmarks. However, these aggregate accuracy scores do not explain the reasons why certain questions or entire domains are more difficult for LLMs. To understand these underlying factors, we conducted a qualitative analysis of the datasets. Inspired by recent work that uses LLMs to describe dataset differences [33], we implement a Proposer-Verifier framework. To the best of the authors’ knowledge, this is the first method that adapts Item Response Theory (IRT) by first identifying groups of questions with varying difficulty [34] and then using LLMs to generate and validate interpretable descriptions of these difficulty groups. The methodology consists of three main steps: 1. Difficulty Grouping: Rank all questions based on their empirical difficulty as average accuracy across all models. Partition them into a set number of distinct difficulty groups, like hard, medium, and easy. 2.Concept Proposal: A LLM (the "Proposer") takes questions from the training split of each difficulty group and generates a natural language description of the distinguishing characteristics of each group. 3. Concept Validation: To validate the quality and utility of these descriptions, we use them as a classification rubric. A LLM (the "Verifier") is tasked with predicting the difficulty group of questions from a held-out test split, based solely on the generated descriptions. This Proposer-Verifier method, as shown in Figure 5, bridges quantitative difficulty signals with qualitative explanations, yielding natural language explanations into model performance. The following section provides a more detailed description of the method. 5.1 Mathematical Formulation LetQ=q 1 , . . . , q N be the set of questions in a benchmark, whereNis the total number of questions. LetM=m 1 , . . . , m M be the set of evaluated LLMs. For each questionq i and model m j , we define the observed binary response r ij = ( 1 if model m j answers q i correctly, 0 otherwise. Question Difficulty Grouping The difficultyp i of a questionq i is its average accuracy across the set of models: p i = 1 |M| |M| X j=1 r ij ∈ [0, 1]. A smallerp i number indicates a harder question. We rank all questions inQbased on theirp i values and partition the sorted list intoGequal-sized quantile groups,Q 1 ,Q 2 , . . . ,Q G , whereQ 1 contains the⌊N/G⌋hardest questions (lowestp i ) andQ G contains the⌈N/G⌉easiest questions (highestp i ). For the analysis, we use G = 3 to create hard, medium, and easy groups. Proposer-Verifier Framework and Baselines To validate the analysis, each difficulty groupQ g is randomly partitioned into a training setQ train g and a held-out test setQ test g with for example an often used 80/20 split. We then generate difficulty descriptions using different "Proposer" methods: 10 MethodISA/IEC 62443 Fortinet NSE Fort. IC- S/SCADA CCNxMMLU Sec Cyber- Metric80 Random Guess33.333.333.333.333.333.3 Baseline44.4±7.239.6±0.6 44.1±14.2 36.5±2.3 28.0±10.4 27.2±0.4 Difficulty45.9±10.3 39.8±0.7 42.0±10.1 43.9±6.8 28.4±4.323.0±4.7 Synthesized 47.7±11.841.9±1.548.9±13.240.2±0.931.5±4.423.9±9.2 Table 3: Mean and standard deviation of F1 score across the different analysis methods (%). Best values are bold. 1.Baseline Proposer (Question-Only): As a baseline, an LLM Φ B generates descriptions D B by analyzing only the question text from the training sets, without being provided the pre-calculated difficulty labels. This tests the ability to infer difficulty from content alone. Φ B (sample(S)| S ∈Q train 1 , . . . ,Q train G )→D B 2.Analysis Proposer (Single-Run): The primary analysis method uses an LLM Φ A that takes samples from the labeled difficulty groups in the training sets to generate a set of natural language descriptions D A . Φ A (sample(S)| S ∈Q train 1 , . . . ,Q train G )→D A This is repeated overkindependent runs to account for model variability, producingksets of descriptions D (1) A , . . . ,D (k) A . 3.Synthesizer Proposer (Multi-Run): To create a more robust set of descriptions, a third LLM mapping Φ S acts as a synthesizer. It takes theksets of descriptions from the Analysis Proposer as input and generates a single, unified set of descriptionsD S . This can also be repeated several times. Φ S (D (1) A , . . . ,D (k) A )→D S For the following experiments, we use k = 3 runs for synthesis. Verifier (Description Validation) The Verifier is an LLM-based classifier Φ V that uses a generated set of descriptionsDto predict the difficulty group of an unseen questionqfrom the test set. Φ V (q,D)→ g ′ ∈1, . . . , G where q ∈ G [ g=1 Q test g The effectiveness of each Proposer method is quantified by the classification performance (F1-score) of the Verifier when using its corresponding descriptions (D B , an averaged overD (i) A , andD S ) on the held-out test sets. For the Proposer, Verifier, and Synthesizer tasks, we use OpenAI’s GPT-5. 5.2 Results and Analysis This section presents the qualitative analysis based on the Proposer-Verifier framework. We first examine how capabilities emerge as model scale increases across different difficulty tiers within the benchmarks. Second, we assess how well the generated difficulty descriptions can be used to classify unseen questions. 11 3B7B 14B30B70B 175B400B 1000B Model Size (B Parameters) 0% 20% 40% 60% 80% 100% Accuracy (%) ISA SA IEC62443 3B7B 14B30B70B 175B400B 1000B Model Size (B Parameters) 0% 20% 40% 60% 80% 100% Accuracy (%) Fortinet NSE full 3B7B 14B30B70B 175B400B 1000B Model Size (B Parameters) 0% 20% 40% 60% 80% 100% Accuracy (%) Fortinet ICS SCADA 3B7B 14B30B70B 175B400B 1000B Model Size (B Parameters) 0% 20% 40% 60% 80% 100% Accuracy (%) MMLU Computer Security 3B7B 14B30B70B 175B400B 1000B Model Size (B Parameters) 0% 20% 40% 60% 80% 100% Accuracy (%) CyberMetric80 3B7B 14B30B70B 175B400B 1000B Model Size (B Parameters) 0% 20% 40% 60% 80% 100% Accuracy (%) Cisco CCNx Difficulty Level Hard Medium Easy Figure 6: Question set accuracy as a function of model size (log scale) across three difficulty tiers (Hard, Medium, Easy) for each benchmark with a regression fitted to show the scaling trend. The top row shows the OT security benchmarks, the bottom row shows the IT-focused benchmarks. Emergent Knowledge Capabilities and Difficulty Scaling To study how cybersecurity knowledge scales with model size, we partitioned the questions in each benchmark into three tiers, hard, medium, and easy, based on average accuracy across models. Figure 6 shows accuracy versus parameter count for these tiers. Performance generally improves with scale, but the scaling behavior and gaps between tiers vary by benchmark. For the three OT-focused benchmarks (top row), the performance gap between easy, medium, and hard questions is wide and persistent. On ISA/IEC 62443 and Fortinet NSE, even the largest models stay below 40% accuracy on the hardest questions. The nearly flat curve on the hardest ISA/IEC 62443 items suggests that increasing model size alone does not add the formal, standard-specific knowledge required; specialized OT knowledge remains a frontier. The general IT security benchmarks (bottom row) show different behavior. MMLU Computer Security and CyberMetric80 exhibit benchmark saturation: accuracy on easy and medium questions clusters near 100%, even for smaller models, so difficulty levels are poorly separated and most items are too simple to differentiate modern LLMs. The Cisco CCNx benchmark lies between these extremes, with clearer separation between all three tiers and consistent gains with model scale, indicating a more informative and learnable domain. Validating the Qualitative Analysis Methodology To evaluate the usefulness of the generated difficulty descriptions, we used them as a rubric for classifying a held-out set of questions. Table 3 reports F1-scores for a Verifier LLM using three description sets: a baseline derived from question text alone, descriptions from single analysis runs, and descriptions synthesized from multiple runs. On average, all methods outperform the random baseline of 33.3%, indicating that the descriptions capture signals of difficulty, but only when the benchmarks have sufficient separation between difficulty 12 tiers (as in Figure 6). For MMLU Computer Security and CyberMetric80, this separation is weak, and clustering by difficulty fails. The baseline “Questions Only” descriptions reach an average F1-score of 36.7%, the single-run analysis 37.2%, and the synthesized descriptions 39.0%. Large standard deviations mean differences are not always statistically clear, but the trend suggests that multi-run synthesis yields the most robust and general descriptions. A notable outlier is CyberMetric80, where all methods achieve F1-scores below the random baseline. This benchmark contains too few genuinely hard questions to support meaningful difficulty tiers. Overall, the validation indicates that the framework produces interpretable descriptions that reflect difficulty patterns, but only for benchmarks with a sufficiently broad and challenging item distribution. Natural Language Descriptions The synthesized difficulty descriptions reveal a progression of required capabilities. Easy questions primarily test factual recall of foundational, single-concept definitions such as standard acronyms or default port numbers. Medium questions emphasize application, requiring models to map principles to scenarios and discriminate between plausible options. The hardest questions demand specialization, including vendor-specific command semantics, formal standards, and operational nuances. These findings show that current LLMs encode a large body of general IT security facts, but their limitations emerge on specialized, context-dependent knowledge that underpins professional practice. Full natural language descriptions for each benchmark are provided in Appendix A. 6 Discussion and Limitations Our findings empirically demonstrate that LLM knowledge in cybersecurity follows a gradient of specificity. Leading models exhibit expert-level knowledge on canonical IT certifications like Cisco CCNx, but their performance degrades on vendor-specific IT benchmarks such as Fortinet NSE and collapses on tasks requiring formal, standards-based OT knowledge like ISA/IEC 62443. This gradient likely reflects the composition of web-scale pre-training corpora, which contain abundant general IT concepts but relatively little proprietary documentation and formal standards [35]. As a result, strong performance on general benchmarks is not a reliable indicator of capability in specialized, context-dependent tasks, especially in critical domains. Beyond the IT/OT knowledge gap, our analysis highlights progress in model efficiency. While performance gains for very large models appear to be saturating, newer models in the 14B–32B parameter range rival the performance of previous-generation models an order of magnitude larger, suggesting ongoing improvements in training methodology and data curation. This increase in parameter efficiency has implications for energy-efficient AI and broader access to capable models. Methodologically, this work introduces a Proposer-Verifier framework that generates and validates interpretable, natural-language descriptions of question difficulty. This provides a way to move beyond aggregate scores and investigate the causes of model successes and failures. A key limitation stems from our data sources for the certification benchmarks. The questions were curated from publicly accessible, community-driven websites. Although these platforms approximate official exam content, their reliance on user-submitted questions and answers introduces a risk of inaccuracies and outdated material. To mitigate this risk, we performed a manual expert review of all questions and answer keys. Items with ambiguous wording, conflicting answers across sources, or insufficiently justified solutions were corrected based on authoritative documentation, or removed. After this expert curation, some residual label noise and selection bias may remain. 13 The Proposer-Verifier framework is also limited by its self-referential nature: we use a frontier LLM to analyze the capabilities of other LLMs. The Proposer may share the same knowledge gaps as the evaluated models, potentially misattributing the source of difficulty. The quality of the analysis is therefore tied to the specific model used for the Proposer task. In addition, the non-deterministic behavior of language models and their sensitivity to prompts can lead to variation between runs, introducing qualitative uncertainty compared to standard quantitative metrics. Future work should address these limitations by improving the robustness and stability of the qualitative analysis, for example through systematic hyperparameter tuning and more constrained prompting. Methodological extensions such as iterative refinement of descriptions or soft prompting could further reduce variance and sensitivity to prompt design. 7 Conclusion The present paper introduces CyberCertBench, a new suite of benchmarks derived from professional certifications, to evaluate LLM knowledge across a gradient of specificity from general IT to specialized OT security. Our central finding is that LLM performance degrades significantly as the required knowledge shifts from web-prevalent concepts to the vendor-specific procedures and formal standards that define professional expertise. While leading models achieve passing grades on general IT certifications, they consistently fail on benchmarks testing deep, specialized knowledge, particularly in vendor-specific procedures and formal OT standards, posing a risk for their use in critical infrastructure environments. Our analysis of scaling and release date trends further reveals that while larger models perform generally better, performance increases are diminishing. Smaller LLMs, on the other hand, have shown a large performance increase in the last year. Therefore, the knowledge capabilities are becoming more parameter-efficient. Methodologically, this work contributes both the publicly available OT benchmarks themselves and a novel Proposer-Verifier framework. The analysis reveals a capability progression, from proficiency in foundational factual recall and applied reasoning to a significant performance drop when faced with the specialized demands of vendor-specific or industrial standards-based knowledge. 14 References [1]Priyan Vaithilingam, Tianyi Zhang, and Elena L. Glassman. Expectation vs. Experience: Evaluating the Usability of Code Generation Tools Powered by Large Language Models. In Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems, CHI EA ’22, pages 1–7. Association for Computing Machinery. ISBN 978-1-4503-9156-6. doi: 10.1145/3491101.3519665. URL https://dl.acm.org/doi/10.1145/3491101.3519665. [2]Shraddha Barke, Michael B. James, and Nadia Polikarpova. Grounded copilot: How programmers interact with code-generating models. Proc. ACM Program. Lang., 7(OOPSLA1):85–111, 2023. doi: 10.1145/3586030. URL https://doi.org/10.1145/3586030. [3]Ranim Khojah, Mazen Mohamad, Philipp Leitner, and Francisco Gomes de Oliveira Neto. Beyond code generation: An observational study of chatgpt usage in software engineering practice. Proc. ACM Softw. Eng., 1(FSE):1819–1840, 2024. doi: 10.1145/3660788. URL https://doi.org/10.1145/3660788. [4] Md Tanvirul Alam, Dipkamal Bhusal, Le Nguyen, and Nidhi Rastogi. CTIBench: A benchmark for evaluating llms in cyber threat intelligence. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors, Ad- vances in Neural Information Processing Systems, volume 37, pages 50805–50825. Cur- ran Associates, Inc. URL https://proceedings.neurips.c/paper_files/paper/2024/file/ 5acd3c628a1819fbf07c39ef73e7285-Paper-Datasets_and_Benchmarks_Track.pdf. [5] Mikel Rodriguez, Raluca Ada Popa, Four Flynn, Lihao Liang, Allan Dafoe, and Anna Wang. A framework for evaluating emerging cyberattack capabilities of AI. arXiv preprint arXiv: 2503.11917, 2025. doi: 10.48550/ARXIV.2503.11917. URL https://doi.org/10.48550/arXiv. 2503.11917. [6]Andy K. Zhang, Neil Perry, Riya Dulepet, Joey Ji, Celeste Menders, Justin W. Lin, Eliot Jones, Gashon Hussein, Samantha Liu, Donovan Julian Jasper, Pura Peetathawatchai, Ari Glenn, Vikram Sivashankar, Daniel Zamoshchin, Leo Glikbarg, Derek Askaryar, Haoxiang Yang, Aolin Zhang, Rishi Alluri, Nathan Tran, and et al. Cybench: A framework for evaluating cybersecurity capabilities and risks of language models. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net, 2025. URL https://openreview.net/forum?id=tc90LV0yRL. [7] Malik Sallam. Chatgpt utility in health care education, research, and practice: Systematic review on the promising perspectives and valid concerns. Healthcare, 11:887, 03 2023. doi: 10.3390/healthcare11060887. [8]Seong Lok Choi, Rishabh Jain, Patrick Emami, Karin Wadsack, Fei Ding, Hongfei Sun, Kenny Gruchalla, Junho Hong, Hongming Zhang, Xiangqi Zhu, et al. egridgpt: Trustworthy ai in the control room. Technical report, National Renewable Energy Laboratory (NREL), Golden, CO (United States), 05 2024. URL https://w.osti.gov/biblio/2352232. [9] Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Yejin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation. ACM Comput. Surv., 55(12):248:1–248:38, 2023. doi: 10.1145/3571730. URL https://doi.org/ 10.1145/3571730. 15 [10]Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL https://openreview.net/forum?id=d7KBjmI3GmQ. [11]Norbert Tihanyi, Mohamed Amine Ferrag, Ridhi Jain, Tamás Bisztray, and Mérouane Debbah. Cybermetric: A benchmark dataset based on retrieval-augmented generation for evaluating llms in cybersecurity knowledge. In IEEE International Conference on Cyber Security and Resilience, CSR 2024, London, UK, September 2-4, 2024, pages 296–302. IEEE, 2024. doi: 10.1109/CSR61664.2024.10679494. URL https://doi.org/10.1109/CSR61664.2024.10679494. [12] Ioannis Zografopoulos, Juan Ospina, Xiaorui Liu, and Charalambos Konstantinou. Cyber- physical energy systems security: Threat modeling, risk assessment, resources, metrics, and case studies. IEEE Access, 9:29775–29818, 2021. doi: 10.1109/ACCESS.2021.3058403. [13] Yijun Xiao and William Yang Wang. On hallucination and predictive uncertainty in conditional language generation. In Paola Merlo, Jorg Tiedemann, and Reut Tsarfaty, editors, Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 2734–2744, Online, April 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.eacl-main.236. URL https://aclanthology.org/2021.eacl-main.236/. [14]Nathaniel Li, Alexander Pan, Anjali Gopal, Summer Yue, Daniel Berrios, Alice Gatti, Justin D. Li, Ann-Kathrin Dombrowski, Shashwat Goel, Gabriel Mukobi, Nathan Helm-Burger, Rassin Lababidi, Lennart Justen, Andrew B. Liu, Michael Chen, Isabelle Barrass, Oliver Zhang, Xiaoyuan Zhu, Rishub Tamirisa, Bhrugu Bharathi, Ariel Herbert-Voss, Cort B. Breuer, Andy Zou, Mantas Mazeika, Zifan Wang, Palash Oswal, Weiran Lin, Adam A. Hunt, Justin Tienken- Harder, Kevin Y. Shih, Kemper Talley, John Guan, Ian Steneker, David Campbell, Brad Jokubaitis, Steven Basart, Stephen Fitz, Ponnurangam Kumaraguru, Kallol Krishna Karmakar, Uday Kiran Tupakula, Vijay Varadharajan, Yan Shoshitaishvili, Jimmy Ba, Kevin M. Esvelt, Alexandr Wang, and Dan Hendrycks. The WMDP benchmark: Measuring and reducing malicious use with unlearning. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. URL https://openreview. net/forum?id=xlr6AUDuJz. [15]Guancheng Li, Yifeng Li, Wang Guannan, Haoyu Yang, and Yang Yu. Seceval: A comprehensive benchmark for evaluating cybersecurity knowledge of foundation models. https://github.com/XuanwuAI/SecEval, 2023. [16] Zefang Liu. Secqa: A concise question-answering dataset for evaluating large language models in computer security. arXiv preprint arXiv: 2312.15838, 2023. doi: 10.48550/ARXIV.2312.15838. URL https://doi.org/10.48550/arXiv.2312.15838. [17]Michael Kouremetis, Marissa Dotter, Alex Byrne, Dan Martin, Ethan Michalak, Gianpaolo Russo, Michael Threet, and Guido Zarrella. OCCULT: evaluating large language models for offensive cyber operation capabilities. arXiv preprint arXiv: 2502.15797, 2025. doi: 10.48550/ ARXIV.2502.15797. URL https://doi.org/10.48550/arXiv.2502.15797. [18]Mary Phuong, Matthew Aitchison, Elliot Catt, Sarah Cogan, Alexandre Kaskasoli, Victoria Krakovna, David Lindner, Matthew Rahtz, Yannis Assael, Sarah Hodkinson, Heidi Howard, Tom Lieberum, Ramana Kumar, Maria Abi Raad, Albert Webson, Lewis Ho, Sharon Lin, Sebastian Farquhar, Marcus Hutter, Grégoire Delétang, Anian Ruoss, Seliem El-Sayed, Sasha Brown, 16 Anca D. Dragan, Rohin Shah, Allan Dafoe, and Toby Shevlane. Evaluating frontier models for dangerous capabilities. arXiv preprint arXiv: 2403.1379, 2024. doi: 10.48550/ARXIV.2403.13793. URL https://doi.org/10.48550/arXiv.2403.13793. [19]Gelei Deng, Yi Liu, Víctor Mayoral Vilches, Peng Liu, Yuekang Li, Yuan Xu, Martin Pinzger, Stefan Rass, Tianwei Zhang, and Yang Liu. Pentestgpt: Evaluating and harnessing large language models for automated penetration testing. In Davide Balzarotti and Wenyuan Xu, editors, 33rd USENIX Security Symposium, USENIX Security 2024, Philadelphia, PA, USA, August 14-16, 2024. USENIX Association, 2024. URL https://w.usenix.org/conference/ usenixsecurity24/presentation/deng. [20]Minghao Shao, Boyuan Chen, Sofija Jancheska, Brendan Dolan-Gavitt, Siddharth Garg, Ramesh Karri, and Muhammad Shafique. An Empirical Evaluation of LLMs for Solving Offensive Security Challenges. arXiv preprint arXiv: 2402.11814, 2024. doi: 10.48550/ARXIV.2402.11814. URL https://arxiv.org/abs/2402.11814. [21]Minghao Shao, Sofija Jancheska, Meet Udeshi, Brendan Dolan-Gavitt, Haoran Xi, Kimberly Milner, Boyuan Chen, Max Yin, Siddharth Garg, Prashanth Krishnamurthy, Farshad Khor- rami, Ramesh Karri, and Muhammad Shafique. NYU CTF bench: A scalable open-source benchmark dataset for evaluating llms in offensive security. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomczak, and Cheng Zhang, editors, Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024, 2024. URL http://papers.nips.c/paper_files/paper/2024/hash/ 69d97a6493fbf016f0a751f253ad18-Abstract-Datasets_and_Benchmarks_Track.html. [22] Jiacen Xu, Jack W. Stokes, Geoff McDonald, Xuesong Bai, David Marshall, Siyue Wang, Adith Swaminathan, and Zhou Li. Autoattacker: A large language model guided system to implement automatic cyber-attacks. arXiv preprint arXiv: 2403.01038, 2024. doi: 10.48550/ARXIV.2403. 01038. URL https://doi.org/10.48550/arXiv.2403.01038. [23]Talor Abramovich, Meet Udeshi, Minghao Shao, Kilian Lieret, Haoran Xi, Kimberly Milner, Sofija Jancheska, John Yang, Carlos E. Jimenez, Farshad Khorrami, Prashanth Krishnamurthy, Brendan Dolan-Gavitt, Muhammad Shafique, Karthik Narasimhan, Ramesh Karri, and Ofir Press. Enigma: Enhanced interactive generative model agent for CTF challenges. arXiv preprint arXiv: 2409.16165, 2024. doi: 10.48550/ARXIV.2409.16165. URL https://doi.org/10.48550/ arXiv.2409.16165. [24] Meet Udeshi, Minghao Shao, Haoran Xi, Nanda Rani, Kimberly Milner, Venkata Sai Charan Putrevu, Brendan Dolan-Gavitt, Sandeep Kumar Shukla, Prashanth Krishnamurthy, Farshad Khorrami, Ramesh Karri, and Muhammad Shafique. D-CIPHER: dynamic collaborative intelligent agents with planning and heterogeneous execution for enhanced reasoning in offensive security. arXiv preprint arXiv: 2502.10931, 2025. doi: 10.48550/ARXIV.2502.10931. URL https://doi.org/10.48550/arXiv.2502.10931. [25]Andrey Anurin, Jonathan Ng, Kibo Schaffer, Jason Schreiber, and Esben Kran. Catastrophic cyber capabilities benchmark (3CB): robustly evaluating LLM agent cyber offense capabilities. arXiv preprint arXiv: 2410.09114, 2024. doi: 10.48550/ARXIV.2410.09114. URL https: //doi.org/10.48550/arXiv.2410.09114. 17 [26]Zefang Liu, Jialei Shi, and John F Buford. Cyberbench: A multi-task benchmark for evaluating large language models in cybersecurity. AAAI-24 Workshop on Artificial Intelligence for Cyber Security (AICS), 2024. [27]Shengye Wan, Cyrus Nikolaidis, Daniel Song, David Molnar, James Crnkovich, Jayson Grace, Manish Bhatt, Sahana Chennabasappa, Spencer Whitman, Stephanie Ding, Vlad Ionescu, Yue Li, and Joshua Saxe. CYBERSECEVAL 3: Advancing the evaluation of cybersecurity risks and capabilities in large language models. arXiv preprint arXiv: 2408.01605, 2024. doi: 10.48550/ARXIV.2408.01605. URL https://doi.org/10.48550/arXiv.2408.01605. [28]Manish Bhatt, Sahana Chennabasappa, Yue Li, Cyrus Nikolaidis, Daniel Song, Shengye Wan, Faizan Ahmad, Cornelius Aschermann, Yaohui Chen, Dhaval Kapil, David Molnar, Spencer Whitman, and Joshua Saxe. Cyberseceval 2: A wide-ranging cybersecurity evaluation suite for large language models. arXiv preprint arXiv: 2404.13161, 2024. doi: 10.48550/ARXIV.2404. 13161. URL https://doi.org/10.48550/arXiv.2404.13161. [29]Wesley Joon-Wie Tann, Yuancheng Liu, Jun Heng Sim, Choon Meng Seah, and Ee-Chien Chang. Using large language models for cybersecurity capture-the-flag challenges and certification questions. arXiv preprint arXiv: 2308.10443, 2023. doi: 10.48550/ARXIV.2308.10443. URL https://doi.org/10.48550/arXiv.2308.10443. [30]Gustav Keppler, Jeremy Kunz, Veit Hagenmeyer, and Ghada Elbez. Evaluating Large Language Models in Cybersecurity Knowledge with Cisco Certificates. In Secure IT Systems: 29th Nordic Conference, NordSec 2024 Karlstad, Sweden, November 6–7, 2024 Proceedings, pages 219–238. Springer-Verlag. ISBN 978-3-031-79006-5. doi: 10.1007/978-3-031-79007-2_12. URL https://doi.org/10.1007/978-3-031-79007-2_12. [31]Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove, Christopher D. Manning, Christopher Ré, Diana Acosta-Navas, Drew A. Hudson, Eric Zelikman, Esin Durmus, Faisal Ladhak, Frieda Rong, Hongyu Ren, Huaxiu Yao, Jue Wang, Keshav Santhanam, Laurel J. Orr, Lucia Zheng, Mert Yüksekgönül, Mirac Suzgun, Nathan Kim, Neel Guha, Niladri S. Chatterji, Omar Khattab, Peter Henderson, Qian Huang, Ryan Chi, Sang Michael Xie, Shibani Santurkar, Surya Ganguli, Tatsunori Hashimoto, Thomas Icard, Tianyi Zhang, Vishrav Chaudhary, William Wang, Xuechen Li, Yifan Mai, Yuhui Zhang, and Yuta Koreeda. Holistic evaluation of language models. Trans. Mach. Learn. Res., 2023, 2023. URL https://openreview.net/forum?id=iO4LZibEqW. [32] Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomczak, and Cheng Zhang, editors, Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024, 2024. URL http://papers.nips.c/paper_files/paper/2024/hash/ ad236edc564f3e3156e1b2feafb99a24-Abstract-Datasets_and_Benchmarks_Track.html. [33] Ruiqi Zhong, Charlie Snell, Dan Klein, and Jacob Steinhardt. Describing Differences between Text Distributions with Natural Language. In Proceedings of the 39th International Conference 18 on Machine Learning, pages 27099–27116. PMLR. URL https://proceedings.mlr.press/v162/ zhong22a.html. [34]Tung-Yu Wu and Melody Lo. U-shaped and inverted-u scaling behind emergent abilities of large language models. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net, 2025. URL https://openreview.net/ forum?id=jjfve2gIXe. [35]Arno Kok, Alberto Martinetti, and Jan Braaksma. The impact of integrating information technology with operational technology in physical assets: A literature review. IEEE Access, 12:111832–111845, 2024. doi: 10.1109/ACCESS.2024.3442443. 19 Appendices A Synthesized Descriptions of Question Difficulty The following tables provides the short and complete, synthesized natural language descriptions of the characteristics of easy, medium, and hard questions for each benchmark, as generated by the qualitative analysis framework. Difficulty Driver Synthesized DescriptionExample Benchmark Factual Recall (Low Difficulty) Tests recognition of single-concept definitions, standard terms, and widely known facts. Success relies on direct knowledge retrieval with minimal reasoning. MMLU Security: Identifying a web proxy’s function. CyberMetric80: Naming the purpose of a WAF. Applied Rea- soning (Medium Diffi- culty) Requires mapping concepts to scenarios, often involving light troubleshooting, multi-select formats, or discriminating between plausible options. CCNx: Interpreting interface counters or compressing an IPv6 address. Vendor-Specific Semantics (High Difficulty) Demands precise knowledge of a specific vendor’s product line, CLI syntax, proprietary features, and nuanced operational behaviors. Fortinet NSE: Knowing SD-WAN policy route precedence rules. Standard- Specific Formal- ism (High Difficulty) Requires familiarity with the specific terminology, structure, and process phases defined within a formal standard like ISA/IEC 62443. ISA/IEC 62443: Identifying which activities belong to the "assess" phase of the IACS security lifecycle. Table 4: Primary Drivers of Question Difficulty Based on Qualitative Analysis. Table 5: Synthesized Difficulty Characteristics for Each Benchmark. BenchmarkSynthesized Difficulty Characteristics Cisco CCNxEasy: Short, definition-focused items with one clear answer covering fun- damentals like RFC1918, VLAN broadcast domains, A/firewalls, and high-level TCP/UDP; minimal vendor syntax or calculations. Medium: Conceptual application with light troubleshooting: interpret simple counters, pick non-overlapping channels, compress IPv6, understand flooding scope, and basic CDP/DTP/PoE nuances. Hard: Multi-constraint, multi-step synthesis requiring exact Cisco defaults and CLI semantics: STP/RSTP states, PortFast effects, DTP/LACP mode interplay, constrained VLSM, and detailed counter-based troubleshooting. 20 Table 5 – continued from previous page BenchmarkSynthesized Difficulty Characteristics MMLU Secu- rity Easy: Short, single-concept, fact-based items on definitions, tools (e.g., Nmap, Tor), CIA triad, OSI/wireless basics, and buffer overflow fundamen- tals; minimal reasoning with clear distractors. Medium: Scenario-oriented questions requiring light application and dis- crimination: forward secrecy, PKI/HTTPS enhancements (DV/EV, stapling, pinning), IPSec vs TLS, and fuzzing styles. Hard: Multi-step, cross-domain reasoning with formal crypto and sys- tem/protocol nuances: TLS/Kerberos semantics, Merkle–Damgård length extension, PRF/CPA subtleties, and system architecture specifics (e.g., NaCl validator rules, OKWS FD passing). Fortinet NSE Easy: Single-concept, factual recall or basic diagnostics with clear cues, minimal ambiguity, and little to no cross-feature dependency; core network- ing/security fundamentals and straightforward Fortinet basics. Medium: Two to three related concepts with moderate scenarios and multi-selects; mapping definitions to product behavior and workflows across SD-WAN, AV modes, FMG ADOMs, SSL VPN, and FSSO/LDAP. Hard: Scenario-heavy, multi-select items requiring precise vendor defaults, precedence rules, and internals across multiple Fortinet products and cloud; integrated troubleshooting (FAZ indexing, FMG HA, SIP ALG vs helper, ZTNA). Fortinet ICS/S- CADA Easy: Single-fact, foundational recall with clear, unambiguous stems and canonical answers (CIA, TCP flags, MTU, basic tool facts) and simple distractors. Medium: Applied recall and conceptual mapping (IPsec modes, default- deny, ICMP, Modbus, tool roles) with moderate ambiguity (negative stems, plausible distractors) but grounded in standard material. Hard: Specialized ICS/SCADA standards and vendor/OS minutiae (IEC 62443, CVSS taxonomy, AH/ESP fields, Windows/nmap defaults) with higher ambiguity (close counts, synonyms, unit mismatches) requiring syn- thesis and elimination. CyberMetric80 Easy: Direct recall of basic definitions and purposes with clear single-correct answers and low-plausibility distractors; core networking/security and gov- ernance terms (e.g., ARP, TLS, MFA, WAF). Medium: Conceptual understanding and light application; select the pri- mary purpose/control/role among plausible options across PKI, networking ports/handshakes, governance/risk/BCP/PCI, and common attacks. Hard: Multi-step application with precise syntax and evolving-standards nuance; includes CIDR/IPv6/XOR, exact tool flags (nmap -sS), specialized crypto/RNG (SRTP AES-GCM, DRBG state protection, DLIES), and policy subtleties. 21 Table 5 – continued from previous page BenchmarkSynthesized Difficulty Characteristics ISA/IEC 62443Easy: Foundational networking and security/ICS facts (e.g., OSI, default deny, phishing, management risk role) with clear answers and obvious distractors; primarily factual recall. Medium: Recall plus light application in ICS contexts (62443 family, TRs like patching, IDS limits, crypto basics, patching priority). Includes best- practice judgments. Hard: Precise, standards-heavy ISA/IEC 62443 topics (zones/conduits, lifecycle assess phase, access control variables), sector-specific standards (API 1164), ISASecure governance, and multi-select with qualifiers and close distractors. B Prompts for Qualitative Difficulty Analysis This appendix contains the full prompts used in the Proposer-Verifier framework for the qualitative analysis of question difficulty. Prompt for Baseline You are an expert in cybersecurity education and assessment . You have been given a large set of multiple choice questions from the exam_name dataset . Your task i s to analyze these questions and create d i f f i c u l t y l e v e l d e s c r i p t i o n s that could be used to c l a s s i f y questions into num_groups d i f f i c u l t y l e v e l s . TRAINING QUESTIONS: all_questions Based on your a n a l y s i s of these questions , create d e s c r i p t i o n s f o r num_groups d i f f i c u l t y l e v e l s that capture the key c h a r a c t e r i s t i c s that would d i s t i n g u i s h questions of d i f f e r e n t complexity . Consider f a c t o r s l i k e : 1. ∗Knowledge Types ∗: What types of knowledge do d i f f e r e n t questions t e s t ? 2. ∗ Complexity Factors ∗: What makes some questions more challenging than others ? 3. ∗Content Areas ∗: Are there topic patterns that c o r r e l a t e with d i f f i c u l t y ? 4. ∗ Question Structure ∗: How do questions d i f f e r in construction and presentation ? At the end , provide a JSON summary with the following structure : 22 " baseline_analysis ": " easiest_group_characteristics ": " Clear d e s c r i p t i o n of what would define the e a s i e s t questions " , " medium_group_characteristics ": " Clear d e s c r i p t i o n of what would define medium d i f f i c u l t y questions " , " hardest_group_characteristics ": " Clear d e s c r i p t i o n of what would define the hardest questions " , " d i f f i c u l t y _ p r o g r e s s i o n ": "How questions would change from easy to medium to hard " , " key_differences ": [" d i f f e r e n c e 1" , " d i f f e r e n c e 2" , " d i f f e r e n c e 3"] , "methodology ": " Brief explanation of how you determined these d i f f i c u l t y l e v e l s " Prompt for Single-Run Difficulty Analysis (Proposer) You are an expert in cybersecurity education and assessment . Analyze the following s e t s of multiple choice questions from the exam_name dataset , grouped by d i f f i c u l t y l e v e l : DIFFICULTY GROUP OVERVIEW: groups_summary HARDEST GROUP ( hardest_group ) : analysis_ results [ hardest_group ] [ ’ formatted_questions ’ ] ############## MEDIUM GROUP (medium_group) : analysis_ results [ medium_group ] [ ’ formatted_questions ’ ] ############## EASIEST GROUP ( easiest_group ) : analysis_ results [ easiest_group ] [ ’ formatted_questions ’ ] Please analyze the key d i f f e r e n c e s across a l l three d i f f i c u l t y groups and provide i n s i g h t s into : 1. ∗Knowledge Types ∗: What types of knowledge do the question groups t e s t ? 2. ∗ Complexity Factors ∗: What makes questions challenging across d i f f i c u l t y l e v e l s ? 3. ∗Content Areas ∗: Are there s p e c i f i c cybersecurity domains or topics that appear more in one group? 23 4. ∗ Question Structure ∗: How do the questions d i f f e r in t h e i r construction ? 5. ∗ D i f f i c u l t y Progression ∗: What patterns emerge as questions progress from easy to hard? At the end , provide a JSON summary with the following structure : "analysis_summary ": " easiest_group_characteristics ": " . . . " , " medium_group_characteristics ": " . . . " , " hardest_group_characteristics ": " . . . " , " d i f f i c u l t y _ p r o g r e s s i o n ": " . . . " , " key_differences ": [ " . . . " , " . . . " , " . . . " ] , "knowledge_gaps ": " . . . " , "recommendations ": " . . . " Prompt for Multi-Run Synthesis (Synthesizer) You are tasked with combining multiple responses into a single , cohesive response . Below , I w i l l provide s e v e r a l responses from d i f f e r e n t a n a l y s i s runs . Your goal i s to i d e n t i f y common themes , r e c o n c i l e d i f f e r e n c e s , and combine the information into a u n i f i e d response . Be sure to preserve a l l key i n s i g h t s from each trace and ensure the f i n a l output i s l o g i c a l l y c on si st e nt and comprehensive . r o l l o u t s Output Format : Combine a l l the provided responses into a new , comprehensive , complete , and u n i f i e d response , prefixed by "# UNIFIED RESPONSE". Your response should not be much longer than the o r i g i n a l responses . At the end , provide a JSON summary with the following structure : "analysis_summary ": 24 " easiest_group_characteristics ": " Clear u n i f i e d d e s c r i p t i o n of what d e f i n e s the e a s i e s t questions " , " medium_group_characteristics ": " Clear u n i f i e d d e s c r i p t i o n of what d e f i n e s the medium d i f f i c u l t y questions " , " hardest_group_characteristics ": " Clear u n i f i e d d e s c r i p t i o n of what d e f i n e s the hardest questions " , " d i f f i c u l t y _ p r o g r e s s i o n ": "How questions change from easy to medium to hard " , " key_differences ": [" d i f f e r e n c e 1" , " d i f f e r e n c e 2" , " d i f f e r e n c e 3"] , "knowledge_gaps ": "Main knowledge gaps revealed across d i f f i c u l t y l e v e l s " , "recommendations ": " Suggestions f o r improving LLM performance across d i f f i c u l t y l e v e l s " Prompt for Difficulty Classification (Verifier) You are an expert in question c l a s s i f i c a t i o n . Given the following three d e s c r i p t i o n s and a question , predict which c l a s s the question belongs to . CLASS DESCRIPTIONS: CLASS 1: easiest_description CLASS 2: medium_description CLASS 3: hardest_description QUESTION TO CLASSIFY: question_text Based on the c h a r a c t e r i s t i c s described above , which c l a s s does t h i s question belong to ? After c a r e f u l consideration , respond with only one of : "CLASS 1" , " CLASS 2" , or "CLASS 3" 25 C Full Benchmark Accuracy Table 6: Mean accuracy (%) for every benchmark and model in alphabetical order. CyMetric 80 MMLU CompSec. Cisco CCNx Fort. NSE Fort. ICS/SCADA ISA/IEC 62443 Claude 3.5 Haiku96.285.076.5 47.581.466.0 Claude 3.5 Sonnet95.087.386.5 59.683.781.1 Claude 4.5 Haiku97.583.082.0 56.083.752.8 Claude 4.5 Sonnet97.591.089.5 62.476.773.6 DeepSeek v3 032495.084.082.0 62.479.177.4 Gemini 2.5 Flash Lite95.078.074.0 52.581.464.2 Gemini 2.5 Pro 050697.589.091.5 61.786.073.6 Gemini Flash 2.097.582.080.8 56.779.174.3 Gemma 2 27B93.879.066.0 50.476.743.4 Gemma 2 9B91.274.054.5 42.674.435.8 Gemma 3 12B92.583.065.0 42.669.843.4 Gemma 3 27B96.281.068.0 45.467.437.7 Gemma 3 4B85.068.051.0 36.953.539.6 Llama2 70B86.273.051.5 44.065.154.7 Llama3.1 405B100.084.080.6 53.980.963.0 Llama3.1 70B92.582.677.4 51.880.064.2 Llama3.1 8B92.581.054.5 41.865.154.7 Llama3.2 3B83.868.053.0 41.874.437.7 Llama4 Maverick93.889.076.5 56.079.166.0 Llama4 Scout87.572.076.0 50.474.462.3 Ministral 3B86.273.057.0 47.560.552.8 Ministral 8B86.276.056.0 48.260.545.3 Mistral 7B v0.388.872.052.0 45.462.860.4 Mistral Large 241193.883.077.0 51.886.056.6 Mistral Medium 3.197.582.084.5 55.381.467.9 Mistral Small 3.1 24B93.882.079.5 53.288.466.0 Mixtral 8x22B95.080.071.0 55.381.466.0 Mixtral 8x7B92.577.061.5 46.160.562.3 Moonshot AI Kimi K297.587.086.5 61.083.777.4 OpenAI GPT 3.5 turbo95.080.067.0 51.881.466.0 OpenAI GPT 4.196.286.084.0 58.286.069.8 OpenAI GPT 4o96.285.085.8 58.286.568.3 OpenAI GPT 597.588.085.5 53.288.469.8 OpenAI GPT 5 Mini98.881.081.0 55.383.752.8 OpenAI GPT 5 Nano92.576.064.0 40.474.432.1 OpenAI GPT OSS 120B98.891.079.0 52.588.462.3 OpenAI GPT OSS 20B97.585.073.0 53.974.460.4 Qwen2 72B98.884.477.0 52.586.075.5 Qwen2.5 72B97.586.678.9 53.987.459.2 Qwen2.5 7B97.577.468.1 48.275.353.6 26 CyMetric 80 MMLU CompSec. Cisco CCNx Fort. NSE Fort. ICS/SCADA ISA/IEC 62443 Qwen3 14B93.887.074.0 53.080.671.1 Qwen3 235B A22B91.289.080.5 51.879.173.6 Qwen3 30B A3B97.583.067.0 51.181.466.0 Qwen3 32B95.083.079.0 49.681.471.7 Qwen3 8B95.079.067.0 50.467.460.4 27