Paper deep dive
Cyber Defense Benchmark: Agentic Threat Hunting Evaluation for LLMs in SecOps
Alankrit Chona, Igor Kozlov, Ambuj Kumar
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 97%
Last extracted: 4/26/2026, 11:29:39 PM
Summary
The Cyber Defense Benchmark is a new evaluation framework designed to measure the effectiveness of LLM agents in performing autonomous threat hunting within a Security Operations Center (SOC) context. Built upon the OTRF Security-Datasets (Mordor), the benchmark uses a Gymnasium-based environment where agents must use SQL queries to identify malicious event timestamps within large Windows event log databases. The study evaluates five frontier models (Claude Opus 4.6, GPT-5, Gemini 3.1 Pro, Kimi K2.5, and Gemini 3 Flash) and finds that all current models fail to meet the operational threshold of 50% recall on MITRE ATT&CK tactics, with even the top performer (Claude Opus 4.6) averaging only 3.8% recall. The benchmark emphasizes the difficulty of open-ended, evidence-driven investigation compared to standard Q&A security tasks.
Entities (11)
Relation Signals (6)
Cyber Defense Benchmark → basedon → MITRE ATT&CK
confidence 100% · spanning 86 MITRE ATT&CK sub-techniques across 12 tactics
Simbian AI → developed → Cyber Defense Benchmark
confidence 100% · Simbian AI · Technical Report v1.0 · April 2026
Cyber Defense Benchmark → evaluates → Claude Opus 4.6
confidence 100% · Evaluating five frontier models - Claude Opus 4.6...
Cyber Defense Benchmark → implements → Gymnasium
confidence 100% · into a Gymnasium reinforcement-learning environment
Cyber Defense Benchmark → uses → OTRF Security-Datasets
confidence 100% · The benchmark wraps 106 real attack procedures from the OTRF Security-Datasets corpus
Cyber Defense Benchmark → usesdataformat → SQLite
confidence 100% · presents the agent with an in-memory SQLite database
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:We introduce the Cyber Defense Benchmark, a benchmark for measuring how well large language model (LLM) agents perform the core SOC analyst task of threat hunting: given a database of raw Windows event logs with no guided questions or hints, identify the exact timestamps of malicious events. The benchmark wraps 106 real attack procedures from the OTRF Security-Datasets corpus - spanning 86 MITRE ATT&CK sub-techniques across 12 tactics - into a Gymnasium reinforcement-learning environment. Each episode presents the agent with an in-memory SQLite database of 75,000-135,000 log records produced by a deterministic campaign simulator that time-shifts and entity-obfuscates the raw recordings. The agent must iteratively submit SQL queries to discover malicious event timestamps and explicitly flag them, scored CTF-style against Sigma-rule-derived ground truth. Evaluating five frontier models - Claude Opus 4.6, GPT-5, Gemini 3.1 Pro, Kimi K2.5, and Gemini 3 Flash - on 26 campaigns covering 105 of 106 procedures, we find that all models fail dramatically: the best model (Claude Opus 4.6) submits correct flags for only 3.8% of malicious events on average, and no run across any model ever finds all flags. We define a passing score as >= 50% recall on every ATT&CK tactic - the minimum bar for unsupervised SOC deployment. No model passes: the leader clears this bar on 5 of 13 tactics and the remaining four on zero. These results suggest that current LLMs are poorly suited for open-ended, evidence-driven threat hunting despite strong performance on curated Q&A security benchmarks.
Tags
Links
- Source: https://arxiv.org/abs/2604.19533v1
- Canonical: https://arxiv.org/abs/2604.19533v1
Trouble viewing inline? Open PDF directly →
Full Text
33,401 characters extracted from source content.
Expand or collapse full text
Simbian AI · Technical Report v1.0 · April 2026 · research@simbian.ai Cyber Defense Benchmark: Agentic Threat Hunting Evaluation for LLMs in SecOps Alankrit Chona Igor Kozlov Ambuj Kumar Simbian AI research@simbian.ai Abstract. We introduce the Cyber Defense Benchmark, a benchmark for measuring how well large language model (LLM) agents perform the core SOC analyst task of threat hunting: given a database of raw Windows event logs with no guided questions or hints, identify the exact timestamps of malicious events. The benchmark wraps 106 real attack procedures from the OTRF Security-Datasets corpus - spanning 86 MITRE ATT&CK sub-techniques across 12 tactics - into a Gymnasium reinforcement-learning environment. Each episode presents the agent with an in-memory SQLite database of 75,000-135,000 log records produced by a deterministic campaign simulator that time-shifts and entity-obfuscates the raw recordings. The agent must iteratively submit SQL queries to discover malicious event timestamps and explicitly flag them, scored CTF-style against Sigma-rule-derived ground truth. Evaluating five frontier models - Claude Opus 4.6, GPT-5, Gemini 3.1 Pro, Kimi K2.5, and Gemini 3 Flash - on 26 campaigns covering 105 of 106 procedures, we find that all models fail dramatically: the best model (Claude Opus 4.6) submits correct flags for only 3.8% of malicious events on average, and no run across any model ever finds all flags. We define a passing score as ≥50% recall on every ATT&CK tactic - the minimum bar for unsupervised SOC deployment. No model passes: the leader clears this bar on 5 of 13 tactics and the remaining four on zero. These results suggest that current LLMs are poorly suited for open-ended, evidence-driven threat hunting despite strong performance on curated Q&A security benchmarks. Cyber Defense Benchmark: Agentic Threat Hunting Evaluation for LLMs in SecOpsPage 2 Simbian AI · Technical Report v1.0 · April 2026 · research@simbian.ai 1. INTRODUCTION Security Operations Center (SOC) analysts face a fundamental challenge: adversaries leave traces scattered across millions of log events, using techniques that evade static rules, and analysts must reason across large, heterogeneous telemetry to reconstruct attack chains. This task - threat hunting - demands hypothesis generation, iterative querying, evidence synthesis, and precise attribution of malicious activity. Large language models have demonstrated strong performance on security knowledge tests and natural-language question answering over pre-segmented log snippets, but their ability to perform open-ended, agentic threat hunting against raw telemetry has not been rigorously measured. Existing security benchmarks fall into one of two categories. Knowledge benchmarks (CTI-Bench [4], SecBench [2], CyberMetric [3]) test factual recall and reasoning over text. Guided log-analysis benchmarks (e.g., Microsoft Research's 2025 benchmark [15] with 44 days of synthetic data and pre-generated Q&A) decompose the analyst task into answerable sub-questions, removing the hypothesis-formation step that defines real threat hunting. Neither captures the difficulty of open-ended investigation. We present the Cyber Defense Benchmark, which differs from prior work on three axes: 1. No guided questions. The agent receives only a terse threat-intel briefing and raw logs; all hypotheses must originate from the agent itself. 2. Real attack telemetry. Logs come from actual adversary-technique executions recorded on Windows hosts, not synthetic generation. 3. Extensive MITRE coverage. 106 procedures spanning 86 ATT&CK sub-techniques across 12 tactics, composed into multi-stage campaigns via a simulator that enforces kill-chain dependencies and entity obfuscation. Our headline finding is stark: even the strongest model (Claude Opus 4.6) finds fewer than 4% of malicious events on average, and all models terminate by giving up rather than completing the hunt. Against a minimal operational bar - ≥50% recall on every ATT&CK tactic - no model passes: the leader clears it on 5 of 13 tactics, the other four on zero. The Cyber Defense Benchmark establishes a rigorous baseline for measuring progress in LLM-based cyber defense. 2. RELATED WORK 2.1 LLM Security Benchmarks A growing body of work evaluates LLMs on cybersecurity tasks. SecBench [2] and CyberMetric [3] assess factual and procedural security knowledge via multiple-choice questions. CTI-Bench [4] tests cyber-threat-intelligence reasoning. CAI-Bench [13] formulates security as jeopardy-style Q&A. These benchmarks measure knowledge but not operational skill in an evidence-gathering environment. CyberAIBench introduces CTF challenges closer in spirit to our work but focused on exploitation rather than defense. 2.2 Log Analysis and Threat Detection Prior work asks LLMs to classify single log entries or answer questions over pre-selected log windows. The Microsoft Research Security Copilot evaluation [15] uses 44 days of synthetic organizational logs with 8 pre-defined attack scenarios and evaluates structured question-answering, decomposing the analyst task and removing the hypothesis-formation step. Our benchmark presents agents with the full, un-segmented log database and no attack-scenario framing. Cyber Defense Benchmark: Agentic Threat Hunting Evaluation for LLMs in SecOpsPage 3 Simbian AI · Technical Report v1.0 · April 2026 · research@simbian.ai 2.3 Agentic Security Evaluation InterCode-CTF [6] and NYU CTF Bench [7] evaluate agents on offensive CTF challenges. PentestBench [8] assesses end-to-end penetration testing against vulnerable machines. The Cyber Defense Benchmark is the first benchmark targeting the defensive counterpart: agentic threat hunting from defender telemetry, requiring active evidence discovery rather than pre-presented attack artifacts. 2.4 Dataset Provenance The Cyber Defense Benchmark builds on OTRF Security-Datasets (Mordor) [1], a public corpus of recorded Windows event logs for 106 attacker procedures executed in controlled lab environments. Previous work uses Mordor logs for Sigma-rule development; we are the first to convert them into a scored, episodic benchmark with deterministic campaign simulation. 3. TASK FORMULATION 3.1 Problem Statement A threat hunt episode is a partially-observable sequential decision problem. The state s is the full set of log records in the database, including which records correspond to malicious events. The agent's belief state is its conversation history. At each turn, the agent may (a) execute a SQL query and observe up to 10 result rows, or (b) submit a list of timestamps it believes are malicious. The episode terminates when the agent submits all correct timestamps, exhausts its 50-query budget, or explicitly gives up. The task is harder than it appears: a typical campaign database contains 75,000-135,000 log records, of which 1,500-6,700 are malicious (roughly 1-5%). Each query reveals at most 10 rows. Finding all malicious events requires covering a large search space with a 50-query budget - an information-retrieval problem that cannot be solved by breadth-first scanning alone. 3.2 Scoring Ground truth is a set F of malicious event timestamps, derived from Sigma-rule detections and LLM-enriched consequence events (Section 6). Timestamps are matched at microsecond precision after normalizing UTC suffixes. The CTF-style reward at turn t is: r t = 10 × |F new,t | + 50 × n[|F cumul,t | = |F|] where F new,t is the set of flags first discovered at turn t. The all-flags bonus was never triggered in any evaluated run. We report two primary metrics: n_flags_in_submitted (correctly submitted timestamps, the primary metric) and n_flags_in_query (flags that appeared in any query result, an upper bound on agent observability). 4. ENVIRONMENT DESIGN 4.1 Gymnasium Interface HolodeckHuntEnv implements the Gymnasium Env API [12]. The action space is spaces.Text(max_length=2000) - a free-form SQL string. The observation space is spaces.Text(max_length=50000) - a structured string containing the current briefing, most recent query, and query result or error message. Episode limits: max_queries=50 (truncation), with a 1.5× safety cap at 75 agent turns. The environment is intentionally minimal: no tool-use API, no vector store, no retrieval augmentation - the only mechanism for information gathering is SQL. Cyber Defense Benchmark: Agentic Threat Hunting Evaluation for LLMs in SecOpsPage 4 Simbian AI · Technical Report v1.0 · April 2026 · research@simbian.ai 4.2 SQLite Log Database Logs are stored in an in-memory SQLite database with a single flat table logs. The schema consists of 505 text columns corresponding to all Windows and Sysmon EventData fields observed across the 106-procedure corpus, plus a raw_json column containing the full original event. Four indexes are built on hot fields: EventID, Computer, Hostname, and User. Query results are truncated to 10 rows; the full row count is always reported so the agent can plan paginated queries. The flat-schema design forces agents to reason about which fields carry signal for which event types - a skill mirroring real-world SIEM query authoring. 4.3 Noise Injection When include_noise=True (default), 500 synthetic benign Sysmon EventID-1 records are shuffled into the attack logs, simulating background process-creation activity across 8 common binaries (explorer.exe, chrome.exe, svchost.exe, etc.), 3 users, and 50 workstations. No flag is associated with noise records, preventing agents from treating any process-creation event as automatically suspicious. 4.4 Agent Interface The UniversalHunter agent receives a system prompt containing: (i) a mission statement, (i) the full 505-column schema, (i) query pagination instructions, and (iv) the three available actions. At each turn the agent must return a structured HunterAction enforced via provider-native JSON-schema constrained decoding: HunterAction reasoning: str # required, non-empty internal monologue tool: Enum # run_sql | submit_flags | give_up sql_query: str | None # required iff tool == run_sql submitted_timestamps: list | None # required iff tool == submit_flags Three tools are available: run_sql (execute SQL, costs one query turn); submit_flags (submit candidate timestamps, free - does not cost a turn); and give_up (terminate early). Models are accessed via LiteLLM [14], supporting Anthropic, OpenAI, Google Vertex AI (with high-thinking mode for Gemini), AWS Bedrock, Fireworks AI, and custom OpenAI-compatible endpoints. 5. DATASET 5.1 Source: OTRF Security-Datasets The benchmark uses 106 procedure recordings from the Open Threat Research Federation (OTRF) Security-Datasets project [1], a public corpus of Windows event logs generated by executing specific attacker techniques in controlled lab environments. Each procedure was executed by a human adversary-simulation operator and captured via Windows Event Forwarding (WEF), Sysmon, or HELK stack collection pipelines. Procedures span credential dumping, lateral movement, privilege escalation, persistence, defense evasion, and other behaviors defined in MITRE ATT&CK. 5.2 Normalization Raw recordings span 9 infrastructure environments and 5 log formats: HELK/Elasticsearch JSON, WEF export, raw_event, Winlogbeat, and ADFS-schema. A custom normalizer enforces a common schema: every record has a TimeCreated field in ISO-8601 format; @timestamp fields are dropped; field aliases (e.g., IpAddress/Ipaddress, ProcessID/ProcessId) are resolved. All 774,218 raw records across 106 procedures parse with zero errors. Cyber Defense Benchmark: Agentic Threat Hunting Evaluation for LLMs in SecOpsPage 5 Simbian AI · Technical Report v1.0 · April 2026 · research@simbian.ai StatisticValue Total procedures106 Total raw log records774,218 Average records per procedure7,304 Record range (min-max)68-9,896 Unique (Channel, EventID) pairs303 Unique schema field keys507 MITRE ATT&CK sub-techniques86 MITRE tactics covered12 Parent-technique coverage37% (of 231) Total malicious event flags23,268 Average flags per procedure219.5 Flag range (min-max per procedure)11-3,068 Table 1: Core dataset statistics. 5.3 MITRE ATT&CK Coverage The 106 procedures cover 86 unique MITRE ATT&CK sub-techniques across 12 tactics of the ATT&CK Enterprise matrix: Initial Access, Execution, Persistence, Privilege Escalation, Defense Evasion, Credential Access, Discovery, Lateral Movement, Collection, Command & Control, Exfiltration, and Impact. This corresponds to 37% coverage of all parent techniques in the covered tactics (86 of 231). The tactic-level analysis in Section 9 adds Resource Development as a thirteenth category for completeness. The benchmark's greedy seed-selection algorithm (Section 7.3) ensures that 26 campaign instances cover 105 of 106 procedures - one procedure is unreachable under the default blueprint's action-tag constraints. 6. FLAG EXTRACTION PIPELINE 6.1 Sigma-Rule Detection For each procedure's normalized logs, we run WithSecureLabs Chainsaw [10] against the SigmaHQ rule repository [9]. Chainsaw produces matches at the (EventID, Channel, record-index) level. Each Chainsaw hit yields one or more sigma flags tagged with the matching rule's identifier and severity (critical, high, medium, low). Rules required patching from yaml.safe_load to yaml.safe_load_all because SigmaHQ uses multi-document YAML with action: global. 6.2 LLM Consequence Enrichment Sigma rules target specific event patterns and miss the causal neighborhood of malicious activity. We use an LLM enrichment step that, for each Sigma-matched event, identifies consequence events: log records immediately preceding or following the detection and causally attributable to the attacker action (e.g., DLL loads into a WMIC process, child process spawns from cmd.exe). These are tagged consequent_sigma or consequent_procedure. An additional enrichment step assigns severity from Sigma YAML metadata, relevance scores (1 = core to the attack, 2 = supporting, 3 = incidental), and narrative step mappings. 6.3 Flag Taxonomy Cyber Defense Benchmark: Agentic Threat Hunting Evaluation for LLMs in SecOpsPage 6 Simbian AI · Technical Report v1.0 · April 2026 · research@simbian.ai The benchmark defines 29 FlagType values across five groups: attacker infrastructure (IP, domain, URL, port), victim identity (host, user, SID, etc.), other entities (host, user, process, DLL, service), malicious artifacts (process path, command, hash, registry key, mutex, etc.), and malicious event timestamps. For the threat-hunting task, all ground-truth flags are of type malicious_event_timestamp, matched at microsecond precision after normalizing UTC suffixes. Across 26 benchmark campaigns, flag counts range from 1,500 to 6,700 per campaign (mean ≈ 3,150). Flag provenance tagDescriptionTotal (26 campaigns) sigma_codeEvent-code level Sigma rule match38,572 consequent_sigmaLLM-identified causal neighbor of Sigma event27,075 sigmaRule-level Sigma match8,691 consequent_procedureProcedure-derived consequence event8,311 procedureOriginal procedure marker1,156 Total83,805 Table 2: Flag provenance tag distribution (a single flag may carry multiple tags). 7. CAMPAIGN SIMULATION 7.1 Motivation Raw Mordor recordings are unsuitable for a generalizable benchmark: they share identical IP addresses, hostnames, and user accounts across procedures, so a model could learn to recognize specific strings rather than detecting genuinely malicious patterns. Our campaign simulator addresses this by composing multiple procedures into a multi-stage kill chain and applying deterministic entity obfuscation. 7.2 Kill-Chain Templates Procedures are tagged with 16 semantic action labels: DELIVERY, EXECUTION, DISCOVERY, CREDENTIAL_HARVESTING, PRIVILEGE_ESCALATION, LATERAL_MOVEMENT, PERSISTENCE, C2, QUIET, and others. Three chain templates define kill-chain structures: • SHORT (4 steps): Delivery → Execution → Quiet → Persistence. • MEDIUM (6 steps): Delivery → Execution → Discovery → Credential Harvesting → Weakening → Quiet. • LONG (9 steps): Delivery → Execution → Weakening → Discovery → Privilege Escalation → Credential Harvesting → Quiet → Lateral Movement → Persistence. The default DIVERSE_INTRUSION blueprint runs one SHORT + one MEDIUM + one LONG chain per campaign (19 total steps), simulating a realistic multi-stage intrusion spanning multiple hosts and credential contexts. 7.3 Procedure Sampling and Seed Selection Each kill-chain slot is filled by softmax sampling over eligible procedures, with temperature-driven suppression of already-used procedures. Eligibility is governed by a requires/adds dependency grammar: a DCSync procedure requiring domain_admin privilege may only be selected if a prior step has escalated to domain administrator level. To maximize MITRE coverage, we sweep seeds 0-999 and compute which procedures each seed's campaign invokes. Greedy set cover selects 26 seeds that collectively cover 105 of 106 procedures. Cyber Defense Benchmark: Agentic Threat Hunting Evaluation for LLMs in SecOpsPage 7 Simbian AI · Technical Report v1.0 · April 2026 · research@simbian.ai 7.4 Deterministic Log Mutation The ReplayProjector transforms raw procedure logs for each campaign: 1. Time shift. All 12 timestamp fields shifted by a seed-derived delta anchored to 2026-01-14T00:00:00Z. Multiple chains are staggered 30 minutes apart with 40% per-step jitter, simulating realistic dwell times. 2. Entity substitution. IPs, hostnames, usernames, domains, and SIDs are rewritten from a seed-derived substitution map. Every substitution is recorded in an audit changelog embedded in the campaign bundle. 3. GUID re-anonymization. Eight GUID-shaped fields (ProcessGuid, ParentProcessGuid, LogonGuid, etc.) are replaced with fresh UUIDs, preventing cross-campaign fingerprinting by string memorization. 4. Flag co-shift. Ground-truth flag timestamps are shifted by the same delta to remain consistent with the mutated logs. The combination of entity obfuscation and GUID re-anonymization ensures that a model cannot exploit memorized string literals from the public Mordor dataset to solve benchmark episodes. Running with the same (seed, blueprint, timestamp) triple produces byte-identical campaign outputs, enabling exact reproducibility. 8. EVALUATION PROTOCOL 8.1 Models Evaluated ModelProviderContext windowCampaigns (N)Notes Claude Opus 4.6Anthropic200K tokens10Strongest model evaluated GPT-5OpenAI128K tokens10Earliest decisive submission Gemini 3.1 Pro PreviewGoogle Vertex AI1M tokens26Thinking level = HIGH Kimi K2.5Fireworks AI128K tokens26Open-weight model Gemini 3 Flash PreviewGoogle Vertex AI1M tokens26Best cost-efficiency Table 3: Models evaluated in Cyber Defense Benchmark v1. 8.2 Experimental Setup Each model is run for one rollout per campaign seed, with a budget of 50 SQL queries (max_queries=50). A 1.5× safety cap at 75 agent turns catches runaway loops. All runs use an identical system prompt across models: a mission statement, the full 505-column schema, pagination instructions, and action descriptions. No attack hints, time windows, or victim host lists are provided. Provider-level parallelism is controlled via per-provider thread pools to respect API rate limits. Total experiment cost across all 98 hunt runs: approximately $243 USD. 8.3 Metrics • n_flags_in_submitted: correctly submitted malicious-event timestamps (primary metric). • n_flags_in_query: flags that appeared in any query result (observability ceiling). • total_flags: ground-truth flag count for the campaign (recall denominator). • total_cost: USD cost of LLM API calls. • total_tokens: total tokens consumed (input + output). 9. RESULTS 9.1 Leaderboard Cyber Defense Benchmark: Agentic Threat Hunting Evaluation for LLMs in SecOpsPage 8 Simbian AI · Technical Report v1.0 · April 2026 · research@simbian.ai ModelNFlags submitted (mean±σ) Submit rate In query (mean) Cost (mean) Tokens (mean) Turns Claude Opus 4.610115.9±11.13.82%156.1$17.893,520,90453.9 GPT-51042.5±19.51.52%52.5$0.741,294,86725.5 Gemini 3.1 Pro2638.9±28.41.36%70.4$1.421,656,09529.5 Kimi K2.52629.1±22.21.05%36.0$1.252,030,82249.4 Gemini 3 Flash2628.2±14.80.98%38.6$0.10299,46913.6 Table 4: Main results. "Submit rate" = mean n_flags_in_submitted / total_flags. Figure 1: Cost-performance Pareto frontier. Each point shows mean ± σ flags submitted vs. mean API cost per run. Claude Opus 4.6 dominates on flags found but at 24× higher cost than GPT-5. 9.2 Zero Passing Scores We define a passing score as ≥50% recall on every MITRE ATT&CK tactic represented in the benchmark. The threshold follows from operational reality: below 50% a model misses more malicious events than it detects in that tactic, which disqualifies it from unsupervised SOC deployment. Thirteen tactics are covered (the twelve kill-chain tactics observed in the Mordor corpus, plus Resource Development). No model passes. Claude Opus 4.6 clears the 50% bar on 5 of 13 tactics (Defense Evasion, Persistence, Execution, Command & Control, Privilege Escalation); the remaining four models clear it on 0 of 13 tactics. GPT-5 additionally has two complete tactic blind spots - Initial Access and Lateral Movement - where it submits no correct flags in any run. The four non-leader models cluster between 10% and 14% mean recall, an order of magnitude below the bar. 9.3 All Models Fail to Complete Hunts The most striking finding is that no run across any model ever triggered the TERMINATED outcome (all flags found). Every run terminates via GAVE_UP (87 of 98 runs), TRUNCATED - query budget exhausted (9 runs), or ERROR (2 runs). On campaigns with the highest flag counts (5,586-6,713 flags), even Claude Opus 4.6 submits at most ∼150 flags - roughly 2% recall. The gap between n_flags_in_query and n_flags_in_submitted reveals a consistent pattern: agents observe more malicious events than they explicitly report. Claude Opus 4.6 Cyber Defense Benchmark: Agentic Threat Hunting Evaluation for LLMs in SecOpsPage 9 Simbian AI · Technical Report v1.0 · April 2026 · research@simbian.ai observes 156 flags on average but submits only 116, suggesting the agent encounters correct evidence but fails to consistently attribute it. Figure 2: Cumulative flags submitted per turn, per LLM group (sample rollouts). Claude Opus 4.6 exploration is deeper before committing; GPT-5 submits earlier. No model reaches the total flags available in any campaign. 9.4 Tactic-Level Recall MITRE TacticClaude Opus 4.6 GPT-5Gemini 3.1 Pro Gemini 3 Flash Kimi K2.5 Defense Evasion0.610.310.230.170.19 Persistence0.570.270.180.130.17 Command & Control0.550.250.210.170.18 Execution0.560.220.200.150.17 Privilege Escalation0.530.240.180.130.16 Resource Development0.460.100.190.150.20 Discovery0.380.150.100.080.09 Collection0.340.020.060.060.05 Lateral Movement0.340.000.100.060.12 Exfiltration0.310.070.050.030.03 Impact0.310.130.120.090.05 Initial Access0.280.000.090.070.08 Credential Access0.210.010.030.030.09 Table 5: Normalized recall per MITRE tactic (high-severity flags only). 1.0 = maximum flags found by any model for that tactic. Cyber Defense Benchmark: Agentic Threat Hunting Evaluation for LLMs in SecOpsPage 10 Simbian AI · Technical Report v1.0 · April 2026 · research@simbian.ai Figure 3: Radar chart of normalized tactic recall for all five models. High-severity (critical/high), high-relevance (1-2) flags only. Claude Opus 4.6 leads across all 13 tactics; all models show near-zero coverage of Credential Access and Initial Access. Claude Opus 4.6 achieves its highest normalized recall on Defense Evasion (0.61) and Persistence (0.57), where common heuristics - PowerShell encoding detection, registry run-key writes, signed-binary proxy execution - are detectable with simple keyword queries. All models show near-zero recall for Credential Access and near-zero or zero recall for Initial Access, where attacks leave subtler traces (LSASS DRSUAPI calls, Kerberoasting service-ticket patterns) that require domain-specific query knowledge not spontaneously generated. 9.5 Hunt Strategy Analysis Across all models, the dominant strategy is breadth-first keyword scanning: scope discovery (event counts, timestamp ranges, channel distributions), then filtering by common malicious event IDs (EventID 1 process creation, EventID 4104 PowerShell script-block logging, EventID 4624 logon events). A representative first-five-turn sequence from Claude Opus 4.6: T1: SELECT COUNT(*), MIN(TimeCreated), MAX(TimeCreated) FROM logs -> 134,867 records, 2026-01-14T00:00:00Z to T01:51:35Z T2: SELECT EventID, Channel, COUNT(*) AS cnt FROM logs GROUP BY EventID, Channel ORDER BY cnt DESC T5: SELECT TimeCreated, EventID, Image, CommandLine FROM logs WHERE EventID='1' AND ( CommandLine LIKE '%encoded%' OR CommandLine LIKE '%bypass%' OR CommandLine LIKE '%invoke%') -> 16 NEW FLAGS DISCOVERED Cyber Defense Benchmark: Agentic Threat Hunting Evaluation for LLMs in SecOpsPage 11 Simbian AI · Technical Report v1.0 · April 2026 · research@simbian.ai GPT-5 commits to submission earlier (mean turn 14 vs. 37 for Claude Opus 4.6), suggesting a different exploration-exploitation tradeoff. Gemini 3 Flash achieves comparable flags-submitted to GPT-5 at 14× lower cost ($0.10 vs. $0.74), suggesting that reasoning quality - not token throughput or context length - is the primary bottleneck for this task. 10. DISCUSSION 10.1 Why Do Models Fail? Three factors likely contribute to the low recall: • Search-space intractability. A 10-row result limit against 75K-135K records forces agents into an information-gathering problem that cannot be solved by breadth-first scanning within 50 queries. Effective threat hunting requires semantic compression: formulating queries that retrieve high-density malicious evidence. • Attribution gap. The consistent gap between n_flags_in_query and n_flags_in_submitted indicates agents observe malicious evidence but fail to attribute it. This may reflect under-confidence in partial information, aversion to submitting wrong timestamps, or limitations in tracking evidence across long conversation histories. • Tactic blind spots. GPT-5 records zero recall on Initial Access and Lateral Movement; all models show near-zero recall on Credential Access. Certain ATT&CK tactics require very specific query patterns - Kerberoasting indicators in Security logs, DRSUAPI call patterns in Sysmon - that current models do not spontaneously generate without explicit detection-engineering training. 10.2 Benchmark Validity Considerations The Sigma-rule-derived ground truth is comprehensive but not perfect: rules may fire on benign lookalike events, and LLM consequence enrichment introduces the evaluator's own model into the annotation pipeline. The 500-record noise injection is a conservative simulation of production log volume; real enterprise environments contain millions of daily events, further amplifying the search challenge. Microsecond-precision timestamp matching is strict by design: coarse-grained submissions (e.g., submitting all events in a one-minute window) should not receive partial credit, as the task requires precise forensic attribution. 11. REPRODUCIBILITY All 26 campaign seeds, procedure labels, chain templates, simulator code, flag extraction pipeline, and evaluation harness are fully deterministic given a (seed, blueprint, timestamp) triple. The 26-seed selection procedure is implemented in research/chain_analysis/seed_simulation.py. All 98 hunt result JSONs are included in the release, containing full conversation traces, per-turn SQL queries and results, flag discovery events, token counts, and costs. Benchmark charts can be regenerated with: python tools/generate_benchmark_page.py \ --output output/benchmark.html \ --artifacts release/artifacts/ LLM API results are nondeterministic at temperature > 0; single-rollout results are reported for most models. Multi-rollout averaging (≥3 rollouts) is recommended for future comparisons. The campaign generator pipeline is proprietary to Simbian AI; pre-built campaign JSONs for the 26 benchmark seeds are distributed with the release to enable independent evaluation without the generator. Cyber Defense Benchmark: Agentic Threat Hunting Evaluation for LLMs in SecOpsPage 12 Simbian AI · Technical Report v1.0 · April 2026 · research@simbian.ai 12. LIMITATIONS AND FUTURE WORK • Single-rollout evaluation. High variance (e.g., Gemini 3.1 Pro: σ=28.4) suggests multi-rollout averaging is needed for statistical reliability. Future evaluations should report mean±σ over ≥3 rollouts. • Windows-only telemetry. All 106 procedures use Windows event logs. Linux auditd, AWS CloudTrail, and network flow data represent a large fraction of real SOC workloads and are not covered. • Alert-augmented setting. A complementary evaluation could pre-surface one Sigma-rule alert and measure the agent's ability to pivot from a known indicator of compromise, testing investigation depth separately from discovery breadth. • Reinforcement-learning agents. All evaluated agents use zero-shot prompting. RL-trained agents optimizing the benchmark’s reward function may yield substantially higher recall. • Per-chain attribution. The current flag set merges all malicious events regardless of sub-chain origin. Future versions will tag flags by kill-chain membership, enabling per-chain and per-stage recall metrics. 13. CONCLUSION The Cyber Defense Benchmark is the first benchmark to evaluate LLM agents on open-ended, evidence-driven threat hunting against real attack telemetry. Our evaluation of five frontier models on 98 runs across 26 multi-procedure campaigns - spanning 105 of 106 MITRE ATT&CK procedures in the dataset - shows uniformly low recall: the best model (Claude Opus 4.6) submits correct malicious event timestamps for only 3.82% of ground-truth flags on average, and no model ever completes a hunt. Against our passing bar (≥50% recall on every ATT&CK tactic), the leader clears 5 of 13 tactics and the other four models clear zero - no model passes. These results do not reflect model quality on security knowledge tasks, where the same models score well; rather, they expose a specific deficiency in open-ended, agentic evidence gathering at scale. We release the benchmark environment, campaign data, and all hunt traces to enable the community to measure and improve LLM capability in cyber defense. ACKNOWLEDGMENTS We thank the Open Threat Research Federation (OTRF) for the Security-Datasets corpus, the SigmaHQ community for the open Sigma rule repository, and WithSecureLabs for the Chainsaw detection engine. This work was conducted at Simbian AI. REFERENCES [1] Rodriguez, J. B., et al. (2020). Mordor: Pre-recorded Security Events. OTRF Security-Datasets. github.com/OTRF/Security-Datasets. [2] Peng, B., et al. (2024). SecBench: A Comprehensive Multi-Dimensional Benchmarking Dataset for LLMs in Cybersecurity. arXiv:2412.20787. [3] Tihanyi, N., et al. (2023). CyberMetric: A Benchmark Dataset for Evaluating LLMs Knowledge in Cybersecurity. arXiv:2402.07688. [4] Liu, Y., et al. (2024). CTI-Bench: Evaluating LLMs in Cyber Threat Intelligence. ACL Findings 2024. [5] Tann, W., et al. (2023). Using LLMs for Cybersecurity CTF Challenges and Certification Questions. IEEE SSCI 2023. Cyber Defense Benchmark: Agentic Threat Hunting Evaluation for LLMs in SecOpsPage 13 Simbian AI · Technical Report v1.0 · April 2026 · research@simbian.ai [6] Yang, J., et al. (2023). InterCode: Standardizing and Benchmarking Interactive Coding with Execution Feedback. NeurIPS 2023. [7] Shao, R., et al. (2024). NYU CTF Bench: A Scalable Open-Source Benchmark for LLMs in Offensive Security. arXiv:2406.05590. [8] Happe, A., et al. (2023). Getting pwn'd by AI: Penetration Testing with Large Language Models. ESEC/FSE 2023. [9] Sigma Project. (2015-2024). SigmaHQ: Generic Signature Format for SIEM Systems. github.com/SigmaHQ/sigma. [10] WithSecureLabs. (2022). Chainsaw: Rapidly Search and Hunt Through Windows Forensic Artefacts. github.com/WithSecureLabs/chainsaw. [11] Strom, B., et al. (2018). MITRE ATT&CK;: Design and Philosophy. MITRE Technical Report MTR180110. [12] Towers, M., et al. (2023). Gymnasium: A Standard Interface for Reinforcement Learning Environments. arXiv:2407.17032. [13] Berrevoets, J., et al. (2024). CAI-Bench: Benchmarks for AI Capabilities in Cybersecurity. Preprint. [14] LiteLLM. (2023). Call all LLM APIs using the OpenAI format. github.com/BerriAI/litellm. [15] Microsoft Research. (2025). Security Copilot Benchmark: Evaluating AI-Assisted Threat Detection. Technical Report, Microsoft Corporation. APPENDIX A: BENCHMARK SEEDS The 26 benchmark seeds selected by greedy MITRE procedure set cover (covering 105 of 106 procedures under the DIVERSE_INTRUSION blueprint): 0, 84, 97, 108, 137, 159, 176, 205, 266, 290, 342, 347, 450, 479, 530, 626, 671, 675, 680, 733, 785, 802, 912, 916, 950, 951 APPENDIX B: DATASET INFRASTRUCTURE ENVIRONMENTS EnvironmentProcedure countPrimary log format(s) shire_v254HELK (53), WEF export (1) wardog_standalone28WEF export (20), raw_event (8) pedro_vbox11WEF export shire_v2_mordor4HELK pandalab3WEF export shire_v13Winlogbeat (2), HELK no-tags (1) aptsimulator1WEF export azsentinel_exchange1WEF export blacksmith_adfs1WEF export Table B.1: Recording infrastructure environments in the Mordor corpus.