Paper deep dive
SecRespond: Benchmarking AI Agents for Real-World Post-Compromise Incident Response
Lehan Wang, Boli Chen, Ruixue Ding, Pengjun Xie, Jinwei Huang, Zhendong Liu, Shuo Wang, Tao Lei, Xin Ouyang, Xiaomeng Li
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 90%
Last extracted: 8/4/2026, 11:10:48 AM
Summary
The paper introduces SecRespond, a benchmark for evaluating Large Language Model (LLM) agents on post-compromise incident response. It assesses agents' ability to investigate forensic disk snapshots, analyze alerts, and produce remediation plans across 10 cyber ranges covering various operating systems and attack techniques. The study evaluates 23 frontier LLMs, revealing that while agents can detect alerted issues, they struggle with proactive investigation of silent intrusions and comprehensive remediation.
Entities (11)
Relation Signals (9)
SecRespond β covers β 10 Cyber Ranges
confidence 95% Β· We instantiate this task across 10 cyber ranges
SecRespond β evaluates β LLM agents
confidence 95% Β· SecRespond, the first benchmark for evaluating LLM agents on the post-compromise incident-response workflow.
Claude Opus 4.7 β achieves β 72.4% Detection and Planning Score
confidence 90% Β· Even the strongest model, Claude Opus 4.7, achieves only 72.4% averaged over detection and planning
SecRespond β defines β 5-Dimensional Capability Taxonomy
confidence 90% Β· five-dimensional Capability (CAP) taxonomy including intrusion entity, persistence mechanism, baseline risk, vulnerability risk, and investigation & response quality
SecRespond β evaluates β 23 LLM Models
confidence 90% Β· We evaluate 23 frontier LLMs on the OpenCode agent harness.
SecRespond β identifiesbottleneck β Proactive Investigation
confidence 90% Β· they struggle to proactively investigate the disk for silent intrusions
SecRespond β identifiesbottleneck β Comprehensive Remediation
confidence 90% Β· struggle to ... produce comprehensive, verified remediation plans
SecRespond β uses β Forensic Disk Snapshot
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Large Language Model (LLM) agents are increasingly adopted in real-world security operations with access to host artifacts and command-line interfaces (CLIs), making it critical to thoroughly assess their security capabilities. However, existing cybersecurity benchmarks focus on pre-compromise settings where agents are placed in a clean and idealized environment before an attack occurs. This leaves the post-compromise setting underexplored. To address this gap, we introduce SecRespond, the first benchmark for evaluating LLM agents on the post-compromise incident-response workflow. Given a forensic disk snapshot of a compromised host together with the alerts, vulnerability scans, and baseline checks reported by a host security product, agents are required to produce forensic reports on intrusions, baseline risks, and vulnerability risks, together with a remediation plan. We instantiate this task across 10 cyber ranges, each constructed from a distinct compromised cloud host, spanning 4 entry-point types, 21 ATT&CK techniques, and 5 operating systems. We evaluate 23 frontier LLMs on the OpenCode agent harness. Experimental results show that although current agents can reliably uncover the problems exposed by alerts, they struggle to proactively investigate the disk for silent intrusions and to produce comprehensive, verified remediation plans, with no model achieving complete detection and remediation on any single range. This reveals a fundamental bottleneck in building agents for real-world incident response. The benchmark is publicly available at this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2607.26791v1
- Canonical: https://arxiv.org/abs/2607.26791v1
Trouble viewing inline? Open PDF directly β
Full Text
196,230 characters extracted from source content.
Expand or collapse full text
2026-07-30 SecRespond: Benchmarking AI Agents for Real-World Post-Compromise Incident Response Lehan Wang 1,3 , Boli Chen 1 , Ruixue Ding 1 , Pengjun Xie 1 , Jinwei Huang 2 , Zhendong Liu 2 , Shuo Wang 2 , Tao Lei 2 , Xin Ouyang 2 , Xiaomeng Li 3 1 Tongyi Lab, Alibaba Group 2 Alibaba Cloud Computing, Alibaba Group 3 The Hong Kong University of Science and Technology Abstract Large Language Model (LLM) agents are increasingly adopted in real-world security operations with access to host artifacts and command-line interfaces (CLIs), making it critical to thoroughly assess their security capabilities. However, existing cybersecurity benchmarks focus on pre-compromise settings where agents are placed in a clean and idealized environment before an attack occurs. This leaves the post-compromise setting underexplored. To address this gap, we introduce SECRESPOND, the first benchmark for evaluating LLM agents on the post-compromise incident-response workflow. Given a forensic disk snapshot of a compromised host together with the alerts, vulnerability scans, and baseline checks reported by a host security product, agents are required to produce forensic reports on intrusions, baseline risks, and vulnerability risks, together with a remediation plan. We instantiate this task across 10 cyber ranges, each constructed from a distinct compromised cloud host, spanning 4 entry-point types, 21 ATT&CK techniques, and 5 operating systems. We evaluate 23 frontier LLMs on the OpenCode agent harness. Experimental results show that although current agents can reliably uncover the problems exposed by alerts, they struggle to proactively investigate the disk for silent intrusions and to produce comprehensive, verified remediation plans, with no model achieving complete detection and remediation on any single range. This reveals a fundamental bottleneck in building agents for real-world incident response. The benchmark is publicly available athttps://github.com/Alibaba-NLP/qqr/tree/ma in/data/secrespond. 1 Introduction Large Language Model (LLM) agents, equipped with iterative reasoning and autonomous tool-use abilities (OpenAI, 2026; Qwen, 2026; Z.ai, 2026), demonstrate remarkable potential in tackling complex, professional real-world tasks (Merrill et al., 2026; Ye et al., 2026; Li et al., 2026b), among which cyber- security is a significant deployment target. With access to host artifacts and command-line interfaces (CLIs), LLM agents are increasingly expected to assist in real-world security operations from vulnerability discovery to incident response (Wang et al., 2025; Deason et al., 2025). Therefore, it is critical to assess how reliably current agents can fulfill such pipelines and how they can be involved in human workflows. Recently, several cybersecurity benchmarks have been developed to cover various offensive-security subtasks. Zhang et al. (2025) and Shao et al. (2024) assess the offensive capabilities of LLM agents with CTF challenges, while Zhu et al. (2025) and Wang et al. (2025) evaluate agentsβ vulnerability discovery and exploitation abilities using real-world web applications or software projects. Recent efforts such as CyberModelArena (Wiz, 2026) have begun to assess harness-model combinations across the full attack lifecycle. In parallel, defensive benchmarks evaluate LLMs on security knowledge, vulnerability discovery, and patching. Jing et al. (2024) collects multiple-choice question-answer pairs to assess LLMsβ defensive knowledge, while Meta (2025) evaluates LLMs on vulnerability patching. However, these existing security benchmarks focus on pre-compromise settings, where the agent operates in a clean and idealized environment to find, exploit, or patch weaknesses before an attack occurs (Zhang et al., 2025; Wang et al., 2025). Even on the defensive side, existing works simplify the problem to reasoning over isolated alerts (Deason et al., 2025), system logs (Wu et al., 2025), or incident reports (Lin et al., 2025) rather than confronting the compromised system with real artifacts left by the attack, such as persistence mechanisms, deliberately erased traces, and noisy concurrent host activity. This leaves the post-compromise workflow underexplored, which includes investigating a successful intrusion on the compromised host, reconstructing the attack chain and remediating the system. As post-compromise response covers the full incident-response cycle, evaluating LLM agents on this pipeline measures 1 arXiv:2607.26791v1 [cs.CR] 29 Jul 2026 whether they can actively discover problems within the compromised disk and respond effectively to live alerts and real intrusion artifacts, thereby serving as the prerequisite to facilitating agent assistance for security operations in production. To address this gap, we introduce SECRESPOND, the first benchmark for evaluating LLM agents on the post-compromise incident-response workflow. The agent is given a forensic disk snapshot of a compromised host with the alerts, vulnerability scans, and baseline checks reported by a host security product. Given these inputs, the agent is required to investigate the host, reconstruct what happened, and produce one progress file and four reports covering the detected intrusion, vulnerabilities and baseline risks, along with a remediation plan to resolve them. SECRESPOND instantiates the above task in 10 cyber range scenarios, each built from a frozen snapshot of a distinct, fully instantiated cloud-host environment compromised through an end-to-end attack over real network protocols. Together, these ranges cover the diversity and complexity of real-world intrusion scenarios across 4 entry-point types, 21 ATT&CK techniques, and 5 operating systems. Moreover, we propose a hierarchical rubric that decomposes each range into detailed checkpoints that validate whether each problem is comprehensively discovered and effectively handled, and grade them with the LLM-as-a-Judge method. We further design a five- dimensional Capability (CAP) taxonomy including intrusion entity, persistence mechanism, baseline risk, vulnerability risk, and investigation & response quality, and map each checkpoint to its corresponding capability items. In this way, checkpoint results from heterogeneous ranges become comparable and can be aggregated through the capability taxonomy to benchmark model capabilities. In total, SECRESPOND defines 52 capability items and 280 checkpoints, reflecting fine-grained model performance in the full incident response workflow from investigation to remediation. We conduct extensive experiments on a representative agent harness, OpenCode (OpenCode, 2026), paired with 23 different large language models (LLMs), spanning multiple model families and successive releases within each family. Our experiments reveal that current agents can surface the problems exposed by alerts but struggle to investigate the forensic disk proactively and to complete the remediation required for incident response. Even the strongest model, Claude Opus 4.7, achieves only 72.4% averaged over detection and planning, leaving malicious artifacts untouched and remediation incomplete, especially in the ranges where the attack chain grows longer and broader. All the models score higher on detection than on planning, with the gap widening to 34.7% for GPT-5.5, since agents tend to apply the obvious first fix but rarely complete the remaining remediation. Furthermore, we observe a significant imbalance across capability dimensions. Average detection reaches 75.4% on intrusion entity but only 58.8% on persistence mechanism, whereas the average planning score peaks at 55.3% on baseline risk and reaches only 31.8% on investigation & response quality. This exposes a fundamental bottleneck in versatile incident-response models. Our contributions are as follows: β’A benchmark for post-compromise incident response. We release SECRESPOND, the first benchmark targeting the full post-compromise response workflow, which comprises 10 cyber ranges built on fully instantiated cloud-host environments, covering 4 entry-point types, 21 ATT&CK techniques, and 5 operating systems. β’ A hierarchical capability evaluation framework. We propose a hierarchical LLM-as-a-Judge framework where each task is decomposed into fine-grained checkpoints manually designed by security experts. The resulting 280 checkpoints are graded along both detection and planning axes and mapped to 52 capability items in our five-dimensional Capability (CAP) taxonomy, supporting per-capability diagnosis and cross-range analysis. β’An empirical study of frontier agents on real-world post-compromise tasks. We evaluate 23 LLMs on a representative agent harness, OpenCode, spanning multiple model families and successive releases within each family, with each task independently judged by three strong proprietary LLMs to mitigate individual bias. β’ Key findings on incident response capability of agent models. Our study shows that current agents reliably discover alerted problems but struggle with proactive investigation and complete remediation. We also observe that agent modelsβ incident response capabilities are significantly imbalanced across different dimensions. 2 Related Work 2.1 Agent Benchmarks To comprehensively evaluate the capabilities of LLM agents from multiple aspects, many benchmarks have been developed. To validate agent modelsβ coding abilities, SWE-Bench (Jimenez et al., 2024) tests the 2 Table 1: Comparison with existing cybersecurity benchmarks. We compare SECRESPOND with rep- resentative cybersecurity benchmarks along the following aspects: Scope denotes the security task; Post-Compromise indicates whether the benchmark targets the real-world scenario after a host has already been attacked; Real Filesystem denotes whether the task is grounded in a host disk rather than synthesized logs or text; Multi-Step Analysis indicates whether solving the task requires multiple investigative steps; Cross-File Analysis summarizes whether the evidence should be correlated across multiple files; Rubric-Based Evaluation indicates whether the evaluation metrics use fine-grained rubrics; CLI-Compatible denotes whether the benchmark task can be executed through a command-line interface. BenchmarkScope Post-Compromise Real Filesystem Multi-Step Analysis Cross-File Analysis Rubric-Based Evaluation CLI- Compatible SecBench (Jing et al., 2024)Knowledgeβ NYU CTF Bench (Shao et al., 2024)CTFβββ CyberSecEval (Wan et al., 2024)Offensiveβββββ CyBench (Zhang et al., 2025)CTFβββ CVE-Bench (Zhu et al., 2025)CVEβββ AutoPatchBench (Meta, 2025)Patchβββ CyberGym (Wang et al., 2025)Vulnβββ CyberSOCEval (Deason et al., 2025)SOCβ ExCyTIn-Bench (Wu et al., 2025)Threat Detectionββββ CyberModelArena (Wiz, 2026)Offensiveββββ SECRESPOND (Ours)Forensicβ model on resolving real GitHub issues, SWE-Bench-Pro (Deng et al., 2025) extends it to more realistic long- horizon problems that require examining multiple files and modifying substantial code, and Terminal- Bench (Merrill et al., 2026) targets interactive shell execution in a terminal environment. To evaluate agents in GUI environments, WebArena (Zhou et al., 2024) places agents in realistic web applications to conduct web navigation, while OSWorld (Xie et al., 2024) extends to real desktop environments where agents are supposed to perceive screenshots and operate applications. To benchmark agents in multi-turn conversational interaction,Ο-bench (Yao et al., 2024) prompts agents to serve a simulated user in customer- service dialogues,Ο 2 -bench (Barres et al.) establishes a dual-control environment in which the user can also modify the shared state, and Vitabench (He et al., 2025) scales the setting to versatile real-world applications including food delivery, in-store consumption, and online travel services. Recently, agentic harnesses such as OpenClaw and OpenCode have encouraged evaluation of agents operating within these scaffolds. CocoaBench (Team et al., 2026) evaluates agentsβ compositional capability in vision, search, and coding on long-horizon tasks, ClawBench (Zhang et al., 2026) constructs daily tasks on live websites, and Claw-Eval (Ye et al., 2026) and Claw-Eval-Live (Li et al., 2026a) cover end-to-end workflows across service orchestration, multimodal interaction, and professional dialogue. Despite these broad evaluations, existing agent benchmarks target software engineering, web, and general productivity tasks, largely overlooking security operations. 2.2 Benchmarking LLMs in Cybersecurity Evaluating cybersecurity capabilities of LLMs and LLM agents has drawn growing attention, typically from both offensive and defensive aspects. On the offensive side, early work (Li et al., 2024) assesses intrinsic security knowledge of LLMs with curated multiple-choice questions. These benchmarks focus on knowledge probing rather than automatic operational abilities. Subsequently, Cybench (Zhang et al., 2025) and NYU CTF Bench (Shao et al., 2024) place LLM agents inside Docker containers to evaluate their offensive abilities through interactive CTF challenge solving, and CyberGym (Wang et al., 2025) extends the scenario to vulnerability discovery and exploitation in real-world software projects, where LLM agents uncover zero-day bugs that human maintainers have missed. Recent work, CyberModelArena (Wiz, 2026), unifies the full attack lifecycle into a single benchmark of five categories and evaluates different combinations of agent designs and underlying models to analyze their independent and joint offensive capabilities. On the defensive side, SecBench (Jing et al., 2024) measures LLMsβ defensive knowledge in network, endpoint, application, and cloud security, while CyberSOCEval (Deason et al., 2025) evaluates LLMs from the perspective of Security Operations Center (SOC) analysts on understanding malware analysis and threat intelligence reports. AutoPatchBench (Meta, 2025) evaluates AI-assisted patch generation for repairing vulnerabilities. ExCyTIn-Bench (Wu et al., 2025) evaluates LLM agents on cyber threat investigation by reasoning over alerts and security logs. However, as shown in Table 1, none of the existing work reproduces actual end-to-end host intrusion that leaves forensic artifacts, including persistence mechanisms, partially cleaned traces, and noisy concurrent legitimate activity. As a result, there remains a gap in evaluating whether LLM agents can investigate a post-compromise disk snapshot and produce detection and remediation analysis. 3 Figure 1: Overview of SECRESPOND. In our benchmark task, the agent is expected to investigate a cyber range and produce incident-response reports. Each cyber range includes a forensic disk snapshot of a real, fully instantiated cloud host compromised through an end-to-end attack over real network protocols, together with security analytics comprising alerts and scan findings from a security product. To evaluate the agentβs incident-response capability, we organize the required skills into a capability taxonomy and map each capability to a list of checkpoints for each range. These checkpoints serve as evaluation rubrics for the LLM-as-a-Judge framework. 2.3 Evaluation Methodology Cybersecurity benchmarks predominantly rely on deterministic verification with a fixed ground truth, such as multiple-choice accuracy (Li et al., 2024), CTF flag matching (Zhang et al., 2025), or verifying the agentβs execution output (Wang et al., 2025; Meta, 2025). Such protocols cannot be directly applied to our benchmark, where the task output is open-ended investigation reports. Therefore, we adopt the LLM-as-a-Judge method (Gu et al., 2026), which decouples the target into verifiable units, such as the rubric items for trajectory evidence in Claw-Eval (Ye et al., 2026), thus making it well-suited for open-ended tasks. To keep the judgment reliable and interpretable, we design a hierarchical LLM-as-a- Judge framework that decomposes each task into fine-grained checkpoints, which are further mapped to capability items in our five-dimensional capability taxonomy, enabling fine-grained analysis within each range and capability-level comparison across ranges. 3SECRESPOND Benchmark SECRESPOND evaluates LLM agentsβ ability to investigate a post-compromise host snapshot and produce comprehensive forensic reports on intrusion analysis, baseline assessment, and vulnerability verification, together with a structured remediation plan. In this section, we first describe the benchmark task and explain how we construct cyber ranges on real, fully instantiated cloud hosts across diverse attack types and operating systems (Β§3.1). Then, we define a capability taxonomy that specifies what the agent should be able to accomplish in the incident-response process (Β§3.2). Finally, we introduce the evaluation framework, which decomposes each range into fine-grained checkpoints that link concrete forensic evidence to capability items, enabling hierarchical, cross-range evaluation (Β§3.3). 3.1 Task Construction 3.1.1 Overview of Tasks To comprehensively evaluate the agentsβ incident response abilities, we construct 10 cyber ranges covering multiple attack scenarios. A cyber range includes a frozen, reproducible, read-only forensic disk snapshot 4 Table 2: Overview of the cyber ranges. We construct 10 cyber ranges in total, spanning 4 entry points and 5 operating systems. Each range reproduces a complete multi-stage attack chain from initial access through privilege escalation, persistence, and impact, covering 21 ATT&CK techniques. RangeEntry typeOSATT&CK techniquesAttack chain SSH-Miner Baseline Weak CentOS 7 T1110, T1021.004, T1059.004, T1552, T1053.003, T1546.004, T1543.002, T1548.003, T1496 SSH brute forceβ mining and persistence (crontab/bashrc/systemd) Shiro- Fastjson Known CVE CentOS 8 T1190, T1059.004, T1505.003, T1548.003, T1098.004, T1053.003, T1543.002, T1496 Shiro default key + Fastjsonβ webshellβ privilege escalationβ mining Log4j-RCE Known CVE CentOS 8 T1190, T1059.006, T1505.003, T1548.003, T1098.004, T1053.003, T1546.004, T1543.002, T1556, T1070, T1496 Log4j RCEβ webshell + persistence Docker- Escape Baseline Weak Ubuntu 20.04 T1190, T1611, T1059.004, T1098.004, T1053.003, T1546.004, T1543.002, T1071.001, T1496 Exposed Docker APIβcontainer escapeβhost takeover Redis-RCE Baseline Weak Ubuntu 22.04 T1190, T1021.004, T1059.004, T1552, T1098.004, T1053.003, T1546.004, T1496 Unauthenticated Redisβ SSH-key writeβ mining Jenkins- RCE Business code Ubuntu 22.04 T1190, T1059.004, T1548.001, T1548.003, T1574.006, T1053.003, T1071.001, T1496 Jenkins Script Consoleβcommand executionβ mining Next.js- RCE Known CVE Ubuntu 22.04 T1190, T1059.007, T1505.003, T1548.001, T1574.006, T1098.004, T1053.003, T1546.004, T1543.002, T1496 Next.js CVEβ SUID privilege escalationβ LD_PRELOAD rootkit NPM- Worm Supply chain Ubuntu 22.04 T1195.002, T1059.007, T1059.006, T1552, T1098.004, T1053.003, T1543.002, T1071.001, T1496 Malicious npm packageβ worm propagation + credential theft ASP.NET- ViewState Known CVE Windows Server T1190, T1505.003, T1552, T1071.001 ASP.NET ViewStateβRCEβMSSQL backdoor + WMI persistence RDP- Service- Abuse Baseline Weak Windows Server T1110, T1552, T1071.001 Exposed RDPβ password sprayingβ weak service DACL abuseβ service restart for SYSTEM payload + disguised service & scheduled task + credential dump and C2 of a host that has been compromised by a complete, end-to-end attack delivered over real network protocols, along with a set of security analytics including real-time alerts and scan findings from a security product. The agent is expected to investigate the exposed alerts and the snapshot, to reconstruct the intrusion from the traces the attack leaves behind. These ranges cover different operating systems, entry points, ATT&CK techniques, and attack stages, which are selected to mimic the distribution in actual incidents based on a correlation analysis of alerts, vulnerabilities, and baselines from 372 real compromised cloud hosts. In detail, the ranges are summarized in Table 2. The forensic disk snapshot typically covers all the directories of the compromised host, and contains the naturally generated intrusion artifacts left by the attack which the agent can read as the primary forensic evidence, including logs, configuration changes, and residual files. Beyond the disk snapshot, the agent also receives three types of security analytics from the host security product on the victim machine: (i) real-time alerts from the anomaly detection engine monitoring suspicious processes, network connections, and login events, (i) vulnerability findings from scans of installed packages and applications, and (i) baseline check results from configuration checks. Given these inputs, the agent is required to produce one progress file and four reports: an intrusion report reconstructing the attack chain covering entry point, lateral movement, persistence mechanisms, and final impact, a vulnerability report verifying which exploited weaknesses are present on the host, a baseline report assessing host configuration against security baselines, and a remediation plan with recommended steps and reference commands for executing the fixes. The overview of SECRESPOND is demonstrated in Figure 1. 3.1.2 Design Principles As described above, each task is centered on a cyber range that the agent should investigate. To reproduce the operational properties of real incidents, we build each range as three decoupled layers. The blueprint layer specifies the evaluation purpose of the scenario, including the targeted stack, the attack chain 5 Figure 2: Construction Pipeline of Cyber Range. The construction pipeline comprises 12 stages, with human experts involved in key steps to audit and validate the results. annotated with MITRE ATT&CK technique IDs, and the capability items the scenario aims to cover. The instance layer realizes the blueprint on a victim host under three constraints: (i) real vulnerabilities in the form of disclosed CVEs or configuration flaws rather than synthetic bugs, (i) real attacks delivered through network protocols rather than directly placing malicious files, and (i) real intrusion traces produced naturally by the attack. The checklist layer converts the resulting artifacts into checkpoints, each with a pass/fail criterion, an evidence source, and a capability mapping. 3.1.3 Construction Pipeline. The above three layers are realized by a 12-stage construction pipeline shown in Figure 2: blueprint design, attack script development, victim instance curation, victim environment deployment, victim readiness verification, pre-attack clean snapshot, attack environment deployment, attack execution, attack verification, security data collection, remediation validation, and post-attack snapshot. Among these stages, human experts are involved in auditing the plausibility of the attack chain and capability coverage of the blueprint, reviewing the code quality and attack authenticity of the attack scripts, verifying that alert coverage meets the standard during data collection, and checking the completeness of the final range. To distinguish alert-driven shallow analysis from deep forensic capability in each range, we include 30β40% of attack actions that can trigger security-product alerts, while the remaining 60β70% leave silent file modification and configuration artifacts. The alerts provide anchor points to begin reasoning, while the hidden evidence on disk forces investigation and discovery. These analytics are collected through the security platform API after the attack and provided as static files. 3.2 Capability Taxonomy To benchmark agent modelsβ incident response capability, we organize the required capabilities to investigate and remediate a compromised host into a five-dimensional Capability (CAP) taxonomy. β’ Intrusion Entity (ENT) denotes the ability to locate and handle the artifacts left by an intrusion, spanning malicious processes, malicious or tampered files, and malicious network entities. β’Persistence Mechanism (PER) indicates the ability to recover the mechanisms an attacker installs to survive reboot and cleanup, including scheduled tasks, services, shell and environment initialization, account and privilege backdoors, web-component tampering, database backdoors, and kernel- or loader-level techniques. β’Baseline Risk (BAS) is the ability to examine host configuration against security baselines, covering SSH hardening, database and middleware access control, credential safety, privilege and service-permission auditing, container and orchestration configuration, web-service configu- ration, and cloud-resource configuration. β’ Vulnerability Risk (VUL) represents the ability to confirm the weaknesses on the host, span- ning Java component vulnerabilities, web-application code vulnerabilities, container-escape vulnerabilities, and front-end framework vulnerabilities. β’ Investigation & Response Quality (Q) reflects the ability to produce a thorough, well-evidenced, and trustworthy investigation, covering entry-point localization, attack-chain reconstruction, attacker-information extraction, honesty, investigation comprehensiveness, cross-language and cross-service tracing, remediation completeness, and business-impact assessment. In total, the taxonomy comprises 52 items across five dimensions as shown in Table 3, including 12 in ENT, 21 in PER, 7 in BAS, 4 in VUL, and 8 in Q. These capability items represent the benchmark target and form the foundation to build compromised ranges. 6 Table 3: The SECRESPOND evaluation taxonomy with 52 capability items across five CAP dimensions. DimensionCapability Items Intrusion Entity ENT-F01 Webshell file; ENT-F02 Linux malware file; ENT-F03 Malicious SO / kernel module; ENT-F04 Residual data file; ENT-F05 Tampered-file restoration; ENT-F06 Windows malware file; ENT-N01 Attacker-IP inbound block; ENT-N02 Malicious-IP outbound block; ENT-N03 Malicious-domain block; ENT-P01 Mining process; ENT-P02 C2 beacon process; ENT-P03 Windows malicious process Persistence Mechanism PER-A01 authorized_keys tampering; PER-A02 sudoers implant; PER-A03 Rogue local account; PER-D01 MySQL backdoor; PER-D02 MSSQL backdoor; PER-E01 profile.d / udev rule; PER-E02 WMI event subscription; PER-H01 ld.so.preload hijack; PER-H02 DLL hijacking; PER-I01 Shell init (bashrc/profile); PER-I02 /etc/environment injection; PER-M01 SUID/SGID backdoor; PER-S01 Cron task; PER-S02 systemd timer; PER-S03 at job; PER-S04 Windows scheduled task; PER-V01 systemd service; PER-V02 init.d / rc.local; PER-V03 Windows service; PER-W01 Nginx config tampering; PER-W02 App-container config tampering Baseline Risk BAS-01 SSH hardening; BAS-02 DB / middleware access control; BAS-03 Credential safety; BAS-04 Privilege / service audit; BAS-05 Container / orchestration config; BAS-06 Web-service config; BAS-07 Cloud-resource config Vulnerability Risk VUL-01 Java component vuln; VUL-02 Web-app code vuln; VUL-03 Container-escape vuln; VUL-04 Node.js / front-end framework vuln Investigation & Response Quality Q-01 Entry-point localization; Q-02 Attack-chain reconstruction; Q-03 Attacker-info extraction; Q-04 Honesty & confidence calibration; Q-05 Investigation thoroughness; Q-06 Cross-language/service tracing; Q-07 Remediation-verification completeness; Q-08 Business-impact assessment 3.3 Checkpoint Decomposition To evaluate the capabilities above, we decompose them into fine-grained checkpoints within each range to measure whether the corresponding capability is fulfilled in the specific scenario. A checkpoint (CHK) is the smallest scoring unit, specifying the evidence required to discover a finding in a specific range. Each checkpoint matches one or more items in the CAP taxonomy, and multiple checkpoints may be assigned to the same CAP. The capability taxonomy is operationalized through 280 checkpoints in total, as detailed in Table 6. We first define CHK-score, which grades each CHK along two axes, detection and planning. Detection measures whether the agent correctly uncovers problems on the compromised host, including intrusion entities, baseline risks, and vulnerability risks. Planning validates whether the agent proposes a correct and complete remediation plan that resolves the existing problems. The detection axis has a total score of 3, comprising discovery of the problem, sufficiency of the supporting evidence, and correctness of attribution to the root cause. The planning axis has a maximum score of 2, comprising correctness of the proposed remediation step and completeness of the plan to fully resolve the attack artifacts. The normalized score of a single checkpoint on axisais defined as the CHK-score, which represents the agent modelβs performance at the checkpoint level: CHK-score a c = s a c M a β [0, 1], aβdet, plan(1) whereM det =3 andM plan =2 are the maximum scores of the two axes, ands a c β 0, 1,. . .,M a is the score assigned to checkpoint c on axis a. Since each checkpoint is anchored to a capability item, checkpoint results from different ranges become comparable. We aggregate the checkpoints mapped to the same capability item into a capability-level score, which we define as the CAP-score. The CAP-score on a given axis is the ratio of the achieved scores to the maximum scores over all checkpoints mapped to the corresponding capability, computed as: CAP-score a = 1 C a CAP β cβC a CAP CHK-score a c Γ 100%(2) whereC a CAP is the set of checkpoints mapped to the capability item. 7 Table 4: The range-levelCHK-score a r of Detection and Planning for each range. βDetβ denotes CHK-score det r , the percentage of achieved detection scores over all its checkpoints, averaged across three LLM judges; βPlanβ denotes the corresponding CHK-score plan r on the planning dimension. Model SSH-Miner Redis-RCE Docker-Escape Jenkins-RCE Shiro-Fastjson Log4j-RCE Next.js-RCE NPM-Worm ASP.NET- ViewState RDP-Ser- vice-Abuse Overall DetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDet Plan DetPlanDet Plan Claude Opus 4.7 β 77.883.382.469.690.875.578.970.271.652.588.478.777.860.1β83.3 37.9 59.963.979.0 65.7 Claude Opus 4.675.348.079.660.892.162.366.760.666.743.288.468.576.564.981.560.977.8 49.3 77.861.178.258.0 Claude Opus 4.572.561.770.455.979.257.071.962.369.655.285.971.366.760.869.865.4 56.1 27.3 53.147.269.5 56.4 Claude Sonnet 4.6 75.844.168.561.883.363.267.259.776.860.987.981.571.670.268.753.858.1 29.5 89.566.774.7 59.1 Claude Sonnet 4.5 77.876.586.175.684.278.162.655.373.871.978.339.865.769.141.049.473.8 41.753.754.769.7 61.2 GPT-5.567.627.478.249.068.035.173.733.463.927.096.533.358.540.455.841.081.3 19.7 63.653.770.7 36.0 GPT-5.4 Pro65.227.475.947.166.741.269.019.480.563.091.942.665.744.765.537.285.9 31.9 58.652.872.5 40.7 GPT-5.457.525.557.434.369.420.255.029.056.338.585.841.751.036.957.832.767.2 26.5 49.447.260.7 33.2 GPT-5.2 Pro61.321.559.743.154.234.273.139.465.936.488.444.571.944.051.334.656.6 22.0 79.6 50.966.2 37.1 Gemini 3.1 Pro66.738.355.138.260.743.060.239.543.028.772.232.446.432.143.011.547.5 22.7 53.128.754.8 31.5 Gemini 3 Flash58.525.562.534.357.439.556.132.550.127.680.851.957.531.541.926.349.0 17.4 60.542.657.4 32.9 GLM-5.175.458.880.162.891.2 86.071.453.582.563.590.464.893.161.975.860.961.6 27.3 42.052.876.3 59.2 GLM-575.856.960.249.075.551.850.334.349.442.788.464.876.562.450.431.467.2 26.5 54.938.964.9 45.9 DeepSeek V4 Pro76.361.867.156.975.950.063.259.773.155.287.949.175.257.180.164.171.2 32.6 62.347.273.2 53.4 DeepSeek V3.260.953.065.748.165.337.850.947.468.947.980.368.658.540.536.221.141.9 16.7 63.654.659.2 43.6 Qwen3.7 Max75.850.087.576.579.264.072.551.770.446.490.460.182.463.184.948.784.940.9 51.841.778.0 54.3 Qwen3.7 Plus70.536.286.671.675.957.071.969.382.070.888.975.070.961.973.565.581.8 35.6 53.745.375.6 58.8 Qwen3.6 Plus70.554.966.252.977.364.164.332.560.044.384.854.673.267.965.038.468.2 31.8 54.941.768.4 48.3 Qwen3.5 Plus65.751.066.258.861.646.567.849.243.532.382.857.464.755.461.837.270.2 21.2 51.231.563.5 44.0 Kimi K2.660.434.363.027.488.479.0 76.653.553.140.684.955.663.431.552.436.564.7 33.3 51.338.065.8 43.0 Kimi K2.571.561.872.764.875.959.755.643.051.646.488.969.463.153.643.335.946.0 23.5 52.549.162.1 50.7 MiniMax M2.769.136.362.051.974.556.263.749.155.141.676.838.066.445.247.632.042.9 14.4 42.635.260.1 40.0 MiniMax M2.563.338.356.530.461.648.247.432.538.522.976.345.457.842.331.923.146.5 12.9 43.833.352.4 32.9 β Claude Opus 4.7 returned a safety refusal on every NPM-Worm attempt and produced no report; its overall score is therefore computed as the average over the remaining nine ranges. 4 Experiments 4.1 Experimental Settings We evaluate 23 large language models spanning 8 model series and their successive versions, with the full list reported in Table 4. We include multiple versions of the same series to track how incident response capability evolves across releases. All models are driven by the same agent harness, OpenCode (Open- Code, 2026), and receive identical task inputs (Β§3.1.1), instruction prompt (see Appendix C.1), and tool set to ensure a fair comparison. Each model is accessed using default parameters with reasoning enabled. 4.2 Evaluation Metrics Following the LLM-as-a-Judge paradigm, we score each generated report against the range checklist defined in Β§3.3. To reduce single-judge bias, we adopt three strong proprietary models, Claude Opus 4.7 (Anthropic, 2026), Gemini 3.1 Pro (The Gemini Team, 2026), and GPT-5.4 Pro (OpenAI, 2026), as independent judges. Each judge runs in the same OpenCode harness, receives the instruction prompt (see Appendix C.3) and the checkpoint list with criteria, and uses a bash tool set to read the reports produced by the assessed model. Each judge then assigns a score for every checkpoint, and theCHK-score a c (Eq. 1) for each checkpoint is averaged over the three judges. We then aggregate the checkpoint-level CHK-scores to the range level. For each rangerand axisa β det,plan, the range-levelCHK-score a r is the mean value of the achieved CHK-score a c over the rangeβs checkpoints: CHK-score a r = β cβC a r CHK-score a c | C a r | Γ 100%,(3) whereC a r is the set of checkpoints in rangerfor axisa. We report the range-level CHK-score (Eq. 3) for range-specific analysis, and the CAP-score (Eq. 2) for range-aggregated analysis. 4.3 Range-Level Analysis across Attack Scenarios Through an in-depth analysis of the agent modelsβ performance across different attack scenarios as demonstrated in Table 4, we derive five primary research findings. Finding 1: Detection consistently outperforms planning, which is the bottleneck across all ranges. All the models achieve a higher range-level CHK-score on Detection than on Planning, as demonstrated in Figure 3 (a). For example, GPT-5.5 scores 70.7% on detection but only 36% on planning, while Claude Sonnet 4.5 presents the narrowest gap of 8.5%. This implies that for a large portion of checkpoints the agent models can successfully detect the problems from the disk snapshot, but fail to provide a comprehensive approach to fixing them. This gap is largely attributed to incomplete fixes instead of 8 Figure 3: (a) Detection consistently outperforms Planning across all models. The x-axis denotes the Detection score and the y-axis presents the Planning score. The dashed line marks βDetection=Planningβ. All models fall below the diagonal, demonstrating that detection always exceeds planning. (b) Version evolution within each model series. Most model families improve across versions, but the progress is not consistent. wrong actions. The agent models often advise correct commands initially, such as terminating a malicious process or removing a dropped file. However, in the follow-up steps, they fail to carry out the remaining remediation, such as rotating the leaked credentials, blocking the outbound channel, or confirming that the business service stays healthy after the cleanup. This suggests that proficiency in detection cannot ensure success in handling an incident thoroughly. In conclusion, current agents are better at finding attack traces or vulnerabilities than producing correct, complete, and verified remediation suggestions. Finding 2: Model performance varies widely, with only a few models leading consistently across attack scenarios and demonstrating superior forensic and incident-response capabilities. Claude Opus 4.7 achieves the highest average range-level CHK-score across ranges (79.0% for detection and 65.7% for planning), followed by Claude Opus 4.6 (78.2%/58.0%), GLM-5.1 (76.3%/59.2%), and Qwen3.7 Plus (75.6%/58.8%), whereas Gemini 3.1 Pro, MiniMax M2.5, and Gemini 3 Flash present the lowest results. Open models such as GLM and DeepSeek outperform the GPT and Gemini series, indicating that proprietary models do not guarantee stronger incident-response capabilities. Overall, performance on this task is highly uneven across models, making model selection an important factor in security incident-response applications. Finding 3: Performance generally improves as models evolve across versions, but the improvement is not universal. As shown in Figure 3 (b), several model families improve steadily across successive releases. For instance, the CHK-scores of the Claude Opus series increase across three releases, from 69.5%/56.4% for Opus 4.5 to 78.2%/58.0% for Opus 4.6 and 79.0%/65.7% for Opus 4.7. Similarly, GLM-5.1 surpasses GLM-5 by 11.4%/13.3% (76.3%/59.2% vs. 64.9%/45.9%), DeepSeek V4 Pro improves over DeepSeek V3.2 by 14.0%/9.8% (73.2%/53.4% vs. 59.2%/43.6%), and Qwen3.7 Plus outperforms Qwen3.6 Plus by 7.2%/10.5% (75.6%/58.8% vs. 68.4%/48.3%). However, this trend is not universal. For example, Kimi K2.6 falls below K2.5 on the planning dimension (43.0% vs. 50.7%), and Gemini 3.1 Pro also underperforms Gemini 3 Flash (54.8%/31.5% vs. 57.4%/32.9%). Finding 4: Range difficulty varies, and model performance declines as the attack chain grows longer and broader. No model achieves complete detection and remediation on any single range. The overall CHK-score across models on each range reveals a clear difficulty stratification that aligns with the structural complexity of each scenario. Agent models generally show strong performance on Log4j-RCE, Docker-Escape, and Redis-RCE ranges with a single entry point, a linear attack chain, and common persistence exposed in alerts such as cron jobs or systemd services. Log4j-RCE is the easiest range because the included CVE is well-known and already familiar to the models. Ranges like SSH-Miner, Next.js-RCE, and Jenkins-RCE additionally include a wider host baseline, multi-step privilege escalation, cross-service correlation and more diverse persistence, leading to a decrease in both detection and planning scores. The hardest ranges are Shiro-Fastjson, RDP-Service-Abuse, NPM-Worm, and ASP.NET-ViewState, which combine a broad attack surface, cross-service or cross-runtime correlation, and deliberately disguised persistence. The complex attack traces in these ranges make it difficult for agent models to uncover potential issues and especially challenging to propose comprehensive remediation methods. Finding 5: Agent models inherently struggle with proactive detection of silent intrusions and gen- erating complete, verified remediation plans. Across all ranges, all the agent models are typically confronted with two primary weaknesses: failure to uncover silent intrusions beyond the exposed attack 9 Table 5: TheCAP-score a of agent models and the traditional agentless scanning method across the five capability dimensions, namely Intrusion Entity (ENT), Persistence Mechanism (PER), Baseline Risk (BAS), Vulnerability Risk (VUL), and Investigation & Response Quality (Q). For each dimension, βDetectβ denotesCAP-score det , the percentage of achieved detection scores over all checkpoints mapped to that capability dimension (following the mapping in Table 6), averaged across three LLM judges; βPlanβ denotes CAP-score plan , the corresponding percentage on the planning axis. Model ENTPERBASVULQ Detect Plan Detect Plan Detect Plan Detect Plan Detect Plan Agentless43.6β2.1β20.7β50.0β Claude Opus 4.785.461.967.848.780.874.876.272.673.653.6 Claude Opus 4.686.058.172.145.878.667.679.263.368.834.4 Claude Opus 4.576.560.856.944.075.863.572.654.462.433.3 Claude Sonnet 4.681.662.758.744.984.269.879.2 67.866.734.4 Claude Sonnet 4.572.858.456.144.166.270.868.166.771.666.7 GPT-5.578.431.171.328.863.646.861.540.061.424.4 GPT-5.4 Pro78.338.376.4 35.273.551.961.545.664.035.5 GPT-5.469.929.247.624.461.846.453.337.855.215.5 GPT-5.2 Pro73.836.067.639.062.444.242.217.862.927.8 Gemini 3.1 Pro65.233.848.724.152.134.537.031.143.812.2 Gemini 3 Flash66.535.347.823.558.536.262.243.345.412.2 GLM-5.184.161.868.446.781.165.778.571.1 75.541.1 GLM-567.842.154.236.667.854.168.958.856.525.5 DeepSeek V4 Pro78.151.769.045.876.159.486.762.266.242.2 DeepSeek V3.269.142.947.230.157.750.069.654.554.836.7 Qwen3.7 Max84.152.479.046.780.963.762.247.869.132.2 Qwen3.7 Plus88.462.3 66.749.482.468.968.960.064.634.4 Qwen3.6 Plus77.147.157.532.274.762.654.852.358.024.5 Qwen3.5 Plus80.244.649.531.561.949.047.444.452.030.0 Kimi K2.672.643.957.933.966.748.655.541.155.726.7 Kimi K2.572.052.449.536.057.752.965.960.054.942.3 MiniMax M2.766.034.143.730.368.651.263.738.952.832.3 MiniMax M2.561.533.338.422.055.238.740.035.542.814.4 traces and to suggest thorough and verified remediation. This represents a common capability ceiling, with detailed evidence for each specific scenario provided in Appendix D.1. On the detection side, the agent models can find the issues directly reflected by the exposed alerts, but fail to discover the hidden problems beyond the obvious trace. Silent intrusions that leave no running process or network footprint are frequently overlooked, and the agent models rarely scan the whole host proactively. This shows that they tend to respond to the given clues instead of actively investigating the system. On the planning side, the agent models usually propose the most obvious fix rather than completing the comprehensive remediation. The remediation plans often stop at removing the visible artifact and leave the secondary risks or undetected persistence mechanisms untouched. More importantly, the models rarely recommend verifying whether the proposed fixes truly work or whether the normal services still run after the cleanup, leaving the remediation partial and unverified. 4.4 Capability-Level Analysis Following our capability taxonomy described in Β§ 3.2, we aggregate CHK-score into CAP-score using the mapping between range checkpoints and capability items (see Table 6). The aggregated results are shown in Table 5, from which we derive the following findings. Finding 1: Models reliably identify intrusion entities but struggle to uncover persistence mechanisms. As observed in Table 5, models are strongest at detecting Intrusion Entity (ENT), where most models exceed a 70% CAP-score. Leading models such as Qwen3.7 Plus and Claude Opus 4.6 achieve 88.4% and 86.0%, respectively. The strength in ENT can be attributed to the nature of malicious entities, which are concrete objects that trigger visible alerts. Models can discover them through routine inspection and by tracing the alerted trail. By contrast, detection collapses on Persistence Mechanism (PER), which has the lowest average detection CAP-score across models. This is because persistence mechanisms can be left in less obvious locations with no active signal, including cron jobs, systemd units, shell-init files, and platform-specific hooks. Uncovering them requires the model to enumerate the whole host rather than simply follow a visible trace. Thus, the ability to conduct a more proactive and systematic investigation represents one of the primary directions for improvement. Finding 2: Models remediate standardized risks effectively but perform weakly on intrusion entity and persistence mechanism cleanup. In terms of planning, models demonstrate superior CAP-scores 10 Figure 4: The CAP-score for each capability (ENT, PER, BAS, VUL, Q) of the representative models. In detection, Qwen3.7 Plus is strongest at entity identification, Claude Sonnet 4.6 at baseline assessment, Qwen3.7 Max at persistence detection, DeepSeek V4 Pro at vulnerability verification, and GLM-5.1 at investigation quality. In planning, Claude Opus 4.7 shows the strongest capabilities on average. when remediating Baseline Risk (BAS) or Vulnerability Risk (VUL), with Claude Opus 4.7 reaching 74.8% on BAS and 72.6% on VUL. These two dimensions are the easiest for remediation planning because their remediation is standardized, such as patching the affected component, disabling an exposed service, or rotating a default credential. Interestingly, models are strongest at detecting ENT but remediate it less successfully. This is because uncovering a malicious entity may require only a single observation, whereas cleaning it up involves a more complex, multi-step process. In particular, modelsβ capability to address PER remains relatively weak in remediation planning because the low detection score implies that many persistence mechanisms are never discovered and therefore cannot be properly remediated. Finding 3: Investigation & Response Quality is consistently among the weakest capabilities across models in both detection and planning, reflecting inadequate and unverified incident handling. The Investigation & Response Quality dimension validates process-level completeness and soundness of the whole incident response procedure, covering entry point localization, attack-chain reconstruction, honesty and confidence calibration, investigation thoroughness, remediation verification completeness, and business impact assessment. The CAP-scores on the quality dimension are low on both the detection and planning axes. Even the highest detection result reaches only 75.5%, falling behind other dimensions, and almost all models fall below 50% on planning except Claude Opus 4.7 and Claude Sonnet 4.5. From the detection perspective, even when models detect a concrete artifact, they still fail to reconstruct the full attack chain because of the challenge of correlating evidence across services and time. In terms of planning, models are inclined to address the immediate problem rather than handle the incident thoroughly, rarely verifying the effectiveness and completeness of the proposed remediation. Finding 4: Different models excel at different capabilities, but none of them dominates across all five dimensions. Figure 4 shows that the best detection model differs for every single dimension. Qwen3.7 Plus is strongest at entity identification with 88.4% on ENT, Qwen3.7 Max at persistence detection with 79.0% on PER, Claude Sonnet 4.6 at baseline assessment with 84.2% on BAS, DeepSeek V4 Pro at vulnerability verification with 86.7% on VUL, and GLM-5.1 at investigation quality with 75.5% on Q. In terms of planning, Claude models achieve the best CAP-score in four of the five dimensions, namely Opus 4.7 on BAS (74.8%) and VUL (72.6%), Sonnet 4.6 on ENT (62.7%), and Sonnet 4.5 on Q (66.7%), while Qwen3.7 Plus reaches the highest planning score of 49.4% on PER. This highlights that there is no universally best model and that model selection for incident response should be driven by the capabilities required for deployment rather than by a single overall ranking. Finding 5: Agents substantially outperform the traditional agentless detection baseline on average, and the gap is largest on detecting persistence mechanism. We also compare the agent models against an agentless baseline 1 used in the traditional incident-response workflow, which is a production scanner that inspects a hostβs disk snapshot without installing any runtime agent. Given the same host disk image, the scanner performs detection purely by static pattern matching driven by a predefined knowledge base, and reports four categories of findings after scanning. Concretely, it flags malicious samples by hash and signature matching against a malware database, identifies software vulnerabilities by 1 https://help.aliyun.com/zh/security-center/user-guide/use-the-agentless-detection-feature 11 Figure 5: Changes in the range-level Detection and Planning CHK-scores of representative models on each range after providing our designed skill. Each value denotes the difference between runs with and without the skill. Figure 6: The Detection and Planning CAP-scores across the five capability dimensions for four represen- tative models with and without our designed skill. enumerating installed packages and matching their versions against the CVE database, detects baseline risks by checking configuration entries against a fixed set of hardening rules, and finds sensitive files by pattern-matching for exposed credentials and keys. Unlike the agent models, agentless detection is detection-only, yielding a static findings list without providing remediation suggestions or reconstructing the attack. We map the agentless-detection findings to our capability dimensions and observe that the scanner reaches 43.6% on ENT and 50.0% on VUL, but only 20.7% on BAS and 2.1% on PER. Agents outperform this traditional security scanner on average across the four comparable dimensions. This is because the scanner reports only isolated static artifacts and fails to analyze persistence mechanisms, reconstruct attack chains, or attribute entry points. In contrast, agents can actively probe the host and connect scattered artifacts across files into an intrusion chain, thereby uncovering persistence mechanisms hidden in the system. Furthermore, unlike the scanner, agent models provide root-cause analyses and remediation plans. 12 Figure 7: (a) Inter-Judge Agreement. For each pair of judges, the value denotes the percentage of checkpoints on which both judges assign the same score, andΟis the Spearman rank correlation between their scores. (b) Each Judgeβs Difference from the Average. For each judge (row) and the representative model of each family (column), we compute the average difference between that judgeβs score and the mean score of the three judges across all checkpoints. 4.5 Augmenting Agents with Procedural Priors A typical failure mode is procedural: agents fail to follow a systematic investigation routine. For example, they follow the alerted trail instead of scanning the whole host, and the remediation plans often stop at the most obvious fix, which is reflected in their low scores on planning and on the Investigation & Response Quality dimension. We therefore distill the operational experience of the security incident-response team into procedural priors encoded as skills and provide them to the agents, allowing us to examine the performance upper bound that such priors alone can achieve. Notably, the designed skills contain only procedural priors rather than range-specific knowledge. Concretely, we encode the general routines by which experts investigate threats and organize fixes, together with report and remediation templates. The skills are distilled from a deployed incident-response product used for real customer cases and do not contain the ground truth of any range. The checklist items, expected findings, malicious file paths, and CVE lists are all excluded, ensuring that integrating such skills does not introduce knowledge leakage. As shown in Figure 5, our designed skills improve the agent modelsβ CHK-scores, particularly on the planning axis. The largest gains occur for models whose planning scores are initially low. For example, GPT-5.4 rises from 26% to 83% on the SSH-Miner CHK-score and increases by 43% on Docker-Escape, while GPT-5.4 Pro gains 45% on Jenkins-RCE and 52% on SSH-Miner. From the capability perspective in Figure 6, our skill can address the blind spots of strong models. For example, although Claude Opus 4.7 performs relatively weakly on persistence, the designed skill improves the PER CAP-score from 68% to 88% in Detection and from 49% to 75% in Planning. Similarly, the skill benefits GLM-5.1 most on the Investigation & Response Quality dimension. However, the skill does not ensure improvement. GLM-5.1 degrades by 12% in detection on Shiro-Fastjson and by 11% on Docker-Escape. This is probably because such broad-scope ranges include attacker traces beyond the skillβs enumerated categories, so the model stops at its boundary rather than continuing to explore and recover these long-tail traces. Overall, these results indicate that a procedural prior mitigates substantial failure, particularly on the planning axis, but cannot eliminate it. Closing this gap requires agents to learn from the outcome of their own investigation rather than executing a predefined routine, which is potentially the next step for this task. 4.6 LLM Judgment Analysis Agreement among LLM Judges. LLM-as-a-Judge can be biased toward assigning higher scores to longer or more structured answers (Zheng et al., 2023). To reduce this effect, we grade each checkpoint on applicable axes with three judges from three different developers, namely Gemini 3.1 Pro, Claude Opus 4.7, and GPT-5.4 Pro, using the same rubric, and compute their average as the final score. However, this average is reliable only if the three judges are consistent with one another and do not share the same bias. We assess this consistency in Figure 7 using all checkpoint scores from the three judges. As shown in Figure 7 (a), the three judges agree on most checkpoints. Each pair assigns the same score on 72% to 75% of all checkpoints, and their rankings are highly correlated (Ο =0.86β0.87). These results indicate that the rubric is interpreted consistently across the three judges and support the reliability of the final scores as measures of model capability. Moreover, Figure 7(b) shows that the main difference among the judges is strictness rather than a preference for specific models. The GPT judge scores the lowest, while the Claude and Gemini judges score slightly above the average. Since this tendency in strictness affects all models in the same way, averaging the scores from three judges can remove this influence, and 13 the model ranking is not affected. More importantly, no judge gives noticeably higher scores to its own model family, indicating that self-preference bias is small. Agreement with Human Experts. To validate the agreement between the LLM judges and human experts, we randomly select 60 checkpoints covering all 10 ranges and ask a security expert to grade 10 trajectories for each checkpoint under the same rubric without seeing the judge results. Detection is scored from 0 to 3, and planning is scored from 0 to 2. The human expert and LLM judges agree closely, with a Pearson correlation of 0.96, a quadratic-weighted Cohenβs kappa (ΞΊ) of 0.94, and a mean absolute error (MAE) of only 0.15. For 98% of the checkpoints, the scores from the human expert and the LLM judges differ by at most one point. Moreover, the LLM judges are slightly stricter than the human expert and are therefore less likely to overstate the evaluation results. 5 Conclusion In this paper, we introduce SECRESPOND, the first benchmark for evaluating LLM agents on the post- compromise incident-response workflow. Each task requires the agent to analyze a forensic disk snapshot of a compromised host together with the alerts, vulnerability scans, and baseline checks reported by a host security product, and deliver forensic reports on intrusions, baseline risks, and vulnerability risks, along with a remediation plan. We instantiate this task across 10 cyber ranges, each built from a distinct compromised cloud host, together covering 4 entry-point types, 21 ATT&CK techniques, and 5 operating systems. By evaluating 23 frontier LLMs on the OpenCode agent harness, we observe that current agents can reliably uncover the problems exposed by alerts but struggle to proactively investigate the disk for silent intrusions and to produce comprehensive, verified remediation plans, with no model achieving complete detection and remediation on any single range. This gap reveals a fundamental bottleneck in building agents for real-world incident response. We hope SECRESPOND provides a foundation for developing security agents that investigate and remediate incidents thoroughly rather than merely reacting to shallow alerts. Ethics Statement Third-Party Model Evaluation.The third-party models included in our evaluation were assessed solely for academic research. All reported results were obtained in a controlled experimental environment and are presented only for capability comparison and technical discussion. The inclusion of model and provider names and scores does not imply any official position or endorsement by the corresponding providers or brands. Neither outputs obtained from these services nor the resulting evaluation records are used to train, fine-tune, or distill any competing model. Security Research and Use Restrictions.All benchmark data were generated in isolated experimental environments under our authorization and control, rather than collected from production systems. Each cyber range is a host that we provisioned, compromised, and snapshotted for this purpose. The released forensic and security-product artifacts are synthetic and contain no data from customer, production, or real-world business systems. We sanitized every disk snapshot by replacing real credentials, keys, and personal data with placeholders, retaining only the forensic artifacts required for analysis. The attack techniques, CVEs, and exploitation methods represented in the scenarios are derived solely from public knowledge bases and vulnerability disclosures, such as MITRE ATT&CK and public vulnerability databases. The benchmark contains forensic evidence rather than runnable exploit chains; it includes no proprietary or internally developed advanced attack techniques and no incremental attack information beyond public sources, and it cannot be directly used to conduct attacks against real-world systems. The benchmark is intended exclusively for defensive security research. Any use for unauthorized intrusion, destructive testing, offensive activity, or any other purpose that violates applicable laws or regulations is strictly prohibited. References Anthropic. Introducing claude opus 4.7, 2026. URLhttps://w.anthropic.com/news/claude-opus-4-7. V Barres, H Dong, S Ray, X Si, and K Narasimhan.Ο 2 -bench: Evaluating conversational agents in a dual-control environment, 2025. URL https://arxiv. org/abs/2506.07982. Lauren Deason, Adam Bali, Ciprian Bejean, Diana Bolocan, James Crnkovich, Ioana Croitoru, Krishna Durai, Chase Midler, Calin Miron, David Molnar, et al. Cybersoceval: Benchmarking llms capabilities for malware analysis and threat intelligence reasoning. arXiv preprint arXiv:2509.20166, 2025. 14 Xiang Deng, Jeff Da, Edwin Pan, Yannis Yiming He, Charles Ide, Kanak Garg, Niklas Lauffer, Andrew Park, Nitin Pasari, Chetan Rane, et al. Swe-bench pro: Can ai agents solve long-horizon software engineering tasks? arXiv preprint arXiv:2509.16941, 2025. Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, et al. A survey on llm-as-a-judge. The Innovation, 7(6), 2026. Wei He, Yueqing Sun, Hongyan Hao, Xueyuan Hao, Zhikang Xia, Qi Gu, Chengcheng Han, Dengchang Zhao, Hui Su, Kefeng Zhang, et al. Vitabench: Benchmarking llm agents with versatile interactive tasks in real-world applications. arXiv preprint arXiv:2509.26490, 2025. Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues? In International Conference on Learning Representations, volume 2024, p. 54107β54157, 2024. Pengfei Jing, Mengyun Tang, Xiaorong Shi, Xing Zheng, Sen Nie, Shi Wu, Yong Yang, and Xiapu Luo. Secbench: A comprehensive multi-dimensional benchmarking dataset for llms in cybersecurity. arXiv preprint arXiv:2412.20787, 2024. Chenxin Li, Zhengyang Tang, Mingxin Huang, Yunlong Lin, Shijue Huang, Shengyuan Liu, Bowen Ye, Rang Li, Lei Li, Benyou Wang, et al. Claw-eval-live: A live agent benchmark for evolving real-world workflows. arXiv preprint arXiv:2604.28139, 2026a. Nathaniel Li, Alexander Pan, Anjali Gopal, Summer Yue, Daniel Berrios, Alice Gatti, Justin D Li, Ann- Kathrin Dombrowski, Shashwat Goel, Long Phan, et al. The wmdp benchmark: Measuring and reducing malicious use with unlearning. arXiv preprint arXiv:2403.03218, 2024. Xiangyi Li, Wenbo Chen, Yimin Liu, Shenghan Zheng, Xiaokun Chen, Yifeng He, Yubo Li, Bingran You, Haotian Shen, Jiankai Sun, et al. Skillsbench: Benchmarking how well agent skills work across diverse tasks. arXiv preprint arXiv:2602.12670, 2026b. Xihuan Lin, Jie Zhang, Gelei Deng, Tianzhe Liu, Tianwei Zhang, Qing Guo, and Riqing Chen. Ircopilot: Automated incident response with large language models. arXiv preprint arXiv:2505.20945, 2025. Mike A Merrill, Alexander G Shaw, Nicholas Carlini, Boxuan Li, Harsh Raj, Ivan Bercovich, Lin Shi, Jeong Yeon Shin, Thomas Walshe, E Kelly Buchanan, et al. Terminal-bench: Benchmarking agents on hard, realistic tasks in command line interfaces. arXiv preprint arXiv:2601.11868, 2026. Meta. Introducing autopatchbench: A benchmark for ai-powered security fixes, 2025. URLhttps: //engineering.fb.com/2025/04/29/ai-research/autopatchbench-benchmark-ai-powered-securit y-fixes/. OpenAI. Introducing gpt-5.4, 2026. URL https://openai.com/index/introducing-gpt-5-4. OpenAI. Introducing gpt-5.4, 2026. URL https://openai.com/index/introducing-gpt-5-4/. OpenCode. Opencode: The open source ai coding agent, 2026. URL https://opencode.ai/. Qwen. Qwen3.7: The agent frontier, 2026. URL https://qwen.ai/blog?id=qwen3.7. Minghao Shao, Sofija Jancheska, Meet Udeshi, Brendan Dolan-Gavitt, Haoran Xi, Kimberly Milner, Boyuan Chen, Max Yin, Siddharth Garg, Prashanth Krishnamurthy, et al. Nyu ctf bench: A scalable open-source benchmark dataset for evaluating llms in offensive security. Advances in Neural Information Processing Systems, 37:57472β57498, 2024. CocoaBench Team, Shibo Hao, Zhining Zhang, Zhiqi Liang, Tianyang Liu, Yuheng Zha, Qiyue Gao, Jixuan Chen, Zilong Wang, Zhoujun Cheng, et al. Cocoabench: Evaluating unified digital agents in the wild. arXiv preprint arXiv:2604.11201, 2026. The Gemini Team. Gemini 3.1 pro: A smarter model for your most complex tasks, 2026. URLhttps: //blog.google/innovation-and-ai/models-and-research/gemini-models/gemini-3-1-pro/. Shengye Wan, Cyrus Nikolaidis, Daniel Song, David Molnar, James Crnkovich, Jayson Grace, Manish Bhatt, Sahana Chennabasappa, Spencer Whitman, Stephanie Ding, et al. Cyberseceval 3: Advanc- ing the evaluation of cybersecurity risks and capabilities in large language models. arXiv preprint arXiv:2408.01605, 2024. Zhun Wang, Tianneng Shi, Jingxuan He, Matthew Cai, Jialin Zhang, and Dawn Song. Cybergym: Evaluating ai agentsβ cybersecurity capabilities with real-world vulnerabilities at scale. arXiv e-prints, p. arXivβ2506, 2025. 15 Wiz. Cyber model arena, 2026. URL https://w.wiz.io/cyber-model-arena. Yiran Wu, Mauricio Velazco, Andrew Zhao, Manuel RaΓΊl MelΓ©ndez LujΓ‘n, Srisuma Movva, Yogesh K Roy, Quang Nguyen, Roberto Rodriguez, Qingyun Wu, Michael Albada, et al. Excytin-bench: Evaluating llm agents on cyber threat investigation. arXiv preprint arXiv:2507.14201, 2025. Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh J Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, et al. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments. Advances in Neural Information Processing Systems, 37: 52040β52094, 2024. Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan.tau-bench: A benchmark for tool-agent-user interaction in real-world domains. arXiv preprint arXiv:2406.12045, 2024. Bowen Ye, Rang Li, Qibin Yang, Yuanxin Liu, Linli Yao, Hanglong Lv, Zhihui Xie, Chenxin An, Lei Li, Lingpeng Kong, et al. Claw-eval: Towards trustworthy evaluation of autonomous agents. arXiv preprint arXiv:2604.06132, 2026. Z.ai. Glm-5.1: Towards long-horizon tasks, 2026. URL https://z.ai/blog/glm-5.1. Andy K Zhang, Neil Perry, Riya Dulepet, Joey Ji, Celeste Menders, Justin Lin, Eliot Jones, Gashon Hussein, Samantha Liu, Donovan Jasper, et al. Cybench: A framework for evaluating cybersecurity capabilities and risks of language models. In International Conference on Learning Representations, volume 2025, p. 25094β25243, 2025. Yuxuan Zhang, Yubo Wang, Yipeng Zhu, Penghui Du, Junwen Miao, Xuan Lu, Wendong Xu, Yunzhuo Hao, Songcheng Cai, Xiaochen Wang, et al. Clawbench: Can ai agents complete everyday online tasks? arXiv preprint arXiv:2604.08523, 2026. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in neural information processing systems, 36:46595β46623, 2023. Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, et al. Webarena: A realistic web environment for building autonomous agents. In International Conference on Learning Representations, volume 2024, p. 15585β15606, 2024. Yuxuan Zhu, Antony Kellermann, Dylan Bowman, Philip Li, Akul Gupta, Adarsh Danda, Richard Fang, Conner Jensen, Eric Ihli, Jason Benn, et al. Cve-bench: a benchmark for ai agentsβ ability to exploit real-world web application vulnerabilities. arXiv preprint arXiv:2503.17332, 2025. 16 A Details on Range Construction SSH-Miner. Thessh-minerrange is built on a CentOS 7 host with weak SSH baseline settings and a weak root password. An external attacker brute-forces the root credential withhydra, drops a mining binary, installs three persistence mechanisms spanning scheduled tasks, shell initialization, and system services, and clears the authentication log. The agent receives this post-attack disk snapshot with (i) five alerts covering the brute-force success, miner execution, suspicious crontab, autorun change, and outbound miner connection, (i) an empty vulnerability list since there is no CVE-driven entry, and (i) baseline checks flagging the SSH, MySQL, and credential-strength baselines. With these inputs, the required reports are expected to cover (i) an intrusion report covering the full attack chain and intrusion entities such as the mining binary, hydra residue, log backup, and attacker IP, (i) a vulnerability report confirming the absence of a CVE-driven entry, (i) a baseline report flagging the SSH and credential baseline weaknesses, and (iv) a remediation plan prioritizing miner termination, persistence cleanup, SSH hardening, and credential rotation. ssh-miner/disk/ |-- usr/ |-- opt/ |-- ... |-- etc/ | |-- ssh/sshd_config | |-- sudoers.d/devops | |-- my.cnf | \-- systemd/system/ | |-- system-update-helper.service | \-- multi-user.target.wants/ | \-- system-update-helper.service |-- home/ | \-- devops/.bash_history |-- root/ | \-- .bashrc |-- tmp/ | \-- .X11-unix/ | |-- .xorg | \-- .xorg.log \-- var/ |-- spool/cron/root \-- w/html/index.html Redis-RCE. Theredis-rcerange is built on an Ubuntu 22.04 host that exposes an unauthenticated Redis server running as root to the public internet. The attacker abuses the Redis server to write an SSH key into the root account, logs in, opens a reverse shell, and deploys a cryptominer disguised as a Redis background-save process, with persistence mechanisms including the injected key, a malicious crontab, an SSH login hook, and a boot script, with the minerβs timestamp forged to resist timeline analysis. The agent receives this snapshot with (i) two alerts covering only the reverse shell and its outbound connection, (i) an empty vulnerability list since the entry is a misconfiguration rather than a CVE, and (i) no baseline findings, leaving the persistence silent on disk. With these inputs, the required reports are expected to cover (i) an intrusion report reconstructing the Redis-to-root chain and intrusion entities such as the miner, injected key, and malicious crontab, (i) a vulnerability report concluding the absence of a CVE-driven entry, (i) a baseline report flagging the Redis unauthorized-access and SSH root-login weaknesses, and (iv) a remediation plan prioritizing miner termination, persistence removal, and Redis and SSH hardening. redis-rce/disk/ |-- opt/ |-- tmp/ |-- usr/ |-- ... |-- etc/ | |-- init.d/redis-watchdog | |-- redis/redis.conf | |-- ssh/sshd_config | \-- sudoers.d/deploy |-- home/ | \-- deploy/.bash_history |-- root/ | \-- .ssh/ 17 | |-- authorized_keys | \-- rc `-- var/ |-- spool/cron/crontabs/root \-- tmp/ |-- .redis-bgsave \-- .redis-bgsave.log Docker-Escape.Thedocker-escaperange is built on an Ubuntu 20.04 host where Docker Remote API is exposed without authentication. The attacker reaches the API, launches a privileged container that mounts the host filesystem to escape to the host, injects an SSH key, and runs a cryptominer hidden as a Docker health tool, with persistence including a crontab, a systemd service, a shell-initialization hook, the injected key, and a long-running backdoor container. The agent receives this snapshot with (i) three alerts covering the mining process, the suspicious crontab, and the anomalous systemd service, (i) an empty vulnerability list since the exposed API is a misconfiguration, and (i) no baseline findings, so the API exposure, container escape, and backdoor container must be recovered from Docker events and host forensics. With these inputs, the required reports are expected to cover (i) an intrusion report reconstructing the API-to-host-takeover chain and entities such as the miner, the escape container, and the backdoor container, (i) a vulnerability report concluding no CVE-driven entry, (i) a baseline report flagging the exposed Docker API and excessive permissions, and (iv) a remediation plan prioritizing miner and backdoor-container removal, persistence cleanup, and firewalling Docker TCP access. docker-escape/disk/ |-- usr/ |-- home/ |-- tmp/ |-- ... |-- etc/systemd/system/ | |-- docker-health-agent.service | \-- docker.service.d/ | \-- override.conf |-- opt/ | |-- .docker/ | | |-- .health-monitor | | \-- .health-monitor.log | \-- app/ | \-- docker-compose.yml |-- var/spool/cron/crontabs/ | \-- root \-- root/ |-- .bashrc \-- .ssh/ \-- authorized_keys Jenkins-RCE. Thejenkins-rcerange is built on an Ubuntu 22.04 host running Jenkins whose au- thorization misconfiguration leaves the Groovy Script Console reachable without authentication. The attacker obtains unauthenticated remote code execution through the console, deploys a cryptominer mas- querading as a Jenkins agent, exfiltrates Jenkins credentials, and installs redundant persistence spanning a cron job, a rogue root-equivalent account, a SUID-root binary, a sudoers backdoor, a library-preload injection, and a Jenkins startup hook. The agent receives this snapshot with (i) five alerts covering the Script Console access, the masqueraded miner, the new privileged account, the cron backdoor, and the outbound mining connection, and (i) no explicit vulnerability or baseline checks. With these inputs, the required reports are expected to cover (i) an intrusion report reconstructing the unauthenticated RCE-to-mining chain and entities such as the miner, the exfiltrated secrets, and the multiple persistence implants, (i) a vulnerability report attributing the entry to a Jenkins authorization misconfiguration, (i) a baseline report flagging the anonymous-administrator setting and weak administration password, and (iv) a remediation plan prioritizing miner termination, removal of all persistence, credential rotation, and revoking anonymous permissions. jenkins-rce/disk/ |-- home/ |-- opt/ |-- tmp/ |-- usr/ |-- ... |-- etc/ 18 | |-- hosts | |-- environment | |-- passwd | |-- sudoers | \-- cron.d/jenkins-update |-- usr/local/bin/.update |-- var/ | |-- cache/jenkins/ | | |-- .update | | |-- .x.so | | \-- .j.tgz | \-- jenkins_home/ | |-- config.xml | |-- init.groovy.d/z-update.groovy | \-- secrets/master.key \-- root/.ssh/authorized_keys Shiro-Fastjson.Theshiro-fastjsonrange is built on a CentOS 8 host running a Java web application on Tomcat that uses a default key and a vulnerable Fastjson library. The attacker gains code execution through Shiro deserialization, drops two JSP webshells hidden among static assets, steals database credentials and dumps a user table, escalates to root by abusing a sudo rule without passwords, attempts to uninstall the security agent, and deploys a miner behind persistence including a systemd service, crontab, shell-initialization hook, injected key, sudoers backdoor, a MySQL UDF backdoor, anatjob, andrc.local. The agent receives this snapshot with (i) fifteen alerts spanning initial access, encoded- command execution, security-agent tampering, privilege escalation, and persistence, (i) an empty vulnerability list since the Shiro and Fastjson flaws are library-level components beyond the scanner, and (i) baseline findings flagging the operating system, Tomcat, and MySQL baselines and several weak-password checks. With these inputs, the required outputs are expected to include (i) an intrusion report reconstructing the Shiro-to-root chain and entities such as the two webshells, the miner, the database backdoor, and the data dump, (i) a vulnerability report identifying the vulnerable Shiro and Fastjson components, (i) a baseline report flagging the Tomcat, MySQL, and credential weaknesses, and (iv) a remediation plan prioritizing removal of the webshells, miner, and all persistence mechanisms, together with version upgrades and database lockdown. shiro-fastjson/disk/ |-- home/ |-- ... |-- etc/ | |-- my.cnf | |-- profile.d/java-env.sh | |-- rc.d/rc.local | |-- sudoers.d/ | | |-- 99-java-ops | | \-- tomcat | \-- systemd/system/java-app-monitor.service |-- opt/ | |-- .cache/ | | |-- .java-updater | | \-- .java-updater.log | |-- ruoyi/application.yml | \-- tomcat/webapps/ROOT/ | |-- WEB-INF/lib/ | | |-- fastjson-1.2.68.jar | | \-- shiro-core-1.7.0.jar | \-- static/ | |-- css/error.jsp | \-- js/analytics.jsp |-- root/.ssh/authorized_keys |-- tmp/.sql_dump |-- usr/lib64/mysql/plugin/lib_mysqludf_json.so \-- var/spool/cron/root Log4j-RCE. Thelog4j-rcerange is built on a CentOS 8 host whose web application bundles a vul- nerable Log4j library exploitable through a JNDI lookup (CVE-2021-44228). The attacker triggers the JNDI injection to launch a webshell, overwrites a deploy script invoked through sudo to escalate to root, deploys a miner, exfiltrates a database dump, and truncates the authentication log to cover its tracks, 19 with persistence including a systemd service, a crontab, an injected key, and a shell-initialization hook. The agent receives this snapshot with (i) six alerts covering the Log4j RCE, the webshell, the privilege escalation, the miner, the crontab backdoor, and the log tampering, (i) a vulnerability check reporting a CVE for the Log4j library, and (i) baseline findings flagging SSH root-login, weak passwords, and the sudo rule without password protection. With these inputs, the required reports are expected to cover (i) an intrusion report reconstructing the Log4j-to-root chain and entities such as the webshell, miner, data dump, and tampered deploy script, (i) a vulnerability report confirming the Log4j RCE and prescribing the upgrade, (i) a baseline report flagging the SSH and sudo weaknesses, and (iv) a remediation plan prioritizing the Log4j upgrade, webshell and miner removal, persistence cleanup, and credential rotation. log4j-rce/disk/ |-- bin/ |-- usr/ |-- lib/ |-- home/ |-- boot/ |-- sbin/ |-- ... |-- etc/ | |-- profile.d/java-env.sh | |-- ssh/sshd_config | |-- sudoers.d/app-deploy | \-- systemd/system/java-gc-helper.service |-- opt/webapp/ | |-- lib/ | | |-- log4j-api-2.14.1.jar | | \-- log4j-core-2.14.1.jar | |-- logs/app.log | |-- VulnWebApp.java | |-- deploy.sh | \-- webshell.py |-- var/ | |-- cache/.java-gc | |-- log/auth.log | \-- spool/cron/crontabs/root |-- tmp/ | |-- .db_dump | |-- .escalation_log | \-- .privesc_proof \-- root/.ssh/authorized_keys Next.js-RCE. Thenextjs-rcerange is built on an Ubuntu 22.04 host running a Next.js application vulnerable to a server-component deserialization RCE. The attacker obtains code execution as the application user, drops a Node.js webshell on a spoofed debugger port, escalates to root through a command injection in a SUID-root helper, installs anLD_PRELOADrootkit, runs a miner, and exfiltrates a database dump, with persistence including the loader-preload rootkit, a shell-initialization hook, a systemd service, a crontab, and an injected key. The agent receives this snapshot with (i) six alerts covering the miner, the suspicious Node child process, the webshell, the crontab, the systemd service, and the outbound mining connection, (i) an empty vulnerability list, requiring the agent to infer vulnerabilities from the disk snapshot, and (i) baseline findings flagging the SSH configuration, database exposure, a non-standard SUID binary, and a loose sudo rule. With these inputs, the required reports are expected to cover (i) an intrusion report reconstructing the RCE-to-rootkit chain and entities such as the malicious shared library, the miner, the webshell, and the data dump, (i) a vulnerability report identifying the framework vulnerability from the dependency manifest, (i) a baseline report flagging the SSH, database, and privilege-escalation weaknesses, and (iv) a remediation plan prioritizing rootkit and miner removal, persistence cleanup, framework upgrade, and SUID and sudo hardening. nextjs-rce/disk/ |-- bin/ |-- boot/ |-- lib/ |-- sbin/ |-- snap/ |-- srv/ |-- usr/lib/ |-- ... |-- etc/ 20 | |-- ld.so.preload | |-- profile.d/node-env.sh | |-- ssh/sshd_config | |-- sudoers.d/node-ops | |-- systemd/system/node-gc-helper.service | \-- postgresql/14/main/ | |-- pg_hba.conf | \-- postgresql.conf |-- opt/webapp/ | |-- .env | |-- package.json | |-- node_modules/next/package.json | \-- .next/static/chunks/debug.js |-- usr/ | |-- lib/x86_64-linux-gnu/.libnode_helper.so | \-- local/bin/backup-tool |-- var/ | |-- cache/.node-gc | |-- cache/.node-gc.log | \-- spool/cron/crontabs/root |-- tmp/ | |-- .pg_dump | |-- .escalation_log | \-- .privesc_proof \-- root/.ssh/authorized_keys NPM-Worm.Thenpm-wormrange is built on an Ubuntu 22.04 host running a Node.js application whose unauthenticated deployment webhook passes its parameter straight intonpm install, letting the attacker install a malicious package from the attacker host. The packageβs installed hook drops a worm that scans the internal network, harvests several classes of credentials, and injects itself into other projects to propagate, after which a leaked cloud access key is used to deploy a miner, with persistence including a crontab, shell and login initialization hooks, a systemd service, injected keys, and the infected project. The agent receives this snapshot with (i) two alerts covering the worm-download command and the web application spawning an abnormal child process, (i) an empty vulnerability list, and (i) baseline findings flagging the Ubuntu host baseline, a weak root password, the Nginx-exposed unauthenticated webhook, and several sensitive-configuration leaks such as a plaintext npm token, an unencrypted SSH key, and loose file permissions. With these inputs, the required reports are expected to cover (i) an intrusion report reconstructing the supply-chain-to-worm chain and entities such as the malicious package, the worm script, the stolen-credential cache, and the miner, (i) a vulnerability report attributing the entry to the unauthenticated webhook and unsigned package install, (i) a baseline report flagging the weak credential, exposed webhook, and sensitive configuration findings, and (iv) a remediation plan prioritizing worm and miner termination, package and propagation removal, credential rotation, webhook authentication, and disabling install scripts. npm-worm/disk/ |-- tmp/ |-- usr/ |-- ... |-- etc/ | |-- profile.d/node-env.sh | |-- sudoers.d/developer | \-- systemd/system/npm-cache-gc.service |-- home/ | \-- developer/ | |-- .bashrc | |-- .git-credentials | |-- .npmrc | \-- .ssh/ | |-- authorized_keys | \-- id_rsa |-- opt/ | |-- .node-helpers/ | | |-- .npm-gc | | \-- .npm-gc.log | |-- internal-tools/package.json | \-- webapp/ | |-- .env | |-- package-lock.json 21 | \-- node_modules/@corp-utils/logger/ | |-- package.json | \-- scripts/setup.js |-- root/ | \-- .ssh/authorized_keys \-- var/ |-- spool/cron/crontabs/developer \-- tmp/.npm-cache/ |-- scan_results.txt |-- sysinfo.json |-- npmrc/.npmrc |-- ssh_keys/id_rsa |-- env_files/.env |-- git_credentials/.git-credentials \-- bash_history/.bash_history ASP.NET-ViewState. Theaspnet-viewstaterange is built on a Windows Server 2019 host running an ASP.NET application whose backup configuration file is downloadable and leaks the hardcoded machineKey. The attacker forges a signed ViewState payload to execute code asSYSTEM, drops an ASPX webshell, dumps and parses LSASS to recover credentials, runs a C2 beacon masqueraded as an audit ser- vice, and clears event logs and shadow copies, with Windows-specific persistence spanning a scheduled task, a WMI event subscription, a sticky-keys/IFEO backdoor, a DLL search-order hijack, and a malicious MSSQL logon trigger. The agent receives this snapshot with (i) three alerts covering the webshell, the bea- con process, and the WMI subscription, (i) a vulnerability check reporting the ViewState deserialization with the backup exposure and application pool misconfigurations, and (i) baseline findings flagging the weak database password, the enabled command execution stored procedure, and the hardcoded machineKey. With these inputs, the required reports are expected to cover (i) an intrusion report recon- structing the ViewState-to-SYSTEMchain and entities such as the webshell, the credential-dump residue, the beacon, and the five persistence objects, (i) a vulnerability report confirming the ViewState RCE and the related misconfigurations, (i) a baseline report flagging the MSSQL andmachineKeyweaknesses, and (iv) a remediation plan prioritizing webshell and beacon removal, teardown of the Windows persistence, machineKey rotation, application-pool de-privileging, and credential rotation. aspnet-viewstate/disk/ |-- Windows/ | |-- System32/ | | |-- Tasks/Microsoft/Windows/Maintenance/AuditTask | | |-- wbem/Repository/OBJECTS.DATA | | \-- ... | \-- Temp/ | |-- lsass.dmp | |-- creds.txt | |-- plant-trigger.sql | \-- crm-app.tar.gz |-- Program Files/ | \-- Notepad++/ | |-- notepad++.exe | \-- version.dll |-- ProgramData/ | |-- WindowsAudit/ | | |-- WindowsAuditSvc.exe | | |-- last-beacon.log | | |-- schtasks-fired.log | | |-- wmi-fired.log | | \-- mssql-trigger-fired.log | \-- ... |-- ... \-- inetpub/wwwroot/CRM/ |-- web.config |-- web.config.bak \-- help.aspx RDP-Service-Abuse.Therdp-service-abuserange is built on a Windows Server 2019 host that exposes RDP to the public internet with a weakhelpdeskpassword. The attacker sprays the exposed RDP service, lands a logon as the low-privilegehelpdeskuser, and abuses a weak service DACL to rewrite the service bin path to a PowerShell payload and restart it, gaining execution asSYSTEM. Then, the attacker installs 22 persistence with a disguised service and a scheduled task, leaves credential dumps, and beacons to a C2 endpoint. The agent receives this snapshot with (i) seven alerts clustered on the post-exploitation stage covering the encoded-PowerShell execution, the LSASS-dump tooling, and the anomalous service registry write, (i) an empty vulnerability list since the entry is an RDP weak-credential and service- misconfiguration chain rather than a CVE, and (i) no baseline findings, leaving the RDP authentication timeline and the DACL only recoverable from the event logs in the disk snapshot. With these inputs, the required reports are expected to cover (i) an intrusion report reconstructing the RDP-spraying-to-SYSTEM chain and discovering the disguised service, scheduled task, credential dump, and C2 indicator, (i) a vulnerability report concluding the absence of a CVE-driven entry, (i) a baseline report flagging the helpdeskweak password and RDP exposure with the weak service DACL, and (iv) a remediation plan prioritizinghelpdeskcredential rotation, a more restrictive service DACL, removal of the persistence, the credential residue, and C2 blocking. rdp-service-abuse/disk/ |-- Windows/System32/ |-- ... |-- ProgramData/ | |-- CorpBackup/ | | |-- corp-backup-svc.ps1 | | \-- system-stage.ps1 | \-- WindowsHealth/ | |-- WindowsHealthSvc.ps1 | \-- last-beacon.log |-- Users/ | \-- helpdesk/NTUSER.DAT |-- Windows/Temp/ | |-- procdump.exe | |-- lsass.dmp | \-- creds.txt \-- Windows/System32/ |-- Tasks/Microsoft/Windows/Maintenance/HealthSync |-- config/ | |-- SYSTEM | \-- SAM \-- winevt/Logs/ |-- Security.evtx |-- System.evtx |-- Microsoft-Windows-TerminalServices-RemoteConnectionManager%4Operational.evtx \-- Microsoft-Windows-TerminalServices-LocalSessionManager%4Operational.evtx 23 B Details on Capability Taxonomy Table 6: The mapping between capability items and range checklist. Each cell lists the checkpoint indices (CHK-n) in that range mapped to the item. ItemCapability NameSSH-Miner Redis-RCE Docker-Escape Jenkins-RCE Shiro-Fastjson Log4j-RCE Next.js-RCE NPM-Worm ASP.NET-ViewState RDP-Service-Abuse Intrusion Entity (ENT) ENT-F01Webshell file1,2,71,43,363,41 ENT-F02Linux malware file2,322,324,6322,6 ENT-F03Malicious SO / kernel module4 ENT-F04Residual data file3,4495,8557,8,9516,17 ENT-F05Tampered-file restoration621 ENT-F06Windows malware file2,4,813,15 ENT-N01Attacker-IP inbound block20231238303863 ENT-N02Malicious-IP outbound block2324113931377 ENT-N03Malicious-domain block10714 ENT-P01Mining process11113211,5 ENT-P02C2 beacon process3 ENT-P03Windows malicious process3 Persistence Mechanism (PER) PER-A01authorized_keys tampering814923 PER-A02sudoers implant10 PER-A03Rogue local account4,22 PER-D01MySQL backdoor6 PER-D02MSSQL backdoor16,17 PER-E01profile.d / udev rule624 PER-E02WMI event subscription12,13,14 PER-H01ld.so.preload hijack4 PER-H02DLL hijacking8,15 PER-I01Shell init (bashrc/profile)121313201520 PER-I02/etc/environment injection7 PER-M01SUID/SGID backdoor5,23 PER-S01Cron task1116123,201913819,25 PER-S02systemd timer7 PER-S03at job21 PER-S04Windows scheduled task9,10,1110,11,12 PER-V01systemd service13,1414,1514,1517,181221,22 PER-V02init.d / rc.local22 PER-V03Windows service6,8,9 PER-W01 Nginx config tampering10 PER-W02 App-container config tampering8,21 Baseline Risk (BAS) BAS-01SSH hardening4β71097,811β1410,11,12 BAS-02DB / middleware access control95β91218,1921 BAS-03Credential safety1012814,2411,13,169,2215,1614,15,17,3621,224,18,19 BAS-04Privilege / service audit81179,1410,21185,20 BAS-05Container / orchestration config5,6,10,2117,2016 BAS-06Web-service config13151319,20 BAS-07Cloud-resource config2311 Vulnerability Risk (VUL) VUL-01Java component vuln26β28,35β376,11,2035 VUL-02Web-app code vuln18 VUL-03Container-escape vuln23 VUL-04Node.js / front-end framework vuln21,22,24,35 Investigation and Response Quality (Q) Q-01Entry-point localization182017,1825162528,294 Q-02Attack-chain reconstruction16,1918,21191529β32,3417,1926,2831,331,2,7 Q-03Attacker-info extraction16,1718,191632,33182730,343 Q-04Honesty & confidence calibration18201819253229 Q-05Investigation thoroughness15171623,243326 Q-06Cross-language/service tracing2927 Q-07Remediation-verification completeness21,2222,2420,2240β453439 Q-08Business-impact assessment17,183732 C Implementation Details C.1 Task Prompt for Linux Operating System # Detection Agent Workspace ## Your Role You are a security incident response analyst working on a forensic analysis. A compromised server's disk snapshot and security platform alert data have been provided. Your job is to analyze them and produce a comprehensive incident response report. ## Data Sources - **Forensic disk snapshot**:`DISK_PATH/` (read-only copy of the server filesystem) - **SAS alerts**:`SAS_PATH/alerts.json` - **SAS vulnerability scan**:`SAS_PATH/vulns.json` - **SAS baseline check**:`SAS_PATH/baselines.json` ## Output Write all outputs to:`OUTPUT_PATH/` Required files: 1.`progress.md` β Investigation timeline and checklist 2.`intrusion-report.md` β Full intrusion analysis with attack chain 3.`vuln-report.md` β Vulnerability verification results 4.`baseline-report.md` β Configuration baseline assessment 5.`remediation-plan.md` β Structured remediation plan (natural language, prioritized P0-P3) 24 ## Constraints - You are running on an analysis machine, NOT the compromised server - The disk snapshot at`DISK_PATH/` is a read-only forensic copy - Do NOT SSH into any machine during analysis - All findings must be based on disk evidence and SAS data - Do NOT fabricate CVEs or vulnerabilities β if unsure, say so with confidence levels - Preserve forensic evidence: recommend backup/quarantine over deletion in remediation-plan.md C.2 Task Prompt for Windows Operating System # Detection Agent Workspace ## Your Role You are a security incident response analyst working on a forensic analysis. A compromised server's disk snapshot and security platform alert data have been provided. Your job is to analyze them and produce a comprehensive incident response report. ## Data Sources - **Forensic disk snapshot**:`DISK_PATH/` β Windows filesystem dumped as a read-only directory tree on this Linux analysis machine - **SAS alerts**:`SAS_PATH/alerts.json` - **SAS vulnerability scan**:`SAS_PATH/vulns.json` - **SAS baseline check**:`SAS_PATH/baselines.json` ### Forensic Access The Windows disk is at`DISK_PATH/` as a directory tree on this Linux machine. Use`find`,`grep`,` cat`,`stat`,`strings` directly against this path. Linux equivalents of the Windows-forensic artifact classes: -`cat` /`grep` β scheduled tasks (XML, Action, RunAs, Trigger; decode any base64`powershell.exe -enc` payloads) -`strings -e l <hive> | grep` β Windows services (path + start account) -`strings -e l <hive> | grep` β Registry Run / RunOnce autostart -`strings -e l <evtx> | grep` and`strings <evtx> | grep` β full Event Log (focus EventID 1102 log clear, 4720 / 4722 account changes, 8224 VSS shadow delete) -`strings -e l <hive> | grep` β any hive key (Run, IFEO Debugger, AppInit, etc.) -`find` /`cat` /`grep` /`strings` β plugin-style artifacts (amcache.*, defender.*, firewall.* . . . ) -`strings <CIM repo> | grep` β WMI permanent subscriptions (`__EventFilter` /` ActiveScriptEventConsumer` /`__FilterToConsumerBinding`) ## Evaluation / Operations Infrastructure Inventory (for forensic dedup) If any of the following IPs / keys / actions appear in the victim host's logs or forensic evidence, **they are traces left by the evaluation platform itself, not attacker activity**. You must dedup them out when identifying the attacker, and **must not mistake infrastructure activity for attacker lateral movement**: - Keys in`authorized_keys` carrying the`BENCHMARK-CONTROL-DO-NOT-REMOVE` comment, and their associated login traces The attacker is the activity on disk that lies **outside this inventory**. Do NOT let`remediation- plan.md` recommend blocking this IP or deleting any key marked`BENCHMARK-CONTROL-DO-NOT-REMOVE`. ## Execution-Risk Responsibility When Writing remediation-plan.md A downstream remediation agent executes the plan you write as authoritative instructionsβ**every command you write is equivalent to authorizing an unattended agent to run it immediately on a production server**. Each remediation item must carry a`risk_class` (auth / network / account / service / data / runtime / none) and an`automation` level (auto / needs_approval / blocked). 25 Any command that would sever the current control plane (immediately expiring a password, stopping sshd, deleting a legitimate administrator key, etc.) **must NOT be written as a concrete command **βinstead, write it as a "risk description + recommended manual-intervention procedure". ## Output Write all outputs to:`OUTPUT_PATH/` Required files: 1.`progress.md` β Investigation timeline and checklist 2.`intrusion-report.md` β Full intrusion analysis with attack chain 3.`vuln-report.md` β Vulnerability verification results 4.`baseline-report.md` β Configuration baseline assessment 5.`remediation-plan.md` β Structured remediation plan (natural language, prioritized P0-P3) ## Constraints - You are running on an analysis machine, NOT the compromised server - The disk snapshot at`DISK_PATH/` is a read-only forensic copy - Do NOT SSH into any machine during analysis - All findings must be based on disk evidence and SAS data - Do NOT fabricate CVEs or vulnerabilities β if unsure, say so with confidence levels - Preserve forensic evidence: recommend backup/quarantine over deletion in remediation-plan.md C.3 Evaluation Prompt # Evaluation Agent Workspace ## Your Role You are a security benchmark evaluation agent for a **single range**. Score the detection agent's analysis reports against the ground-truth checklist for this range. ## Input - **Checklist** (scoring rubric):`CHECKLIST_PATH` - **Detection reports**:`DETECTION_REPORTS_PATH/` ## Methodology Follow the evaluation SKILL at:`EVALUATION_SKILL_PATH` ## How to Work ### Step 1: Read the Checklist Read the checklist thoroughly. Each CHK item defines: - **Detection score** (max 3): Discovery(1) + Evidence(1) + Attribution(1) -- scored by you (LLM-as-a -Judge) - **Plan score** (max 2): Correctness(1) + Completeness(1) -- scored by you (LLM-as-a-Judge) Items marked "detection only" have N/A for plan score. ### Step 2: Read All Reports Read all detection agent outputs. 1.`progress.md` β Investigation timeline and checklist 2.`intrusion-report.md` β Full intrusion analysis with attack chain 3.`vuln-report.md` β Vulnerability verification results 4.`baseline-report.md` β Configuration baseline assessment 5.`remediation-plan.md` β Structured remediation plan ### Step 3: Score Each CHK Item **Detection score (0-3):** (skip for plan-only items) - Discovery (0/1): Did the agent identify this issue? - Evidence (0/1): Did it cite specific evidence (paths, log lines, config values)? - Attribution (0/1): Did it correctly attribute the finding? **Plan score (0-2):** (skip for detection-only items) - Correctness (0/1): Does remediation-plan.md contain correct remediation steps for this issue? 26 - Completeness (0/1): Does the plan include verification, edge cases, side effects? ### Step 4: Write Output Write to:`OUTPUT_PATH/` 1. **evaluation-report.md**: Per-CHK scoring with rationale and evidence quotes. End with strengths/ weaknesses analysis. 2. **scores.json**: Structured JSON with per-checkpoint, scores and overall summary. ## Scoring Constraints - Be strict and objective. Only give credit for what is explicitly present in the reports. - Quote specific evidence when justifying scores. - Do NOT give credit for things the agent "probably checked but didn't report." - Note environment limitations (e.g., Docker lacking NET_ADMIN) in the report but still score as 0. C.4 Evaluation Skill # Security Evaluation SKILL ## Overview This SKILL drives the evaluation Agent to quantitatively assess the outputs of the detection Agent and the remediation Agent. Core mechanism: β Assess the quality of the analysis report and the remediation plan (judged item by item by the evaluation Agent) Each checkpoint (CHK) is split into two independent scoring dimensions, so that a problem can be pinpointed to either the analysis stage or the plan stage: | Dimension | What is scored | Scoring source | Scoring method | |------|--------|---------|---------| | **Detection score** | Whether the issue was correctly identified, backed by sufficient evidence, and correctly attributed | The detection Agent's analysis report | LLM-as-a-Judge | | **Plan score** | Whether the remediation plan is technically correct, complete in its steps, and mindful of verification and safety | The detection Agent's remediation plan | LLM-as-a-Judge | The two dimensions are **never merged into a single total**; they are always reported independently. **You only need to evaluate one range.** Output that range's item-by-item scores and structured data. ## Input Specification | Input | Description | Source | |------|------|------| | Checklist | The range's ground truth, containing the CHK to CAP mapping and per-item scoring criteria |`ranges/<range>/checklist.md` | | Analysis report | The report produced by the detection Agent (intrusion-report / vuln-report / baseline-report) | Detection Agent output | | Remediation plan | The structured remediation plan produced by the detection Agent | Detection Agent output`remediation-plan.md` | ## Scoring Process ### Step 1: Read the checklist Read through the checklist and understand, for each CHK item: - The detection pass criteria and the plan pass criteria - Whether it is a detection-only item (plan score marked N/A) or plan-only item (detection score marked N/A) ### Step 2: Read all reports Read the detection Agent's complete output (analysis report + remediation-plan.md). 27 ### Step 3: Score item by item Score each CHK item independently: #### Detection score (max 3 points) β LLM-as-a-Judge | Sub-item | Points | Criteria | |------|------|---------| | Discovery | 0/1 | Whether the report identifies the issue | | Evidence | 0/1 | Whether concrete evidence is cited (file paths, log lines, config values, process information, etc.) | | Attribution | 0/1 | Whether the attribution is correct (the finding is correctly linked to the attack chain / root cause) | Plan-only items are marked N/A and excluded from the detection score aggregation. Scoring constraints: - **Strictly based on the report's content**; award points only for what the report explicitly states - Do not award points for things that "may have been checked but were not written in the report" - Quote the report verbatim as the basis for the score #### Plan score (max 2 points) β LLM-as-a-Judge | Sub-item | Points | Criteria | |------|------|---------| | Correctness | 0/1 | Whether the remediation plan in remediation-plan.md or the report is technically correct | | Completeness | 0/1 | Whether the plan includes verification steps, edge-case handling, and consideration of side effects | Detection-only items (e.g., attack timeline reconstruction, entry-point determination) are marked N/A and excluded from the plan score aggregation. ### Step 4: Negative deductions The following cases require deductions from the detection score (annotate them in the evaluation report): | Negative rule | Deduction | Description | |---------|------|------| | Fabrication | Detection score set to 0 | The report invents non-existent CVEs, files, logs, etc. | | Wrong attribution | Attribution sub-score set to 0 | The finding is attributed to the wrong attack chain or root cause | | Dangerous advice | Plan score set to 0 | The plan contains operations that could cause data loss or service disruption without flagging the risk | ### Step 5: Determine the diagnostic mode Determine the overall diagnostic mode from the combination of dimension scores: | Mode | Detection | Plan | Meaning | Optimization direction | |------|------|------|------|---------| | All-round | Strong | Strong | Ideal state | Maintain | | Plan bottleneck | Strong | Weak | Finds the issue but cannot produce a correct plan | Improve the detection Agent's plan-generation ability | | Analysis blind spot | Weak | Weak | Core analysis capability is insufficient | Improve the detection Agent's foundational capability | | Blind fix | Weak | β | Happens to fix it correctly, but the report shows no analysis | Untrustworthy; the analysis pipeline needs strengthening | Strong/weak threshold: β₯70% is strong, <70% is weak. Annotate both the overall and the per-dimension diagnostic mode in the report. ## Output Specification ### 1. evaluation-report.md A detailed evaluation report in Markdown, containing: #### Per-item scoring table 28 One section per CHK item, including: - CHK ID, name - Detection score: discovery/evidence/attribution, 0-1 each, with the scoring rationale and verbatim quotes from the report - Plan score: correctness/completeness, 0-1 each, with the scoring rationale - Deductions (if any) #### Diagnostic analysis - Overall diagnostic mode - Per-dimension diagnostic modes - Most frequent point losses (Top N detection misses / Top N plan defects) - Improvement suggestions ### 2. scores.json Structured scoring data, for pipeline consumption and cross-run comparison: ```json "meta": "range": "ssh-miner", "run_id": "20260411-001", "model_under_test": "claude-opus-4-7", "evaluator_model": "claude-opus-4-7", "timestamp": "2026-04-11T10:00:00Z" , "checkpoints": [ "chk_id": "CHK-01", "name": "Mining process detection", "detection": "discovery": 1, "evidence": 1, "attribution": 1, "total": 3, "max": 3, "rationale": "..." , "plan": "correctness": 1, "completeness": 0, "total": 1, "max": 2, "rationale": "..." , "penalties": [] ], "summary": "total_checkpoints": 23, "detection_only_count": 6, "detection": "score": 60, "max": 69, "pct": 87.0 , "plan": "score": 28, "max": 34, "pct": 82.4 , "diagnosis": "..." ``` ## Scoring Constraints 29 1. **Strictly objective**: award points only for what the report explicitly states, quoting the original text as the basis for the judgment 2. **No speculation**: do not award points for things the Agent "may have checked but did not report" 3. **Dimensions stay independent**: the detection score and plan score are always reported separately , never merged into a single total D Supplementary Experimental Results D.1 Per-Range Performance Analysis SSH-Miner. The results for thessh-minerrange are shown in Table 7. Regarding detection, most models reliably recover the mining process, determine that the entry point is SSH brute force, and surface the SSH misconfigurations together with the persistence mechanisms. The primary weaknesses are failures to investigate silent attacks beyond the alerts, such as the brute-force tool residue (CHK-03) and thebashrcpersistence (CHK-12). Notably, the scores on the comprehensive persistence scan (CHK-15), the brute-force evidence recovery (CHK-16), and the restart-and-verify confirmation (CHK-22) are all near zero. In terms of planning, models can successfully harden the SSH baseline, revoke the entry, and disable remote MySQL root access, as evidenced by high planning scores on CHK-04, CHK-05, CHK-08, and CHK-09. However, the cleanup remains incomplete: models fail to close thebashrcbackdoor (CHK-12), block the attacker IP (CHK-20), verify the SSH hardening (CHK-21), or restart the services (CHK-22). Redis-RCE. The results for theredis-rcerange are shown in Table 8. For detection, most models successfully identify the Redis baseline misconfigurations, including the exposed bind address, the missingrequirepasssetting, disabled protected mode, unrestricted dangerous commands, the injected SSH public key, and the malicious startup service. They also correctly attribute the intrusion to unau- thorized Redis access and distinguish it from SSH brute force or a CVE exploit. However, most models overlook artifacts that coincide with the attack but are not part of the Redis-generated attack trace, such as CHK-08, CHK-11, CHK-12, and CHK-17. This indicates that models are inclined to discover artifacts directly produced by the attack but overlook other host-wide baseline issues. On the planning axis, models partially harden the Redis baseline and clean up the explicit persistence, achieving high success rates in removing the injected SSH key and the malicious startup service. Nevertheless, their Redis hardening remains incomplete, often missing the change from the root run-as user (CHK-08) and the final verification of the applied hardening (CHK-22). Docker-Escape. The results for thedocker-escaperange are shown in Table 9. Most models correctly detect the surface of the chain, flagging the mining process (CHK-01), the backdoor container (CHK- 03), the unauthenticated Docker TCP API exposure (CHK-05), and the host-level persistence (CHK-12, CHK-13, CHK-14). However, nearly every model scores close to zero on the proactive comprehensive persistence scan (CHK-16), showing that they recover only the specific persistence artifacts along the alerted trail and do not sweep the host thoroughly. Moreover, several models, including models in the Gemini series, fail to detect the full escape mechanism because they do not recognize that the developer user βs membership in the Docker group is a root-equivalent escape primitive (CHK-07). On the planning dimension, models can harden and securely restart the Docker daemon (CHK-20), but they almost never verify that the legitimate business container survives the cleanup (CHK-22), and they leave the Docker-group escape (CHK-07) and the miner βs outbound channel (CHK-24) open. This risks disrupting production while the escape route remains intact. Jenkins-RCE. The results for thejenkins-rcerange are shown in Table 10. On the detection side, most models reliably identify the visible attacks from alerts. They successfully flag the explicit intrusion entities, such as the mining process and its mining-pool outbound connection, the anonymous entry point misconfiguration, and the conspicuous persistence artifacts such as the cron entry and the rogue account. Stronger proprietary models additionally reconstruct the full process chain from the Jenkins application layer and the mining process back to the Groovy payload. The weaknesses are concentrated in three aspects. First, inconspicuous persistence is widely missed, including the SUID-root backdoor, theLD_PRELOADinjection, and the sudoers tampering. This indicates that the models are inclined to follow the alerted attack trail rather than proactively examine the system state. Second, the silent configuration weakness, such as the weak Jenkins admin password, is rarely recovered. Third, models report the direct cause of the incident but hardly assess its collateral effects, failing to flag that revoking anonymous Administer also severs any legitimate unauthenticated access path, which makes business- impact assessment among the weakest dimensions. On the planning side, models commonly perform well on stopping the miner, deleting the cron entry and rogue account, blocking the attacker and mining- pool IPs, and revoking anonymous administrative access. However, the models fail to remediate the 30 same inconspicuous persistence that detection misses. In addition, the remediation plans rarely reach a complete and verified state, as shown by the low planning scores on CHK-10, CHK-21, and CHK-23. Shiro-Fastjson. The results for theshiro-fastjsonrange are shown in Table 11. Detection on this range shows a high success rate. Most models recover both JSP webshells, the mining process and binary, and the OS-level persistence. They also reconstruct the root cause in depth, attributing the entry to Shiro deserialization (CHK-25), tracing the Java process chain (CHK-29), and rebuilding the sudo privilege-escalation path and the attack timeline (CHK-30, CHK-31). However, two common weaknesses can be observed. First, the database- and service-layer attack surface is largely missed, as reflected by failures on CHK-06, CHK-12, and CHK-15. This suggests that models inspect the host filesystem and process state but not the database engine or service network exposure. Second, although models identify the headline Shiro and Fastjson vulnerabilities (CHK-26, CHK-27), few trace the deserialization down to the underlying program that makes it exploitable (CHK-28). On the planning side, models can propose remediation plans for the explicit entities and baselines, with high scores on stopping the miner, reducing Tomcatβs privileges, and rotating the database credentials. However, the database-resident UDF persistence is left untouched, and the plans rarely reach a verified state (CHK-41, CHK-43, CHK-45). Log4j-RCE. The results for thelog4j-rcerange are shown in Table 12. Detection on this range is broadly strong, and most models reliably discover the active intrusion entities and the Log4j vulnerability, indicating that the models have sufficient knowledge of this CVE. However, most models still demonstrate weaknesses in two scenarios. First, latent login-triggered persistence is a systematic blind spot: the profile.dbackdoor (CHK-15) and the injected SSH key (CHK-14) leave no trace in the running process or network state and are missed by every model except the GPT series. Second, exfiltrated data on disk and the JNDI remote-class-loading misconfiguration in the application source are identified by only a few models, with most models earning partial credit. In terms of planning, models perform reliably on explicit baseline-hardening tasks such as fixing the SSH configuration, assessing the vulnerability, and upgrading Log4j. However, they fail to suggest complete fixes or cleanups, performing poorly on the source-code repair and its recompile-and-restart steps (CHK-11), the cleanup of undetected login-triggered persistence (CHK-14, CHK-15), and the full removal of the mining payload (CHK-03). Next.js-RCE.The results for thenextjs-rcerange are shown in Table 13. For detection, models inspect the web application thoroughly but rarely pivot to the surrounding host. They can flag the disguised miner and hidden webshell and trace the entry point to the Next.js RSC deserialization RCE. However, most models ignore artifacts outside the web applicationβs own directory, such as CHK-04, CHK-05, CHK-06, and CHK-09, which are recovered only by a few models, including GLM-5.1 and Claude Opus 4.6. Furthermore, few models trace the RSC flaw back to its React root (CHK-22). On the planning dimension, the remediation plan covers the application baseline, but the undetected host persistence is left in place (CHK-04, CHK-06, CHK-09), the leaked secrets are not fully rotated (CHK-20), and few models recommend replaying the RSC payload to confirm that the endpoint rejects it after the Next.js upgrade (CHK-24). The headline vulnerability is therefore patched but never verified. NPM-Worm. The results for thenpm-wormrange are shown in Table 14. On detection, most models reliably identify the active intrusion entities and the entry point, including the worm process for CHK-01, the malicious npm package for CHK-04, the JS dropper for CHK-03, the unauthenticated webhook endpoint for CHK-13, and the attacker IP for CHK-30. Notably, the runtime attack chain is consistently investigated despite spanning two language runtimes (CHK-27), and the attack timeline is broadly reconstructed (CHK-33). Overall detection coverage on this range, however, is lower than on other ranges. Models collapse on the wormβs disguised persistence, namely the masqueraded systemd service, theprofile.dbackdoor, and the comprehensive persistence sweep. This indicates that current models remain limited under a wider attack surface and the wormβs deliberate use of disguise. On the planning dimension, models propose reliable remediation only for the explicit entry point, with high success rates in uninstalling the malicious npm package for CHK-04 and hardening the webhook endpoint for CHK-13. The cleanup, however, remains incomplete. Agents still fail to remove the worm script and mining binary or eradicate the wormβs lateral infection and disguised persistence. Moreover, most models broadly recommend rotating leaked credentials without specifying how to handle each credential category. ASP.NET-ViewState. The results for theaspnet-viewstaterange are shown in Table 15. For this Windows range, detection performance differs across artifact types. Models typically recover the visible IIS-layer surface and attribute the correct entry point. However, Windows-specific stealth mechanisms are challenging to discover. The side-loading hijack is missed by almost all models (CHK-08, CHK-15), and the encoded payloads (CHK-09/10/11) are recovered only occasionally. This indicates that recovering such artifacts depends on whether a model proactively inspects the host rather than following the alerted ViewState trail. On the planning dimension, models successfully fix the headline vulnerability, rotate 31 themachineKey, repair the misconfigurations, and rotate the SQL credential. Nevertheless, the MSSQL logon trigger andxp_cmdshellbackdoor are detected but never cleaned up, with planning scores close to zero on CHK-16 and CHK-17. In addition, the attacker IP is rarely blocked (CHK-06), and the undetected scheduled-task persistence is left untouched. RDP-Service-Abuse.The results for therdp-service-abuserange are shown in Table 16. The obvious post-compromise footprint is recovered almost universally, whereas the initial-access and privilege-abuse path that defines the scenario is largely overlooked. Almost every model reconstructs the SYSTEM payload execution chain (CHK-07) and flags the disguised service (CHK-08), the scheduled task (CHK-10, CHK-11), the credential dump (CHK-15/16/17), and the C2 indicators (CHK-14). In contrast, the entry point is missed by nearly all models as evidenced by CHK-01, CHK-05, and CHK-06. On the planning dimension, models can remediate the parts of the chain that they detect, such as CHK-09, CHK-12, and CHK-18. However, remediation of the unseen entry vector is similarly poor. Only Claude Opus 4.6 and Claude Sonnet 4.6 propose fixing the weakhelpdeskpassword (CHK-19) and hardening the DACL (CHK-20). Most agents finish with the persistence cleaned up but leave the exploitable service and its weak credential unchanged. 32 Table 7: Per-model checkpoint scores forssh-miner, averaged over three LLM judges (Detection: 0β3; Planning: 0β2). Model CHK-01 Mining process detection CHK-02 Mining binary cleanup CHK-03 Hydra/ attack-tool residue CHK-04 SSH Permit RootLogin CHK-05 SSH Password Authentication CHK-06 SSH MaxAuth Tries CHK-07 Root password strength CHK-08 devops sudoers NOPASSWD DetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlan Claude Opus 4.73.002.003.002.000.001.003.002.003.002.003.002.002.331.673.002.00 Claude Opus 4.63.001.673.001.330.000.003.001.333.002.003.001.003.002.003.001.00 Claude Opus 4.53.002.003.001.000.000.003.001.333.002.003.001.000.332.003.001.00 Claude Sonnet 4.63.002.003.001.000.000.003.001.333.002.003.001.003.002.003.001.00 Claude Sonnet 4.53.002.003.001.670.000.003.002.003.002.003.002.003.002.003.002.00 GPT-5.53.000.333.000.330.000.003.001.003.001.333.001.001.671.333.001.00 GPT-5.4 Pro3.000.333.000.330.000.003.001.003.001.673.001.000.330.672.331.00 GPT-5.43.000.333.000.670.000.003.001.003.001.003.001.000.000.673.001.00 GPT-5.2 Pro2.330.333.000.330.000.003.001.003.001.003.001.000.330.333.001.00 Gemini 3.1 Pro3.001.673.001.000.000.003.001.003.001.003.001.002.331.673.001.00 Gemini 3 Flash2.671.332.670.670.000.003.001.003.001.003.001.002.331.003.001.00 GLM-5.13.002.003.001.000.000.003.001.333.002.003.001.003.002.003.002.00 GLM-53.001.003.001.330.000.003.001.673.002.003.001.333.002.003.002.00 DeepSeek V4 Pro3.002.003.001.000.000.003.002.003.002.003.002.003.002.003.001.00 DeepSeek V3.23.001.673.002.000.000.333.001.003.001.003.001.001.672.003.002.00 Qwen3.7 Max3.002.003.001.000.330.003.001.333.001.333.001.002.332.003.001.00 Qwen3.7 Plus3.000.673.000.330.000.003.001.003.001.333.001.003.002.003.001.00 Qwen3.6 Plus3.002.003.002.000.000.003.001.333.002.003.001.002.002.003.002.00 Qwen3.5 Plus3.000.333.001.000.000.003.002.003.002.003.002.000.330.673.002.00 Kimi K2.62.671.673.001.000.000.003.001.003.001.003.001.002.001.673.001.00 Kimi K2.53.001.673.000.670.670.333.002.003.001.673.002.002.671.673.002.00 MiniMax M2.73.000.333.001.330.000.003.001.003.002.003.001.002.672.003.001.00 MiniMax M2.53.001.333.000.670.000.003.001.003.001.003.001.002.671.673.001.00 Model CHK-09 MySQL remote root access CHK-10 MySQL creds in bash_history CHK-11 Crontab persistence CHK-12 Bashrc persistence CHK-13 Systemd service persistence CHK-14 Systemd service mas- querade ID CHK-15 Comprehensive persistence scan CHK-16 Brute-force evidence recovery DetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlan Claude Opus 4.73.002.003.002.003.002.003.002.003.002.003.00β0.67β0.00β Claude Opus 4.63.001.333.001.003.001.000.000.003.001.003.00β0.00β0.33β Claude Opus 4.53.002.003.001.673.001.330.000.003.002.003.00β0.67β0.33β Claude Sonnet 4.63.001.333.000.333.001.000.000.003.000.673.00β0.00β0.33β Claude Sonnet 4.53.002.000.000.003.001.330.000.003.002.003.00β1.00β0.67β GPT-5.52.001.000.000.003.000.673.000.673.000.332.67β0.33β0.00β GPT-5.4 Pro0.330.332.670.333.001.003.001.003.000.332.67β0.00β0.33β GPT-5.41.000.330.000.003.000.332.671.673.000.330.33β0.00β0.00β GPT-5.2 Pro2.330.332.330.003.000.672.330.673.000.330.33β0.00β0.33β Gemini 3.1 Pro2.671.000.000.003.001.003.001.003.001.672.67β0.00β0.33β Gemini 3 Flash3.000.332.670.003.001.000.000.003.000.330.67β0.00β0.00β GLM-5.13.002.003.002.003.001.000.000.003.002.003.00β0.00β0.00β GLM-53.001.673.001.003.001.330.000.003.001.673.00β0.00β0.33β DeepSeek V4 Pro3.001.672.671.003.000.673.001.003.001.673.00β0.00β0.00β DeepSeek V3.22.002.000.000.003.001.000.000.003.001.001.33β0.33β0.33β Qwen3.7 Max3.001.333.000.003.002.003.002.003.001.003.00β0.67β0.00β Qwen3.7 Plus3.001.003.001.003.001.000.000.003.001.332.33β0.00β0.00β Qwen3.6 Plus3.001.673.002.003.001.330.000.003.001.332.33β0.00β0.00β Qwen3.5 Plus3.002.001.670.003.001.000.000.003.002.000.33β0.00β0.33β Kimi K2.63.001.003.000.003.001.000.000.003.001.330.67β0.00β0.33β Kimi K2.53.002.000.000.003.001.330.000.003.002.003.00β0.33β1.00β MiniMax M2.73.001.003.000.673.000.670.000.003.001.331.33β0.00β0.00β MiniMax M2.53.001.000.000.003.001.000.000.003.001.670.00β0.00β0.33β Model CHK-17 Attacker-IP identification CHK-18 Entry-point determination CHK-19 Attack- timeline reconstruction CHK-20 Attacker-IP firewall block CHK-21 SSH hardening verification CHK-22 Service restart and verification CHK-23 Mining outbound- connection block DetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlan Claude Opus 4.72.33β3.00β1.67β1.000.003.002.000.670.333.001.33 Claude Opus 4.62.67β3.00β2.33β3.001.001.330.000.330.003.000.67 Claude Opus 4.52.67β3.00β2.67β3.002.001.330.330.330.332.671.00 Claude Sonnet 4.62.33β3.00β2.33β3.000.332.330.670.330.002.670.33 Claude Sonnet 4.52.67β3.00β2.33β3.002.003.002.002.001.003.002.00 GPT-5.52.67β3.00β2.33β1.330.000.330.000.330.003.000.33 GPT-5.4 Pro2.33β3.00β2.67β0.670.000.670.000.330.002.670.33 GPT-5.43.00β3.00β2.67β0.000.000.000.000.000.003.000.33 GPT-5.2 Pro2.67β3.00β2.00β0.330.000.330.000.000.002.670.33 Gemini 3.1 Pro2.67β3.00β2.33β0.330.000.330.000.000.002.330.00 Gemini 3 Flash2.33β3.00β1.33β0.670.000.000.000.000.001.000.00 GLM-5.12.67β3.00β2.33β1.670.002.330.671.000.673.000.33 GLM-53.00β2.67β2.00β3.001.332.331.000.000.003.000.00 DeepSeek V4 Pro2.67β3.00β2.33β0.000.003.002.000.000.003.001.00 DeepSeek V3.22.67β3.00β2.00β0.330.670.670.670.670.673.001.00 Qwen3.7 Max2.67β3.00β3.00β0.330.000.670.330.330.003.000.67 Qwen3.7 Plus2.33β2.67β2.33β3.000.330.000.000.000.003.000.33 Qwen3.6 Plus2.33β3.00β2.67β1.330.001.670.000.330.003.000.00 Qwen3.5 Plus2.00β3.00β2.33β2.330.003.001.330.330.332.670.67 Kimi K2.62.67β3.00β2.33β0.330.000.330.000.000.000.330.00 Kimi K2.53.00β3.00β2.33β1.000.672.672.001.000.671.670.33 MiniMax M2.73.00β3.00β2.33β0.000.002.000.000.330.003.000.00 MiniMax M2.52.67β3.00β2.00β3.001.000.000.000.000.003.000.67 33 Table 8: Per-model checkpoint scores forredis-rce, averaged over three LLM judges (Detection: 0β3; Planning: 0β2). Model CHK-01 Mining process detection CHK-02 Mining binary cleanup CHK-03 Attacker SSH public-key identification CHK-04 Redis-written crontab residue CHK-05 Redis bind configuration CHK-06 Redis require pass CHK-07 Redis protected mode CHK-08 Redis run-as user DetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlan Claude Opus 4.71.331.672.001.333.002.003.001.333.002.003.001.673.002.003.002.00 Claude Opus 4.63.000.673.000.673.002.003.002.003.001.673.001.333.002.001.331.67 Claude Opus 4.51.332.002.001.003.002.001.001.333.001.003.001.003.002.000.000.00 Claude Sonnet 4.61.672.002.001.333.002.003.001.673.002.003.002.003.002.001.330.00 Claude Sonnet 4.51.330.672.331.333.001.673.001.673.002.003.001.673.002.003.002.00 GPT-5.51.330.672.330.673.002.003.001.673.001.333.001.333.001.671.001.00 GPT-5.4 Pro1.000.001.330.003.001.673.001.003.001.673.001.003.002.003.002.00 GPT-5.40.330.002.001.003.002.003.001.003.001.333.001.003.001.331.331.00 GPT-5.2 Pro1.330.001.330.003.001.671.000.003.001.673.001.673.002.000.001.00 Gemini 3.1 Pro1.001.671.671.003.001.673.001.003.001.673.001.003.001.330.331.00 Gemini 3 Flash1.331.002.331.003.001.003.000.673.001.673.001.003.001.000.000.00 GLM-5.12.002.002.671.673.002.003.002.003.001.333.001.003.002.000.330.00 GLM-51.331.672.331.332.331.002.000.333.002.003.001.673.002.000.000.00 DeepSeek V4 Pro1.332.002.331.333.002.000.000.003.001.333.001.333.001.670.000.00 DeepSeek V3.21.330.002.001.003.002.003.001.673.001.673.001.003.001.332.331.00 Qwen3.7 Max3.002.003.001.003.002.003.002.003.002.003.002.003.002.000.000.00 Qwen3.7 Plus3.002.003.001.003.002.003.001.673.001.673.001.673.001.672.672.00 Qwen3.6 Plus1.671.002.331.333.002.000.000.003.002.003.001.673.002.000.000.00 Qwen3.5 Plus2.672.002.671.002.672.002.671.333.002.003.002.003.001.330.000.00 Kimi K2.60.000.001.670.003.001.003.000.673.002.003.001.003.001.330.000.00 Kimi K2.51.672.002.001.003.002.003.002.003.001.002.671.673.001.670.000.00 MiniMax M2.70.670.332.001.002.671.672.671.003.001.333.001.333.002.002.330.33 MiniMax M2.50.672.002.001.002.671.001.330.333.001.003.001.003.001.000.330.00 Model CHK-09 Redis rename command CHK-10 SSH Permit RootLogin CHK-11 deploy sudoers NOPASSWD CHK-12 Redis session creds in bash_history CHK-13 SSH .ssh/rc persistence CHK-14 init.d malicious script CHK-15 init.d service masquerade CHK-16 Redis crontab file signature ID DetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlan Claude Opus 4.73.002.003.001.330.000.003.001.333.001.333.001.672.67β3.00β Claude Opus 4.63.001.673.001.000.000.000.000.003.000.673.001.673.00β3.00β Claude Opus 4.53.001.003.001.000.000.003.000.003.001.003.002.002.67β0.00β Claude Sonnet 4.63.001.673.001.000.000.000.000.003.002.003.001.672.33β3.00β Claude Sonnet 4.53.002.002.332.000.000.003.000.673.001.673.002.002.67β3.00β GPT-5.53.001.673.001.000.000.003.000.333.001.333.001.002.67β3.00β GPT-5.4 Pro3.001.333.001.000.000.002.001.333.001.003.001.002.33β3.00β GPT-5.43.001.002.001.000.000.000.000.000.000.000.000.000.00β3.00β GPT-5.2 Pro3.001.333.001.000.000.000.000.003.001.003.001.330.00β1.00β Gemini 3.1 Pro0.000.000.000.000.000.000.000.000.000.003.001.002.67β3.00β Gemini 3 Flash2.331.332.331.000.000.000.000.000.000.003.001.002.00β3.00β GLM-5.13.001.673.001.000.000.003.000.003.001.003.002.002.33β3.00β GLM-53.001.333.001.000.000.000.000.002.671.003.002.000.00β0.67β DeepSeek V4 Pro3.002.003.001.000.000.000.000.003.001.003.002.002.33β0.00β DeepSeek V3.23.001.672.331.000.000.000.000.002.001.000.000.000.00β3.00β Qwen3.7 Max3.001.673.001.333.002.003.001.003.001.003.002.002.67β3.00β Qwen3.7 Plus3.001.673.001.673.001.000.000.002.671.003.002.002.67β3.00β Qwen3.6 Plus3.001.673.001.000.000.003.000.673.001.003.002.002.67β0.00β Qwen3.5 Plus3.002.002.671.000.000.000.000.000.000.003.002.000.33β0.67β Kimi K2.63.001.000.000.000.000.000.000.002.671.003.000.331.33β3.00β Kimi K2.53.001.673.002.000.000.000.000.003.001.003.001.672.00β3.00β MiniMax M2.73.001.673.001.330.000.000.000.000.000.003.002.000.33β1.00β MiniMax M2.53.000.002.671.000.000.000.000.000.000.003.001.002.33β1.33β Model CHK-17 Compre- hensive persistence scan CHK-18 Redis unauthorized access evidence CHK-19 Attacker-IP identification CHK-20 Entry-point determination CHK-21 Attack-timeline reconstruction CHK-22 Redis hardening verification CHK-23 Port access restriction CHK-24 Service restart and verification DetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlan Claude Opus 4.71.67β3.00β2.67β3.00β2.67β1.000.673.001.000.330.33 Claude Opus 4.60.00β3.00β2.67β2.67β3.00β1.671.333.002.001.000.33 Claude Opus 4.50.00β2.67β2.67β2.67β3.00β2.331.333.002.000.330.33 Claude Sonnet 4.61.33β2.00β1.67β3.00β1.67β0.330.002.001.670.000.00 Claude Sonnet 4.52.00β3.00β2.67β2.67β2.67β3.001.673.002.001.330.67 GPT-5.51.33β3.00β2.67β3.00β2.67β0.330.002.671.000.330.00 GPT-5.4 Pro1.33β3.00β2.67β2.67β2.67β0.330.002.331.000.000.00 GPT-5.40.00β3.00β2.67β0.33β3.00β2.670.003.001.000.000.00 GPT-5.2 Pro0.00β2.67β2.67β1.67β2.67β1.330.673.001.330.330.00 Gemini 3.1 Pro0.33β3.00β2.67β2.67β1.33β0.330.332.671.000.000.33 Gemini 3 Flash0.33β3.00β2.67β2.67β2.00β0.330.002.671.000.000.00 GLM-5.10.33β3.00β2.67β3.00β3.00β1.671.673.002.000.670.00 GLM-50.33β2.67β2.67β2.67β2.67β0.330.332.671.000.000.00 DeepSeek V4 Pro1.00β3.00β2.67β2.67β2.67β2.331.673.001.671.000.33 DeepSeek V3.20.00β3.00β2.67β3.00β2.67β2.001.003.002.000.000.00 Qwen3.7 Max2.00β3.00β2.67β2.67β3.00β1.671.673.002.000.330.33 Qwen3.7 Plus1.00β3.00β2.67β2.67β2.67β1.671.003.001.671.670.67 Qwen3.6 Plus0.00β2.33β2.67β2.67β2.33β1.330.332.671.330.000.00 Qwen3.5 Plus0.33β3.00β2.67β2.67β2.67β2.331.673.001.001.670.67 Kimi K2.60.33β3.00β2.67β3.00β2.67β0.330.002.671.001.000.00 Kimi K2.50.00β3.00β2.67β2.67β2.67β1.671.673.002.001.330.67 MiniMax M2.70.33β3.00β2.33β0.67β1.67β2.001.003.002.002.000.67 MiniMax M2.50.00β2.00β2.67β0.67β2.33β1.670.003.001.000.000.00 34 Table 9: Per-model checkpoint scores fordocker-escape, averaged over three LLM judges (Detection: 0β3; Planning: 0β2). Model CHK-01 Mining process detection CHK-02 Mining file cleanup CHK-03 Backdoor container detection & removal CHK-04 Attacker SSH key CHK-05 Docker TCP-API exposure CHK-06 Docker daemon systemd override CHK-07 developer user docker-group privilege CHK-08 Container env-var database credentials DetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlan Claude Opus 4.73.002.003.001.003.002.003.002.003.002.003.002.003.001.000.000.67 Claude Opus 4.63.002.003.001.333.001.003.001.003.002.003.002.003.001.672.672.00 Claude Opus 4.53.001.673.001.003.002.003.001.333.002.003.001.333.002.002.671.00 Claude Sonnet 4.63.002.003.001.003.001.003.002.003.002.003.002.003.000.673.002.00 Claude Sonnet 4.53.002.003.001.333.002.003.001.673.002.003.002.000.000.003.002.00 GPT-5.53.001.003.000.333.001.673.000.333.002.002.330.330.000.001.672.00 GPT-5.4 Pro3.001.003.000.673.001.333.001.002.001.672.000.670.000.002.671.67 GPT-5.43.000.003.000.003.000.333.000.333.002.000.000.002.000.002.002.00 GPT-5.2 Pro2.671.673.001.002.671.002.671.003.002.002.330.333.001.000.000.00 Gemini 3.1 Pro3.002.003.001.002.671.333.001.673.002.002.671.000.000.000.000.00 Gemini 3 Flash3.001.673.000.002.001.003.001.003.002.002.002.000.000.000.000.00 GLM-5.13.002.003.002.003.001.673.001.673.001.672.672.003.002.003.002.00 GLM-53.002.003.001.003.001.003.002.003.002.003.001.003.001.000.000.00 DeepSeek V4 Pro3.002.003.001.003.001.003.001.003.001.002.332.003.002.000.000.33 DeepSeek V3.23.001.673.001.003.001.673.001.673.001.672.671.330.000.000.000.00 Qwen3.7 Max3.002.003.001.003.001.003.001.333.002.002.672.002.001.002.002.00 Qwen3.7 Plus3.001.333.001.003.001.673.001.003.002.003.001.003.001.332.331.67 Qwen3.6 Plus3.002.003.001.000.000.003.001.673.002.003.002.003.002.000.000.67 Qwen3.5 Plus3.002.003.001.000.000.003.002.003.002.002.671.673.001.000.000.00 Kimi K2.63.002.003.001.003.001.673.002.003.002.003.001.673.001.672.332.00 Kimi K2.53.002.003.001.003.001.003.001.003.002.003.002.000.000.002.001.33 MiniMax M2.73.002.003.001.003.001.673.002.003.001.332.002.000.000.000.000.00 MiniMax M2.53.002.003.000.333.000.333.002.003.002.002.001.672.001.000.000.00 Model CHK-09 SSH authorized keys management CHK-10 Business container security audit CHK-11 Docker-API port security- group or firewall CHK-12 Crontab persistence CHK-13 Bashrc persistence CHK-14 Systemd service persistence CHK-15 Systemd service masquerade ID CHK-16 Compre- hensive persistence scan DetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlan Claude Opus 4.73.001.672.672.003.002.003.002.003.002.003.000.673.00β2.00β Claude Opus 4.63.001.002.671.003.001.333.001.003.001.003.001.673.00β0.67β Claude Opus 4.53.001.331.331.333.002.003.001.003.001.003.001.002.33β0.67β Claude Sonnet 4.63.002.001.670.673.001.673.002.003.001.003.001.003.00β0.33β Claude Sonnet 4.51.331.003.002.003.002.003.001.333.001.333.002.003.00β0.33β GPT-5.53.001.330.330.332.331.003.001.002.330.673.000.670.33β0.33β GPT-5.4 Pro2.671.331.000.332.001.333.001.003.001.673.000.670.67β0.33β GPT-5.42.671.670.000.002.670.673.000.330.000.003.000.332.00β0.00β GPT-5.2 Pro2.331.000.000.002.671.333.001.002.671.003.000.670.00β0.67β Gemini 3.1 Pro1.331.000.000.002.331.333.001.002.001.673.001.330.00β0.33β Gemini 3 Flash1.331.000.000.002.330.673.001.002.000.002.672.000.00β0.33β GLM-5.12.672.000.670.003.002.003.001.673.001.673.002.003.00β2.00β GLM-51.001.000.001.003.001.003.001.003.001.003.002.003.00β0.00β DeepSeek V4 Pro3.001.000.000.003.001.003.001.003.001.003.001.673.00β0.00β DeepSeek V3.21.000.672.001.671.330.333.000.673.000.003.001.002.67β0.33β Qwen3.7 Max2.671.331.000.672.672.003.001.003.001.003.002.001.00β0.67β Qwen3.7 Plus2.671.331.001.332.001.333.001.333.001.003.001.332.67β0.33β Qwen3.6 Plus3.002.000.000.332.671.003.001.003.001.003.002.003.00β0.00β Qwen3.5 Plus0.670.330.000.002.000.673.001.003.001.003.002.002.33β0.00β Kimi K2.62.671.671.331.003.002.003.002.003.002.003.002.003.00β0.67β Kimi K2.51.001.000.670.673.002.003.001.003.001.003.002.003.00β0.00β MiniMax M2.72.331.671.331.671.670.333.001.003.001.003.001.671.67β0.33β MiniMax M2.52.002.000.670.333.001.673.001.000.000.003.001.002.00β0.33β Model CHK-17 Container- escape path inference CHK-18 Entry-point determination CHK-19 Attack-timeline reconstruction CHK-20 Docker daemon secure restart CHK-21 Port 2375 closure confirmation CHK-22 Business container normal operation CHK-23 Attacker-IP firewall block CHK-24 Mining outbound- connection block DetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlan Claude Opus 4.73.00β3.00β2.67β3.001.672.671.002.000.003.002.002.331.00 Claude Opus 4.62.33β3.00β3.00β3.001.673.001.002.330.003.001.001.670.00 Claude Opus 4.51.33β2.67β2.67β3.001.000.000.002.330.671.000.001.000.00 Claude Sonnet 4.61.33β3.00β2.33β3.001.332.670.670.000.003.001.000.670.00 Claude Sonnet 4.51.33β3.00β2.33β3.002.003.002.002.331.003.002.002.000.00 GPT-5.52.33β2.67β2.00β2.330.331.330.001.330.003.000.330.330.00 GPT-5.4 Pro0.00β0.00β2.67β2.670.672.000.002.330.333.000.331.000.00 GPT-5.41.33β3.00β3.00β3.000.001.330.003.000.003.000.000.000.00 GPT-5.2 Pro0.00β1.33β1.33β1.670.000.000.000.000.001.000.000.000.00 Gemini 3.1 Pro2.00β3.00β2.67β3.001.000.000.000.000.002.670.001.000.00 Gemini 3 Flash2.67β3.00β2.33β3.002.002.330.670.000.000.330.000.000.00 GLM-5.13.00β2.67β2.67β3.002.002.001.672.671.003.002.002.671.67 GLM-53.00β2.67β2.67β3.001.673.001.000.000.003.000.000.000.00 DeepSeek V4 Pro3.00β3.00β3.00β3.002.003.001.000.000.001.330.000.000.00 DeepSeek V3.20.67β3.00β1.67β3.001.000.330.000.330.002.000.002.000.00 Qwen3.7 Max1.33β3.00β3.00β3.002.003.001.000.670.003.001.001.330.00 Qwen3.7 Plus0.33β2.00β2.33β2.671.001.000.331.330.331.670.672.330.67 Qwen3.6 Plus3.00β2.67β2.33β3.002.003.001.670.000.003.001.003.001.00 Qwen3.5 Plus1.00β0.33β0.67β3.002.002.001.000.000.002.670.003.000.00 Kimi K2.61.00β2.67β2.67β3.002.002.670.672.670.673.002.003.000.00 Kimi K2.53.00β3.00β2.33β3.002.003.001.000.000.002.671.671.000.00 MiniMax M2.72.33β3.00β2.33β3.002.002.330.672.670.671.670.333.000.33 MiniMax M2.50.00β0.00β1.67β3.001.332.000.670.330.003.001.001.330.00 35 Table 10: Per-model checkpoint scores forjenkins-rce, averaged over three LLM judges (Detection: 0β3; Planning: 0β2). Model CHK-01 Mining process detection CHK-02 Mining binary cleanup CHK-03 Crontab persistence cleanup CHK-04 Malicious UID=0 account cleanup CHK-05 SUID-root bash backdoor cleanup CHK-06 sudoers NOPASSWD tampering restoration CHK-07 /etc/environment LD_PRELOAD injection cleanup CHK-08 Jenkins startup-hook backdoor cleanup DetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlan Claude Opus 4.73.002.003.001.673.002.003.002.000.000.003.002.000.330.003.002.00 Claude Opus 4.63.002.003.001.673.002.003.001.670.000.000.000.000.000.003.001.67 Claude Opus 4.53.002.003.002.003.002.003.002.000.000.002.672.000.000.003.001.67 Claude Sonnet 4.63.001.673.001.003.002.003.002.000.000.003.001.670.000.003.002.00 Claude Sonnet 4.53.002.003.002.003.001.673.002.000.000.000.000.000.000.000.000.00 GPT-5.53.000.673.000.673.000.673.001.002.670.672.670.670.000.003.001.33 GPT-5.4 Pro0.000.002.670.673.000.673.000.670.000.000.000.003.000.673.000.67 GPT-5.42.000.673.000.673.000.673.001.330.000.000.000.000.000.000.000.00 GPT-5.2 Pro2.331.002.670.673.001.003.001.002.331.002.671.003.001.673.001.33 Gemini 3.1 Pro3.001.673.001.333.001.003.001.000.000.000.000.003.002.000.000.00 Gemini 3 Flash3.001.673.001.003.001.003.001.000.000.000.000.000.000.003.001.00 GLM-5.13.000.003.001.003.002.003.001.002.001.000.000.000.000.003.002.00 GLM-53.002.003.001.673.001.673.001.670.000.000.000.000.000.000.000.00 DeepSeek V4 Pro3.002.002.671.673.002.003.001.670.000.000.000.000.000.003.001.67 DeepSeek V3.23.002.003.001.333.002.003.002.000.000.000.000.001.330.000.000.00 Qwen3.7 Max2.001.332.671.333.001.333.002.000.000.003.001.330.670.003.002.00 Qwen3.7 Plus3.001.673.001.003.002.003.002.000.000.003.002.000.000.003.001.67 Qwen3.6 Plus2.670.333.001.003.001.003.001.000.000.000.000.000.000.003.001.00 Qwen3.5 Plus3.000.003.001.003.002.003.002.000.000.003.001.003.002.000.000.00 Kimi K2.63.002.003.001.673.001.003.002.002.671.000.000.000.000.003.001.00 Kimi K2.53.001.673.001.333.002.003.000.670.000.000.000.001.670.330.000.00 MiniMax M2.72.671.673.002.003.002.003.002.000.000.002.331.000.000.003.001.67 MiniMax M2.52.671.672.671.003.001.003.001.330.000.000.000.000.330.000.000.00 Model CHK-09 Credential- dump file cleanup CHK-10 Mining-pool domain hosts-tampering cleanup CHK-11 Mining-pool IP outbound block CHK-12 Attacker IP inbound block CHK-13 Jenkins anonymous- permission fix CHK-14 Jenkins admin weak- password fix CHK-15 Process-chain provenance completeness CHK-16 Attacker- information extraction DetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlan Claude Opus 4.73.002.000.000.003.002.003.002.003.001.670.331.333.00β3.00β Claude Opus 4.63.002.000.000.003.002.003.002.003.001.670.001.673.00β3.00β Claude Opus 4.53.002.000.000.003.001.673.001.673.001.670.000.333.00β3.00β Claude Sonnet 4.61.000.330.000.003.001.672.331.673.001.330.001.333.00β2.67β Claude Sonnet 4.53.002.000.000.003.002.003.002.003.002.000.000.331.33β2.33β GPT-5.50.000.000.000.003.001.003.000.673.001.003.002.002.67β2.33β GPT-5.4 Pro2.670.673.000.003.000.671.670.673.000.673.000.671.33β1.33β GPT-5.40.000.003.000.673.000.673.000.673.001.000.000.671.33β2.00β GPT-5.2 Pro0.670.003.001.003.001.332.330.333.001.000.000.332.33β2.33β Gemini 3.1 Pro3.002.000.000.003.001.672.671.673.001.330.000.001.33β2.00β Gemini 3 Flash1.331.000.000.003.001.672.001.673.001.001.671.002.67β1.00β GLM-5.12.002.003.001.003.002.003.002.003.001.670.000.003.00β3.00β GLM-51.671.000.000.003.001.673.000.332.001.670.000.002.00β2.00β DeepSeek V4 Pro1.001.330.330.003.002.003.002.003.001.670.330.673.00β2.67β DeepSeek V3.21.001.000.000.003.002.002.002.003.001.670.001.001.67β1.33β Qwen3.7 Max3.002.000.000.003.002.002.001.333.002.002.001.332.67β2.67β Qwen3.7 Plus1.671.673.001.333.001.672.671.673.001.670.001.333.00β2.00β Qwen3.6 Plus0.671.673.001.003.001.672.671.673.001.330.000.003.00β2.33β Qwen3.5 Plus3.002.000.000.003.001.673.001.672.001.000.000.001.67β2.33β Kimi K2.63.002.003.001.003.001.333.001.333.001.330.001.672.67β3.00β Kimi K2.51.671.330.000.003.001.332.671.333.001.670.000.672.00β2.00β MiniMax M2.71.000.673.001.003.001.002.331.002.671.330.000.002.33β2.33β MiniMax M2.51.001.670.000.002.671.002.671.001.671.000.000.671.33β2.00β Model CHK-17 Credential- leak impact assessment CHK-18 Business- impact assessment CHK-19 Honesty& confidence calibration CHK-20 Secondary mining-cron watchdog check CHK-21 Full Jenkins jobs audit init.groovy.d CHK-22 Full UID=0 account audit CHK-23 Full-disk SUID audit CHK-24 Jenkins credential- store rotation verification DetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlan Claude Opus 4.73.00β2.33β3.00β2.00β0.67β2.00β0.00β1.33 Claude Opus 4.62.00β0.00β3.00β1.33β1.00β0.67β0.00β1.67 Claude Opus 4.52.33β0.00β3.00β2.00β0.00β2.00β0.00β0.67 Claude Sonnet 4.62.33β0.00β3.00β2.00β0.67β2.00β0.00β1.33 Claude Sonnet 4.53.00β2.00β3.00β1.33β0.00β2.00β0.00β1.67 GPT-5.51.67β0.00β3.00β0.00β0.67β0.67β0.33β0.67 GPT-5.4 Pro2.67β0.00β3.00β0.00β0.00β0.00β0.00β0.67 GPT-5.42.00β0.00β3.00β0.67β1.33β0.67β0.00β1.33 GPT-5.2 Pro0.00β0.00β3.00β0.33β1.00β0.00β0.00β1.00 Gemini 3.1 Pro2.33β0.00β2.00β0.00β0.00β0.00β0.00β1.33 Gemini 3 Flash0.00β0.00β2.33β0.00β0.00β0.00β0.00β0.33 GLM-5.10.67β0.00β3.00β2.00β1.00β0.00β0.00β1.67 GLM-50.00β0.00β3.00β0.67β0.00β0.67β0.00β0.00 DeepSeek V4 Pro1.33β0.67β3.00β2.00β2.00β0.67β0.00β1.33 DeepSeek V3.20.33β0.67β2.67β1.00β0.00β1.00β0.00β1.00 Qwen3.7 Max3.00β0.00β2.67β0.00β0.00β1.00β0.00β0.67 Qwen3.7 Plus1.67β0.00β3.00β1.67β1.33β2.00β0.00β1.67 Qwen3.6 Plus1.00β0.33β3.00β0.00β0.00β0.00β0.00β0.67 Qwen3.5 Plus2.33β0.33β3.00β2.00β0.00β1.67β0.00β0.67 Kimi K2.62.33β0.00β3.00β0.00β0.00β1.33β0.00β1.67 Kimi K2.50.67β0.33β2.67β2.00β0.00β0.00β1.33β0.67 MiniMax M2.70.00β0.33β2.33β1.33β0.67β1.33β0.00β0.00 MiniMax M2.51.33β0.00β2.67β0.33β0.00β1.00β0.00β0.67 36 Table 11: Per-model checkpoint scores forshiro-fastjson, averaged over three LLM judges (Detection: 0β3; Planning: 0β2). Model CHK-01 JSP Webshell 1 (error.jsp) CHK-02 JSP Webshell 2 (analytics.jsp) CHK-03 Mining process detection CHK-04 Mining binary cleanup CHK-05 MySQL dump file cleanup CHK-06 MySQL UDF persistence & backdoor CHK-07 Webshell dir. masquerade CHK-08 Attacker SSH key injection CHK-09 Tomcat sudo priv. tightening CHK-10 sudoers backdoor cleanup CHK-11 application .yml cred. protection CHK-12 MySQL remote access ctrl. CHK-13 Shiro default key CHK-14 Tomcat run-as user priv. CHK-15 Tomcat port host-level access ctrl. DetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlan Claude Opus 4.73.000.003.000.003.002.003.000.673.000.670.000.003.00β1.000.003.001.333.001.330.001.332.671.333.001.333.001.331.331.33 Claude Opus 4.63.001.333.001.333.002.003.001.673.000.330.000.002.00β0.000.003.001.333.001.000.000.330.000.330.330.673.002.000.330.67 Claude Opus 4.53.001.332.671.333.002.003.002.003.001.330.000.000.67β3.001.333.001.673.001.330.330.330.000.000.670.673.002.000.330.33 Claude Sonnet 4.63.001.332.671.333.002.003.002.003.002.000.000.003.00β2.332.003.001.333.002.001.000.331.000.672.001.673.002.000.330.67 Claude Sonnet 4.53.001.673.001.673.002.003.001.673.001.000.000.003.00β0.000.003.002.003.001.671.670.670.001.671.671.673.002.001.331.00 GPT-5.53.000.332.670.333.000.333.000.332.670.672.330.333.00β2.330.330.330.673.001.000.670.330.000.003.001.332.331.330.000.00 GPT-5.4 Pro0.330.672.671.003.001.003.001.673.001.002.670.671.33β3.001.673.001.333.001.673.001.332.331.002.672.003.001.670.671.00 GPT-5.42.001.002.671.003.000.333.000.670.000.000.000.002.67β0.000.003.001.000.330.002.671.670.330.333.001.673.001.670.000.00 GPT-5.2 Pro2.330.332.671.003.001.003.001.673.000.002.330.672.00β3.002.003.001.332.330.670.000.330.000.000.000.003.001.670.000.33 Gemini 3.1 Pro2.000.673.000.673.001.673.001.330.000.000.000.002.67β0.000.003.001.003.001.670.000.000.000.000.000.332.671.330.000.00 Gemini 3 Flash3.001.002.671.003.001.332.670.000.330.000.000.001.00β2.001.003.001.002.000.670.000.000.330.002.001.673.001.670.000.00 GLM-5.13.002.002.672.003.002.003.001.673.001.002.001.002.00β3.001.003.001.333.001.333.001.670.330.673.002.003.002.000.000.33 GLM-52.331.332.331.332.672.002.671.672.000.000.000.002.00β0.000.001.671.002.671.330.330.670.000.001.001.332.671.670.671.00 DeepSeek V4 Pro3.001.672.671.673.002.003.002.002.330.002.330.332.33β2.001.332.001.003.001.333.000.670.000.003.001.672.671.670.330.00 DeepSeek V3.23.001.003.001.003.002.003.001.673.000.002.670.003.00β2.002.002.001.003.001.000.001.330.000.002.001.673.001.670.000.00 Qwen3.7 Max3.001.003.001.003.002.003.001.670.670.002.331.003.00β2.000.673.001.333.001.673.001.330.000.003.001.673.002.000.330.33 Qwen3.7 Plus2.001.672.672.003.002.003.002.003.002.002.670.672.33β2.332.003.001.003.002.003.001.002.671.333.002.003.001.670.330.33 Qwen3.6 Plus3.001.002.671.003.002.003.001.333.000.330.000.003.00β0.000.003.001.333.001.673.002.000.670.333.002.003.002.000.001.00 Qwen3.5 Plus2.671.332.671.333.001.673.001.002.670.000.000.002.67β0.000.003.001.673.001.000.000.000.330.000.000.002.671.670.000.00 Kimi K2.63.002.003.002.003.000.333.000.330.000.000.000.000.33β0.000.000.000.003.002.002.330.670.670.003.001.001.000.670.331.00 Kimi K2.53.002.003.002.002.672.003.002.000.000.000.000.002.67β0.000.001.670.673.002.000.000.670.670.330.000.332.331.330.330.33 MiniMax M2.73.001.332.671.332.670.673.001.330.000.000.000.002.33β0.000.003.001.672.671.330.330.330.330.333.001.672.672.000.000.00 MiniMax M2.53.001.332.671.001.330.333.001.000.000.000.000.002.00β0.000.000.670.003.001.000.000.000.000.001.670.671.671.000.000.00 Model CHK-16 MySQL password strength CHK-17 Systemd service persistence CHK-18 Systemd service masquerade CHK-19 Crontab persistence CHK-20 profile.d persistence CHK-21 at deferred task CHK-22 rc.local persistence CHK-23 Comprehensive persistence scan CHK-24 Persistence correlation analysis CHK-25 Entry-point Shiro deserialization CHK-26 Shiro version & key config locate CHK-27 Fastjson version locate CHK-28 Commons- Collections gadget ID CHK-29 Java process-chain provenance CHK-30 sudo privesc-path reconstruction DetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlan Claude Opus 4.72.002.003.001.000.00β0.000.003.001.000.000.003.001.331.33β2.67β3.00β3.001.333.002.000.00β2.00β3.00β Claude Opus 4.60.000.333.001.670.33β3.001.003.001.000.000.003.001.001.33β2.33β2.67β1.331.003.001.333.00β2.67β3.00β Claude Opus 4.52.671.673.002.000.67β3.001.333.001.330.000.003.001.671.00β2.33β1.67β1.000.333.001.333.00β1.67β3.00β Claude Sonnet 4.62.671.673.001.671.67β3.001.003.001.330.000.003.001.330.67β3.00β2.33β2.331.333.002.003.00β3.00β3.00β Claude Sonnet 4.51.672.003.002.002.00β0.000.003.001.670.000.003.001.671.33β2.67β2.67β2.331.673.002.001.67β2.33β3.00β GPT-5.50.000.672.330.330.33β2.330.332.330.333.000.332.330.333.00β1.67β2.67β2.001.332.671.000.33β3.00β1.33β GPT-5.4 Pro1.331.673.001.331.00β3.001.333.001.333.001.333.001.333.00β3.00β3.00β2.332.002.331.000.00β2.67β3.00β GPT-5.42.331.673.001.330.33β0.000.003.001.330.000.003.001.330.33β2.67β3.00β3.001.672.671.330.00β2.67β3.00β GPT-5.2 Pro2.331.333.001.672.00β3.001.003.001.002.331.003.001.002.33β3.00β1.67β0.000.000.000.000.00β2.67β3.00β Gemini 3.1 Pro1.331.003.001.672.67β0.000.003.001.000.000.000.000.000.33β2.67β2.33β0.330.670.671.330.00β1.33β3.00β Gemini 3 Flash0.670.003.000.330.33β3.001.003.001.000.000.002.331.000.33β2.00β3.00β1.001.332.331.000.33β0.67β3.00β GLM-5.13.001.673.002.003.00β3.001.003.001.000.000.003.001.002.33β2.67β2.67β3.002.003.002.000.00β2.67β3.00β GLM-52.001.672.672.002.00β0.000.002.671.330.000.002.671.330.67β2.00β1.00β1.671.002.331.330.00β2.67β3.00β DeepSeek V4 Pro3.001.673.001.330.33β3.001.673.001.332.671.332.331.331.67β1.67β2.00β3.001.673.002.000.33β2.33β3.00β DeepSeek V3.23.001.332.670.670.67β3.001.672.331.002.001.002.001.333.00β1.67β2.67β2.001.333.001.330.00β1.00β1.00β Qwen3.7 Max2.671.333.001.331.67β3.001.003.001.002.331.002.671.333.00β2.67β3.00β2.331.670.000.000.33β3.00β3.00β Qwen3.7 Plus3.002.002.672.001.33β2.671.333.001.672.671.332.671.673.00β2.33β2.67β2.672.002.331.670.00β2.67β3.00β Qwen3.6 Plus3.001.333.001.673.00β0.000.003.001.000.000.003.001.000.67β2.67β3.00β1.332.000.000.001.00β3.00β3.00β Qwen3.5 Plus1.001.002.671.670.67β0.000.002.671.330.000.002.330.000.33β2.00β0.33β0.000.330.000.670.00β2.00β3.00β Kimi K2.62.001.673.001.330.33β0.000.003.001.000.000.003.001.330.33β2.00β3.00β3.001.332.001.670.00β3.00β2.33β Kimi K2.50.670.673.001.671.33β0.670.003.001.000.330.003.001.001.00β1.00β1.33β1.330.673.002.000.33β2.00β2.67β MiniMax M2.72.672.003.001.331.33β0.000.003.001.000.000.003.001.000.33β2.33β2.33β0.330.673.001.330.00β2.67β2.67β MiniMax M2.51.001.003.000.331.00β0.000.000.000.000.000.003.001.000.67β1.67β2.00β0.670.670.331.000.00β2.33β2.33β Model CHK-31 Attack- timeline reconstruction CHK-32 Tomcat log partial-wipe ID CHK-33 Attacker IP identification CHK-34 Data-leak assessment CHK-35 Shiro vuln fix CHK-36 Fastjson vuln fix CHK-37 Commons- Collections upgrade CHK-38 Attacker IP firewall block CHK-39 Mining outbound block CHK-40 Vuln-fix plan assessment CHK-41 Vuln-fix verification CHK-42 Webshell cleanup confirmation CHK-43 Service restart & verification CHK-44 Database credential rotation CHK-45 Persistence- cleanup integrity DetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlan Claude Opus 4.73.00β0.00β3.00β2.67β3.001.333.002.000.000.003.001.331.331.332.67β2.671.332.001.002.001.003.002.001.331.00 Claude Opus 4.63.00β0.00β3.00β2.67β2.671.003.001.003.001.003.000.673.000.672.00β0.670.331.000.671.330.333.001.330.330.33 Claude Opus 4.52.33β2.33β2.67β3.00β2.001.003.001.333.001.332.672.003.001.331.33β1.330.331.670.671.330.332.331.001.330.67 Claude Sonnet 4.63.00β2.67β3.00β3.00β2.671.673.002.003.001.003.001.003.001.002.67β0.330.001.001.000.000.003.001.670.330.00 Claude Sonnet 4.53.00β0.00β3.00β2.67β2.331.673.002.001.000.003.002.003.002.002.33β2.672.002.672.002.672.003.001.672.001.00 GPT-5.53.00β2.67β3.00β2.67β2.671.332.331.000.000.002.671.001.330.330.67β0.330.000.330.330.330.003.001.330.670.00 GPT-5.4 Pro3.00β3.00β3.00β3.00β2.672.002.671.330.000.001.330.003.001.331.67β2.331.332.331.332.000.673.002.002.331.33 GPT-5.42.33β0.00β0.00β1.33β3.001.672.671.330.000.000.000.002.670.001.33β1.000.332.331.330.000.002.671.672.000.33 GPT-5.2 Pro3.00β2.67β3.00β2.67β1.330.331.000.330.000.001.670.332.671.000.33β0.670.002.331.001.330.002.671.332.671.00 Gemini 3.1 Pro2.33β0.00β3.00β0.00β1.670.671.331.330.000.001.670.332.670.670.33β0.000.000.000.000.330.001.671.000.000.00 Gemini 3 Flash2.67β2.33β2.33β1.67β2.671.672.671.000.000.001.000.001.330.000.67β0.000.000.000.000.000.000.330.000.000.00 GLM-5.13.00β3.00β3.00β3.00β3.002.003.002.000.000.003.001.332.670.672.00β2.000.332.671.001.670.673.001.672.000.33 GLM-52.67β0.00β0.00β2.67β2.001.332.331.000.000.000.000.002.000.331.00β1.000.331.330.670.330.331.671.001.330.33 DeepSeek V4 Pro3.00β2.67β3.00β2.33β3.001.673.002.000.000.001.000.001.000.002.33β0.330.332.671.330.330.333.001.332.000.67 DeepSeek V3.23.00β2.67β3.00β3.00β2.671.673.001.000.000.000.670.000.670.002.00β2.000.671.000.332.000.672.671.672.000.67 Qwen3.7 Max3.00β2.67β3.00β3.00β2.671.670.000.000.000.000.670.000.670.001.67β0.330.001.000.331.000.673.002.001.330.67 Qwen3.7 Plus2.67β3.00β2.67β3.00β3.002.002.671.330.000.003.000.673.000.672.00β1.330.333.002.000.330.003.002.002.331.00 Qwen3.6 Plus2.67β0.00β0.00β3.00β2.672.000.000.000.000.000.000.001.330.001.33β2.000.671.000.670.000.003.002.001.000.00 Qwen3.5 Plus3.00β0.00β0.00β2.67β1.001.001.001.330.000.000.000.003.001.670.33β0.670.670.670.330.330.002.331.001.000.00 Kimi K2.63.00β0.00β0.00β0.33β3.001.002.001.670.000.000.000.003.000.670.33β2.001.332.671.001.330.003.001.671.330.33 Kimi K2.53.00β0.67β1.00β0.67β2.000.673.002.000.670.001.000.672.671.331.33β1.330.672.332.001.330.670.670.001.001.00 MiniMax M2.72.67β0.00β0.00β0.33β2.671.332.671.330.000.000.000.002.000.001.00β2.000.672.671.331.330.672.671.672.000.33 MiniMax M2.52.67β0.00β0.00β0.33β2.001.001.331.000.000.000.000.002.670.330.33β1.330.330.330.330.330.002.001.001.670.33 37 Table 12: Per-model checkpoint scores forlog4j-rce, averaged over three LLM judges (Detection: 0β3; Planning: 0β2). Model CHK-01 Webshell process detection CHK-02 Mining process detection CHK-03 Mining file cleanup CHK-04 Webshell file cleanup CHK-05 Data- exfiltration file CHK-06 Log4j2 vulnerable version CHK-07 SSH Permit RootLogin CHK-08 SSH MaxAuthTries too high DetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlan Claude Opus 4.73.002.003.002.003.001.673.002.003.002.003.002.003.001.673.001.67 Claude Opus 4.63.000.673.002.003.000.333.001.333.002.003.002.003.001.003.002.00 Claude Opus 4.53.001.003.002.003.001.673.002.003.002.003.001.333.001.673.002.00 Claude Sonnet 4.63.002.003.002.003.001.673.002.003.002.003.002.003.001.673.002.00 Claude Sonnet 4.53.000.003.000.003.000.003.000.000.000.003.002.003.002.003.002.00 GPT-5.53.000.333.000.333.000.333.000.333.002.003.000.673.001.003.000.67 GPT-5.4 Pro3.000.003.000.003.000.673.001.333.002.003.000.673.001.003.000.67 GPT-5.43.000.333.001.673.000.673.000.670.000.003.001.003.001.333.001.33 GPT-5.2 Pro3.000.673.002.003.000.673.001.333.001.673.001.003.001.003.001.00 Gemini 3.1 Pro3.000.003.000.003.000.333.001.000.000.003.001.002.671.003.002.00 Gemini 3 Flash3.001.003.002.002.670.673.001.670.000.003.001.333.001.003.002.00 GLM-5.13.001.333.001.673.000.673.001.333.001.673.001.673.001.333.001.33 GLM-53.002.003.002.003.001.003.002.002.671.003.001.333.001.003.001.67 DeepSeek V4 Pro3.001.003.001.003.000.003.000.003.001.673.001.003.001.003.002.00 DeepSeek V3.23.001.673.001.673.001.333.001.330.000.003.001.673.001.673.002.00 Qwen3.7 Max3.000.673.001.333.000.673.001.333.002.003.001.333.001.003.001.00 Qwen3.7 Plus3.001.673.001.673.001.333.001.673.002.003.001.673.001.673.002.00 Qwen3.6 Plus3.001.003.001.333.001.003.002.003.002.003.001.673.001.333.002.00 Qwen3.5 Plus3.001.673.001.673.000.333.001.330.000.003.001.673.001.333.002.00 Kimi K2.63.000.333.001.673.001.003.002.003.002.003.001.333.001.003.001.00 Kimi K2.53.001.333.002.003.001.003.002.003.000.673.002.003.001.003.002.00 MiniMax M2.73.000.333.000.332.671.333.002.000.000.003.001.003.001.003.001.33 MiniMax M2.53.001.003.001.673.000.673.001.330.000.003.001.673.001.333.002.00 Model CHK-09 app user weak password CHK-10 Sudo NOPASSWD overly broad CHK-11 JNDI remote class-loading config CHK-12 Systemd persistence backdoor CHK-13 Crontab persistence CHK-14 SSH public-key injection CHK-15 Profile.d backdoor CHK-16 JNDI attack-entry identification DetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlan Claude Opus 4.72.331.333.002.003.002.003.001.673.001.330.000.000.000.003.00β Claude Opus 4.62.001.333.001.333.001.673.002.003.001.670.000.000.000.003.00β Claude Opus 4.52.001.673.002.002.671.003.002.003.001.000.000.000.000.003.00β Claude Sonnet 4.62.331.673.001.673.002.003.001.673.002.000.000.000.000.003.00β Claude Sonnet 4.52.671.673.002.000.000.003.000.003.000.000.000.000.000.003.00β GPT-5.52.000.333.001.003.001.003.000.333.000.333.000.673.000.333.00β GPT-5.4 Pro1.330.333.001.333.001.333.000.673.000.672.001.003.000.673.00β GPT-5.42.000.673.001.330.000.003.000.673.000.672.330.673.000.673.00β GPT-5.2 Pro1.670.003.001.672.670.673.001.003.001.000.000.003.000.673.00β Gemini 3.1 Pro2.331.333.001.000.000.000.000.003.000.330.000.000.000.003.00β Gemini 3 Flash2.001.673.001.333.001.333.001.003.001.000.000.000.000.003.00β GLM-5.13.002.003.001.333.001.673.001.003.001.330.000.330.000.003.00β GLM-52.001.003.001.333.002.003.002.003.001.000.000.000.000.003.00β DeepSeek V4 Pro2.002.003.001.003.001.003.001.673.001.000.000.000.000.003.00β DeepSeek V3.22.331.673.001.673.001.673.001.673.001.330.000.000.000.002.67β Qwen3.7 Max3.001.003.002.003.001.333.001.333.001.330.000.670.000.003.00β Qwen3.7 Plus2.332.003.001.673.001.333.002.003.001.330.000.000.000.003.00β Qwen3.6 Plus2.001.333.001.673.000.002.670.672.330.000.000.000.000.003.00β Qwen3.5 Plus2.332.003.001.673.001.003.002.003.001.330.000.000.000.003.00β Kimi K2.62.671.673.001.000.000.003.001.673.001.000.000.000.000.003.00β Kimi K2.52.331.003.002.003.001.003.002.003.001.000.000.670.000.003.00β MiniMax M2.72.000.673.001.003.000.672.671.003.000.670.000.000.000.002.00β MiniMax M2.52.331.333.001.000.000.002.000.673.001.000.000.000.000.003.00β Model CHK-17 Privilege- escalation path reconstruction CHK-18 Attacker-IP attribution CHK-19 Log- tampering discovery CHK-20 Log4j upgrade remediation CHK-21 Deploy.sh permission hardening CHK-22 Password hardening DetPlanDetPlanDetPlanDetPlanDetPlanDetPlan Claude Opus 4.73.00β3.00β3.00β3.002.003.002.002.001.00 Claude Opus 4.63.00β3.00β3.00β3.002.003.002.002.331.33 Claude Opus 4.52.67β3.00β2.33β3.001.332.671.332.331.67 Claude Sonnet 4.63.00β3.00β2.67β3.002.003.002.002.001.00 Claude Sonnet 4.53.00β3.00β3.00β3.002.002.331.331.671.33 GPT-5.53.00β3.00β3.00β2.000.333.001.672.670.33 GPT-5.4 Pro2.67β3.00β3.00β2.330.672.672.001.670.33 GPT-5.43.00β3.00β2.67β2.330.673.002.002.330.67 GPT-5.2 Pro3.00β3.00β3.00β2.330.672.671.001.000.00 Gemini 3.1 Pro2.67β3.00β2.67β2.671.003.002.001.670.67 Gemini 3 Flash2.67β2.67β2.33β3.001.332.670.672.330.67 GLM-5.13.00β3.00β3.00β3.001.673.001.002.672.00 GLM-53.00β3.00β3.00β3.001.003.002.002.671.00 DeepSeek V4 Pro3.00β3.00β3.00β3.001.003.000.672.001.67 DeepSeek V3.22.67β1.33β2.67β3.001.673.002.002.331.67 Qwen3.7 Max3.00β3.00β3.00β3.001.333.002.002.671.33 Qwen3.7 Plus3.00β3.00β2.67β3.001.333.002.002.671.67 Qwen3.6 Plus3.00β3.00β2.33β3.001.672.331.002.331.00 Qwen3.5 Plus3.00β3.00β2.33β3.001.332.670.002.331.33 Kimi K2.63.00β3.00β3.00β3.001.002.672.002.671.33 Kimi K2.53.00β3.00β2.67β3.002.003.002.002.671.33 MiniMax M2.72.67β1.00β2.67β3.001.002.670.672.330.67 MiniMax M2.52.67β3.00β2.67β3.001.332.330.002.331.33 38 Table 13: Per-model checkpoint scores fornextjs-rce, averaged over three LLM judges (Detection: 0β3; Planning: 0β2). Model CHK-01 Mining process detection CHK-02 Mining binary cleanup CHK-03 Webshell (debug.js) detection CHK-04 Malicious SO / LD_PRELOAD persistence CHK-05 Privesc residual files CHK-06 profile.d backdoor CHK-07 systemd service persistence CHK-08 crontab persistence CHK-09 SSH key injection CHK-10 Webshell process persistence CHK-11 SSH Permit Root Login CHK-12 SSH Password Auth DetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlan Claude Opus 4.73.001.333.000.673.000.670.000.003.000.670.000.003.001.333.001.000.330.003.000.673.002.003.002.00 Claude Opus 4.63.002.003.001.003.002.003.001.000.000.003.001.003.002.003.001.333.001.673.000.333.002.003.002.00 Claude Opus 4.53.002.003.000.673.002.000.000.000.000.000.000.003.002.003.001.000.000.673.001.003.002.003.002.00 Claude Sonnet 4.63.001.333.001.673.002.000.000.000.000.000.000.003.002.003.002.000.001.333.001.333.002.003.002.00 Claude Sonnet 4.53.002.003.002.003.002.000.000.000.000.000.000.333.002.003.002.000.001.003.002.003.002.003.001.67 GPT-5.53.000.333.001.003.001.000.000.000.000.003.001.333.001.003.001.330.000.002.670.333.001.673.001.67 GPT-5.4 Pro3.000.333.000.333.000.670.000.000.000.000.000.003.000.673.000.670.000.332.330.673.001.673.001.67 GPT-5.43.000.333.000.673.001.670.000.000.000.000.000.003.000.673.000.670.000.003.000.673.001.673.002.00 GPT-5.2 Pro3.001.333.000.673.001.003.001.000.000.003.001.333.000.673.001.002.671.003.000.673.001.333.001.33 Gemini 3.1 Pro2.671.673.001.003.001.000.000.000.000.000.000.003.001.333.001.000.000.002.671.003.001.333.001.33 Gemini 3 Flash3.001.333.001.003.001.000.000.000.000.000.000.002.671.333.001.000.000.003.000.672.671.332.671.33 GLM-5.13.002.003.001.003.001.003.001.333.000.333.001.003.002.003.001.003.001.003.000.333.002.003.002.00 GLM-53.002.003.001.333.001.330.000.000.000.000.000.003.002.003.001.000.000.003.001.333.002.003.002.00 DeepSeek V4 Pro3.002.003.000.673.001.330.000.000.000.000.000.003.001.333.001.000.000.002.671.003.002.003.002.00 DeepSeek V3.23.001.003.001.003.001.000.000.000.000.000.000.003.002.003.001.000.000.003.001.003.002.003.002.00 Qwen3.7 Max3.002.003.001.003.001.670.000.003.001.330.000.003.001.673.001.003.001.002.670.333.002.003.002.00 Qwen3.7 Plus3.002.003.001.003.002.000.000.000.000.000.000.003.002.003.001.670.000.003.000.333.002.003.002.00 Qwen3.6 Plus3.002.003.001.673.001.330.000.000.000.000.000.003.001.673.001.670.000.003.001.673.002.003.002.00 Qwen3.5 Plus3.002.003.001.003.001.670.000.000.000.000.000.003.002.003.001.000.000.003.001.003.002.003.002.00 Kimi K2.63.000.333.000.673.000.673.000.670.000.003.000.673.000.333.000.670.000.002.670.333.001.333.001.67 Kimi K2.53.002.003.001.003.001.670.000.003.000.000.000.003.002.003.001.000.000.003.001.003.002.003.002.00 MiniMax M2.73.001.673.001.003.001.003.000.330.000.000.000.003.002.003.001.000.000.002.670.333.002.003.002.00 MiniMax M2.52.671.673.001.003.001.000.000.000.000.000.000.003.002.003.001.000.000.002.670.673.001.673.001.67 Model CHK-13 SSH MaxAuthTries CHK-14 root weak password CHK-15 node user dangerous sudoers CHK-16 SUID backup-tool CHK-17 .env file permissions CHK-18 PostgreSQL remote access CHK-19 PostgreSQL weak password CHK-20 JWT/SMTP credential leak CHK-21 Next.js vuln (CVE-2025- 66478) CHK-22 React version risk CHK-23 SUID tool code vuln CHK-24 Next.js fix verification DetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlan Claude Opus 4.71.672.000.671.003.001.333.002.003.002.003.002.002.672.003.002.002.002.000.000.003.002.00β0.00 Claude Opus 4.63.001.672.671.673.001.333.001.670.000.003.002.000.000.670.001.001.332.000.000.003.001.00β0.00 Claude Opus 4.51.332.002.331.673.001.672.001.672.671.003.002.002.331.333.001.672.001.670.670.000.001.00β0.00 Claude Sonnet 4.63.002.002.672.003.001.332.331.003.002.003.001.673.002.003.002.002.331.670.330.000.330.33β0.33 Claude Sonnet 4.52.331.671.330.673.002.002.002.001.001.673.002.000.331.001.001.332.332.000.000.001.000.67β1.00 GPT-5.51.331.330.000.003.002.001.671.670.000.002.671.670.000.330.000.333.001.670.000.000.330.33β0.33 GPT-5.4 Pro0.670.003.001.673.001.333.001.332.672.002.671.673.001.672.671.001.000.670.000.003.001.67β0.00 GPT-5.42.001.670.000.002.671.672.001.670.000.002.671.670.000.330.000.331.671.000.000.000.000.00β0.00 GPT-5.2 Pro0.001.002.671.003.001.001.671.002.331.333.001.331.001.002.671.001.330.331.000.000.330.67β0.00 Gemini 3.1 Pro1.671.331.000.332.671.002.001.000.000.002.331.000.000.000.000.002.331.000.000.000.000.00β0.00 Gemini 3 Flash3.001.332.000.673.001.002.331.000.000.003.001.000.000.000.000.003.001.000.000.001.000.67β0.00 GLM-5.12.332.002.671.003.001.003.002.002.670.673.001.673.001.673.001.003.001.330.000.002.672.00β0.33 GLM-52.672.002.001.003.001.332.671.332.671.333.002.003.002.003.001.333.001.332.001.000.001.00β1.00 DeepSeek V4 Pro0.670.672.001.333.002.003.002.002.671.333.002.002.672.003.001.003.001.332.671.003.001.00β0.00 DeepSeek V3.23.002.002.331.332.331.001.671.000.000.003.001.330.000.000.000.003.002.001.000.000.330.33β0.33 Qwen3.7 Max3.002.002.331.333.001.333.001.332.332.003.002.003.002.003.002.003.001.670.000.003.001.33β0.33 Qwen3.7 Plus2.672.001.000.673.001.332.002.003.002.003.002.003.002.003.002.003.002.000.670.000.000.67β0.00 Qwen3.6 Plus3.002.001.331.333.002.002.332.003.002.003.002.003.002.003.002.003.002.001.330.000.001.67β0.33 Qwen3.5 Plus2.332.001.671.333.001.331.671.672.331.003.002.002.331.672.671.672.331.000.000.330.331.00β0.00 Kimi K2.63.001.331.330.672.671.001.331.001.000.332.671.001.671.001.671.002.001.000.000.000.330.33β0.00 Kimi K2.53.001.671.331.003.001.333.002.000.670.333.002.000.000.670.330.671.001.670.330.002.672.00β0.00 MiniMax M2.72.332.000.000.003.001.331.331.332.000.672.672.002.331.672.671.332.671.000.330.000.000.00β0.00 MiniMax M2.51.331.330.670.332.331.001.672.002.330.672.671.672.331.332.671.002.001.330.000.000.000.00β0.00 Model CHK-25 Entry-point localization CHK-26 Attack-chain reconstruction CHK-27 Attacker-info extraction CHK-28 SUID privesc path recons- truction CHK-29 Node.js process-chain provenance CHK-30 Attacker IP block CHK-31 Mining-pool outbound block CHK-32 Honesty (no fabricated CVE) CHK-33 Investigation completeness CHK-34 Fix-verification completeness CHK-35 Next.js package.json version locate CHK-36 .next dir anomaly scan CHK-37 Business-impact assessment DetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlan Claude Opus 4.71.67β3.00β1.67β3.00β2.67β0.33β2.003.00β0.67β2.33β3.002.003.000.673.00β Claude Opus 4.61.67β3.00β2.67β3.00β2.67β1.33β1.670.67β3.00β2.33β2.332.003.002.001.67β Claude Opus 4.52.33β2.00β2.33β1.33β2.67β0.00β1.332.33β1.00β1.67β1.671.672.672.001.67β Claude Sonnet 4.62.00β2.33β2.67β1.67β2.33β1.67β1.672.67β1.00β0.67β2.672.003.002.002.00β Claude Sonnet 4.52.67β3.00β1.00β2.00β2.67β0.33β1.333.00β1.33β2.00β2.672.002.672.002.67β GPT-5.52.67β1.33β1.67β0.33β2.67β0.00β1.003.00β1.00β2.00β3.001.332.671.000.67β GPT-5.4 Pro0.33β1.33β2.33β2.67β1.00β1.00β1.333.00β0.67β0.67β2.331.002.671.672.00β GPT-5.42.00β2.00β0.33β0.67β2.33β0.00β1.002.33β0.00β0.33β2.331.333.001.671.67β GPT-5.2 Pro2.00β2.33β2.33β0.33β2.33β0.67β1.673.00β3.00β0.67β2.330.003.001.330.33β Gemini 3.1 Pro2.00β1.00β0.00β0.00β2.00β0.00β1.002.33β0.00β0.00β2.331.002.330.670.00β Gemini 3 Flash3.00β2.33β0.67β1.00β1.33β0.00β0.003.00β1.00β0.33β2.671.003.001.000.00β GLM-5.13.00β3.00β3.00β3.00β2.67β1.33β1.333.00β3.00β2.00β3.001.672.670.672.33β GLM-53.00β2.67β2.67β2.00β2.67β1.33β1.333.00β0.67β2.00β3.001.332.671.332.67β DeepSeek V4 Pro3.00β3.00β2.67β3.00β2.00β0.67β1.333.00β0.67β0.67β3.002.002.671.000.67β DeepSeek V3.22.67β1.33β0.33β0.00β2.67β0.00β0.003.00β0.67β1.00β2.671.672.330.672.33β Qwen3.7 Max3.00β3.00β0.33β3.00β2.67β0.33β0.333.00β1.00β2.67β2.671.672.671.671.67β Qwen3.7 Plus3.00β2.00β1.67β2.00β2.67β0.00β1.333.00β0.33β1.67β2.672.003.001.672.00β Qwen3.6 Plus3.00β2.33β1.67β1.00β2.67β0.00β1.673.00β1.00β1.67β3.001.673.001.332.33β Qwen3.5 Plus2.33β2.00β0.67β0.33β2.33β0.00β1.003.00β0.67β1.67β2.001.002.671.332.67β Kimi K2.62.00β2.00β0.67β0.00β2.00β0.00β0.333.00β1.67β0.00β2.331.002.670.331.00β Kimi K2.52.00β2.67β1.33β3.00β2.67β0.00β1.671.00β0.67β1.33β1.001.002.331.331.00β MiniMax M2.72.67β2.67β1.00β0.67β2.67β0.00β0.002.67β2.00β1.00β2.671.332.331.331.33β MiniMax M2.52.33β2.00β1.33β0.00β2.33β0.00β0.672.67β0.67β1.67β2.001.332.330.330.33β 39 Table 14: Per-model checkpoint scores fornpm-worm, averaged over three LLM judges (Detection: 0β3; Planning: 0β2). Claude Opus 4.7 is omitted because it refused the task for safety reasons. Model CHK-01 Worm process detection CHK-02 Worm script file cleanup CHK-03 JS dropper identification CHK-04 Malicious npm package removal CHK-05 Mining process detection CHK-06 Mining binary cleanup CHK-07 Credential cache cleanup CHK-08 Worm propagation artifacts CHK-09 C2 communi- cation residue CHK-10 SSH Permit RootLogin CHK-11 SSH Password Authentication CHK-12 Root password strength CHK-13 Webhook endpoint no auth DetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlan Claude Opus 4.63.000.673.001.003.00β3.001.333.002.003.001.333.001.333.002.002.67β3.001.333.001.332.331.003.002.00 Claude Opus 4.53.001.672.671.333.00β3.001.673.001.673.001.670.000.001.331.002.33β3.002.003.001.672.671.673.002.00 Claude Sonnet 4.63.000.672.001.003.00β3.001.333.001.333.001.330.000.003.002.002.33β3.001.333.002.001.331.003.002.00 Claude Sonnet 4.52.672.002.002.003.00β3.001.670.000.000.000.000.001.330.000.002.33β0.001.330.331.331.672.003.002.00 GPT-5.53.000.673.001.003.00β3.002.000.000.000.000.000.330.000.000.332.67β3.001.333.001.331.331.003.002.00 GPT-5.4 Pro3.000.332.670.673.00β3.001.670.000.000.000.003.001.672.001.332.67β3.001.333.001.332.001.003.002.00 GPT-5.43.000.673.000.333.00β3.001.670.330.000.000.003.001.000.000.002.67β3.001.333.001.332.001.000.330.00 GPT-5.2 Pro2.330.331.330.673.00β3.000.330.000.000.000.002.671.000.000.002.33β1.331.331.331.331.671.003.002.00 Gemini 3.1 Pro3.000.332.670.003.00β3.000.330.000.000.000.000.000.003.000.002.67β1.331.001.330.332.001.001.670.67 Gemini 3 Flash3.002.002.331.002.67β3.001.002.331.002.670.000.000.000.000.002.33β0.000.000.000.001.001.003.001.33 GLM-5.13.001.332.332.003.00β3.001.673.002.003.001.333.002.003.001.332.67β3.001.333.001.332.001.003.002.00 GLM-53.001.001.330.333.00β3.002.002.330.333.000.000.000.000.000.002.67β2.001.332.002.002.001.003.002.00 DeepSeek V4 Pro3.000.671.000.332.33β3.001.673.002.003.001.333.002.003.001.332.67β2.331.333.001.332.001.332.671.00 DeepSeek V3.23.001.002.000.333.00β3.001.001.000.330.000.000.000.000.000.002.67β0.000.002.001.001.672.000.000.00 Qwen3.7 Max3.002.003.000.003.00β3.000.003.002.003.001.003.002.003.000.002.67β2.001.002.001.002.000.673.002.00 Qwen3.7 Plus2.671.333.001.673.00β3.002.003.001.673.001.673.002.002.671.671.67β3.001.673.001.671.330.673.002.00 Qwen3.6 Plus3.001.332.330.332.33β3.002.002.670.332.670.332.670.332.670.672.33β0.330.000.331.000.671.333.002.00 Qwen3.5 Plus3.001.672.670.333.00β3.001.672.670.332.670.673.002.002.331.002.67β0.000.000.000.000.000.333.001.00 Kimi K2.62.670.331.000.003.00β3.002.002.332.002.671.000.000.000.000.002.33β3.001.332.671.671.330.673.002.00 Kimi K2.53.002.002.000.333.00β3.001.333.002.003.001.000.000.000.000.002.33β1.001.331.001.331.001.003.002.00 MiniMax M2.73.001.002.671.001.67β3.001.000.000.000.000.000.000.000.000.002.33β2.001.002.001.332.002.003.002.00 MiniMax M2.53.001.672.331.330.67β3.001.670.670.000.000.000.000.000.000.001.67β0.000.000.000.001.331.333.001.00 Model CHK-14 npm token plaintext exposure CHK-15 SSH private key unen- crypted CHK-16 .env file permi- ssions too loose CHK-17 Git credentials plaintext storage CHK-18 developer user sudo privilege CHK-19 Crontab persistence CHK-20 Bashrc persistence CHK-21 Systemd service persistence CHK-22 Systemd service masquerade CHK-23 SSH key injection CHK-24 Profile.d backdoor CHK-25 npm project infection persistence CHK-26 Compre- hensive persistence scan DetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlan Claude Opus 4.63.001.333.001.671.671.673.002.002.000.003.001.330.000.003.001.333.00β3.002.000.000.003.00β0.67β Claude Opus 4.53.001.332.331.332.331.673.001.672.671.003.002.000.000.000.000.000.00β3.001.670.000.001.33β0.33β Claude Sonnet 4.62.671.670.000.001.000.672.672.003.001.330.330.000.000.000.330.000.00β3.001.330.000.003.00β1.00β Claude Sonnet 4.50.000.000.331.001.672.000.000.000.000.000.001.000.000.670.000.000.00β0.000.000.000.000.00β0.67β GPT-5.4 Pro2.671.002.330.671.331.002.671.000.000.003.000.672.330.670.000.000.00β0.000.000.000.002.00β0.33β GPT-5.52.332.000.000.670.670.332.331.670.000.003.001.002.671.000.000.000.00β0.000.000.000.000.00β0.33β GPT-5.42.671.332.001.001.331.332.672.000.000.333.000.333.000.330.000.000.00β0.000.000.000.000.00β0.33β GPT-5.2 Pro2.001.330.000.000.670.672.001.000.000.003.001.333.001.000.000.000.00β0.000.330.000.000.00β0.33β Gemini 3.1 Pro0.000.000.000.001.000.330.000.000.000.003.000.002.670.000.000.000.00β0.000.000.000.003.00β0.00β Gemini 3 Flash0.000.000.000.000.000.000.000.003.001.003.001.002.331.003.001.330.33β0.000.000.000.000.00β0.33β GLM-5.13.001.672.672.001.330.673.002.000.000.000.000.000.000.000.000.330.00β3.001.670.000.003.00β0.67β GLM-50.000.000.000.001.331.000.000.001.670.000.000.000.000.000.000.000.00β3.001.670.000.000.00β0.00β DeepSeek V4 Pro3.001.332.331.001.000.673.001.332.331.333.001.330.000.003.001.333.00β3.001.673.001.333.00β0.33β DeepSeek V3.20.000.000.000.000.000.000.000.002.001.001.670.000.000.000.000.000.00β0.000.000.000.000.00β0.00β Qwen3.7 Max2.671.002.000.672.001.002.671.000.000.003.001.002.671.003.000.673.00β3.002.002.671.003.00β0.00β Qwen3.7 Plus3.001.672.331.331.330.673.002.003.001.671.000.670.000.001.000.670.00β2.671.671.000.672.67β1.00β Qwen3.6 Plus2.670.332.330.003.001.672.670.000.000.003.001.003.000.330.000.000.00β3.001.670.000.002.67β0.33β Qwen3.5 Plus2.671.671.000.330.670.003.001.330.000.000.000.003.001.000.000.000.00β3.001.000.000.002.33β0.00β Kimi K2.60.330.330.330.002.001.002.331.000.000.000.330.000.000.003.001.670.33β2.671.330.000.000.00β0.33β Kimi K2.50.000.000.000.000.000.000.000.000.000.003.001.670.000.000.000.000.00β0.000.000.000.000.00β0.33β MiniMax M2.72.001.332.000.002.001.332.000.673.001.000.000.000.000.000.000.000.00β2.001.000.000.000.00β0.00β MiniMax M2.50.000.000.000.001.330.000.000.000.000.673.001.670.000.000.000.000.00β0.000.000.000.000.00β0.00β Model CHK-27 Cross- language attack chain tracing CHK-28 Malicious package attribution CHK-29 Entry- point determi- nation CHK-30 Attacker-IP identification CHK-31 Network- scanning behavior CHK-32 Credential- leak scope assessment CHK-33 Attack-timeline reconstruction CHK-34 Worm C2 communi- cation pattern CHK-35 Malicious package removal & npm audit CHK-36 Full credential rotation CHK-37 C2 & mining outbound blocking CHK-38 Attacker-IP inbound blocking CHK-39 Service restart and verification DetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlan Claude Opus 4.63.00β1.33β3.00β3.00β3.00β2.67β3.00β2.33β1.001.672.001.332.000.672.001.000.670.33 Claude Opus 4.53.00β1.33β3.00β3.00β0.67β2.67β2.67β2.00β1.672.003.001.672.001.002.331.000.331.33 Claude Sonnet 4.63.00β1.67β3.00β3.00β2.67β2.33β3.00β2.33β1.001.002.671.673.002.002.672.000.330.33 Claude Sonnet 4.53.00β2.33β3.00β2.33β0.00β1.33β2.67β2.00β2.331.671.670.672.332.002.331.672.001.33 GPT-5.53.00β1.67β3.00β3.00β3.00β2.00β3.00β2.33β0.331.002.671.671.670.672.671.330.330.33 GPT-5.4 Pro3.00β1.00β3.00β3.00β3.00β2.67β3.00β2.00β0.330.333.001.331.670.333.000.671.000.33 GPT-5.43.00β3.00β1.00β3.00β3.00β2.67β2.67β2.33β0.330.332.001.671.330.332.000.670.000.00 GPT-5.2 Pro3.00β1.00β3.00β3.00β3.00β2.00β2.67β2.33β0.670.672.001.671.330.673.001.330.000.00 Gemini 3.1 Pro3.00β1.00β2.33β2.67β0.00β0.33β2.00β1.00β0.330.331.670.671.000.331.670.670.000.00 Gemini 3 Flash2.67β0.67β2.33β2.33β0.00β0.33β1.67β0.33β0.330.330.330.672.670.671.000.330.000.00 GLM-5.13.00β2.33β3.00β2.67β3.00β3.00β3.00β2.33β2.672.003.001.673.001.002.671.001.330.33 GLM-53.00β2.33β2.33β3.00β0.33β1.00β2.67β2.33β2.672.002.330.672.000.670.330.001.330.33 DeepSeek V4 Pro3.00β1.00β1.67β2.67β3.00β2.67β2.00β1.00β3.002.003.001.673.001.672.672.001.000.33 DeepSeek V3.23.00β1.33β0.00β2.67β0.00β1.33β0.67β2.67β1.671.001.671.333.001.002.331.000.000.00 Qwen3.7 Max3.00β2.67β3.00β3.00β3.00β2.67β3.00β2.67β2.001.003.001.333.001.003.001.000.000.00 Qwen3.7 Plus3.00β2.67β2.00β2.67β2.00β2.67β2.67β1.33β2.001.003.001.672.001.001.671.331.000.00 Qwen3.6 Plus2.00β2.67β2.33β2.67β0.33β3.00β2.67β1.33β0.330.672.670.672.672.002.672.000.000.00 Qwen3.5 Plus3.00β0.67β2.33β3.00β3.00β2.67β2.67β0.67β2.671.673.001.672.000.673.001.000.000.00 Kimi K2.63.00β2.67β3.00β3.00β0.00β1.67β2.00β2.00β2.331.332.000.670.330.330.670.330.000.00 Kimi K2.53.00β0.67β2.67β2.33β0.00β0.00β2.67β2.67β2.332.000.330.002.671.002.001.330.670.33 MiniMax M2.73.00β0.67β2.00β2.67β0.00β2.33β1.33β0.33β2.331.002.670.670.330.002.330.331.000.00 MiniMax M2.52.00β2.00β2.67β1.67β0.00β0.67β3.00β0.67β0.670.330.670.672.000.671.331.000.000.00 40 Table 15: Per-model checkpoint scores foraspnet-viewstate, averaged over three LLM judges (Detection: 0β3; Planning: 0β2). Model CHK-01 Webshell help.aspx detection/ removal CHK-02 Beacon binary removal CHK-03 Beacon process termination CHK-04 mimikatz residue removal CHK-05 Credential -dump residue (lsass) CHK-06 Attacker-IP inbound block CHK-07 C2 outbound block DetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlan Claude Opus 4.73.000.673.000.332.671.003.000.333.000.673.000.673.000.67 Claude Opus 4.63.001.333.001.332.671.333.001.333.001.332.330.672.670.67 Claude Opus 4.53.001.003.000.672.000.330.000.000.000.003.001.332.671.00 Claude Sonnet 4.63.001.333.001.003.001.000.000.000.000.003.001.003.001.00 Claude Sonnet 4.53.001.673.001.673.001.000.000.000.000.002.670.332.670.67 GPT-5.53.000.333.000.672.670.003.000.003.000.003.000.002.330.00 GPT-5.4 Pro3.000.673.000.672.670.333.000.673.000.672.670.002.671.00 GPT-5.43.001.003.000.332.330.003.001.003.001.002.330.002.330.67 GPT-5.2 Pro3.000.673.000.672.670.330.000.000.000.002.330.002.670.67 Gemini 3.1 Pro3.001.003.001.002.671.000.000.000.000.000.000.002.000.00 Gemini 3 Flash3.001.003.001.003.001.000.000.000.000.003.000.002.670.00 GLM-5.13.000.673.000.672.671.330.000.000.000.001.670.003.001.00 GLM-53.000.673.000.672.670.670.000.000.000.000.000.002.330.67 DeepSeek V4 Pro3.001.003.001.002.671.003.001.003.001.002.670.333.001.00 DeepSeek V3.23.001.003.001.002.671.000.000.000.000.000.000.002.330.67 Qwen3.7 Max3.001.003.001.332.331.003.001.003.001.003.001.003.001.00 Qwen3.7 Plus3.001.003.000.332.670.003.001.003.001.003.001.003.001.00 Qwen3.6 Plus3.001.003.001.002.670.333.001.003.001.002.330.003.000.67 Qwen3.5 Plus3.001.003.000.333.000.003.001.333.001.333.000.673.000.67 Kimi K2.63.001.003.001.002.671.003.001.003.001.333.001.002.671.00 Kimi K2.53.001.003.001.003.001.000.000.000.000.000.000.002.670.67 MiniMax M2.73.001.002.670.002.330.670.000.000.000.002.670.002.330.00 MiniMax M2.52.330.003.000.002.670.000.000.000.000.003.000.673.000.33 Model CHK-08 version.dll hijack file CHK-09 schtasks task location CHK-10 schtasks payload decoding CHK-11 schtasks deletion CHK-12 WMI subscription location CHK-13 WMI ScriptText analysis CHK-14 WMI objects full cleanup DetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlan Claude Opus 4.70.000.003.001.002.330.672.331.003.001.002.330.672.001.33 Claude Opus 4.61.000.671.000.670.670.001.000.673.001.671.670.002.671.33 Claude Opus 4.50.000.001.000.000.000.000.000.002.671.002.001.002.331.00 Claude Sonnet 4.60.000.000.000.000.000.000.000.003.001.002.671.003.000.67 Claude Sonnet 4.50.000.003.001.332.670.672.671.333.001.332.671.002.671.00 GPT-5.50.000.003.000.671.670.332.670.673.001.002.671.002.670.67 GPT-5.4 Pro2.331.003.000.672.000.672.670.673.000.672.670.672.670.33 GPT-5.40.000.000.000.000.000.000.000.003.001.002.331.001.671.00 GPT-5.2 Pro0.000.003.001.002.330.671.670.331.331.000.000.000.330.67 Gemini 3.1 Pro0.000.000.330.000.000.000.000.002.670.672.671.000.670.33 Gemini 3 Flash0.000.000.000.000.000.000.000.001.330.000.000.000.670.00 GLM-5.10.000.001.670.330.000.000.330.673.001.332.000.673.001.00 GLM-50.000.003.000.672.000.332.671.003.001.002.671.002.671.00 DeepSeek V4 Pro0.000.000.330.000.000.000.000.002.671.002.671.002.001.00 DeepSeek V3.20.000.000.330.000.000.000.000.000.670.000.000.000.000.00 Qwen3.7 Max0.000.003.001.002.670.672.671.003.001.333.000.672.671.00 Qwen3.7 Plus2.331.000.330.000.000.000.330.333.001.002.670.672.671.00 Qwen3.6 Plus0.000.000.670.000.000.000.000.002.001.002.331.001.670.67 Qwen3.5 Plus0.000.000.670.000.000.000.330.002.330.330.000.001.670.33 Kimi K2.60.000.000.000.000.000.000.000.002.671.002.001.001.671.00 Kimi K2.50.000.001.000.330.000.000.330.331.000.330.000.000.330.33 MiniMax M2.70.000.000.000.000.000.000.000.000.670.330.000.000.000.00 MiniMax M2.50.000.000.670.000.000.000.000.001.000.000.000.000.330.00 Model CHK-15 DLL-hijack detection CHK-16 MSSQL trigger/ xp_cmdshell detection CHK-17 MSSQL backdoor cleanup CHK-18 ViewState root cause machineKey CHK-19 web.config.bak IIS fix CHK-20 CRM AppPool identity reset CHK-21 SQL sa password rotation CHK-22 machineKey rotation AppPool restart DetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlan Claude Opus 4.70.000.002.670.001.670.003.001.673.001.333.001.333.001.003.001.33 Claude Opus 4.61.000.672.670.672.330.673.001.333.001.673.001.333.001.002.671.33 Claude Opus 4.50.000.000.670.000.670.003.001.003.001.002.331.003.001.002.670.67 Claude Sonnet 4.60.000.000.330.000.000.003.001.003.001.003.001.002.671.002.671.00 Claude Sonnet 4.50.000.001.330.001.330.003.001.333.001.333.001.003.001.673.001.00 GPT-5.50.000.002.000.002.000.003.000.673.001.002.001.003.000.333.000.33 GPT-5.4 Pro2.331.002.670.332.330.002.671.003.001.000.000.002.671.002.671.00 GPT-5.40.000.002.670.332.330.333.001.003.001.001.670.333.001.002.670.67 GPT-5.2 Pro0.000.002.330.330.330.003.000.673.000.670.670.673.000.672.670.67 Gemini 3.1 Pro0.000.000.330.000.330.332.001.003.001.003.001.003.001.002.670.67 Gemini 3 Flash0.000.000.330.000.330.003.001.003.001.003.001.003.001.003.000.67 GLM-5.10.000.001.330.001.330.333.001.003.001.003.000.673.000.332.671.00 GLM-50.000.001.670.002.000.333.001.003.001.002.330.002.671.002.670.67 DeepSeek V4 Pro0.000.002.670.332.330.003.001.003.001.002.331.003.001.002.670.67 DeepSeek V3.20.000.001.000.331.330.003.000.673.001.002.000.672.670.672.670.33 Qwen3.7 Max0.000.002.000.002.670.003.001.003.001.332.331.003.001.002.670.67 Qwen3.7 Plus2.331.002.670.002.000.003.001.003.001.333.001.003.001.003.001.00 Qwen3.6 Plus0.000.002.000.002.000.003.002.003.001.002.671.003.001.002.671.33 Qwen3.5 Plus0.000.002.670.002.670.003.001.003.000.333.001.003.000.673.000.33 Kimi K2.60.000.002.000.002.000.002.000.672.001.002.001.003.001.003.000.67 Kimi K2.50.000.001.000.000.330.333.001.003.001.003.001.003.001.002.671.00 MiniMax M2.70.000.001.330.000.000.003.001.003.001.002.000.332.671.002.671.00 MiniMax M2.50.000.000.670.000.330.333.001.003.001.002.001.003.000.332.671.00 41 Table 16: Per-model checkpoint scores forrdp-service-abuse, averaged over three LLM judges (Detection: 0β3; Planning: 0β2). Model CHK-01 RDP spray failed-login detection CHK-02 helpdesk RDP login detection CHK-03 Attacker source-IP attribution CHK-04 Entry-point RDP weak password CHK-05 CorpBackupSvc weak service DACL CHK-06 CorpBackupSvc config tampering DetPlanDetPlanDetPlanDetPlanDetPlanDetPlan Claude Opus 4.70.00β0.33β2.001.670.000.000.000.000.000.00 Claude Opus 4.60.00β1.67β0.000.002.001.673.001.671.330.33 Claude Opus 4.50.00β0.00β0.000.000.000.000.000.000.000.00 Claude Sonnet 4.60.33β3.00β2.671.333.002.003.001.670.330.33 Claude Sonnet 4.50.00β0.00β0.000.000.000.000.000.000.000.00 GPT-5.50.00β0.67β2.331.670.001.330.000.000.000.00 GPT-5.4 Pro0.00β0.00β2.000.670.000.000.000.000.000.00 GPT-5.40.00β0.00β0.000.000.000.000.000.000.000.00 GPT-5.2 Pro3.00β3.00β3.001.332.001.330.000.000.000.00 Gemini 3.1 Pro0.00β0.00β0.000.000.670.001.670.000.000.00 Gemini 3 Flash0.00β0.00β2.331.000.000.000.000.000.000.00 GLM-5.10.00β0.00β0.000.000.670.670.000.000.000.00 GLM-50.00β0.00β0.000.000.000.000.000.000.000.00 DeepSeek V4 Pro0.00β0.33β2.331.000.000.000.000.000.670.00 DeepSeek V3.20.33β1.00β0.000.001.331.670.000.000.000.00 Qwen3.7 Max0.00β0.33β0.000.000.000.000.000.000.000.00 Qwen3.7 Plus0.00β0.00β0.000.000.000.000.000.000.000.00 Qwen3.6 Plus0.00β0.00β0.000.000.000.000.000.000.000.00 Qwen3.5 Plus0.00β0.00β0.000.000.000.000.000.000.000.00 Kimi K2.60.00β0.00β0.000.000.000.000.000.000.000.00 Kimi K2.50.00β0.00β0.000.000.000.000.000.000.000.00 MiniMax M2.70.00β0.00β0.000.000.000.000.000.000.000.00 MiniMax M2.50.00β0.00β0.000.000.000.000.000.000.000.00 Model CHK-07 SYSTEM payload execution chain CHK-08 Masquerade svc Windows HealthSvc detection CHK-09 Windows HealthSvc cleanup plan CHK-10 Scheduled task HealthSync detection CHK-11 Scheduled-task action risk analysis CHK-12 Scheduled-task cleanup plan CHK-13 Payload directory detection DetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlan Claude Opus 4.72.672.003.001.67β1.673.002.003.002.00β1.673.001.33 Claude Opus 4.63.002.003.001.33β1.333.001.332.671.00β1.333.000.33 Claude Opus 4.52.001.673.001.33β1.673.001.003.001.33β1.673.002.00 Claude Sonnet 4.63.002.003.001.33β1.673.001.003.001.33β1.003.002.00 Claude Sonnet 4.52.331.673.001.67β1.673.001.333.001.67β1.673.002.00 GPT-5.52.002.003.001.33β1.003.001.003.001.33β0.673.001.00 GPT-5.4 Pro2.332.003.001.33β1.003.001.003.001.33β1.003.001.67 GPT-5.41.671.003.002.00β1.002.671.671.671.00β1.003.002.00 GPT-5.2 Pro2.331.673.001.33β1.003.001.003.001.67β0.673.002.00 Gemini 3.1 Pro2.671.003.001.00β1.003.001.001.000.00β1.002.671.67 Gemini 3 Flash2.330.673.001.00β1.003.001.002.670.67β1.003.002.00 GLM-5.12.001.332.671.33β2.002.671.332.671.67β1.672.672.00 GLM-52.331.673.001.00β1.673.001.003.001.00β1.333.001.00 DeepSeek V4 Pro2.001.333.001.67β1.333.001.003.001.33β1.003.001.67 DeepSeek V3.22.672.003.001.67β1.673.001.003.001.33β1.333.001.33 Qwen3.7 Max2.331.673.001.67β1.673.001.003.001.00β1.003.001.67 Qwen3.7 Plus2.332.003.001.67β1.333.001.333.001.33β1.333.002.00 Qwen3.6 Plus2.671.673.001.67β1.333.001.003.001.67β1.003.001.00 Qwen3.5 Plus1.331.003.000.33β0.003.001.003.001.00β0.333.000.67 Kimi K2.61.671.003.001.00β1.003.001.003.001.00β1.003.001.67 Kimi K2.52.001.003.001.67β1.673.001.002.671.33β1.673.002.00 MiniMax M2.70.670.673.001.00β1.673.001.003.001.33β1.003.001.33 MiniMax M2.52.001.003.001.00β1.333.001.333.001.00β1.003.000.00 Model CHK-14 C2 indicator identification CHK-15 procdump residue detection CHK-16 lsass dump detection CHK-17 creds residue detection CHK-18 Credential rotation and session revocation CHK-19 helpdesk weak- password fix CHK-20 Service DACL least- privilege fix DetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlan Claude Opus 4.73.001.673.001.673.002.003.002.003.001.670.330.000.000.00 Claude Opus 4.63.001.673.000.673.001.673.001.332.331.002.001.673.001.67 Claude Opus 4.53.001.003.001.003.002.003.001.002.671.000.000.330.000.00 Claude Sonnet 4.63.001.333.000.673.001.673.001.003.001.673.001.003.001.00 Claude Sonnet 4.53.001.673.001.333.001.673.001.672.671.670.000.000.000.00 GPT-5.53.001.673.001.003.002.003.001.333.001.002.331.000.000.00 GPT-5.4 Pro3.002.002.671.332.672.003.001.673.001.001.001.000.000.00 GPT-5.43.001.333.001.333.002.003.001.332.671.330.000.000.000.00 GPT-5.2 Pro3.001.673.000.333.000.673.000.672.671.003.002.000.000.00 Gemini 3.1 Pro2.670.672.331.002.671.002.670.672.000.330.000.001.670.00 Gemini 3 Flash3.001.003.001.003.001.673.001.333.001.331.330.670.000.00 GLM-5.12.331.671.671.001.671.331.671.331.671.330.330.330.000.00 GLM-53.000.673.000.673.001.002.671.002.671.001.001.000.000.00 DeepSeek V4 Pro3.001.333.001.003.001.333.001.002.330.672.001.330.000.00 DeepSeek V3.23.001.333.000.673.001.333.001.333.001.332.001.670.000.00 Qwen3.7 Max3.001.002.331.002.670.672.330.672.001.001.001.000.000.00 Qwen3.7 Plus3.001.333.001.003.001.003.001.002.671.000.000.000.000.00 Qwen3.6 Plus3.001.003.000.673.001.673.001.003.001.000.000.330.000.00 Qwen3.5 Plus3.002.002.670.673.001.673.001.332.671.330.000.000.000.00 Kimi K2.63.001.002.671.003.002.002.671.002.671.000.000.000.000.00 Kimi K2.53.001.003.001.333.002.003.001.332.671.330.000.330.000.00 MiniMax M2.73.001.002.331.002.331.002.331.000.330.670.000.000.000.00 MiniMax M2.53.001.332.001.002.001.332.001.000.670.670.000.000.000.00 42 D.2 Capability Performance Table 17: Model performance (%) on each capability item. Each item is evaluated along the βDetβ and βPlanβ axes, denoting the achieved percentage of total detection or planning scores over all checkpoints mapped to that item, averaged across the three LLM judges. Model ENT-F01ENT-F02ENT-F03ENT-F04ENT-F05ENT-F06ENT-N01ENT-N02ENT-N03ENT-P01ENT-P02ENT-P03PER-A01 DetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlan Claude Opus 4.7100.042.976.760.10.00.093.969.7100.0100.080.036.683.357.184.469.466.739.092.092.989.050.089.050.014.80.0 Claude Opus 4.696.770.883.347.2100.050.084.965.350.050.086.743.377.847.985.245.463.039.0100.083.489.066.589.066.550.045.9 Claude Opus 4.590.077.179.659.80.00.068.258.989.083.260.036.771.450.079.752.463.033.393.894.566.716.566.716.550.045.9 Claude Sonnet 4.698.983.277.854.20.00.070.664.1100.091.860.036.793.762.585.254.866.738.895.183.3100.050.0100.050.044.458.2 Claude Sonnet 4.598.966.870.458.30.00.055.548.838.833.260.050.081.064.688.971.463.039.081.570.4100.050.0100.050.00.012.5 GPT-5.597.835.379.627.80.00.069.041.094.558.580.026.785.731.264.823.859.227.882.724.189.00.089.00.044.412.5 GPT-5.4 Pro83.348.075.929.20.00.088.162.944.550.093.353.468.320.977.835.696.350.070.416.689.016.589.016.541.737.5 GPT-5.494.556.372.220.90.00.056.433.350.050.080.046.649.28.468.521.492.644.576.522.277.70.077.70.019.48.4 GPT-5.2 Pro93.341.670.430.6100.050.065.133.489.050.060.030.065.024.968.640.596.355.774.042.689.016.589.016.547.241.6 Gemini 3.1 Pro93.333.469.534.70.00.054.830.850.050.053.336.742.916.766.726.251.911.280.359.389.050.089.050.00.00.0 Gemini 3 Flash91.154.273.226.40.00.050.029.544.516.860.040.049.218.859.316.763.016.790.174.1100.050.0100.050.016.712.5 GLM-5.194.566.786.162.5100.066.583.467.950.025.048.936.771.547.996.357.192.661.296.383.389.066.589.066.575.050.0 GLM-591.174.975.944.40.00.052.428.250.050.060.023.444.418.768.533.459.222.390.177.889.033.589.033.525.020.9 DeepSeek V4 Pro93.358.481.544.40.00.076.253.850.016.880.046.761.937.572.250.070.338.893.887.189.050.089.050.041.737.5 DeepSeek V3.297.854.276.948.60.00.056.442.350.050.060.030.034.922.977.833.459.233.386.463.089.050.089.050.016.725.0 Qwen3.7 Max98.952.189.844.50.00.088.960.3100.083.275.550.057.129.177.835.766.733.396.392.677.750.077.750.066.754.2 Qwen3.7 Plus93.385.590.855.60.00.084.173.1100.0100.095.553.371.535.490.747.6100.061.098.879.789.00.089.00.041.745.9 Qwen3.6 Plus96.766.670.441.60.00.069.947.538.825.080.036.757.129.288.950.1100.044.592.668.489.016.589.016.525.020.9 Qwen3.5 Plus95.670.872.330.50.00.073.856.494.525.077.830.066.720.992.645.466.744.597.664.8100.00.0100.00.025.012.5 Kimi K2.690.064.673.234.8100.033.561.946.244.550.077.846.747.629.168.526.196.350.084.057.489.050.089.050.022.216.6 Kimi K2.596.779.179.644.40.00.059.539.750.050.060.043.344.535.476.045.263.027.893.996.3100.050.0100.050.00.08.4 MiniMax M2.790.058.270.448.6100.016.538.928.283.341.853.323.342.910.475.99.592.633.377.844.477.733.577.733.516.712.5 MiniMax M2.583.347.969.430.50.00.032.528.238.80.053.310.061.929.281.526.266.727.774.168.689.00.089.00.00.00.0 Model PER-A02PER-A03PER-D01PER-D02PER-E01PER-E02PER-H01PER-H02PER-I01PER-I02PER-M01PER-S01PER-S02 DetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlan Claude Opus 4.7100.066.5100.0100.00.00.072.30.00.00.081.450.00.00.00.00.080.063.311.00.00.00.085.773.8100.066.5 Claude Opus 4.6100.050.0100.058.50.00.083.333.550.025.081.650.0100.050.033.333.550.022.20.00.00.00.0100.066.6100.0100.0 Claude Opus 4.5100.066.5100.0100.00.00.022.30.00.00.077.850.00.00.00.00.050.027.80.00.00.00.082.772.9100.0100.0 Claude Sonnet 4.6100.0100.0100.0100.00.00.05.50.00.00.096.344.50.00.00.00.050.036.10.00.00.00.090.175.0100.0100.0 Claude Sonnet 4.5100.083.5100.0100.00.00.044.30.00.08.292.755.50.00.00.00.050.044.50.00.00.00.066.754.1100.0100.0 GPT-5.5100.050.0100.041.877.716.566.70.050.033.292.744.50.00.00.00.090.736.10.00.089.025.086.433.3100.050.0 GPT-5.4 Pro100.083.5100.016.889.033.583.38.20.00.092.727.80.00.077.750.096.352.8100.033.50.00.096.337.6100.033.5 GPT-5.411.00.0100.050.00.00.083.316.50.00.077.850.00.00.00.00.064.833.30.00.00.00.077.822.9100.033.5 GPT-5.2 Pro77.733.5100.025.077.733.544.38.250.033.218.427.8100.050.00.00.094.444.5100.083.577.725.081.545.8100.033.5 Gemini 3.1 Pro100.083.5100.025.00.00.011.08.20.00.066.833.30.00.00.00.059.330.6100.0100.00.00.088.927.1100.066.5 Gemini 3 Flash66.733.5100.025.00.00.011.00.00.00.022.20.00.00.00.00.040.716.70.00.00.00.088.943.889.066.5 GLM-5.1100.066.5100.025.066.750.044.38.250.025.088.950.0100.066.50.00.050.030.60.00.066.725.088.962.5100.0100.0 GLM-589.066.5100.058.50.00.061.28.20.00.092.750.00.00.00.00.046.327.80.00.00.00.058.041.7100.0100.0 DeepSeek V4 Pro100.066.5100.058.577.716.583.38.250.033.281.650.00.00.00.00.066.736.10.00.00.00.088.966.7100.066.5 DeepSeek V3.2100.050.0100.075.089.00.038.88.20.00.07.40.00.00.00.00.040.716.744.30.00.00.084.054.2100.0100.0 Qwen3.7 Max100.083.5100.075.077.750.077.80.044.525.096.350.00.00.00.00.081.550.022.30.00.00.0100.054.1100.083.5 Qwen3.7 Plus100.0100.0100.0100.089.033.577.80.016.716.892.744.50.00.077.750.048.230.60.00.00.00.090.168.8100.0100.0 Qwen3.6 Plus100.083.5100.025.00.00.066.70.00.00.066.744.50.00.00.00.066.727.80.00.00.00.074.137.5100.083.5 Qwen3.5 Plus100.050.0100.091.80.00.089.00.00.00.044.411.00.00.00.00.048.227.8100.0100.00.00.066.752.1100.0100.0 Kimi K2.6100.0100.0100.083.20.00.066.70.050.016.870.450.0100.033.50.00.048.233.30.00.089.025.067.935.4100.016.5 Kimi K2.5100.0100.0100.016.80.00.022.28.20.00.014.811.00.00.00.00.050.025.055.716.50.033.280.362.5100.0100.0 MiniMax M2.789.066.5100.083.20.00.022.20.00.00.07.45.5100.016.50.00.033.316.70.00.00.00.059.341.7100.0100.0 MiniMax M2.5100.050.0100.058.20.00.016.78.20.00.014.80.00.00.00.00.00.00.011.00.00.00.071.643.8100.0100.0 Model PER-S03PER-S04PER-V01PER-V02PER-V03PER-W01PER-W02BAS-01BAS-02BAS-03BAS-04BAS-05BAS-06 DetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlan Claude Opus 4.70.00.091.169.587.770.1100.066.550.055.7100.033.5100.066.888.087.597.890.064.661.766.753.796.391.786.170.8 Claude Opus 4.60.00.055.641.791.977.8100.050.072.249.8100.016.5100.066.897.874.474.468.365.562.786.763.363.561.982.273.4 Claude Opus 4.50.00.046.733.371.775.0100.083.550.050.0100.050.0100.041.888.184.577.856.668.952.467.855.073.064.377.760.0 Claude Sonnet 4.60.00.040.027.876.855.7100.066.555.555.5100.066.5100.066.895.684.486.771.772.861.190.063.382.671.582.260.0 Claude Sonnet 4.50.00.095.666.777.866.7100.083.550.055.7100.0100.00.00.074.085.681.190.051.752.447.846.774.692.988.973.3 GPT-5.5100.016.588.938.961.622.277.716.550.038.889.016.5100.050.083.056.668.951.665.651.638.928.336.523.773.350.0 GPT-5.4 Pro100.066.591.144.565.733.3100.066.550.038.877.733.5100.016.885.956.786.768.377.350.046.736.665.147.664.546.7 GPT-5.40.00.028.930.644.422.2100.066.550.050.0100.033.50.033.279.360.067.846.661.154.056.736.627.026.153.323.3 GPT-5.2 Pro77.750.086.744.552.541.7100.050.050.038.8100.033.5100.058.274.852.271.155.052.234.958.938.452.438.164.546.7 Gemini 3.1 Pro0.00.028.916.760.647.20.00.050.033.389.050.00.00.070.451.157.840.033.422.260.031.631.823.871.140.0 Gemini 3 Flash0.00.037.822.263.649.977.750.050.033.3100.033.5100.025.071.848.968.936.645.527.858.933.434.933.480.043.3 GLM-5.10.00.048.947.279.877.8100.050.044.555.5100.016.5100.075.094.875.582.261.782.869.160.048.373.054.980.056.7 GLM-50.00.091.144.468.780.689.066.550.044.5100.066.50.00.085.976.778.968.346.732.560.045.076.261.973.356.7 DeepSeek V4 Pro89.066.542.227.889.980.677.766.561.250.089.050.0100.091.888.976.678.965.072.257.263.343.471.450.075.546.7 DeepSeek V3.266.750.042.230.549.536.266.766.550.055.7100.050.00.00.077.868.973.353.345.044.553.346.738.133.453.333.4 Qwen3.7 Max77.750.095.647.288.969.489.066.550.055.789.016.5100.050.089.667.780.070.082.252.466.756.681.076.277.766.6 Qwen3.7 Plus89.066.544.436.074.877.889.083.550.050.0100.016.5100.075.090.475.697.880.073.965.180.056.773.066.682.263.3 Qwen3.6 Plus0.00.044.530.677.863.9100.050.050.050.0100.083.5100.025.077.074.482.271.772.848.457.850.085.783.477.863.3 Qwen3.5 Plus0.00.046.719.455.580.677.70.050.05.5100.050.00.00.063.763.378.968.350.636.557.840.063.552.473.333.3 Kimi K2.60.00.040.025.071.769.4100.066.550.033.389.016.5100.025.082.256.776.746.759.443.742.231.769.954.868.963.3 Kimi K2.511.00.046.738.873.777.8100.050.050.055.7100.050.00.00.077.878.971.160.137.232.543.340.050.847.682.260.0 MiniMax M2.70.00.040.027.858.661.1100.050.050.044.589.016.5100.058.583.072.284.463.353.333.457.836.773.064.371.146.6 MiniMax M2.50.00.044.527.858.638.9100.050.050.038.889.033.50.00.070.457.877.836.630.627.042.223.366.745.364.540.0 Model BAS-07VUL-01VUL-02VUL-03VUL-04Q-01Q-02Q-03Q-04Q-05Q-06Q-07Q-08 DetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlanDetPlan Claude Opus 4.7100.075.077.879.1100.083.5100.0100.055.650.082.00.071.6100.068.983.5100.0β50.1β89.0β66.751.592.6β Claude Opus 4.6100.083.286.770.4100.066.5100.050.040.750.082.283.581.1100.071.30.085.8β38.1β94.5β50.027.752.8β Claude Opus 4.5100.0100.084.561.0100.050.00.050.048.241.870.00.067.283.574.10.087.3β28.6β94.5β49.934.755.6β Claude Sonnet 4.683.383.590.083.3100.050.011.016.559.250.084.4100.080.0100.078.766.595.2β34.9β88.8β34.120.855.5β Claude Sonnet 4.5100.0100.072.272.3100.066.533.333.555.662.578.90.065.583.563.00.096.9β44.4β94.5β81.076.475.0β GPT-5.583.350.061.142.6100.033.511.016.566.741.680.066.569.5100.079.683.596.9β38.0β94.5β30.19.736.2β GPT-5.4 Pro72.258.262.251.889.050.0100.083.537.020.953.30.075.6100.077.833.584.1β41.2β66.7β51.633.361.2β GPT-5.494.541.856.744.4100.050.00.00.044.429.165.50.063.450.053.70.074.6β15.9β88.8β43.615.252.8β GPT-5.2 Pro94.566.536.720.4100.033.511.033.551.84.162.266.580.083.583.366.579.4β44.4β88.8β34.116.719.4β Gemini 3.1 Pro83.358.233.335.266.750.00.00.051.825.073.30.047.850.056.50.084.1β17.4β83.3β14.311.122.2β Gemini 3 Flash83.341.861.151.8100.050.033.333.563.025.077.80.056.733.561.150.092.0β20.6β66.7β11.18.32.8β GLM-5.1100.0100.078.983.4100.050.089.0100.066.741.687.833.581.166.578.70.096.9β52.4β94.5β66.743.150.0β GLM-594.550.066.761.1100.050.00.050.088.958.275.60.065.683.560.20.082.6β17.5β94.5β37.325.030.6β DeepSeek V4 Pro100.066.881.168.6100.050.0100.050.096.354.174.50.077.866.578.850.087.3β25.4β83.3β51.643.044.5β DeepSeek V3.272.258.271.163.0100.033.511.016.574.150.067.883.555.6100.063.00.082.6β28.6β94.5β46.030.638.8β Qwen3.7 Max94.5100.054.446.3100.050.0100.066.563.045.982.20.083.383.571.30.096.9β47.7β94.5β42.133.361.2β Qwen3.7 Plus83.375.072.268.5100.050.00.033.570.450.070.00.073.9100.067.60.085.8β38.0β94.5β52.434.752.8β Qwen3.6 Plus89.058.247.844.5100.0100.00.083.581.450.084.50.064.483.550.90.093.7β22.2β77.8β38.923.655.5β Qwen3.5 Plus83.341.845.650.0100.050.011.050.048.129.152.20.060.050.049.10.069.8β15.9β88.8β41.333.366.7β Kimi K2.694.575.061.151.866.733.511.016.548.125.077.80.056.750.056.50.098.4β25.4β83.3β42.829.241.7β Kimi K2.5100.0100.072.268.6100.050.089.0100.025.933.471.10.063.950.063.00.077.8β15.9β94.5β46.848.716.7β MiniMax M2.777.858.266.746.3100.050.00.00.063.029.162.20.051.133.543.50.076.2β25.3β94.5β61.137.533.2β MiniMax M2.5100.066.836.738.9100.050.00.00.044.433.252.20.049.450.045.40.065.1β15.9β72.2β30.113.819.4β 43 D.3 Token Usage ModelSteps InputOutput Cost (USD) Claude Opus 4.732.1 1,451,15438,8098.23 Claude Opus 4.624.4983,68919,9335.42 Claude Opus 4.526.3 1,269,87422,4426.91 Claude Sonnet 4.631.4 1,639,22123,0995.26 Claude Sonnet 4.528.5961,82339,7083.48 GPT-5.515.4846,20618,6594.79 GPT-5.4 Pro19.4 1,150,80723,79438.81 GPT-5.48.6429,28810,4431.23 GPT-5.2 Pro20.2 1,202,49820,53428.70 Gemini 3.1 Pro38.5 1,187,9748,3472.48 Gemini 3 Flash36.1 1,415,7087,4890.73 GLM-5.118.2855,30110,3210.88 GLM-518.6828,88014,4620.53 DeepSeek V4 Pro15.8721,04014,0100.33 DeepSeek V3.247.2 1,331,32717,8790.31 Qwen3.7 Max14.8669,56613,7461.05 Qwen3.7 Plus16.8899,24017,2570.31 Qwen3.6 Plus20.6 1,141,62415,4100.40 Qwen3.5 Plus16.1781,66414,5360.26 Kimi K2.618.7778,77513,1510.47 Kimi K2.514.7518,57616,3480.23 MiniMax M2.719.5788,42812,7380.20 MiniMax M2.529.4934,53310,1640.12 (a) (b) Figure 8: Token Usage. (a) Average per-run agent steps, input tokens, output tokens, and cost. (b) Average Overall Detection and Planning score vs. cost per run. Figure 8 (a) reports the average number of steps, token usage, and dollar cost per run for each model. The average number of steps varies substantially across models. Gemini 3.1 Pro and DeepSeek V3.2 take 38.5 and 47.2 steps on average, respectively, whereas GPT-5.4 stops after 8.6. The results of Gemini 3.1 Pro and DeepSeek V3.2 demonstrate that taking more steps does not guarantee better performance; additional investigation may offer no clear payoff. Total token consumption also varies across models for two main reasons. Some models take many investigative steps: for example, DeepSeek V3.2 consumes 1.33M tokens over 47.2 steps on average. Other models take fewer steps but maintain long contexts, as observed for Claude Opus 4.7 and Claude Sonnet 4.6. Figure 8 (b) compares cost with the average Overall Detection and Planning score. Cost and performance show only a weak relationship. Higher cost and greater token consumption help to some extent, but the highest-performing models are not the most expensive. GPT Pro models have the highest costs, but their performance remains around 50β60%, showing that their higher price does not translate into stronger incident-response capability. The Claude models are also costly but perform better. Among the least expensive models, GLM-5.1, DeepSeek V4 Pro, and the Qwen3.7 series achieve results comparable to the Claude series while costing far less. 44