Paper deep dive
CyberClear: A Benchmark for LLM Agent Systems on APT Attack Chain Provenance
Qi Chen, Fushuo Huo, Hangli Shen, Jingcai Guo, Shuhao Li, Guang Cheng
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Large language model agents have demonstrated promising capabilities in cybersecurity tasks, yet their ability to reconstruct complete Advanced Persistent Threat attack campaigns from complex security logs remains largely unexplored. Existing cybersecurity benchmarks for agents mainly focus on vulnerability discovery, exploitation, and security analysis tasks, leaving the evaluation of attack chain provenance under realistic security logs insufficiently studied. To address this gap, we introduce CyberClear, a benchmark for evaluating LLM agents and advanced agent systems on APT attack chain provenance from long-context security logs. CyberClear covers both single-step attacks and multi-stage attack chains, requiring agents to identify attack evidence, infer attack progression, and generate provenance graphs containing entities, causal relationships, MITRE ATT&CK techniques, and forensic evidence. To enable comprehensive evaluation, we develop an evaluation method tailored to APT attack chain provenance. Unlike conventional text similarity metrics that focus on surface-level matching, our evaluation examines whether reconstructed graphs preserve the semantics of attack chains across single-step behavior correctness, multi-step behavior identification, temporal and causal consistency, entity and relationship fidelity, and overall attack narrative consistency. Advanced multi-agent systems powered by state-of-the-art LLMs still struggle on CyberClear, motivating us to propose CyberProvenance, an agent cyber harness designed for multi-agents that augments LLM agents with evidence accumulation, execution-based validation, and feedback-guided refinement mechanisms for reliable attack-chain provenance. Extensive evaluations on CyberClear demonstrate the effectiveness of CyberProvenance in improving evidence reasoning, execution-grounded validation, and complete APT attack chain reconstruction.
Tags
Links
- Source: https://arxiv.org/abs/2609.32424v1
- Canonical: https://arxiv.org/abs/2609.32424v1
Trouble viewing inline? Open PDF directly →
Full Text
135,497 characters extracted from source content.
Expand or collapse full text
Under review as a conference paper at ICLR 2027 CYBERCLEAR: A BENCHMARK FOR LLM AGENT SYSTEMS ON APT ATTACK CHAIN PROVENANCE Qi Chen 1 , Fushuo Huo 1∗ , Hangli Shen 1 , Jingcai Guo 2 , Shuhao Li 3 , Guang Cheng 1 1 School of Cyber Science and Engineering, Southeast University 2 The Hong Kong Polytechnic University 3 Zhongguancun Laboratory fushuohuo@seu.edu.cn ABSTRACT Large language model (LLM) agents have demonstrated promising capabilities in cybersecurity tasks, yet their ability to reconstruct complete Advanced Persistent Threat (APT) attack campaigns from complex security logs remains largely unex- plored. Existing cybersecurity benchmarks for agents mainly focus on vulnerabil- ity discovery, exploitation, and security analysis tasks, leaving the evaluation of attack chain provenance under realistic security logs insufficiently studied. To ad- dress this gap, we introduce CyberClear, a benchmark for evaluating LLM agents and advanced agent systems on APT attack chain provenance from long-context security logs. CyberClear covers both single-step attacks and multi-stage at- tack chains, requiring agents to identify attack evidence, infer attack progression, and generate provenance graphs containing entities, causal relationships, MITRE ATT&CK techniques, and forensic evidence. To enable comprehensive evalua- tion, we develop an evaluation method tailored to APT attack chain provenance. Unlike conventional text similarity metrics that focus on surface-level matching, our evaluation examines whether reconstructed graphs preserve the semantics of attack chains across single-step behavior correctness, multi-step behavior identifi- cation, temporal and causal consistency, entity and relationship fidelity, and over- all attack narrative consistency. Advanced multi-agent systems powered by state- of-the-art LLMs still struggle on CyberClear, motivating us to propose Cyber- Provenance, an agent cyber harness designed for multi-agents that augments LLM agents with evidence accumulation, execution-based validation, and feedback- guided refinement mechanisms for reliable attack-chain provenance. Extensive evaluations on CyberClear demonstrate the effectiveness of CyberProvenance in improving evidence reasoning, execution-grounded validation, and complete APT attack chain reconstruction. The full benchmark and source code are available at https://cyberclear-bench.github.io/cyberclear/. 1INTRODUCTION Large language model (LLM) agents are taking on increasingly complex tasks through multi-step reasoning, tool use, and interaction with external environments (Yao et al., 2022; Schick et al., 2023; Wang et al., 2025). Their capabilities have extended to cybersecurity, including vulnerability discovery, exploitation, and security analysis (Deng et al., 2023; Fang et al., 2024). As agents assume a greater role in security work, comprehensive benchmarks are needed to evaluate their capabilities under realistic conditions. Existing cybersecurity benchmarks evaluate agents through classic capture-the-flag (CTF) chal- lenges (Shao et al., 2025; Zhang et al., 2025a) and vulnerabilities drawn from real software projects (Carlini et al., 2025; Zhu et al., 2025; Zhang et al., 2026; Lee et al., 2025; Wang et al., 2026). These benchmarks test whether agents can solve security challenges, discover or reproduce vulnerabilities, and repair vulnerable code. However, these evaluations do not directly assess agents’ ability to reason about attack progression and reconstruct complete attack chains. ∗ Corresponding author. 1 arXiv:2609.32424v1 [cs.CR] 26 Sep 2026 Under review as a conference paper at ICLR 2027 Advanced persistent threat (APT) attack chain provenance addresses this task by correlating secu- rity evidence to recover how an attack campaign unfolded (Li et al., 2021; Zipperle et al., 2022; Zhang et al., 2025b). This task presents three main challenges: (i) identifying relevant evidence from long and noisy security logs; (i) establishing relationships among processes, files, users, and network activities; and (i) inferring dependencies and temporal progression across multiple attack stages (Phan & Bauschert, 2026; Zhang et al., 2025c). However, existing evaluations largely focus on attack detection or investigation guided by prior attack clues, and lack a unified setting for assess- ing whether agents can independently discover dispersed evidence and reconstruct complete attack chains without prior attack knowledge. DiverseAttackBehaviors 450BenchmarkInstancesAttackChainProvenance Discovery Reconnaissance Stealth Credential Access PersistenceImpact Initial Access Execution CollectionCommand and Control Privilege Escalation ... ... Defender Logs No priorattackclues ... Step NStep 2Step 1 AttackProgression Entitiesandcausalrelationship Avg. 531K characters Figure 1: Overview of CyberClear. Given long-context defender logs without prior attack clues, agents are required to identify attack evidence, infer attack progression, and reconstruct complete APT attack chains across multiple attack stages. To address this gap, we introduce CyberClear, a benchmark for evaluating LLM agents and ad- vanced agent systems on APT attack chain provenance. As illustrated in Figure 1, CyberClear evaluates whether agents can reconstruct complete APT attack chains from long-context defender observations without prior attack clues. CyberClear contains 450 instances covering both single-step attacks and multi-stage attack chains, with each instance containing an average of 531K characters of security logs. Agents are required to identify attack evidence, infer attack progression, and gener- ate provenance graphs containing attack entities, causal relationships, MITRE ATT&CK techniques, and forensic evidence. These elements identify the involved entities, executed actions, and attack strategies, enabling the complete reconstruction of APT attack chains. Since provenance graphs describing the same attack may differ in representation, we develop a DOT-based LLM judge to evaluate the semantic quality of reconstructed attack chains. The proposed evaluation method as- sesses generated graphs from five perspectives: single-step behavior correctness, multi-step behav- ior identification, temporal and causal consistency, entity and relationship fidelity, and overall attack narrative consistency, while ignoring irrelevant differences in graph identifiers and visual styles. Our evaluation reveals that current agents still face significant challenges in long-context evidence reasoning, causal attack progression inference, and complete APT attack chain reconstruction. To address these challenges, we propose CyberProvenance, an agent cyber harness for improving the reliability of LLM agents in reconstructing attack chains from long-context security logs in Cy- berClear. The harness accumulates evidence across logs, validates predicted attack behaviors in isolated execution environments, and refines inconsistent steps using execution feedback and log evidence. We evaluate CyberProvenance alongside representative agent frameworks on CyberClear using the proposed metrics. Experimental results demonstrate that CyberProvenance effectively performs continuous validation of attack behaviors and reconstructs complete attack chains. Our main contributions are as follows: • We introduce CyberClear, a comprehensive benchmark for evaluating LLM agents and advanced agent systems on APT attack chain provenance from long-context defender logs without supplied attack clues. • We develop a semantic evaluation method that assesses reconstructed provenance graphs from multiple perspectives, including single-step behavior correctness, multi-step behavior identification, temporal and causal consistency, entity and relationship fidelity, and overall attack consistency. 2 Under review as a conference paper at ICLR 2027 • We propose CyberProvenance, an agent cyber harness designed for multi-agents, which augments LLM agents with evidence accumulation, execution-based validation, and feedback-guided refinement. • We conduct extensive evaluations of representative agent frameworks and LLM backbones on CyberClear, showing that the primary limitations of current agents for APT analysis lie not merely in long context reasoning, but in iterative refinement, evidence consistency, and the effective use of cybersecurity specific verification tools. 2RELATED WORK APT Attack Chain Provenance. APT attack chain provenance aims to reconstruct complete attack campaigns by tracing and correlating suspicious activities from large volumes of security logs (Li et al., 2021; Zipperle et al., 2022; Zhang et al., 2025b). Unlike traditional intrusion de- tection (Du et al., 2017; Mirsky et al., 2018; Pendlebury et al., 2019) that only identifies isolated malicious events, attack chain provenance focuses on discovering the relationships among entities, behaviors, and attack stages to recover the evolution of multi-step attacks (Phan & Bauschert, 2026; Zhang et al., 2025c). Early methods such as SLEUTH (Hossain et al., 2017), HOLMES (Milajerdi et al., 2019), and CONAN (Xiong et al., 2020) rely on predefined expert rules or TTP mappings to identify suspicious activities, but their limited rule coverage often leads to a large number of false positives. Subsequent graph learning-based approaches (Li et al., 2023b; Rehman et al., 2024; Cheng et al., 2024) leverage provenance graphs to capture complex dependencies among system entities for improved detection, but provide either insufficient contextual information or overly coarse-grained results for attack investigation. With the rapid development of LLMs, recent studies have increas- ingly applied LLM-based agents to attack provenance tasks (Mukherjee & Kantarcioglu, 2025; Yan et al., 2026; Zhao et al., 2026; Ock et al., 2026), enabling more flexible reasoning over security logs and attack behaviors. However, these approaches either identify only entity relationships associated with attack indicators (Mukherjee & Kantarcioglu, 2025; Ock et al., 2026) or rely on predefined attack clues during evaluation (Yan et al., 2026; Zhao et al., 2026), making them difficult to ap- ply to realistic APT attack chain provenance scenarios where the complete attack process must be reconstructed from security logs. Table 1: Comparison with existing cybersecurity benchmarks for Agents. BenchmarkScopeInstance Attack Behaviors ATT&CK Logs Attack Chain NYU CTF Bench (Shao et al., 2025)CTF2006✗ Cybench (Zhang et al., 2025a)CTF406✗ AutoAdvExBench (Carlini et al., 2025)CTF + Real-world752✗ CVE-Bench (Zhu et al., 2025)Real-world408✗ BountyBench (Zhang et al., 2026)Real-world4036✗ SEC-bench (Lee et al., 2025)Real-world20016✗ CyberGym (Wang et al., 2026)Real-world1,50728✗ DiagChain-Bench (Liu et al., 2026)Real-world69N/R✗✓ CyberClearReal-world45089✓ Cybersecurity Benchmarks for Agents. The increasing adoption of agents in cybersecurity has motivated the development of benchmarks for evaluating their autonomous security capabilities (Lin et al., 2026; Sanz-Gómez et al., 2025; Carlini et al., 2025). As shown in Table 1, existing benchmarks mainly focus on offensive security tasks, including CTF-based problem solving and real-world vul- nerability assessment, such as Cybench (Zhang et al., 2025a), CVE-Bench (Zhu et al., 2025), Boun- tyBench (Zhang et al., 2026), and CyberGym (Wang et al., 2026). While these benchmarks pro- vide realistic environments for evaluating agent capabilities in vulnerability discovery, exploitation, and remediation, they primarily focus on software vulnerability-related tasks, overlooking whether agents can analyze attack evidence from security logs and reason over provenance information to reconstruct complete attack chains. To address this gap, we introduce a dedicated benchmark, Cy- berClear, that better emulates realistic APT provenance scenarios featuring long-context security logs and complex multi-stage attacks. These characteristics make CyberClear substantially more challenging for existing agents, as evidenced by our evaluation results in Section 5. 3 Under review as a conference paper at ICLR 2027 3CYBERCLEAR BENCHMARK 3.1TASK FORMULATION We formulate CyberClear as an APT attack chain provenance task that requires agents to recon- struct complete attack processes from large-scale security logs. Given a collection of defender logs L = L 1 ,L 2 ,...,L n collected during an attack campaign, the agent performs provenance analy- sis without access to prior attack clues. The agent is required to identify attack-related entities, infer their relationships, and reconstruct the complete attack process from dispersed log evidence. The final output is a provenance graph G = (E,R), where E represents attack-related entities such as processes, accounts, files, services, and network resources, while R captures their interactions and dependencies. Beyond entity relationships, the graph organizes the reconstructed attack process into an attack narrative timeline, where each step contains its attack behavior, corresponding MITRE ATT&CK techniques (Jiang et al., 2025), and supporting forensic evidence. 3.2BENCHMARK CONSTRUCTION Figure 2 presents an overview of our four-stage data construction pipeline, including attack evidence extraction, provenance graph generation, automatic verification, and human expert review. Each sample consists of defender logs collected from a continuous attack campaign and a provenance graph that represents relationships among attack entities and the evolution of multi-stage attacks. Open-sourceDataset Defender LogsAttacker Logs 123 LLM AccountIPProcessFileService Attack Evidence ExtractionProvenance Graph Generation Compiler LLM Structured Evidence ATT&CK Mappings digraph G a [label="Host"]; b [label="Account"]; a -> b [label="runs"]; ... 2 4 4 MLLM AutomaticVerification Graph Readability Technical Completeness Factual Consistency Format Compliance CyberClear HumanExpertReview Causal and Temporal Consistency Entity Normalization Attack-Step Segmentation Graph Structure Clarity Attack Narrative Timeline ... Step 1 Step NStep 2 Figure 2: The construction pipeline of CyberClear. Stage I: Attack Evidence Extraction. We collect raw attack logs from publicly available APT datasets, including PROVCON (Yusof et al., 2025) and CAM-LDS (Landauer et al., 2026), which are constructed from real-world attack scenarios. While designed to support APT behavior analysis and provenance research, these datasets primarily focus on attack detection and security log analysis. We design attack evidence extraction rules based on attacker logs. For each sample, we provide Claude Opus 4.8 (Anthropic, 2026) with extraction rules to extract attack evidence, including key attack entities, temporal events, entity interactions, step-level attack descriptions, and corresponding MITRE ATT&CK techniques. The extracted attack evidence is further organized into structured representations for subsequent processing. Stage I: Provenance Graph Generation. We employ Claude Opus 4.8 to generate Graphviz DOT code for each sample based on the extracted attack evidence and MITRE ATT&CK annotations. To ensure consistent entity representation in generated DOT graphs, we define unified graph repre- sentations that assign predefined node formats to different entity types, including network entities, accounts, processes, files, and services. The generated DOT code is compiled into visual prove- nance graphs, and failed compilations are returned to the model for iterative correction. As shown in Figure 2 (2), the resulting provenance graphs preserve entity relationships, and summarize the evolution of multi-stage attacks through attack timelines. Stage I: Automatic Verification. The generated graphs only require evidence consistency verifi- cation, so we employ GPT-4o (Hurst et al., 2024) to automatically verify them using attacker logs as 4 Under review as a conference paper at ICLR 2027 the ground truth. The verification process consists of four sequential checks: (1) Graph readability, ensuring that the generated graphs are clear, complete, and visually understandable; (2) Technical completeness, verifying that all required ATT&CK techniques are correctly included; (3) Factual consistency, checking whether the reconstructed entities, relationships, and attack timeline are con- sistent with the actual attack behaviors; and (4) Format compliance, ensuring that the generated graphs follow predefined visualization and representation requirements. If any verification criterion fails, the feedback generated by GPT-4o is sent back to step 2 for graph regeneration. Stage IV: Human Expert Review. To further ensure dataset reliability, security experts manually review the provenance graphs that pass automatic verification. The experts evaluate the generated graphs based on predefined quality criteria and filter out low quality samples from open-source datasets, including incomplete attack traces, insufficient evidence coverage, and inconsistent prove- nance information. Only samples that satisfy both automatic verification and expert review are included in CyberClear. 3.3DATA ANALYSIS Table 2: Statistics of the CyberClear. StatisticEasy MediumHard Overall Total Samples Sample Count31812210450 Attack Complexity Single-step Attack31800318 Multi-step Attack012210132 Input Log Volume (count) Minimum1179– Maximum122492– Average3.265.9983.25.78 Input Log Length (characters) Minimum33611671,222– Maximum5.18e6 5.44e6 5.76e7– Average7.07e5 6.69e5 7.80e5 5.31e5 Dataset Scale. CyberClear contains 450 APT at- tack chain provenance samples with approximately 2.1GB of data, including 318 easy instances, 122 medium instances, and 10 hard instances catego- rized by attack complexity. Table 2 summarizes the dataset statistics, including sample distribution, at- tack complexity, input log volume, and log length. The easy subset consists entirely of single-step at- tacks, while medium and hard subsets contain multi- step attack chains requiring models to reason over dependencies among multiple attack stages. Com- pared with easy and medium instances, hard samples involve substantially larger input log volumes, with an average of 83.2 log files per sample and longer contexts containing approximately 7.80× 10 5 characters, providing a more challenging setting for evaluating APT attack chain provenance capabilities. Attack Behaviors. Figure 3 shows the distribution of attack behaviors in CyberClear. The bench- mark covers 89 distinct attack behaviors spanning Credential Access, Defense Evasion, Persistence, Execution, Impact, and other attack categories. Discovery and Reconnaissance are the two most frequent categories, accounting for 24.00% and 15.56% of all instances, respectively. Among them, Composite Attacks are constructed by combining multiple attack behaviors, accounting for 6.44% of all instances. This diverse distribution provides broad coverage of APT behaviors across different attack stages and enables comprehensive evaluation of attack chain provenance capabilities. 3.4EVALUATION METRICS Discovery Reconnaissance Credential Access Stealth Persistence Execution Impact Initial Access Command and Control Collection Privilege Escalation Composite Attacks Figure 3: Distribution of attack behaviors. Code-Level Evaluation. The generated DOT code is evaluated from both textual similarity and func- tional correctness perspectives. BLEU and ROUGE- L are used to measure the similarity between gen- erated and reference DOT code, capturing token- level overlap and sequence-level consistency, re- spectively. CodeBLEU is further adopted to eval- uate structural consistency by considering code syn- tax and semantic information beyond surface-level matching. Pass@1 is used to assess whether the gen- erated DOT code can be successfully compiled, re- flecting practical generation reliability. Semantic Evaluation via LLM-as-a-Judge. To complement lexical matching and code structure metrics, we develop a DOT-based LLM judge to evaluate the semantic quality of reconstructed attack chains. The judge directly compares reference and predicted DOT code while ignoring ir- 5 Under review as a conference paper at ICLR 2027 relevant differences in node identifiers, statement order, graph layouts, and visual styles. We use GLM-5.2 (Zeng et al., 2026) as the judge model to perform evidence-based semantic comparison between reference and predicted provenance graphs. The semantic evaluation framework assesses reconstructed provenance graphs from five perspectives. Each dimension is scored from 0 to 1, and the reported scores are averaged over all samples. Detailed illustrations are in Appendix C. • Single-Step Judge. Evaluates whether individual attack behaviors are correctly reconstructed by comparing the actor, core action, target, and overall attack meaning. • Loose Judge. Measures whether multi-step attack behaviors are correctly identified without con- sidering temporal ordering among attack steps. • Strict Judge. Evaluates whether the reconstructed attack chain preserves the temporal progression and causal dependencies among different attack stages. • Detail Judge. Measures the consistency of graph-level elements, including entities, directed rela- tionships, and attributes between reference and predicted provenance graphs. • Scenario Judge. Assesses whether the reconstructed graph conveys the same overall attack pro- cess, target, method, progression, and outcome as the reference attack campaign. Attack Descriptions ATT&CK ... Log Analysis Agent Attack Validation Agent Chunk Check Agent LogFile EvidenceMemory DOT Evidence DefenderLogs ATT&CK Technique IDs Structured Attack Metadata Single Multiple Single-step Regeneration Figure 4: Overview of CyberProvenance, an agent cyber harness for reliable attack-chain prove- nance. Designed for multi-agent collaboration, the harness assigns specialized capabilities to dif- ferent agents, enabling evidence accumulation, execution-based validation, and feedback-guided refinement. 4METHODOLOGY We present CyberProvenance, an agent cyber harness designed for multi-agent collaboration to im- prove the reliability of LLM-based agents. The harness assigns specialized capabilities to different agent roles, including Log Analysis Agent, Attack Validation Agent, and Check Agent, which col- laboratively perform evidence accumulation, execution-based validation, and iterative provenance graph refinement, as illustrated in Figure 4. Log Analysis Agent. To identify and accumulate attack evidence from large volumes of logs and establish an initial provenance structure, the Log Analysis Agent is responsible for evidence accu- mulation in the CyberProvenance harness. It receives defender security logs L =L 1 ,...,L n and progressively analyzes logs to identify attack related evidence, including temporal events, processes, commands, network activities, and potential MITRE ATT&CK techniques. To handle long context logs, each log file L i is divided into multiple chunks C 1 i ,...,C K i i , and the extracted evidence from each chunk is continuously accumulated into an Evidence Memory M . Based on the accumu- lated evidence, the agent performs global reasoning to generate an initial provenance hypothesis: H 0 =F(M) = (A, S, T, G 0 ), where A denotes the attack scenario, S represents attack stages and step dependencies, T denotes ATT&CK technique mappings, and G 0 represents the reconstructed provenance graph. The recon- structed provenance graph G 0 is finally serialized into structured DOT code, providing an explicit graph representation for subsequent attack validation. Attack Validation Agent. The Attack Validation Agent provides execution-grounded validation by verifying the executability and reliability of reconstructed attack steps through real environment reproduction. This reduces the risk of relying on plausible but unverified attack inferences. Given the 6 Under review as a conference paper at ICLR 2027 predicted ATT&CK techniques T =τ 1 ,...,τ m and structured attack steps generated by the Log Analysis Agent, the agent retrieves the corresponding execution specifications for each technique: q i =R(τ i ) = (p i ,x i ,r i ,d i ), where p i , x i , r i , and d i represent the required platform, executor, privilege, and dependency infor- mation of technique τ i , respectively. Using these specifications, the agent selects appropriate execu- tion strategies, where independent techniques are reproduced using Atomic Red Team (Red Canary) and dependency aware multi step attack chains are orchestrated through CALDERA (Apache Soft- ware Foundation) in isolated environments built with Ludus (Bad Sector Labs). The generated exe- cution traces collected by Sysmon (Microsoft), Linux Audit (Linux Audit Project), and Zeek (Zeek Project) are used to produce structured validation reports, indicating whether each reconstructed attack step can be successfully reproduced. Check Agent. The Check Agent closes the refinement loop of the harness by integrating the re- constructed provenance graph and validation feedback to assess reliability through graph structures, attack-step dependencies, and ATT&CK technique mappings. When all attack steps are successfully reproduced and consistent with the collected evidence, the current provenance graph is preserved as the final result. Otherwise, unreliable steps are refined while preserving validated nodes, and the updated hypothesis is revalidated to form an iterative prediction, validation, and refinement loop. 5EXPERIMENTS 5.1EXPERIMENTAL SETUP LLMs. We evaluate DeepSeek-V4-Flash (Xu et al., 2026),Qwen3.8-Max (Qwen Team, 2026), andGPT-5.6-Terra (OpenAI, 2026) in the main experiments. These models exhibit different rea- soning and generation characteristics, allowing us to examine the performance of agent frameworks across different LLM backbones. Additional results withGPT-4o-mini (Hurst et al., 2024) and Gemini-3.5-Flash-Lite (Google DeepMind, 2026) are reported in Appendix A.1. Methods. We compare CyberProvenance against two categories of baselines: (1) Single-Agent Methods including OpenClaw (Single-Agent) (OpenClaw); (2) Multi-Agent Systems including OpenClaw (Multi-Agent), AgentVerse (Chen et al., 2024), LLM-Debate (Du et al., 2023a), Multi- Persona (Wang et al., 2024b), LLM-Blender (Jiang et al., 2023a), DyLAN (Liu et al., 2024), Mac- Net (Qian et al., 2025), CAMEL (Li et al., 2023a), and AgentScope-V2 (Dawei Gao, 2024; 2025). Additional details are provided in Appendix B.1. OpenClaw (Single) OpenClaw (Multiple) AgentVerse LLM-Debate MultiPersona LLM-Blender DyLAN MacNet CAMEL AgentScope-V2 CyberProvenance GPT-5.6-Te r r aDeepSeek-V4-FlashQwen-3.8-Max Easy Difficulty 1.0 0.8 0.6 0.4 0.2 OpenClaw (Single) OpenClaw (Multiple) AgentVerse LLM-Debate MultiPersona LLM-Blender DyLAN MacNet CAMEL AgentScope-V2 CyberProvenance 1.0 0.8 0.6 0.4 0.2 MediumDifficultyHardDifficulty OpenClaw (Single) OpenClaw (Multiple) AgentVerse LLM-Debate MultiPersona LLM-Blender DyLAN MacNet CAMEL AgentScope-V2 CyberProvenance 0 .06 0. 0 5 0. 0 4 0. 0 3 0. 02 Figure 5: Average Semantic Metrics scores of different agent frameworks across the Easy, Medium, and Hard subsets of CyberClear. 5.2MAIN RESULTS Code-Level Metrics. Table 3 shows that existing agent frameworks can generate valid provenance graphs, but their performance varies markedly across methods and LLM backbones, indicating that stable attack-graph reconstruction remains difficult. AgentScope-V2 achieves the highest BLEU and CodeBLEU scores of 0.5035 and 0.7240, while CyberProvenance obtains the best ROUGE-L score of 0.6968 and Pass@1 score of 0.9933. Compared with general-purpose agent frameworks, Cy- berProvenance maintains consistently strong code-level performance across multiple metrics. These 7 Under review as a conference paper at ICLR 2027 Table 3: Performance comparison on CyberClear across agent frameworks and LLM backbones using code-level and semantic metrics. Blue cells indicate the best result for each metric. Methods Code-Level MetricsSemantic Metrics BLEU ROUGE-L CodeBLEU Pass@1Single Loose Strict Detail Scenario OpenClaw (Single) 0.33170.48630.37950.82220.6794 0.6006 0.4387 0.4051 0.4805 0.35560.51170.38680.72000.7812 0.7125 0.5490 0.4898 0.5823 0.36920.52970.38870.84670.7726 0.7214 0.5008 0.4536 0.5862 OpenClaw (Multiple) 0.41830.56540.56480.92000.5513 0.4749 0.3580 0.3301 0.3679 0.44220.59080.57210.81780.7160 0.6629 0.5034 0.4090 0.5847 0.45160.58480.56430.93330.7136 0.6684 0.5178 0.4053 0.5821 AgentVerse 0.36640.54460.60740.94000.5080 0.4337 0.3273 0.3039 0.3269 0.39030.57000.61470.83780.7304 0.6743 0.5150 0.4248 0.5886 0.40150.59420.60270.94220.7113 0.6897 0.5011 0.4179 0.6135 LLM-Debate 0.40890.52780.65020.86440.6135 0.5344 0.4089 0.3758 0.4334 0.43280.55320.65750.76220.7622 0.6853 0.5368 0.4581 0.6061 0.44130.57890.64940.88440.7436 0.7082 0.5216 0.4704 0.6047 MultiPersona 0.37750.55700.42170.92890.6843 0.6033 0.4418 0.4037 0.4842 0.40140.58240.42900.82670.7800 0.7107 0.5519 0.4825 0.5961 0.41360.53140.42470.94220.7131 0.7198 0.5289 0.5002 0.5235 LLM-Blender 0.41960.54950.57610.87330.5632 0.4869 0.3660 0.3384 0.3789 0.44350.57490.58340.77110.7216 0.6668 0.5077 0.4151 0.5864 0.45190.59760.61120.89560.7317 0.6761 0.5248 0.4465 0.5969 DyLAN 0.34970.52410.50990.88440.8454 0.7514 0.5421 0.4769 0.6223 0.38080.55280.54910.86670.85110.7705 0.6224 0.5433 0.6545 0.38840.52580.53750.87110.8350 0.7502 0.5592 0.5214 0.6771 MacNet 0.36330.47280.53260.92670.6186 0.5590 0.3928 0.3977 0.4241 0.36300.47840.54220.78890.7708 0.7129 0.5226 0.5090 0.5061 0.33890.45150.50860.80880.7375 0.6809 0.4758 0.5070 0.5022 CAMEL 0.35770.56470.56290.92220.3754 0.3048 0.2371 0.2217 0.2043 0.40570.58690.57900.79780.6710 0.6322 0.4673 0.3618 0.5656 0.38640.57810.42010.93770.5371 0.4557 0.2955 0.3025 0.4117 AgentScope-V2 0.49210.64770.68050.86220.8412 0.7373 0.6291 0.5485 0.7021 0.49770.65230.68540.78890.8343 0.6786 0.5864 0.5169 0.6936 0.50350.63340.72400.98660.8525 0.6597 0.6106 0.5710 0.7115 CyberProvenance (Ours) 0.49710.68730.68670.99330.8627 0.7325 0.6340 0.5981 0.6710 0.48100.68270.70520.89110.8764 0.70270.6477 0.5776 0.7087 0.49520.69680.70160.93330.8897 0.7438 0.63370.61240.7263 results suggest that iterative refinement and cybersecurity-specific validation tools contribute to more reliable provenance graph reconstruction. Semantic Metrics. The challenge is more evident under semantic metrics, where most existing frameworks struggle to preserve complete attack progression, entity dependencies, and fine-grained evidence. Although DyLAN achieves the best Loose score of 0.7705, its advantage does not extend consistently to stricter semantic criteria. CyberProvenance achieves the highest Single, Strict, Detail, and Scenario scores of 0.8897, 0.6477, 0.6124, and 0.7263, respectively. The remaining gap on 8 Under review as a conference paper at ICLR 2027 Strict and Detail further indicates that complete APT attack-chain reconstruction is still challenging for current agent systems. 5.3PERFORMANCE ANALYSIS ACROSS DIFFICULTY LEVELS We evaluate different agent frameworks across the Easy, Medium, and Hard subsets of CyberClear using the average Semantic Metrics score over Single, Loose, Strict, Detail, and Scenario. As shown in Figure 5, all frameworks exhibit consistent performance degradation as the number of input log files increases rapidly. This trend indicates that larger log volumes make it increasingly difficult to identify relevant evidence and maintain coherent attack progression during provenance. These results demonstrate that CyberClear effectively captures the increasing difficulty of APT attack chain reconstruction across different log scales. LogFiles E1: python3 -> stty type=SYSCALL msg=audit(1758531876.085:4573): ... ppid=2803 pid=2806 auid=4294967295 uid=33 gid=33 euid=33 suid=33 fsuid=33 egid=33 sgid=33 fsgid=33 tty=(none) ses=4294967295 comm="python3" exe="/usr/bin/python3.9" ... suid=33 fsuid=33 egid=33 sgid=33 fsgid=33 tty=pts0 ses=4294967295 comm="bash" exe="/usr/bin/bash" ... suid=33 fsuid=33 egid=33 sgid=33 fsgid=33 tty=pts0 ses=4294967295 comm="stty" exe="/usr/bin/stty" E2: stty-> chmod ...type=EXECVE msg=audit(1758531925.041:4594): argc=5 a0="stty" a1="rows" a2="38" a3="columns" a4="116"... E3: stty-> linpeas.sh ...type=EXECVE msg=audit(1758531957.641:4612): argc=3 a0="chmod" a1="+x" a2="linpeas.sh" type=PATH msg=audit(1758531957.641:4612): item=0 name="/usr/bin/chmod" inode=811 dev=fe:01 mode=0100755 ouid=0 ogid=0 rdev=00:00 nametype=NORMAL cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0OUID="root" OGID="root" ... Process Entity File Entity Attack Step Figure 6: Case study comparing AgentVerse and CyberProvenance on attack provenance using GPT- 5.6-Terra. CyberProvenance corrects unsupported causal relations by aligning the reconstructed graph with execution evidence. 5.4PERFORMANCE ANALYSIS ACROSS ATTACK STAGE LEVELS 0.60 0.63 0.68 0.74 0.64 0.72 0.84 0.77 0.65 0.85 0.87 0.42 0.56 0.68 0.54 0.45 0.57 0.64 0.55 0.35 0.66 0.73 0.58 0.62 0.66 0.69 0.62 0.67 0.85 0.61 0.63 0.84 0.84 0.40 0.53 0.66 0.59 0.42 0.57 0.61 0.46 0.26 0.64 0.69 0.57 0.64 0.70 0.71 0.64 0.70 0.85 0.74 0.67 0.84 0.85 0.45 0.56 0.70 0.52 0.46 0.61 0.66 0.58 0.52 0.68 0.73 GPT-5.6-TerraDeepSeek-V4-FlashQwen-3.8-Max SingleMultiSingleMultiSingle Multi OpenClaw (Single) OpenClaw (Multiple) AgentVerse LLM-Debate MultiPersona LLM-Blender DyLAN MacNet CAMEL AgentScope-V2 CyberProvenance 0.0–0.20.2–0.40.4–0.60.6–0.80.8–1.0 Figure 7: Average Semantic Metrics scores across single-stage and multi-stage attacks. Attack stage complexity exposes a clear perfor- mance gap in current agent frameworks. As shown in Figure 7, we report the average Semantic Met- rics score across Single, Loose, Strict, Detail, and Scenario, where nearly all methods perform substan- tially worse on multi-stage attacks than on single- stage attacks across the three LLM backbones. The decline is especially pronounced for CAMEL, MacNet, and LLM-Debate, whereas CyberProve- nance remains more robust and consistently achieves stronger multi-stage performance.These results highlight the importance of iterative validation and evidence-guided refinement for maintaining coher- ent attack chain reconstruction as attack complexity increases. 5.5CASE STUDY Figure 6 presents a representative example using GPT-5.6-Terra, where the ground-truth graph shows that python3 launches bash, which subsequently invokes both stty and chmod, while chmod modifies linpeas.sh. AgentVerse incorrectly associates python3 directly with stty, although the audit log shows that stty is spawned by the Bash process rather than Python. It further introduces dependencies from stty to chmod and linpeas.sh, even though chmod is indepen- dently invoked by Bash and the permission change on linpeas.sh is directly caused by chmod +x. In contrast, CyberProvenance removes these unsupported dependencies and reconstructs the causal structure more consistently with the ground truth. More cases in the Appendix E. 9 Under review as a conference paper at ICLR 2027 6CONCLUSION In this work, we introduce CyberClear, a benchmark for evaluating LLM agents on APT attack chain provenance from long-context security logs. CyberClear covers both single-step and multi- stage attacks and provides a semantic evaluation method that assesses attack behaviors, temporal and causal consistency, entity relationships, and overall attack narrative consistency. Extensive evalua- tions show that even advanced agent systems still face substantial challenges in evidence reasoning and complete attack chain reconstruction. To address these limitations, we propose CyberProve- nance, demonstrates the effectiveness of an agent cyber harness that combines evidence accumu- lation, execution-based validation, and feedback-guided refinement for reliable APT attack chain reconstruction. We hope CyberClear and CyberProvenance will provide a useful foundation for developing more capable and trustworthy agents for complex cybersecurity analysis. AI USE STATEMENT In this work, we used generative AI tools to assist with refining the research methodology and ex- perimental design, implementing and debugging parts of the experimental code, and interpreting intermediate experimental results. We also used generative AI tools for literature search and sum- marization, language polishing, improving the organization and readability of the manuscript, and assisting with the preparation of figures, tables, and captions. All AI-assisted code and experimental procedures were manually reviewed and tested by the authors, and the reported results were checked against the corresponding execution outputs and evaluation scripts. Literature-related suggestions were verified against the original papers, and all AI-assisted text, figures, and technical claims were reviewed and revised by the authors before inclusion. We take full responsibility for the final con- tent of this work, including all text, claims, code, experiments, and artifacts produced with the aid of generative AI. ETHICS STATEMENT This work studies defensive cybersecurity analysis and APT attack provenance. CyberClear is con- structed from publicly available security datasets and does not involve human subjects or private user data collection. All execution-based validation is conducted only in authorized and isolated environ- ments, with private network ranges and safeguards that prevent destructive or unauthorized actions against external systems. While the studied techniques are inherently dual-use, the benchmark and framework are intended for defensive analysis, provenance, and evaluation of security agents, and we will clearly document the intended use and safety constraints of the released artifacts. REPRODUCIBILITY STATEMENT We have taken several steps to support the reproducibility of this work. The main paper describes the benchmark construction, evaluation protocol, and overall experimental setup, while the appendix provides the complete configuration details for all evaluated agent frameworks. Additional imple- mentation details, data processing procedures, prompts, and evaluation methods are included in the appendix. These materials are intended to provide the information necessary to reproduce the re- ported experiments and results. REFERENCES Ahmed Aly, Essam Mansour, and Amr Youssef. Ocr-apt: Reconstructing apt stories from audit logs using subgraph anomaly detection and llms. In Proceedings of the 2025 ACM SIGSAC Conference on Computer and Communications Security, p. 261–275, 2025. Anthropic. System card: Claude opus 4.8, 2026. URL https://w-cdn.anthropic.com/ 0b4915911b0d19eca5b5e635c80fef830a37ea.pdf. Apache Software Foundation. Apache caldera. https://github.com/apache/caldera. GitHub repository. Bad Sector Labs. Ludus. https://github.com/badsectorlabs/ludus. GitHub reposi- tory. 10 Under review as a conference paper at ICLR 2027 Sidahmed Benabderrahmane, Petko Valtchev, James Cheney, and Talal Rahwan.Apt-llm: Embedding-based anomaly detection of cyber advanced persistent threats using large language models. In 2025 13th International Symposium on Digital Forensics and Security (ISDFS), p. 1–6. IEEE, 2025. Yoshua Bengio and Yann LeCun. Scaling learning algorithms towards AI. In Large Scale Kernel Machines. MIT Press, 2007. Xavier Cadet, Aditya Vikram Singh, Harsh Mamania, Edward Koh, Alex Fitts, Dirk Van Bruggen, Simona Boboila, Peter Chin, and Alina Oprea. Retrieval-augmented llms for security incident analysis. In Proceedings of the ACM Conference on AI and Agentic Systems, p. 103–123, 2026. Nicholas Carlini, Javier Rando, Edoardo Debenedetti, Milad Nasr, and Florian Tramèr. Autoad- vexbench: Benchmarking autonomous exploitation of adversarial example defenses. arXiv preprint arXiv:2503.01811, 2025. Weize Chen, Yusheng Su, Jingwei Zuo, Cheng Yang, Chenfei Yuan, Chi-Min Chan, Heyang Yu, Yaxi Lu, Yi-Hsin Hung, Chen Qian, Yujia Qin, Xin Cong, Ruobing Xie, Zhiyuan Liu, Maosong Sun, and Jie Zhou. AgentVerse: Facilitating multi-agent collaboration and exploring emergent behaviors. arXiv preprint arXiv:2308.10848, 2023. Weize Chen, Yusheng Su, Jingwei Zuo, Cheng Yang, Chenfei Yuan, Chi-Min Chan, Heyang Yu, Yaxi Lu, Yi-Hsin Hung, Chen Qian, et al. Agentverse: Facilitating multi-agent collaboration and exploring emergent behaviors. In International Conference on Learning Representations, volume 2024, p. 20094–20136, 2024. Zijun Cheng, Qiujian Lv, Jinyuan Liang, Yan Wang, Degang Sun, Thomas Pasquier, and Xueyuan Han. Kairos: Practical intrusion detection and investigation using whole-system provenance. In 2024 IEEE Symposium on Security and Privacy (SP), p. 3533–3551. IEEE, 2024. Xuchen Pan Weirui Kuang Zhijian Ma Bingchen Qian Fei Wei Wenhao Zhang Yuexiang Xie Daoyuan Chen Liuyi Yao Hongyi Peng Zeyu Zhang Lin Zhu Chen Cheng Hongzhu Shi Yaliang Li Bolin Ding Jingren Zhou Dawei Gao, Zitao Li. Agentscope: A flexible yet robust multi-agent platform. CoRR, abs/2402.14034, 2024. Yuexiang Xie Weirui Kuang Liuyi Yao Bingchen Qian Zhijian Ma Yue Cui Haohao Luo Shen Li Lu Yi Yi Yu Shiqi He Zhiling Luo Wenmeng Zhou Zhicheng Zhang Xuguang He Ziqian Chen Weikai Liao Farruh Isakulovich Kushnazarov Yaliang Li Bolin Ding Jingren Zhou Dawei Gao, Zi- tao Li. Agentscope 1.0: A developer-centric framework for building agentic applications. CoRR, abs/2508.16279, 2025. Gelei Deng, Yi Liu, Víctor Mayoral-Vilches, Peng Liu, Yuekang Li, Yuan Xu, Tianwei Zhang, Yang Liu, Martin Pinzger, and Stefan Rass. Pentestgpt: An llm-empowered automatic penetration testing tool. arXiv preprint arXiv:2308.06782, 2023. Min Du, Feifei Li, Guineng Zheng, and Vivek Srikumar. Deeplog: Anomaly detection and diagnosis from system logs through deep learning. In Proceedings of the 2017 ACM SIGSAC conference on computer and communications security, p. 1285–1298, 2017. Yilun Du, Shuang Li, Antonio Torralba, Joshua B Tenenbaum, and Igor Mordatch.Improv- ing factuality and reasoning in language models through multiagent debate. arXiv preprint arXiv:2305.14325, 2023a. Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, and Igor Mordatch. Improv- ing factuality and reasoning in language models through multiagent debate. arXiv preprint arXiv:2305.14325, 2023b. Richard Fang, Rohan Bindu, Akul Gupta, and Daniel Kang. Llm agents can autonomously exploit one-day vulnerabilities. arXiv preprint arXiv:2404.08144, 2024. Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. Deep learning, volume 1. MIT Press, 2016. 11 Under review as a conference paper at ICLR 2027 GoogleDeepMind.Gemini3.5flashmodelcard,2026.URL https: //storage.googleapis.com/deepmind-media/Model-Cards/ Gemini-3-5-Flash-Model-Card.pdf. Mark Handley, Vern Paxson, and Christian Kreibich. Network intrusion detection: evasion, traf- fic normalization, and End-to-End protocol semantics. In 10th USENIX Security Symposium (USENIX Security 01), 2001. Geoffrey E. Hinton, Simon Osindero, and Yee Whye Teh. A fast learning algorithm for deep belief nets. Neural Computation, 18:1527–1554, 2006. Md Nahid Hossain, Sadegh M Milajerdi, Junao Wang, Birhanu Eshete, Rigel Gjomemo, R Sekar, Scott Stoller, and VN Venkatakrishnan. SLEUTH: Real-time attack scenario reconstruction fromCOTS audit data. In 26th USENIX security symposium (USENIX Security 17), p. 487– 504, 2017. Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024. Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. Llm-blender: Ensembling large language models with pairwise ranking and generative fusion. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 14165–14178, 2023a. Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. LLM-Blender: Ensembling large language models with pairwise ranking and generative fusion. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics, 2023b. Yuning Jiang, Qiaoran Meng, Feiyang Shang, Nay Oo, Le Thi Hong Minh, Hoon Wei Lim, and Biplab Sikdar. Mitre att&ck applications in cybersecurity and the way forward. arXiv preprint arXiv:2502.10825, 2025. Max Landauer, Wolfgang Hotwagner, Thorina Boenke, Florian Skopik, and Markus Wurzenberger. Cam-lds: Cyber attack manifestations for automatic interpretation of system logs and security alerts. arXiv preprint arXiv:2603.04186, 2026. Hwiwon Lee, Ziqi Zhang, Hanxiao Lu, and Lingming Zhang. Sec-bench: Automated benchmarking of llm agents on real-world software security tasks, 2025. URL https://arxiv.org/abs/ 2506.11791. Guohao Li, Hasan Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. Camel: Com- municative agents for" mind" exploration of large language model society. Advances in neural information processing systems, 36:51991–52008, 2023a. Shaofei Li, Feng Dong, Xusheng Xiao, Haoyu Wang, Fei Shao, Jiedong Chen, Yao Guo, Xiangqun Chen, and Ding Li. Nodlink: An online system for fine-grained apt attack detection and investi- gation. arXiv preprint arXiv:2311.02331, 2023b. Zhenyuan Li, Qi Alfred Chen, Runqing Yang, Yan Chen, and Wei Ruan. Threat detection and investigation with system-level provenance graphs: A survey. Computers & Security, 106:102282, 2021. Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text Summarization Branches Out, p. 74–81. Association for Computational Linguistics, 2004. Justin Lin, Eliot Jones, Donovan Jasper, Ethan Ho, Anna Wu, Arnold Yang, Neil Perry, Andy Zou, Matt Fredrikson, Zico Kolter, et al. Comparing ai agents to cybersecurity professionals in real- world penetration testing. In International Conference on Learning Representations, volume 2026, p. 1872–1902, 2026. LinuxAuditProject.Linuxaudit. https://github.com/linux-audit/ audit-userspace. GitHub repository. 12 Under review as a conference paper at ICLR 2027 Xuyang Liu, Yibin Han, Zhenwei Zhang, Kai Chang, Zhiwei Xu, Tian Qiu, Weixian Deng, Jiabao Gao, Xiaolin Peng, Hai Wan, and Xibin Zhao. Diagchain: A diagnostic benchmark for evaluating llm agents on evidence-grounded attack chain reconstruction, 2026. URL https://arxiv. org/abs/2608.03591. Zijun Liu, Yanzhe Zhang, Peng Li, Yang Liu, and Diyi Yang. A dynamic llm-powered agent net- work for task-oriented agent collaboration, 2024. URL https://arxiv.org/abs/2310. 02170. Microsoft.Sysmon:System monitor. https://learn.microsoft.com/en-us/ sysinternals/downloads/sysmon. Sadegh M Milajerdi, Rigel Gjomemo, Birhanu Eshete, Ramachandran Sekar, and VN Venkatakr- ishnan. Holmes: real-time apt detection through correlation of suspicious information flows. In 2019 IEEE symposium on security and privacy (SP), p. 1137–1152. IEEE, 2019. Yisroel Mirsky, Tomer Doitshman, Yuval Elovici, and Asaf Shabtai. Kitsune: An ensemble of autoencoders for online network intrusion detection. arXiv preprint arXiv:1802.09089, 2018. Kunal Mukherjee and Murat Kantarcioglu. Llm-driven provenance forensics for threat investigation and detection. arXiv preprint arXiv:2508.21323, 2025. Jiwon Ock, Jungsik Lee, and Changhee Choi. Distributed multi-llm provenance graph construction for scalable correlation of apt logs. In 2026 IEEE 46th International Conference on Distributed Computing Systems Workshops (ICDCSW), p. 13–16. IEEE, 2026. OpenAI. Gpt-5.6 system card. https://deploymentsafety.openai.com/gpt-5-6/ gpt-5-6.pdf, 2026. OpenAI Deployment Safety Hub. OpenClaw.OpenClaw:Personal ai assistant. https://github.com/openclaw/ openclaw. Software repository. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: A method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, p. 311–318. Association for Computational Linguistics, 2002. doi: 10.3115/1073083.1073135. Feargus Pendlebury, Fabio Pierazzi, Roberto Jordaney, Johannes Kinder, and Lorenzo Cavallaro. TESSERACT: Eliminating experimental bias in malware classification across space and time. In 28th USENIX security symposium (USENIX Security 19), p. 729–746, 2019. Trung V Phan and Thomas Bauschert. Learning the apt kill chain: Temporal reasoning over prove- nance data for attack stage estimation. In ICC 2026-IEEE International Conference on Commu- nications, p. 1–6. IEEE, 2026. Chen Qian, Zihao Xie, YiFei Wang, Wei Liu, Kunlun Zhu, Hanchen Xia, Yufan Dang, Zhuoyun Du, Weize Chen, Cheng Yang, et al. Scaling large language model-based multi-agent collaboration. In International Conference on Learning Representations, volume 2025, p. 41488–41505, 2025. Xuebo Qiu, Mingqi Lv, Yimei Zhang, Tieming Chen, Tiantian Zhu, Qijie Song, and Shouling Ji. Apt-cglp: advanced persistent threat hunting via contrastive graph-language pre-training. In Pro- ceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 1, p. 1228–1239, 2026. Qwen Team. Qwen3.8-max: A new bar for coding and cowork, August 2026. URL https: //qwen.ai/blog?id=qwen3.8. RedCanary.Atomicredteam. https://github.com/redcanaryco/ atomic-red-team. GitHub repository. Mati Ur Rehman, Hadi Ahmadi, and Wajih Ul Hassan. Flash: A comprehensive approach to in- trusion detection via provenance graph representation learning. In 2024 IEEE Symposium on Security and Privacy (SP), p. 3552–3570. IEEE, 2024. 13 Under review as a conference paper at ICLR 2027 Shuo Ren et al. Codebleu: A method for automatic evaluation of code synthesis. arXiv preprint arXiv:2009.10297, 2020. Martin Roesch et al. Snort: Lightweight intrusion detection for networks. In Lisa, volume 99, p. 229–238, 1999. María Sanz-Gómez, Víctor Mayoral-Vilches, Francesco Balassone, Luis Javier Navarrete-Lozano, Cristóbal RJ Chavez, and Maite del Mundo de Torres. Cybersecurity ai benchmark (caibench): a meta-benchmark for evaluating cybersecurity ai agents. arXiv preprint arXiv:2510.24317, 2025. Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. Advances in neural information processing systems, 36:68539– 68551, 2023. Minghao Shao, Sofija Jancheska, Meet Udeshi, Brendan Dolan-Gavitt, Haoran Xi, Kimberly Milner, Boyuan Chen, Max Yin, Siddharth Garg, Prashanth Krishnamurthy, Farshad Khorrami, Ramesh Karri, and Muhammad Shafique. Nyu ctf bench: A scalable open-source benchmark dataset for evaluating llms in offensive security, 2025. URL https://arxiv.org/abs/2406.05590. Xingyao Wang, Boxuan Li, Yufan Song, Frank F Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, et al. Openhands: An open platform for ai software de- velopers as generalist agents. In International Conference on Learning Representations, volume 2025, p. 65882–65919, 2025. Zhenhailong Wang, Shaoguang Mao, Wenshan Wu, Tao Ge, Furu Wei, and Heng Ji. Unleashing the emergent cognitive synergy in large language models: A task-solving agent through multi-persona self-collaboration. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics, 2024a. Zhenhailong Wang, Shaoguang Mao, Wenshan Wu, Tao Ge, Furu Wei, and Heng Ji. Unleashing the emergent cognitive synergy in large language models: A task-solving agent through multi-persona self-collaboration. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), p. 257–279, 2024b. Zhun Wang, Tianneng Shi, Jingxuan He, Matthew Cai, Jialin Zhang, and Dawn Song. Cybergym: Evaluating ai agents’ real-world cybersecurity capabilities at scale. In International Conference on Learning Representations, volume 2026, p. 123341–123386, 2026. Chunlin Xiong, Tiantian Zhu, Weihao Dong, Linqi Ruan, Runqing Yang, Yueqiang Cheng, Yan Chen, Shuai Cheng, and Xutong Chen. Conan: A practical real-time apt detection system with high accuracy and efficiency. IEEE Transactions on Dependable and Secure Computing, 19(1): 551–565, 2020. Anyi Xu, Bangcai Lin, Bing Xue, Bingxuan Wang, Bingzheng Xu, Bochao Wu, Bowei Zhang, Chaofan Lin, Chen Dong, Chenchen Ling, et al. Deepseek-v4: Towards highly efficient million- token context intelligence. arXiv preprint arXiv:2606.19348, 2026. Wenhao Yan, Ning An, Linxu Li, Bingsheng Bi, Bo Jiang, Zhigang Lu, Baoxu Liu, Junrong Liu, and Cong Dong. Provagent: Threat detection based on identity-behavior binding and multi-agent collaborative attack investigation. arXiv preprint arXiv:2603.09358, 2026. John Yang, Carlos Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. Swe-agent: Agent-computer interfaces enable automated software engineering. Advances in Neural Information Processing Systems, 37:50528–50652, 2024. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629, 2022. Anis Yusof, Shaofei Li, Arshdeep Singh Kawatra, Ding Li, Ee-Chien Chang, and Zhenkai Liang. From observations to insights: Constructing effective cyberattack provenance with provcon. In Proceedings of the Workshop on SOC Operations and Construction (WOSOC), 2025. 14 Under review as a conference paper at ICLR 2027 Zeek Project. Zeek: The zeek network security monitor. https://github.com/zeek/zeek. GitHub repository. Aohan Zeng, Xin Lv, Zhenyu Hou, Zhengxiao Du, Qinkai Zheng, Bin Chen, Da Yin, Chendi Ge, Chenghua Huang, Chengxing Xie, et al. Glm-5: from vibe coding to agentic engineering. arXiv preprint arXiv:2602.15763, 2026. Andy Zhang, Joey Ji, Celeste Menders, Riya Dulepet, Thomas Qin, Ron Wang, Junrong Wu, Kyleen Liao, Jiliang Li, Jinghan Hu, et al. Bountybench: Dollar impact of ai agent attackers and defenders on real-world cybersecurity systems. Advances in Neural Information Processing Systems, 38, 2026. Andy K Zhang, Neil Perry, Riya Dulepet, Joey Ji, Celeste Menders, Justin W Lin, Eliot Jones, Gashon Hussein, Samantha Liu, Donovan Julian Jasper, Pura Peetathawatchai, Ari Glenn, Vikram Sivashankar, Daniel Zamoshchin, Leo Glikbarg, Derek Askaryar, Haoxiang Yang, Aolin Zhang, Rishi Alluri, Nathan Tran, Rinnara Sangpisit, Kenny O Oseleononmen, Dan Boneh, Daniel E. Ho, and Percy Liang. Cybench: A framework for evaluating cybersecurity capabilities and risks of language models. In The Thirteenth International Conference on Learning Representations, 2025a. URL https://openreview.net/forum?id=tc90LV0yRL. Bo Zhang, Yansong Gao, Boyu Kuang, Changlong Yu, Anmin Fu, and Willy Susilo. A survey on advanced persistent threat detection: A unified framework, challenges, and countermeasures. ACM Computing Surveys, 57(3):62, 2025b. Zhao Zhang, Senlin Luo, Yingdan Guan, and Limin Pan. Mpca: Constructing the apts prove- nance graphs through multi-perspective confidence and association. Information and Soft- ware Technology, 180:107670, 2025c. ISSN 0950-5849. doi: https://doi.org/10.1016/j.infsof. 2025.107670. URL https://w.sciencedirect.com/science/article/pii/ S0950584925000096. Guangze Zhao, Yongzheng Zhang, Weilin Gai, Hongri Liu, Yuliang Wei, and Bailing Wang. Hunteragent: Neuro-symbolic attack trace reconstruction under anti-forensics. arXiv preprint arXiv:2605.29269, 2026. Yuxuan Zhu, Antony Kellermann, Dylan Bowman, Philip Li, Akul Gupta, Adarsh Danda, Richard Fang, Conner Jensen, Eric Ihli, Jason Benn, et al. Cve-bench: A benchmark for ai agents’ ability to exploit real-world web application vulnerabilities. arXiv preprint arXiv:2503.17332, 2025. Michael Zipperle, Florian Gottwalt, Elizabeth Chang, and Tharam Dillon. Provenance-based intru- sion detection systems: A survey. ACM Computing Surveys, 55(7):1–36, 2022. 15 Under review as a conference paper at ICLR 2027 APPENDIX OVERVIEW This appendix provides supplementary experiments, analyses, and case studies, organized as fol- lows: • Appendix A: Additional Experiments. We report additional evaluations with different LLM backbones, conduct ablation studies, analyze consistency with human judgments, and investigate performance across difficulty and attack-stage levels. • Appendix B: Experimental Details. We provide detailed descriptions of baseline configurations, validation environments, and experimental settings. • Appendix C: Details of Semantic Evaluation via LLM-as-a-Judge. We describe the scoring criteria, multi-dimensional evaluation protocol, and prompts used for semantic assessment. • Appendix D: Attack Behavior Taxonomy and Boundaries. We present the complete attack be- havior taxonomy, discuss validation boundaries, and summarize future extensions of CyberClear. • Appendix E: More Cases. We provide additional qualitative examples of attack-chain prove- nance reconstruction and error analysis. AADDITIONAL EXPERIMENTS A.1EVALUATION WITH ADDITIONAL LLM BACKBONES Table 4 reports additional results with GPT-4o-mini and Gemini-3.5-Flash-Lite to examine the ro- bustness of different agent frameworks across additional LLM backbones. The results show sub- stantial performance variation across backbones, indicating that provenance is sensitive to the un- derlying model capability. With Gemini-3.5-Flash-Lite, CyberProvenance achieves the strongest performance across all five Semantic Metrics metrics while maintaining competitive code-level sim- ilarity. In contrast, performance with GPT-4o-mini is generally lower, particularly on Strict, Detail, and Scenario, highlighting the difficulty of reconstructing complete attack chains with weaker back- bones. These results further show that both the agent framework and the underlying LLM contribute substantially to reliable APT attack chain provenance. A.2ABLATION STUDIES To quantify the contribution of each harness component, we remove Evidence Memory, Execution Validation, and Feedback Refinement individually while keeping all other settings unchanged. As shown in Table 5, removing Execution Validation causes the largest degradation, reducing the av- erage Semantic Metrics score from 0.7212 to 0.6472, with especially large drops in Strict, Detail, and Scenario. Removing Evidence Memory yields a comparable decline to 0.6516 and notably re- duces Strict and Detail by 0.0759 and 0.0888, respectively, showing the importance of maintaining consistent evidence across long-context logs. Removing Feedback Refinement leads to a smaller but consistent decrease, with the average semantic score falling to 0.6897 and all evaluation metrics degrading relative to the full model. These results demonstrate that Evidence Memory and Execu- tion Validation provide the main gains, while Feedback Refinement further improves reconstruction quality through iterative correction. A.3CONSISTENCY WITH HUMAN JUDGMENTS To evaluate the reliability of our semantic evaluation method, we compare the model-based scores with human judgments. Two human experts independently assess the generated provenance graphs, and each expert repeats the evaluation three times; the final human score is obtained by averaging all judgments. We use the GPT-5.6-Terra evaluation results for this analysis and measure the correlation between automatic scores and human assessments. As shown in Figure 8, the scores produced by our evaluator exhibit strong consistency with human judgments across all five dimensions. The Pearson correlation coefficients reach 0.9425, 0.9351, 0.9445, 0.9426, and 0.9376 for Single, Loose, Strict, Detail, and Scenario, respectively. These results indicate that our evaluation method can reliably approximate human assessments while providing a scalable alternative for evaluating attack-chain provenance reconstruction. 16 Under review as a conference paper at ICLR 2027 Table 4: Additional results of representative agent frameworks on CyberClear with different LLM backbones. Methods Code-Level MetricsSemantic Metrics BLEU ROUGE-L CodeBLEU Pass@1Single Loose Strict Detail Scenario OpenClaw (Single) 0.36170.52360.30190.81130.4509 0.3588 0.2253 0.2602 0.3209 0.34280.49750.36590.82670.6521 0.5748 0.4146 0.3917 0.4615 OpenClaw (Multiple) 0.44830.60270.48720.90850.5694 0.4592 0.2982 0.3279 0.4293 0.42370.57180.55260.91780.5894 0.5127 0.3812 0.3526 0.4098 AgentVerse 0.39640.58190.52980.92890.6383 0.5314 0.3510 0.3650 0.4869 0.37490.55210.59630.93780.5486 0.4725 0.3497 0.3264 0.3618 LLM-Debate 0.43890.56510.57260.85330.7407 0.6388 0.4284 0.4178 0.5727 0.41460.53870.63850.87110.6418 0.5589 0.4236 0.3895 0.4587 MultiPersona 0.40750.59430.34410.91780.4299 0.3581 0.2263 0.2476 0.2945 0.38930.56850.41480.92670.6635 0.5864 0.4281 0.3972 0.4716 LLM-Blender 0.44960.58680.49850.86220.5911 0.4816 0.3147 0.3397 0.4475 0.42610.55840.56890.88000.5962 0.5187 0.3864 0.3615 0.4219 DyLAN 0.43620.57620.61440.86220.7863 0.6842 0.4583 0.4366 0.6127 0.45840.57830.66430.88440.7583 0.6273 0.4258 0.4538 0.6029 MacNet 0.39080.55590.40660.84670.4413 0.2981 0.1763 0.2564 0.3343 0.39330.55780.54440.84660.5742 0.4674 0.3567 0.4523 0.4196 CAMEL 0.39020.58730.33840.92220.4264 0.3596 0.2278 0.2454 0.2894 0.38870.58420.42190.92220.4753 0.3606 0.2563 0.2505 0.3009 AgentScope-V2 0.46420.60730.65130.98670.6646 0.5702 0.3934 0.4035 0.4998 0.50350.64350.73110.86220.7095 0.6889 0.4785 0.5557 0.6278 CyberProvenance (Ours) 0.48710.62460.66730.98220.6748 0.5792 0.3974 0.4046 0.5098 0.48960.67380.69410.99110.8485 0.7186 0.6179 0.5843 0.6897 Table 5:Ablation study of CyberProvenance on CyberClear using GPT-5.6-Terra. Red values indicate performance drops relative to the full model. Variants Code-Level MetricsSemantic Metrics BLEUROUGE-LCodeBLEUPass@1SingleLooseStrictDetailScenario CyberProvenance0.49520.69680.70160.93330.88970.74380.63370.61240.7263 w/o Evidence Memory0.4510↓0.0442 0.6423↓0.0545 0.6554↓0.0462 0.9044↓0.02890.8421↓0.0476 0.6845↓0.0593 0.5578↓0.0759 0.5236↓0.0888 0.6502↓0.0761 w/o Execution Validation0.4598↓0.0354 0.6492↓0.0476 0.6615↓0.0401 0.8978↓0.03550.8470↓0.0427 0.6780↓0.0658 0.5485↓0.0852 0.5200↓0.0924 0.6425↓0.0838 w/o Feedback Refinement0.4825↓0.0127 0.6811↓0.0157 0.6889↓0.0127 0.9178↓0.01550.8734↓0.0163 0.7169↓0.0269 0.5983↓0.0354 0.5716↓0.0408 0.6885↓0.0378 A.4ADDITIONAL PERFORMANCE ANALYSIS ACROSS DIFFICULTY LEVELS The additional results examine how code-level similarity varies with task difficulty across different agent frameworks and LLM backbones. BLEU, ROUGE-L, and CodeBLEU remain relatively sta- ble for most methods on easy and medium instances, but decrease substantially on hard instances. As shown in Figure 9, the decline becomes more pronounced as the complexity of attack-chain re- construction increases. Hard instances also produce larger performance gaps among different agent frameworks, indicating greater differences in their ability to preserve the structure of the target 17 Under review as a conference paper at ICLR 2027 0.50.60.70.80.9 Human Judgment Score 0.5 0.6 0.7 0.8 0.9 GLM-5.2 Score r= 0.9425 Single 0.400.450.500.550.600.650.700.750.80 Human Judgment Score 0.40 0.45 0.50 0.55 0.60 0.65 0.70 0.75 0.80 GLM-5.2 Score r= 0.9351 Loose 0.30.40.50.6 Human Judgment Score 0.3 0.4 0.5 0.6 GLM-5.2 Score r= 0.9445 Strict 0.300.350.400.450.500.550.600.65 Human Judgment Score 0.30 0.35 0.40 0.45 0.50 0.55 0.60 0.65 GLM-5.2 Score r= 0.9426 Detail 0.400.450.500.550.600.650.700.75 Human Judgment Score 0.40 0.45 0.50 0.55 0.60 0.65 0.70 0.75 GLM-5.2 Score r= 0.9376 Scenario Model Legend OpenClaw-Single OpenClaw-Multiple AgentVerse LLM-Debate MultiPersona LLM-Blender DyLAN MacNet CAMEL AgentScope-V2 CyberProvenance Figure 8: Consistency between GPT-5.6-Terra-based evaluation and human judgments. Two human experts independently evaluate generated provenance graphs, with each evaluation repeated three times and averaged. provenance graphs. This trend is consistently observed across the three LLM backbones, although the extent of degradation varies across models. These results further show that increasing task diffi- culty challenges both the semantic reconstruction of attack chains and the structural consistency of the generated graphs. A.5ADDITIONAL PERFORMANCE ANALYSIS ACROSS ATTACK STAGE LEVELS Across the three code-level metrics, agent frameworks exhibit clear differences between single- stage and multi-stage attack-chain reconstruction. Most methods show relatively stable BLEU and ROUGE-L scores across the two settings, while larger variations can be observed in CodeBLEU. As shown in Figure 10, AgentScope-V2 and CyberProvenance maintain consistently high scores across different LLM backbones, indicating stronger structural fidelity in the generated provenance graphs. In contrast, several general-purpose agent frameworks achieve noticeably lower similarity, especially when reconstructing more complex multi-stage attack chains. The comparison across GPT-5.6-Terra, DeepSeek-V4-Flash, and Qwen-3.8-Max further shows that the same agent frame- work can exhibit different levels of robustness depending on the underlying LLM. These results suggest that reliable provenance graph generation depends not only on the backbone model, but also on whether the agent framework provides effective support for iterative refinement, evidence consistency, and cybersecurity-oriented validation. BEXPERIMENTAL DETAILS B.1BASELINES For all baselines, we set the temperature to 0.2 and use top-p = 1.0. The remaining configuration details for each baseline are provided below: OpenClaw (Single) (OpenClaw). We configure OpenClaw with a single LLM agent for security log analysis, attack chain reconstruction, and DOT generation, using Web Search when needed and allowing up to three rounds of self verification and refinement. We utilize the official implementation available at https://github.com/openclaw/openclaw. OpenClaw (Multiple). We build three collaborative OpenClaw agents for attack evidence ex- traction, DOT code generation, and output verification, respectively. Web Search is restricted to ATT&CK identifier and name verification, and the system performs up to three rounds of gener- 18 Under review as a conference paper at ICLR 2027 OpenClaw (Single) OpenClaw (Multiple) AgentVerse LLM-Debate MultiPersona LLM-Blender DyLAN MacNet CAMEL AgentScope-V2 CyberProvenance GPT-5.6-Te r r aDeepSeek-V4-FlashQwen-3.8-Max Easy Difficulty OpenClaw (Single) OpenClaw (Multiple) AgentVerse LLM-Debate MultiPersona LLM-Blender DyLAN MacNet CAMEL AgentScope-V2 CyberProvenance MediumDifficultyHardDifficulty OpenClaw (Single) OpenClaw (Multiple) AgentVerse LLM-Debate MultiPersona LLM-Blender DyLAN MacNet CAMEL AgentScope-V2 CyberProvenance 1.0 0.8 0.6 0.4 0.2 1.0 0.8 0.6 0.4 0.2 0 .06 0. 0 5 0. 0 4 0. 0 3 0. 02 OpenClaw (Single) OpenClaw (Multiple) AgentVerse LLM-Debate MultiPersona LLM-Blender DyLAN MacNet CAMEL AgentScope-V2 CyberProvenance OpenClaw (Single) OpenClaw (Multiple) AgentVerse LLM-Debate MultiPersona LLM-Blender DyLAN MacNet CAMEL AgentScope-V2 CyberProvenance OpenClaw (Single) OpenClaw (Multiple) AgentVerse LLM-Debate MultiPersona LLM-Blender DyLAN MacNet CAMEL AgentScope-V2 CyberProvenance 1.0 0.8 0.6 0.4 0.2 1.0 0.8 0.6 0.4 0.2 0 .06 0. 0 5 0. 0 4 0. 0 3 0. 02 OpenClaw (Single) OpenClaw (Multiple) AgentVerse LLM-Debate MultiPersona LLM-Blender DyLAN MacNet CAMEL AgentScope-V2 CyberProvenance OpenClaw (Single) OpenClaw (Multiple) AgentVerse LLM-Debate MultiPersona LLM-Blender DyLAN MacNet CAMEL AgentScope-V2 CyberProvenance OpenClaw (Single) OpenClaw (Multiple) AgentVerse LLM-Debate MultiPersona LLM-Blender DyLAN MacNet CAMEL AgentScope-V2 CyberProvenance 1.0 0.8 0.6 0.4 0.2 1.0 0.8 0.6 0.4 0.2 0 .06 0. 0 5 0. 0 4 0. 0 3 0. 02 BlEU ROUGE-L CodeBLEU Figure 9: Code-Level similarity across difficulty levels using BLEU, ROUGE-L, and CodeBLEU. Columns correspond to easy, medium, and hard instances, and each radar chart compares agent frameworks under three LLM backbones. 0.360.400.330.35 0.34 0.39 0.45 0.45 0.420.42 0.44 0.45 0.39 0.42 0.360.38 0.38 0.41 0.44 0.45 0.410.42 0.43 0.44 0.41 0.43 0.370.39 0.39 0.42 0.45 0.45 0.420.42 0.440.45 0.37 0.43 0.340.37 0.370.42 0.33 0.37 0.370.36 0.34 0.41 0.41 0.43 0.350.39 0.41 0.40 0.51 0.48 0.440.44 0.50 0.50 0.50 0.48 0.500.49 0.48 0.48 0.520.550.480.500.500.55 0.580.590.560.580.580.61 0.590.590.540.560.560.59 0.580.580.520.540.540.58 0.520.550.550.570.570.60 0.600.590.550.560.570.60 0.510.570.520.550.540.60 0.440.480.470.480.460.52 0.580.580.570.580.580.59 0.640.610.620.630.650.66 0.710.670.680.700.680.69 0.370.430.380.380.370.42 0.560.590.560.570.560.59 0.600.610.600.620.610.63 0.650.650.640.670.650.67 0.410.470.420.420.420.46 0.610.620.570.590.580.60 0.520.590.500.530.540.59 0.490.550.540.520.520.59 0.570.570.560.590.590.56 0.730.710.670.690.720.74 0.710.690.680.700.700.72 OpenClaw (Single) OpenClaw (Multiple) AgentVerse LLM-Debate MultiPersona LLM-Blender DyLAN MacNet CAMEL AgentScope-V2 CyberProvenance GPT-5.6-Te r r aDeepSeek-V4-FlashQwen-3.8-Max SingleMultiSingleMultiSingle Multi GPT-5.6-Te r r aDeepSeek-V4-FlashQwen-3.8-Max SingleMultiSingleMultiSingle Multi 0.0–0.20.2–0.40.4–0.60.6–0.80.8–1.0 GPT-5.6-Te r r aDeepSeek-V4-FlashQwen-3.8-Max SingleMultiSingleMultiSingle Multi BlEUROUGE-L CodeBLEU Figure 10: Code-Level similarity across single-stage and multi-stage instances using BLEU, ROUGE-L, and CodeBLEU. Columns correspond to three LLM backbones, and each heatmap com- pares agent frameworks under single-stage and multi-stage settings. ation, review, and refinement. We use the official code released at https://github.com/ openclaw/openclaw. AgentVerse (Chen et al., 2024). We configure AgentVerse with four expert agents that collaborate under a predefined protocol for log analysis and attack chain reconstruction. The agents jointly analyze evidence, propose candidate reconstructions, and consolidate the final DOT graph. The source code is available at https://github.com/OpenBMB/AgentVerse/. 19 Under review as a conference paper at ICLR 2027 LLM-Debate (Du et al., 2023a). We configure LLM-Debate with three agents and two debate rounds. Each agent independently generates an attack-chain candidate, then revises it after review- ing the other agents’ outputs, with the final results consolidated into a single reconstruction. We use the official implementation available at https://github.com/composable-models/ llm_multiagent_debate. MultiPersona (Wang et al., 2024b). In our experiments, MultiPersona uses a single LLM in- stance to simulate three task-specific personas for attack-chain reconstruction. The personas interact over multiple rounds to analyze attack evidence and collaboratively refine the reconstructed chain. The experiments are conducted using the implementation provided at https://github.com/ MikeWangWZHL/Solo-Performance-Prompting. LLM-Blender (Jiang et al., 2023a). For LLM-Blender, three generator LLMs produce 11 candidate reconstructions, which are pairwise ranked by PairRanker. The top three candidates (K = 3) are retained and fused by GenFuser to produce the final DOT graph. Our experiments build on the implementation released at https://github.com/yuchenlin/LLM-Blender. DyLAN (Liu et al., 2024). We enable DyLAN’s dynamic agent selection, multi-round interaction, and early stopping for attack-chain reconstruction. The corresponding implementation is obtained from https://github.com/SALT-NLP/DyLAN. MacNet (Qian et al., 2025). In the MacNet setting, agents are organized as a directed acyclic collaboration network and interact following the topological order to derive the final reconstruction. We conduct the experiments with the MacNet implementation from https://github.com/ OpenBMB/ChatDev/tree/macnet. CAMEL (Li et al., 2023a). For CAMEL, we assign predefined roles to the participating agents and allow them to collaboratively reconstruct the attack chain through autonomous role-playing dialogue. The experimental setup is implemented with the codebase at https://github.com/ camel-ai/camel. AgentScope-V2 (Dawei Gao, 2024; 2025). AgentScope-V2 is configured with role-specialized agents that exchange intermediate evidence and reconstruction results through structured messages, allowing later agents to refine earlier outputs before producing the final graph. We implement this baseline using the AgentScope repository at https://github.com/agentscope-ai/ agentscope. B.2ATTACK VALIDATION ENVIRONMENT For environment-based validation, we select execution procedures using the predicted ATT&CK Technique identifiers, while obtaining platform and runtime settings from predefined environment configurations. The configurations of the execution and telemetry tools are provided below: Atomic Red Team (Red Canary). We use Atomic Red Team to reproduce individual attack steps. For each predicted Technique identifier, we retrieve candidate tests from a predefined catalog and select compatible implementations according to their supported platforms, execution requirements, and dependencies. Execution results and the corresponding telemetry are retained for subsequent comparison with the observed evidence. We use the official implementation available at https: //github.com/redcanaryco/atomic-red-team. CALDERA (Apache Software Foundation). We use CALDERA to orchestrate multi-step at- tack chains with execution dependencies. Predicted Technique identifiers are mapped to prede- fined abilities, whose requirements and parsers specify execution prerequisites and extract facts from procedure outputs. The execution agents and planner are specified by the experiment con- figuration. Operation records and execution traces are retained to assess step completion and examine dependencies between steps. We use the official implementation available at https: //github.com/apache/caldera. Ludus (Bad Sector Labs). We use Ludus with Proxmox to provision isolated virtual machines from predefined environment templates. The range configuration specifies the operating systems, network topology, services, and telemetry components required by the selected procedures. Testing mode is configured to block communication outside the experimental range, and snapshots provide 20 Under review as a conference paper at ICLR 2027 reproducible starting states for validation runs. The official documentation is available at https: //docs.ludus.cloud/. Sysmon (Microsoft). We deploy Sysmon on virtual machines to collect host activity under a pre- defined event-filtering configuration. The collected events cover process creation, network con- nections, and file activity, providing host-level observations for comparison with the reconstructed attack steps. We use the official distribution available at https://learn.microsoft.com/ en-us/sysinternals/downloads/sysmon. Linux Audit (Linux Audit Project). We deploy Linux Audit on virtual machines and configure audit rules for the system calls and file operations relevant to the selected procedures. The audit daemon collects the resulting records, including timestamps, process identifiers, user identifiers, and operation outcomes where available. These records provide execution evidence for validating attack steps on Linux hosts. We use the official implementation available at https://github. com/linux-audit/audit-userspace. Zeek (Zeek Project). We use Zeek to monitor traffic visible on the experimental range’s des- ignated capture interface. Connection records and protocol-specific logs, including DNS and HTTP records where applicable, complement the host telemetry collected by Sysmon and Linux Audit. The resulting network observations are associated with validation runs and incorporated into the structured validation reports. We use the official implementation available at https: //github.com/zeek/zeek. CDETAILS OF SEMANTIC EVALUATION VIA LLM-AS-A-JUDGE C.1SCORING CRITERIA We evaluate each reference attack step by aligning it with the predicted content and assessing the degree of semantic agreement. Based on this comparison, each step is categorized as Correct, Minor Partial, Moderate Partial, Major Partial, Incorrect, or Missing. We then assign an integer raw score in [0, 100] according to the step-level scoring criteria below and normalize it to [0, 1] by dividing by 100. • (80,100] (Correct): The reference attack event is accurately reconstructed. Its actor, core action, target, and overall attack meaning are preserved. A slightly lower score within this range may be assigned when only limited non-critical details are omitted or harmlessly imprecise. • (60,80] (Minor Partial): The same event and its core behavior are clearly identifiable, but a sec- ondary behavioral qualifier, execution method, local outcome, or causal explanation is incomplete. These differences do not alter the identity, attack stage, or principal meaning of the event. • (40,60] (Moderate Partial): The same event remains identifiable, but at least one core semantic component of the actor, action, target, or outcome is missing, incorrect, or ambiguous. Conse- quently, the attack process is incomplete, although it has not changed into a different event. • (20,40] (Major Partial): The prediction is substantially incomplete and preserves only a broad core behavior, with most important information missing or inaccurate. • (0,20] (Incorrect): The predicted event substantively conflicts with the reference event or exhibits only a very weak semantic correspondence. • 0 (Missing): The prediction contains no behavior with a meaningful semantic correspondence to the reference attack event. C.2MULTI-DIMENSIONAL SEMANTIC ASSESSMENT In this section, we provide detailed descriptions of our multi-dimensional semantic evaluation method. Given a reference provenance graph G and a predicted provenance graph ˆ G, the judge directly compares their DOT code, treating the reference graph as authoritative. Differences in node identifiers, DOT statement order, graph layout, colors, shapes, and other purely visual properties are ignored, while entities and attributes, such as commands, file paths, IP addresses, timestamps, PIDs, services, and ATT&CK identifiers, are evaluated when they carry forensic significance. We 21 Under review as a conference paper at ICLR 2027 use GLM-5.2 as the judge model with the temperature set to 0, and require each judgment to provide supporting evidence from both the reference and predicted graphs whenever available. Single-Step Judge. The Single-Step Judge evaluates the semantic agreement between the refer- ence event and the predicted content for single-step samples. It directly assigns a real-valued score in the range [0, 1] based on whether the predicted event preserves the core action, target, and overall meaning of the reference event. Loose Judge. The Loose Judge evaluates multi-step samples by independently measuring the se- mantic coverage of each reference attack step without considering temporal order. The sample-level score is computed as the mean of all step-level scores. A predicted step may cover multiple ref- erence steps only when it contains distinct sub-actions corresponding to these steps, and the same predicted sub-action cannot be reused to match multiple reference steps. Unsupported predicted actions cannot match reference steps and do not contribute to the score. Strict Judge. The Strict Judge evaluates whether the predicted attack chain preserves the temporal progression of the reference chain. Let K denote the number of reference steps in the longest consecutively correct prefix, and let N denote the total number of reference steps. The prefix is counted in reference-step units rather than predicted-step positions, allowing one predicted step to cover multiple consecutive reference steps when its distinct sub-actions appear in the correct semantic order. The prefix terminates at the first partial or incorrect reference event, or when an unsupported attack action is inserted before the reference chain is completed. If all reference steps are correctly covered, U denotes the number of unsupported attack actions appended after the completed chain. The Strict Judge score is defined as: S strict = K N ,K < N, 2N 2N + U , K = N. (1) A completely correct chain without unsupported appended actions receives a score of 1. Unsup- ported entities, relations, and attributes are evaluated separately by the Detail Judge. Detail Judge. The Detail Judge evaluates the fidelity of graph-level details, including entities, directed relations, and semantic attributes. Purely visual properties, such as node colors, shapes, and layout information, are ignored. Reference and predicted elements are aligned one-to-one within each category. A fully correct match receives a weight of 1.0, while partial matches receive weights of 0.75, 0.50, or 0.25 according to the preserved semantic information. Incorrect matches, uncovered reference elements, and unsupported predicted elements receive zero weight. For each category c∈ent, rel, attr, we compute: F c = 2TP w c N ref c + N pred c , S detail = X c w c F c ,(2) where TP w c denotes the weighted semantic matches, and N ref c and N pred c denote the numbers of reference and predicted elements in category c. We set w ent = 0.35, w rel = 0.45, and w attr = 0.20. The final Detail-Judge score is normalized to [0, 1]. Scenario Judge. The Scenario Judge evaluates the overall consistency between the reference and predicted attack descriptions extracted from DOT files. It assigns a real-valued score in the range [0, 1] according to whether the two descriptions convey the same attack process, target, method, progression, and outcome. The judge allows synonymous expressions, semantic paraphrases, and harmless generalizations. However, contradictions, unsupported core attack stages, and fabricated attack objectives reduce the final score. 22 Under review as a conference paper at ICLR 2027 C.3LLM-AS-A-JUDGE PROMPTS System Prompt You are an evidence−grounded and semantically tolerant cybersecurity evaluation judge specializing in ,→ APT attack provenance. Return exactly one valid JSON object that follows the requested schema. Use only evidence encoded in ,→ the supplied reference and predicted DOT graphs. Do not use markdown fences or add text outside the JSON object. Before responding, verify that all quotes, backslashes, newlines, commands, paths, and evidence excerpts inside JSON strings are correctly escaped. Figure 11: System prompt defining the evaluator’s role, evidence constraints, and JSON-only re- sponse requirements. General Evaluation Protocol You are evaluating an APT provenance graph encoded in Graphviz DOT. Compare ,→ REFERENCE_DOT with PREDICTED_DOT. REFERENCE_DOT is authoritative. I. GENERAL EVALUATION PRINCIPLES 1. Judge semantic equivalence rather than lexical overlap. 2. Ignore harmless differences in node IDs, DOT statement order, layout, rank direction, font, color, ,→ shape, and other purely visual attributes. 3. Do not reward additional cybersecurity knowledge merely because it is plausible. Ignore harmless ,→ paraphrases or explanatory wording that adds no factual claim. A substantive predicted action, ,→ participating entity, relation, objective, timestamp, or ATT&CK technique that is not supported by ,→ the reference is an error. 4. For Single−Step, Loose, and Strict, judge the semantic attack event rather than exact forensic ,→ identifiers. If the semantic subject/actor role, core action, object/target role, and overall meaning ,→ agree, allow synonyms, paraphrases, active/passive rewrites, and reasonable generalization. An ,→ omitted or generalized ATT&CK ID, PID, timestamp, exact IP address, command, path, process/ ,→ file/service name, or other non−behavioral detail does not by itself prevent a step−level correct ,→ judgment. Detail Judge audits those entities, relations, and attributes separately. 5. Equivalent timezone expressions are acceptable only when they denote the same event and preserve ,→ temporal order. 6. Quote or concisely identify evidence from both DOT graphs for every judgment. Prefer step numbers ,→ , node IDs, edge endpoints, labels, commands, paths, timestamps, and ATT&CK IDs. 7. Do not infer an unstated fact merely because it would be plausible in a real attack. I. EVALUATION ROUTING Extract the numbered attack−narrative steps from REFERENCE_DOT and let N be the number of ,→ reference steps. If N = 1, use evaluation_mode = "single_step" and evaluate: Single−Step Judge, Detail Judge, and Scenario Judge. If N >= 2, use evaluation_mode = "multi_step" and evaluate: Loose Judge, Strict Judge, Detail Judge, and Scenario Judge. Timeline step boxes are evaluated by the step metrics. Do not count step boxes again as system entities under Detail Judge. Step−level judges evaluate attack−step coverage and temporal organization from the narrative timeline. They must not separately inventory or score system nodes, provenance edges, edge attributes, or ATT&CK annotations; Detail Judge owns that structural audit. A step is covered when its semantic subject/actor role, core action, object/target role, and overall event meaning agree. These roles need 23 Under review as a conference paper at ICLR 2027 General Evaluation Protocol (continued) not use the same words or exact identifiers. This semantic check is not a second edge−level score. I. COMMON 0−100 SCORING PROTOCOL 81−100: Faithful 61−80: Mostly Faithful 41−60: Partially Faithful 21−40: Weakly Aligned 0−20: Minimally Aligned or Failed For Strict and Detail, these bands only interpret the computed score. For Single−Step, every Loose ,→ reference step, and Scenario Judge, they guide direct score selection and are refined in Sections VI, ,→ VII, and X. Reserve 100 for complete semantic equivalence under the content evaluated by the metric. Use 0 when ,→ no meaningful semantic correspondence exists. IV. SEMANTIC ALIGNMENT AND FALSE−POSITIVE ACCOUNTING Strict−specific simplification: a reference behavior in the wrong chronological order, the absence of a corresponding next predicted sub−action, and a prediction that ends early are all incorrect. Do not use separate missing or out_of_order labels in strict.step_checks. Use unsupported_inserted only when an additional reference−unsupported attack action is inserted into the predicted narrative before the reference chain has been completed. Prediction−only label: unsupported: A substantive predicted action, participating entity, relation, objective, timestamp, or ATT ,→ &CK technique cannot be supported by or matched to any reference content. A named entity is ,→ substantive when the prediction asserts that it participates in the attack. Harmless paraphrases, ,→ connective wording, legends, and formatting that introduce no new factual claim are not ,→ unsupported. Correct reference coverage and unsupported prediction must be evaluated independently. Do not invalidate a correctly aligned reference item merely because other prediction content is ,→ fabricated. Single−Step and Loose score only their reference steps and do not count prediction−side false positives ,→ . Unsupported content is audited separately by Detail, Scenario, and unsupported_content. Scenario does not count units or compute precision/recall; it considers unsupported content's semantic ,→ impact in its holistic score. For Detail Judge, use weighted semantic precision, recall, and F1. Within the category being scored, let: − N_ref be the number of unique substantive reference semantic units; − N_pred be the number of unique substantive predicted semantic units; − TP_w be the weighted semantic true−positive count. A correct one−to−one alignment contributes ,→ 1.0. A partial alignment must additionally receive a partial_severity and credit: minor partial = 0.75, moderate partial = 0.50, and major partial = 0.25. Incorrect, missing, and unsupported_inserted contribute 0. Then compute: semantic_recall = TP_w / N_ref semantic_precision = TP_w / N_pred semantic_f1 = 2 * semantic_precision * semantic_recall/ (semantic_precision + semantic_recall) The equivalent count form is: 24 Under review as a conference paper at ICLR 2027 General Evaluation Protocol (continued) semantic_f1 = 2 * TP_w / (N_ref + N_pred) raw_score = 100 * semantic_f1 final_score = raw_score Each reference unit may align to at most one predicted unit, and each predicted unit may align to at ,→ most one reference unit. A compound prediction step may cover multiple reference steps only after ,→ being decomposed into distinct sub−actions. Do not create many−to−one credit from the same sub− ,→ action. For Detail rows, identify a matched predicted unit canonically with the pair prediction_match and ,→ prediction_evidence. Within one Detail category, do not reuse the same normalized pair for two ,→ different reference items. If the same display name occurs in different predicted nodes or edges, ,→ include enough node/edge evidence to distinguish the two units. Partial−severity calibration: Minor Partial (credit 0.75): the core fact is faithful and only a limited investigation−relevant qualifier is ,→ omitted, generalized, or locally inaccurate; Moderate Partial (credit 0.50): the same event or item is clearly recognizable, but one essential ,→ participant, object, method, result, or relation detail is missing or inaccurate; Major Partial (credit 0.25): only the central behavior or role remains recognizable and most ,→ distinguishing information is absent, but the match is still stronger than keywords or category ,→ overlap. For correct, incorrect, missing, and unsupported_inserted, set partial_severity to null and credit to 1.0, ,→ 0, 0, and 0 respectively. Do not use a partial credit merely to express uncertainty; select the credit ,→ from the semantic evidence. V. ORDER AND COMPOUND−STEP RULES One predicted step may match multiple reference steps when it contains multiple distinguishable attack sub−actions. Segmentation differences are not errors. Order−of−mention convention: 1. Explicit temporal evidence has highest priority, including "before", "after", "first", "then", "subsequently", timestamps, numbered actions, and directed temporal arrows. 2. If no explicit temporal evidence exists, textual order of mention is chronological order. Thus "A and ,→ B" means A followed by B, and "A, B, and C" means A followed by B followed by C. 3. Actions on separate lines are ordered from top to bottom. 4. Explicit temporal meaning overrides surface mention order. For example, "B occurred after A" ,→ means A followed by B. 5. If the text explicitly says actions are simultaneous, do not impose a sequential order from mention ,→ order alone. 6. These rules apply to attack actions, not lists of entities or attributes. "Files A and B" does not ,→ describe two ordered attack actions. Loose Judge uses these semantic correspondences and ignores their order. Strict Judge uses ordered sub−action alignment rather than fixed Step−number alignment. A compound predicted step may satisfy several consecutive reference steps when it contains distinct ,→ sub−actions in the same order as the reference. Several Strict rows may therefore share one ,→ prediction_step_index, but each must cite a different prediction_subaction. The same sub−action ,→ cannot receive credit twice, and a compound prediction cannot skip an unmatched intermediate ,→ reference step. Figure 12: General evaluation protocol specifying shared principles for evidence-grounded semantic comparison of reference and predicted provenance graphs. 25 Under review as a conference paper at ICLR 2027 Single-Step and Loose Judge Prompt SINGLE−STEP JUDGE 1.Apply only when N = 1. Compare the single reference attack step with the predicted attack content. Judge whether the semantic subject/actor role, core action, object/target role, and overall attack ,→ meaning agree. 2.Allow synonymous or generalized expression and ignore ATT&CK annotations and exact forensic identifiers at this step level. Supporting evidence may occur in one compound prediction step, but it ,→ must clearly express the reference behavior. 3.Assign exactly one reference−alignment label from Section IV: correct, partial, incorrect, or missing. Apply the entity boundaries already incorporated into those definitions when the action matches but an ,→ entity is omitted, generalized, or contradictory. 4.Assign one direct integer step_score from 0 to 100 to the only reference step. Use exactly the same ,→ semantic anchors as Loose Judge The judgment, partial_severity, and step_score must agree. The Single−Step score is exactly this step ,→ score: single_step_judge_score = step_score Single−Step has no temporal−order requirement and does not use N_pred, precision, recall, or F1. ,→ Extra prediction−only actions, entities, relations, and attributes do not reduce this reference−step score; record them in unsupported_content and evaluate them under Detail and Scenario. If no meaningful predicted event corresponds to the reference step, use judgment=missing and step_score=0. LOOSE JUDGE Apply only when N >= 2. Evaluate unordered semantic coverage of reference attack steps. Do not ,→ penalize reordering. For every reference step, locate a semantically corresponding predicted sub−action if one exists. Assign exactly one reference−alignment label from Section IV to every reference step: correct, partial, ,→ incorrect, or missing. Apply the entity boundaries already incorporated into those definitions when the action matches but an ,→ entity is omitted, generalized, or contradictory. A single predicted step may correctly cover several reference steps through distinct sub−actions. Under ,→ Loose Judge, those reference steps need not be consecutive and their predicted order is not ,→ penalized. Never use one predicted sub−action to satisfy two different reference steps. Score every reference step independently with one integer step_score from 0 to 100. Judge whether the ,→ matched predicted sub−action preserves the reference step's semantic subject/actor role, core action ,→ , object/target role, behavioral direction, and overall event meaning. Allow synonymous or ,→ generalized wording. Do not lower the step score merely for omitted ATT&CK annotations or exact forensic identifiers that ,→ do not change the event. Use these anchors: 81−100, correct: The same semantic event is reconstructed: its actor/subject role, core action, object/ ,→ target role, and overall meaning agree. Use 100 when this behavioral meaning is fully preserved. ,→ Use 81−99 for harmless semantic imprecision, not for missing ATT&CK IDs or exact forensic ,→ identifiers alone. 61−80, minor partial: The same event and its core behavior are clear, but a limited investigation− ,→ relevant component is omitted, generalized, or locally inaccurate. 41−60, moderate partial: The same event remains identifiable, but one essential participant, object, ,→ action component, result, or causal detail is missing or wrong. 26 Under review as a conference paper at ICLR 2027 Single-Step and Loose Judge Prompt (continued) 21−40, major partial: Only the broad core event is preserved; most distinguishing information is ,→ missing or inaccurate, although the match is stronger than isolated keywords or entity overlap. 1−20, incorrect or minimally aligned: A candidate sub−action has only a weak event−level connection ,→ and materially conflicts with the reference action, actor/object, direction, causal meaning, or ,→ outcome. 0, missing or wholly incompatible: No meaningful predicted event corresponds to the reference step. ,→ Keyword, entity−name, or ATT&CK−label overlap alone receives 0. The categorical judgment, partial_severity, and step_score must agree with these ranges. Extra ,→ prediction−only actions do not lower a matched reference step's Loose score and do not enter a ,→ denominator; record them in unsupported_content for the other applicable judges. Let s_j be the step_score for reference step j and N be the total number of reference steps. Compute the ,→ sample score without integer rounding: loose_step_score_sum = sum(s_j), j = 1,...,N loose_judge_score = loose_step_score_sum / N At dataset level, let M be the number of multi−step samples and first calculate the sample mean above ,→ for each sample i. Then compute the macro−average: dataset_loose_score = (1 / M) * sum(loose_judge_score_i), i = 1,...,M Every sample therefore has equal weight regardless of its number of reference steps. Do not pool all steps across the dataset, and do not round per−sample means before computing the ,→ dataset macro−average. Figure 13: Single-Step and Loose Judge prompt for evaluating individual attack steps and unordered semantic coverage of multi-step attack chains. Strict Judge Prompt Apply only when N >= 2. Evaluate the consecutive correct prefix using ordered sub−action alignment. ,→ Maintain a reference cursor and a prediction cursor. Decompose compound predicted steps into ,→ distinguishable ordered sub−actions. One predicted step may advance the reference cursor through multiple consecutive reference steps, ,→ provided those sub−actions are semantically correct and appear in the required order. A reference step is strict−correct when: 1.its semantic subject/actor role matches; 2.its core action/predicate matches; 3.its semantic object/target role and overall attack meaning are preserved; 4.it occurs at the next required chronological position; and 5.no unsupported substantive narrative−timeline action appears before it or between matched actions. The semantic roles need not use the same words or exact identifiers. Synonyms, paraphrases, active/ ,→ passive rewrites, and reasonable generalization are strict−correct. ATT&CK annotations and exact ,→ PIDs, timestamps, IPs, commands, paths, or entity names do not affect Strict unless they change ,→ which attack event is described. Assign each Strict comparison exactly one label: correct, partial, incorrect, or unsupported_inserted. − Use incorrect when the next predicted sub−action has the wrong behavior, actor/object, or attack ,→ meaning; when the expected behavior appears in the wrong chronological order; when no ,→ corresponding predicted sub−action exists; or when the prediction has ended early. 27 Under review as a conference paper at ICLR 2027 Strict Judge Prompt (continued) − Use unsupported_inserted only when the next predicted sub−action is an additional substantive attack ,→ action unsupported by the reference before the reference chain is complete. Also record that action ,→ in unsupported_content with strict_effect=terminates_prefix and one canonical step_actions unit. Let K be the number of consecutive correct reference steps from the beginning. Start with K = 0 and ,→ increase K once for each consecutively correct reference step. Distinct ordered sub−actions from ,→ one compound predicted step may increase K multiple times. Stop immediately at the first partial, ,→ incorrect, or unsupported_inserted result. Later correct content may be audited but must not ,→ increase K or restore the prefix. If the predicted timeline ends early, label the first unmatched reference step incorrect, set prediction_step_index to null, and explain "prediction ended early" in reason. If a reference ,→ behavior exists but appears in the wrong chronological order, label it incorrect and explain the order mismatch in reason. An extra entity or provenance edge that appears only in the graph structure and is absent from the narrative timeline is audited by Detail Judge and does not by itself terminate the strict prefix. Strict score computation is deterministic: If K < N: strict_raw_score = 100 * K / N If K = N but U unsupported substantive attack actions are appended after the complete correct ,→ reference timeline, combine ordered recall = 1 with ordered precision = N / (N + U) using F1: strict_raw_score = 100 * 2 * N / (2 * N + U) U excludes harmless restatements, legends, entity details, and visual annotations. Round half up: strict_judge_score = strict_raw_score Use 100 only when K = N and U = 0. Use 0 when the first chronological comparison already fails. Required consistency examples: N=2: K=0 −> 0; K=1 −> 50; K=2 −> 100. N=3: K=0 −> 0; K=1 −> 33; K=2 −> 67; K=3 −> 100. N=4: K=0 −> 0; K=1 −> 25; K=2 −> 50; K=3 −> 75; K=4 −> 100. Interpretive bands, which must not replace the formula: 81−100: faithful or nearly complete consecutive timeline; 61−80: long correct prefix, but failure before completion; 41−60: meaningful intermediate prefix, often around half; 21−40: short initial attack fragment; 0−20: failure at the beginning or negligible correct prefix. Figure 14: Strict Judge prompt for evaluating the consecutive correct prefix of an attack chain under ordered sub-action alignment. Detail Judge Prompt Inventory and evaluate reference−to−prediction coverage in three categories. Every inventory item ,→ must be atomic enough for one clear judgment. Avoid double−counting the same fact. Before writing any Detail rows, make a one−to−one allocation separately for entities, relations, and ,→ attributes: assign each predicted unit to only its single best−matching reference unit. 28 Under review as a conference paper at ICLR 2027 Detail Judge Prompt (continued) Write that matched reference row first. After a predicted unit has been allocated, never use it for ,→ another reference row. For every remaining reference item, either use a different real predicted unit or write judgment=" ,→ missing" with an empty prediction_match and empty prediction_evidence. A missing row must ,→ never cite a prediction. 1. Entities: Semantic graph nodes representing attack participants or objects, including accounts/users, processes, tools, commands represented as explicit objects, files/artifacts, hosts/IPs, services, sockets, network objects, and other system entities. Identify an entity from its semantic label and role, such as [user], [process], [tool], [file], [network], or [service], rather than from color, shape, or fill style, because generated graphs may use different visual conventions. 2. Relations: Important directed provenance relations between system entities. Direction and action semantics both matter. A line is an edge, not an entity. An edge between two system entities that expresses an attack action such as executed, created, read, downloaded, accessed, or connected_to remains a provenance relation whether rendered as a solid, dashed, or dotted line. 3. Attributes: Investigation−relevant PID, timestamp/order, executable, command, host/IP, file path, service name, evidence/log source, and ATT&CK ID explicitly encoded in the reference. Do not classify the following as system entities or provenance relations: − timeline step boxes, graph titles, top−level scenario descriptions, legends, cluster boundaries, and ,→ layout anchors; − a dotted step−to−entity edge, which is an evidence−alignment or traceability link and may be used to ,→ locate supporting evidence but is not a provenance relation; − a dashed step−to−step edge, which is a narrative or temporal−order edge used only by Strict Judge; − a dashed cluster boundary or an edge that exists only to control layout. Visual style alone never determines edge type. Classify an edge from the semantic types of its ,→ endpoints and its label. A solid edge is usually a provenance relation only when it connects system ,→ entities and expresses a system or attack behavior. PIDs, timestamps, ATT&CK IDs, executable paths, commands, and host/IP values embedded inside ,→ another entity node are normally attributes of that entity, not separate entities. A command, tool, ,→ host, or IP is a separate entity only when the DOT explicitly represents it as a semantic participant ,→ node. The entities, relations, and attributes arrays must perform bidirectional inventory. First list every unique substantive reference item and align it one−to−one to at most one predicted item using correct, partial, incorrect, or missing. Then list every remaining unique substantive predicted entity, relation, or attribute as an unsupported false positive in unsupported_content. A fabricated attack event may legitimately contribute separate false positives to the entity and relation categories because these are distinct structural units; within each category, however, deduplicate repeated or paraphrased ,→ instances of the same semantic item. Use the correct, partial, incorrect, and missing definitions from Section IV for every reference detail; do not create a separate Detail−specific meaning for these labels. Treat a reversed or contradictory relation aligned to a reference relation as incorrect rather than partial ,→ when it changes the attack meaning. Record an independent fabricated relation in ,→ unsupported_content and include it in the relation N_pred count, thereby lowering relation ,→ precision and F1. Do not apply an additional discretionary deduction or score cap. For small graphs, use the same atomic ,→ inventory and formulas; do not invent extra nodes to enlarge the denominator or ignore a ,→ substantive item merely because the graph is small. Compute one weighted semantic F1 for each category c in entities, relations, attributes: TP_w_c = sum of correct and partial credits in category c recall_c = TP_w_c / N_ref_c precision_c = TP_w_c / N_pred_c 29 Under review as a conference paper at ICLR 2027 Detail Judge Prompt (continued) f1_c = 2 * TP_w_c / (N_ref_c + N_pred_c) Use default category weights entities=0.35, relations=0.45, and attributes=0.20. Relations receive the largest weight because attack provenance depends primarily on recovering directed behavioral dependencies, not merely naming nodes. If a category has N_ref_c=N_pred_c=0, mark it inactive and renormalize the weights of the remaining active categories to sum to 1. detail_f1 = sum(active_weight_c * f1_c) detail_raw_score = 100 * detail_f1 detail_judge_score = detail_raw_score Do not round category F1 values, active weights, detail_f1, or detail_raw_score before normalization and aggregation. If all categories are inactive, set Detail score to 0 and flag the sample for schema review, because a provenance graph is expected to contain at least one detail unit. Figure 15: Detail Judge prompt for evaluating entity, relation, and attribute coverage through one- to-one reference–prediction matching. Scenario Judge Prompt Compare only REFERENCE_SCENARIO with PREDICTED_SCENARIO. Judge whether the two descriptions communicate the same overall attack scenario. Ignore wording, ,→ sentence order, formatting, and harmless differences in detail. Do not infer unstated intent. An unsupported attack stage, objective, or ATT&CK technique is a ,→ substantive error. Do not decompose the descriptions into atomic claims, do not count fields, and do not calculate ,→ precision, recall, or F1. Read each description as one complete semantic unit and directly assign an integer score from 0 to 100. ,→ Consider, only when explicitly stated and relevant, whether they preserve the same core attack ,→ behavior or intent, principal target or object, important method/tool, high−level progression, and ,→ outcome. These are interpretive cues, not mandatory fields; never penalize a description for ,→ omitting a field that the reference itself does not state. Use the following holistic rubric: 81−100 (Equivalent or Nearly Equivalent): Both descriptions communicate the same attack scenario. ,→ Use 100 only when their substantive meanings are fully equivalent and the prediction adds no ,→ material unsupported meaning. Scores 81−99 allow only minor generalization, omission, or ,→ harmless extra detail. 61−80 (Mostly Equivalent): The same overall scenario and principal attack process are clearly ,→ recognizable, but one or more important specifics are omitted, generalized, or locally inaccurate. ,→ There is no contradiction that changes the central attack story. 41−60 (Partially Equivalent): A substantial portion of the scenario matches and the central behavior ,→ may remain recognizable, but a major stage, target, method, objective, progression, or outcome is ,→ missing, uncertain, or wrong. 21−40 (Weakly Related): The descriptions share only a broad attack category, isolated behavior, or ,→ limited fragment, while most of the scenario meaning is absent or different. 0−20 (Minimally Related or Failed): Scores 0−20 indicate that only a trivial semantic fragment ,→ overlaps and the descriptions largely communicate different scenarios. Use 0 when the predicted ,→ scenario is absent, unrelated, or fundamentally contradicts the reference scenario. Unsupported additions are judged by their effect on the overall meaning, not by a fixed deduction. A peripheral plausible detail may have little effect; a fabricated core stage, target, objective, or outcome must reduce the score substantially. Do not award unsupported content merely because it is plausible cybersecurity knowledge. 30 Under review as a conference paper at ICLR 2027 Scenario Judge Prompt (continued) Return a concise qualitative judgment together with matched overall meaning, important omissions or generalizations, contradictions, unsupported additions, evidence from both descriptions, and the reason for the selected score. If the reference scenario is absent, use judgment="unscorable", score 0, and request schema review. Figure 16: Scenario Judge prompt for holistic evaluation of semantic agreement between reference and predicted attack scenarios. Required JSON Output Schema Return JSON only. Do not return an overall average; the caller computes the arithmetic mean of ,→ applicable metric scores. For evaluation_mode = "single_step", return exactly these top−level fields: "sample": "<sample_name>", "evaluation_mode": "single_step", "scores": "single_step_judge_score": 0, "detail_judge_score": 0, "scenario_judge_score": 0 , "score_calculations": "single_step_judge": "step_score": 0, "final_score": 0 , "detail_judge": "entities": "reference_unit_count": 0, "predicted_unit_count": 0, "weighted_true_positive": 0.0, "recall": 0.0, "precision": 0.0, "f1": 0.0, "active_weight": 0.0 , "relations": "reference_unit_count": 0, "predicted_unit_count": 0, "weighted_true_positive": 0.0, "recall": 0.0, "precision": 0.0, "f1": 0.0, "active_weight": 0.0 , "attributes": "reference_unit_count": 0, "predicted_unit_count": 0, "weighted_true_positive": 0.0, "recall": 0.0, "precision": 0.0, "f1": 0.0, "active_weight": 0.0 , "detail_f1": 0.0, "raw_score": 0.0, 31 Under review as a conference paper at ICLR 2027 Required JSON Output Schema (continued) "final_score": 0 , "scenario_judge": "scoring_method": "holistic_llm_judgment", "selected_band": "0−25", "final_score": 0 , "score_reasons": "single_step_judge": "...", "detail_judge": "...", "scenario_judge": "..." , "unsupported_content": [], "reference_timeline": [ "index": 1, "summary": "...", "evidence": "..." ], "prediction_timeline": [ "index": 1, "summary": "...", "evidence": "..." ], "single_step_match": "gold_step_index": 1, "prediction_step_index": 1, "prediction_subaction": "...", "judgment": "correct", "partial_severity": null, "credit": 1.0, "step_score": 100, "gold_evidence": "...", "prediction_evidence": "...", "reason": "..." , "detail": "entities": [], "relations": [], "attributes": [] , "scenario": "judgment": "equivalent", "reference_summary": "...", "prediction_summary": "...", "matched_meaning": [], "missing_or_generalized": [], "contradictions": [], "unsupported_additions": [], "reference_evidence": "...", "prediction_evidence": "...", "reason": "..." For evaluation_mode = "multi_step", return the common top−level fields plus loose_step_matches and 32 Under review as a conference paper at ICLR 2027 Required JSON Output Schema (continued) strict. The scores object contains loose_judge_score, strict_judge_score, detail_judge_score, and ,→ scenario_judge_score. The score_calculations object contains loose_judge, detail_judge, and scenario_judge. The strict object contains reference_step_count, prediction_step_count, ,→ consecutive_correct_prefix_count, first_failure_reference_index, first_failure_prediction_index, ,→ first_failure_type, unsupported_suffix_step_count, strict_prefix_ratio, strict_raw_score, ,→ strict_judge_score, and step_checks. SAMPLE_NAME: __SAMPLE_NAME__ REFERENCE_SCENARIO: __REFERENCE_SCENARIO__ PREDICTED_SCENARIO: __PREDICTED_SCENARIO__ REFERENCE_DOT: __REFERENCE_DOT__ PREDICTED_DOT: __PREDICTED_DOT__ Figure 17: Required JSON output schema for reporting applicable metric scores, score calculations, and supporting evaluation details. DATTACK BEHAVIOR TAXONOMY AND BOUNDARIES D.1COMPLETE LIST OF ATTACK BEHAVIORS Table 6 presents the complete attack behaviors in CyberClear. We normalize the annotated attack behaviors according to their MITRE ATT&CK techniques and merge instances with identical tech- nique sets. This process yields 89 distinct attack behaviors from 450 instances, including 46 single- technique behaviors and 43 composite behaviors. Composite behaviors capture attacks involving multiple ATT&CK techniques and thus reflect more complex attacker activity patterns. We further evaluate the attack-behavior coverage of different agent frameworks on CyberClear. For each behavior, we compute the compilation success rate over all associated instances and regard a generated provenance graph as successful when its DOT code can be compiled correctly. As shown in Table 7, a behavior is considered supported when at least 80% of its instances are successfully compiled. This analysis provides a finer-grained view of model capability by showing which attack behaviors can be handled reliably beyond the aggregate Pass@1 score. D.2METHOD BOUNDARIES We ground the concrete validation implementation in the observed log evidence, while using ATT&CK Techniques to specify the corresponding attack behavior categories. Because the same Technique may admit multiple implementations across operating systems, tools, and versions, the selected implementation must be compatible with the execution context and evidence reflected in the logs. We obtain infrastructure-dependent runtime parameters from predefined configurations and validate inter-step dependencies exclusively using facts and evidence generated during the cur- rent execution. After each local refinement, we replay the attack chain from the initial snapshot to prevent previously generated facts from affecting subsequent dependency verification. We conduct all validation in authorized private environments with external network access blocked, predefined target scopes, and non-destructive execution constraints. D.3FUTURE WORK Future work will explore adaptive cyber harnesses that dynamically optimize evidence retrieval, val- idation strategies, tool selection, and agent collaboration based on accumulated experience. Mean- while, we plan to extend the scale and scope of CyberClear by incorporating more APT campaigns, 33 Under review as a conference paper at ICLR 2027 ATT&CK behaviors, and longer multi-stage attack chains. We will broaden the benchmark to cover more diverse environments, including host, network, cloud, and container settings. We also aim to include more challenging observations with noisy logs, missing evidence, and cross-source incon- sistencies. In addition, we will enrich the benchmark with finer-grained annotations for attack-step dependencies and evidence alignment. Finally, we will evaluate emerging agent frameworks and foundation models to provide a more comprehensive understanding of their capabilities in complete attack-chain reconstruction. EMORE CASES This section presents additional cases from CyberClear. 34 Under review as a conference paper at ICLR 2027 LogFiles E1: judy-> xclip->clipboard type=SYSCALL msg=audit(1765381153.620:136565): ... ppid=28278 pid=30585 auid=1002 uid=1002 gid=1002 euid=1002 suid=1002 fsuid=1002 egid=1002 sgid=1002 fsgid=1002 tty=pts2 ses=1 comm="xclip" exe="/usr/bin/xclip" subj=unconfined key="user_commands"ARCH=x86_64 SYSCALL=execveAUID="judy" UID="judy" GID="judy" EUID="judy" SUID="judy" FSUID="judy" EGID="judy" SGID="judy" FSGID="judy“ type=EXECVE msg=audit(1765381153.620:136565): argc=4 a0="xclip" a1="-selection“ a2="clipboard" a3="-o" type=SOCKADDR msg=audit(1765381153.632:136566): SADDR= saddr_fam=local path=/tmp/.X11-unix/X0 Figure 18: Case study of attack-chain provenance. LLM-Debate incorrectly omits the bash → stty dependency and its terminal-configuration step, despite explicit audit evidence highlighted in yellow: comm="stty", exe="/usr/bin/stty", and stty rows 38 columns 116. CyberProvenance correctly aligns this evidence with the corresponding entity relation and attack step, producing a provenance graph consistent with the observed execution. 35 Under review as a conference paper at ICLR 2027 LogFiles E1: bash -> stty ... type=SYSCALL msg=audit(1758549308.756:4447): ... ppid=2634 pid=2641 auid=4294967295 uid=33 gid=33 euid=33 suid=33 fsuid=33 egid=33 sgid=33 fsgid=33 tty=pts0 ses=4294967295 comm="stty" exe="/usr/bin/stty” ... SYSCALL=execveAUID="unset" UID="w-data" ... type=EXECVE msg=audit(1758549308.756:4447): argc=5 a0="stty" a1="rows" a2="38" a3="columns"a4="116" Figure 19:Case study comparing provenance results. LLM-Debate omits the bash → stty dependency, although the yellow-highlighted audit evidence explicitly records /usr/bin/stty and stty rows 38 columns 116. CyberProvenance correctly aligns this evidence with the corresponding entity relation and attack step. 36 Under review as a conference paper at ICLR 2027 LogFiles E1: john-> auditf.service file audit(1765796989.444): ... ppid=4162 pid=4182 auid=1002 uid=0 comm="vim" exe="/usr/bin/vim.basic“ AUID="john" UID="root" SYSCALL=unlink PROCTITLE: vim /etc/systemd/system/auditf.service PATH:name="/etc/systemd/system/.auditf.service.swp“ nametype=DELETE E2: sudo-> auditf.service ... inetfwsudo:COMMAND=/usr/bin/systemctl... audit: ppid=4162 pid=4185 auid=1002 comm="sudo“ exe="/usr/bin/sudo“ argc=3 a0="sudo" a1="systemctl" a2="daemon-reload" inetfwsudo:COMMAND=/usr/bin/systemctl... audit: ppid=4162 pid=4215 auid=1002comm="sudo" exe="/usr/bin/sudo"argc=4 a0="sudo" a1="systemctl"a2="enable" a3="auditf.service" E3: auditf.service-> ens3 ... inetfwsudo:COMMAND=/usr/bin/systemctl... audit: ppid=4162 pid=4245 auid=1002 comm="sudo" exe="/usr/bin/sudo" argc=4 a0="sudo" a1="systemctl“ a2="start" a3="auditf.service" audit: ppid=1 pid=4249 comm="auditf" exe="/usr/bin/auditf“ argc=1 a0="/usr/bin/auditf" Host inetfw Host inetfw auditf.service ens3 Figure 20: Case study of attack-chain provenance. MacNet fails to preserve three key evidence- supported relations, including the modification of auditf.service, its privileged activation through sudo, and the subsequent execution of the malicious service, while also omitting impor- tant host, service, and network entities. As highlighted in yellow, the logs explicitly record vim /etc/systemd/system/auditf.service, systemctl enable auditf.service, and the execution of /usr/bin/auditf. CyberProvenance retains these dependencies and re- constructs a more complete attack progression consistent with the observed audit evidence. 37 Under review as a conference paper at ICLR 2027 LogFiles E1: python3 -> stty type=SYSCALL msg=audit(1758701779.952:13696): ... ppid=17415 pid=17418 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=7 comm="python3" exe="/usr/bin/python3.9“ type=SYSCALL msg=audit(1758701779.988:13697): ... ppid=17418 pid=17419 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts1 ses=7 comm="bash" exe="/usr/bin/bash“ type=SYSCALL msg=audit(1758701828.800:13738): ... ppid=17419 pid=17438 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts1 ses=7 comm="stty" exe="/usr/bin/stty" E2: python -> xterm256-color type=EXECVE msg=audit(1758701828.800:13738): argc=5 a0="stty" a1="rows" a2="38" a3="columns" a4="116" ... type=PATH msg=audit(1758701828.800:13738): item=2 name="/lib64/ld-linux-x86-64.so.2" inode=1964 dev=fe:01 mode=0100755 ouid=0 ogid=0 rdev=00:00 nametype=NORMAL cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0OUID="root" OGID="root"type=PROCTITLE msg=audit(1758701828.800:13738): proctitle=7374747900726F777300333800636F6C756D6E7300313136... Figure 21: Case study of attack-chain provenance. OpenClaw fails to faithfully preserve the process dependencies associated with terminal configuration. As highlighted in yellow, the logs explicitly record python3 spawning bash, followed by the execution of /usr/bin/stty with rows 38 columns 116. CyberProvenance correctly reconstructs these evidence-supported process re- lations and the corresponding attack-step progression. 38 Under review as a conference paper at ICLR 2027 LogFiles E1: puppet-> download linuxsharepuppet-agent[2029]: Exec[curl http://192.42.1.174:8888/install.sh | bash] executed successfully ... ppid=2029 pid=2100 comm="sh" exe="/usr/bin/dash" ...... ppid=2100 pid=2101 comm="curl" exe="/usr/bin/curl" ...argc=2 a0="curl“ a1="http://192.42.1.174:8888/install.sh" TCP 192.168.100.23:46314 -> 192.42.1.174:8888 HTTP GET /install.sh User-Agent: curl/7.88.1 E2: cron-> healthcheck_cron.sh ... audit(1765543801.860): ... ppid=552 pid=2003 uid=0 comm="cron" exe="/usr/sbin/cron" ... audit(1765543801.864): ... ppid=2003 pid=2004 comm="sh" exe="/usr/bin/dash" ... argc=3 a0="/bin/sh" a1="-c“ a2="/media/health/healthcheck_cron.sh" audit(1765543801.864): ... ppid=2004 pid=2005 comm="healthcheck_cro" exe="/usr/bin/bash" ... argc=2 a0="/bin/bash“ a1="/media/health/healthcheck_cron.sh" E3: healthcheck_cron.sh -> FTP ... audit(1765543801.876): ... ppid=2007 pid=2009comm="lftp" exe="/usr/bin/lftp"... argc=4 a0="/usr/bin/lftp" a1="-u“a2="aecid,***" a3="172.17.100.122" Incorrect entity type Figure 22: Case study of attack-chain provenance. AgentScope-V2 misclassifies a key entity and fails to faithfully align the reconstructed steps with the observed execution chain. As highlighted in yellow, the logs explicitly record the Puppet-triggered curl download, the subsequent execution of healthcheck_cron.sh by cron, and the final lftp connection to 172.17.100.122. CyberProvenance correctly reconstructs these evidence-supported entities and attack-step dependen- cies. 39 Under review as a conference paper at ICLR 2027 LogFiles E1: zmcontroller -> external host type=SYSCALL msg=audit(1759319071.428:5289): ... ppid=1 pid=3064 auid=4294967295 uid=0 gid=21 euid=0 suid=0 fsuid=0 egid=21 sgid=21 fsgid=21 tty=(none) ses=4294967295 comm="zmcontroller" exe="/usr/bin/zmcontroller"... type=SOCKADDR msg=audit(1759319071.444:5291): saddr=020001BBC02A01AE0000000000000000SADDR= saddr_fam=inetladdr=192.42.1.174 lport=443 E2: chmod-> file ... ppid=3064 pid=3183 comm="chmod" exe="/usr/bin/chmod" ... argc=3 a0="chmod" a1="755" a2="/tmp/nmap" E3: nmapprocess -> network scan type=SYSCALL msg=audit(1759319101.544:5324): arch=c000003e syscall=59 success=yes exit=0 a0=c0009355e0 a1=c00038bd40 a2=c00093b300 a3=0 items=2 ppid=3064 pid=3188 auid=4294967295 uid=0 gid=21 euid=0 suid=0 fsuid=0 egid=21 sgid=21 fsgid=21 tty=(none) ses=4294967295 comm="nmap" exe="/tmp/nmap“ type=EXECVE msg=audit(1759319101.544:5324): argc=2 a0="/tmp/nmap" a1="172.17.100.0/24" Figure 23:Case study of attack-chain provenance. LLM-Blender misses several key evidence- supported relations and fails to faithfully preserve the reconnaissance chain. As highlighted in yellow, the logs explicitly record the zmcontroller connection to 192.42.1.174:443, the chmod 755 /tmp/nmap operation, and the subsequent nmap scan of 172.17.100.0/24. CyberProvenance correctly reconstructs these observed entities, relations, and attack-step depen- dencies. 40 Under review as a conference paper at ICLR 2027 Table 6: All attack behaviors in CyberClear. The 450 instances yield 89 distinct technique sets: 46 single-technique and 43 composite behaviors. Attack behaviorATT&CK ID(s) B01 Abuse Elevation Control Mechanism: Sudo and Sudo Caching T1548.003 B02 Account Access RemovalT1531 B03 Account Discovery: Local Account T1087.001 B04 Active Scanning: Vulnerability Scanning T1595.002 B05 Active Scanning: Wordlist Scanning T1595.003 B06 Application Window Discovery T1010 B07 Boot or Logon Autostart Execution T1547 B08 Clipboard DataT1115 B09 Command and Scripting Interpreter T1059 B10 Command and Scripting Interpreter: Unix Shell T1059.004 B11 Create or Modify System Process: Systemd Service T1543.002 B12 Data DestructionT1485 B13 Data from Information Repositories T1213 B14 Data from Local SystemT1005 B15 Data from Network Shared Drive T1039 B16 Exfiltration Over C2 Channel T1041 B17 Exploitation for Privilege Escalation T1068 B18 File and Directory Discovery T1083 B19 File and Directory Permissions Modification: Linux and Mac Permissions T1222.002 B20 Gather Victim Host Information: Software T1592.002 B21 Hide Artifacts: Hidden Files and Directories T1564.001 B22 Indicator Removal: Clear Command History T1070.003 B23 Indicator Removal: File Deletion T1070.004 B24 Ingress Tool TransferT1105 B25 Inhibit System RecoveryT1490 B26 Network Boundary Bridging T1599 B27 Network SniffingT1040 B28 OS Credential DumpingT1003 B29 OS Credential Dumping: /etc/passwd and /etc/shadow T1003.008 B30 Password Policy Discovery T1201 B31 Peripheral Device Discovery T1120 B32 Remote ServicesT1021 B33 Remote Services: SSHT1021.004 B34 Remote System DiscoveryT1018 B35 Scheduled Task/JobT1053 B36 Scheduled Task/Job: CronT1053.003 B37 Service StopT1489 B38 Software Deployment Tools T1072 B39 System Information Discovery T1082 B40 System Network Configuration Discovery T1016 B41 System Network Connections Discovery T1049 B42 System Owner/User Discovery T1033 B43 System Time DiscoveryT1124 B44 Use Alternate Authentication Material: Application Access Token T1550.001 B45 Valid Accounts: Local Accounts T1078.003 B46 Virtualization/Sandbox Evasion: System Checks T1497.001 B47 OS Credential Dumping: /etc/passwd and /etc/shadow + Exfiltration Over C2 Channel T1003.008 +T1041 B48 Masquerading: Match Legitimate Resource Name or Location + Traffic Signaling: Port Knocking T1036.005 +T1205.001 B49 Masquerading: Match Legitimate Resource Name or Location + Hide Artifacts: Hidden Files and Directories T1036.005 +T1564.001 B50 Network Sniffing + Steal Application Access Token T1040 +T1528 B51 Command and Scripting Interpreter + Hijack Execution Flow T1059 +T1574 B52 Command and Scripting Interpreter: Unix Shell + Remote Access Tools T1059.004 +T1219 B53 Command and Scripting Interpreter: Python + Exploit Public-Facing Application T1059.006 +T1190 B54 Data Staged: Local Data Staging + Archive Collected Data: Archive via Utility T1074.001 +T1560.001 B55 Account Manipulation: SSH Authorized Keys + Ingress Tool Transfer T1098.004 +T1105 B56 Ingress Tool Transfer + Create Account: Local Account T1105 +T1136.001 Attack behaviorATT&CK ID(s) B57 Ingress Tool Transfer + Data Encrypted for Impact T1105 +T1486 B58 Ingress Tool Transfer + Modify Authentication Process: Pluggable Authentication Modules T1105 +T1556.003 B59 Data Encrypted for Impact + Data Manipulation: Stored Data Manipulation T1486 +T1565.001 B60 Event Triggered Execution + Hijack Execution Flow: Path Interception by Search Order Hijacking T1546 +T1574.008 B61 Boot or Logon Autostart Execution + System Services T1547 +T1569 B62 Gather Victim Network Information: DNS + Gather Victim Org Information T1590.002 +T1591 B63 Gather Victim Host Information: Software + Active Scanning T1592.002 +T1595 B64 Rootkit + Hide Artifacts + Hijack Execution Flow T1014 +T1564 +T1574 B65 Remote System Discovery + Network Service Discovery + Ingress Tool Transfer T1018 +T1046 +T1105 B66 Masquerading: Match Legitimate Resource Name or Location + Application Layer Protocol: Web Protocols + Traffic Signaling: Port Knocking T1036.005 +T1071.001 +T1205.001 B67 Scheduled Task/Job: Cron + Application Layer Protocol: Web Protocols + Ingress Tool Transfer T1053.003 +T1071.001 +T1105 B68 Command and Scripting Interpreter: Unix Shell + Software Deployment Tools + Ingress Tool Transfer T1059.004 +T1072 +T1105 B69 Command and Scripting Interpreter: Unix Shell + Non-Application Layer Protocol + Exploit Public-Facing Application T1059.004 +T1095 +T1190 B70 Valid Accounts: Domain Accounts + Brute Force: Password Guessing + External Remote Services T1078.002 +T1110.001 +T1133 B71 Valid Accounts: Local Accounts + Ingress Tool Transfer + Traffic Signaling: Port Knocking T1078.003 +T1105 +T1205.001 B72 Account Manipulation: SSH Authorized Keys + Ingress Tool Transfer + System Binary Proxy Execution T1098.004 +T1105 +T1218 B73 Ingress Tool Transfer + Create Account: Local Account + System Binary Proxy Execution T1105 +T1136.001 +T1218 B74 Ingress Tool Transfer + User Execution: Malicious Link + Phishing: Spearphishing Voice T1105 +T1204.001 +T1566.004 B75 Ingress Tool Transfer + System Binary Proxy Execution + Modify Authentication Process: Pluggable Authentication Modules T1105 +T1218 +T1556.003 B76 Ingress Tool Transfer + Remote Access Tools + Hide Artifacts: Hidden Files and Directories T1105 +T1219 +T1564.001 B77 Remote Services: SMB/Windows Admin Shares + Exfiltration Over C2 Channel + Command and Scripting Interpreter: PowerShell + Application Layer Protocol: Web Protocols T1021.002 +T1041 +T1059.001 +T1071.001 B78 Scheduled Task/Job: Cron + Exploitation of Remote Services + Implant Internal Image + Deploy Container T1053.003 +T1210 +T1525 +T1610 B79 Input Capture: Keylogging + Input Capture: Credential API Hooking + Clipboard Data + Software Extensions T1056.001 +T1056.004 +T1115 +T1176 B80 Exfiltration Over C2 Channel + Command and Scripting Interpreter: PowerShell + Indicator Removal: File Deletion + Phishing: Spearphishing Attachment + Acquire Infrastructure: Server T1041 +T1059.001 +T1070.004 +T1566.001 +T1583.004 B81 Exfiltration Over C2 Channel + Exploit Public-Facing Application + Create or Modify System Process: Windows Service + Phishing: Spearphishing Attachment + Acquire Infrastructure: Server T1041 +T1190 +T1543.003 +T1566.001 +T1583.004 Attack behaviorATT&CK ID(s) B82 OS Credential Dumping: LSASS Memory + OS Credential Dumping: Cached Domain Credentials + Input Capture: Keylogging + Email Collection: Remote Email Collection + Create or Modify System Process: Windows Service + Phishing: Spearphishing Attachment T1003.001 +T1003.005 +T1056.001 +T1114.002 +T1543.003 +T1566.001 B83 OS Credential Dumping: LSASS Memory + OS Credential Dumping: Cached Domain Credentials + Input Capture: Keylogging + Command and Scripting Interpreter + Email Collection: Local Email Collection + Email Collection: Remote Email Collection + Phishing: Spearphishing Attachment T1003.001 +T1003.005 +T1056.001 +T1059 +T1114.001 +T1114.002 +T1566.001 B84 OS Credential Dumping: LSASS Memory + OS Credential Dumping: Cached Domain Credentials + Input Capture: Keylogging + Email Collection: Local Email Collection + Email Collection: Remote Email Collection + Create or Modify System Process: Windows Service + Phishing: Spearphishing Attachment T1003.001 +T1003.005 +T1056.001 +T1114.001 +T1114.002 +T1543.003 +T1566.001 B85 OS Credential Dumping: LSASS Memory + OS Credential Dumping: Cached Domain Credentials + Remote Services + Input Capture: Keylogging + Email Collection: Local Email Collection + Email Collection: Remote Email Collection + Create or Modify System Process: Windows Service + Phishing: Spearphishing Attachment T1003.001 +T1003.005 +T1021 +T1056.001 +T1114.001 +T1114.002 +T1543.003 +T1566.001 B86 Remote Services: Windows Remote Management + Exfiltration Over C2 Channel + Command and Scripting Interpreter: PowerShell + Web Service: Bidirectional Communication + Ingress Tool Transfer + Exfiltration Over Web Service: Exfiltration to Cloud Storage + Lateral Tool Transfer + Acquire Infrastructure: Server T1021.006 +T1041 +T1059.001 +T1102.002 +T1105 +T1567.002 +T1570 +T1583.004 B87 OS Credential Dumping: LSASS Memory + OS Credential Dumping: Cached Domain Credentials + Remote Services + Input Capture: Keylogging + Command and Scripting Interpreter + Application Layer Protocol: Web Protocols + Email Collection: Local Email Collection + Email Collection: Remote Email Collection + Phishing: Spearphishing Attachment T1003.001 +T1003.005 +T1021 +T1056.001 +T1059 +T1071.001 +T1114.001 +T1114.002 +T1566.001 B88 Exfiltration Over C2 Channel + Command and Scripting Interpreter: PowerShell + Application Layer Protocol: Web Protocols + Account Discovery: Local Account + Drive-by Compromise + System Binary Proxy Execution: Mshta + System Binary Proxy Execution: Msiexec + Create or Modify System Process: Windows Service + Boot or Logon Autostart Execution: Registry Run Keys / Startup Folder + Acquire Infrastructure: Web Services T1041 +T1059.001 +T1071.001 +T1087.001 +T1189 +T1218.005 +T1218.007 +T1543.003 +T1547.001 +T1583.006 B89 System Service Discovery + System Network Configuration Discovery + System Owner/User Discovery + System Network Connections Discovery + Process Discovery + Permission Groups Discovery + System Information Discovery + File and Directory Discovery + Account Discovery + Password Policy Discovery + Software Discovery + System Location Discovery + Group Policy Discovery T1007 +T1016 +T1033 +T1049 +T1057 +T1069 +T1082 +T1083 +T1087 +T1201 +T1518 +T1614 +T1615 41 Under review as a conference paper at ICLR 2027 Table 7: Attack behavior coverage of different agent frameworks under GPT-5.6-Terra. A check- mark indicates that the framework successfully handles the corresponding attack behavior, while a cross indicates failure. Behavior OpenClaw (Single) OpenClaw (Multiple) AgentVerse LLM-Debate MultiPersona LLM-Blender DyLAN MacNet CAMEL AgentScope-V2 CyberProvenance B01✓ B02✓ B03✓ B04✓ B05✓ B06✓ B07✓ B08✓ B09✓ B10✓ B11✓ B12✓ B13✓ B14✓ B15✓ B16✓ B17✗✓✗✓✗✓ B18✓ B19✓ B20✓ B21✓ B22✓ B23✓ B24✓ B25✓✗✓ B26✓ B27✓ B28✗✓✗✓ B29✓ B30✓ B31✓ B32✓ B33✓ B34✓ B35✓ B36✓ B37✓ B38✗✓✗✓✗✓ B39✓ B40✓ B41✓ B42✓ B43✓ B44✗✓✗✓✗✓ B45✓ B46✓ B47✓✗ B48✗✓✗✓✗✓✗✓ B49✓✗✓ B50✗✓✗✓✗✓ B51✓✗✓✗✓ B52✓ B53✗✓✗✓ B54✓✗ B55✓✗✓ B56✗✓✗✓✗✓ B57✗✓✗✓✗✓ B58✗✓✗✓✗✓✗✓ B59✗✓✗✓✗✓ B60✗✓ B61✓✗✓✗✓✗✓ B62✗✓✗✓✗✓✗✓ B63✓✗✓✗✓ B64✗✓✗✓✗✓ B65✓✗✓✗✓ B66✗✓✗✓✗✓✗ B67✓✗✓✗✓ B68✓✗✓ B69✗✓ B70✓✗✓✗✓ B71✗✓✗✓✗✓ B72✓✗✓✗✓ B73✗✓✗✓✗ B74✗✓✗✓✗✓ B75✗✓✗✓✗✓ B76✗✓✗✓ B77✗✓✗ B78✗✓✗✓✗✓ B79✗✓✗ B80✗✓✗✓✗✓ B81✗✓✗✓✗✓✗ B82✗ B83✗ B84✗ B85✗ B86✗ B87✗ B88✗ B89✗ 42