Paper deep dive
COHORT: Collaborative Orchestration for Hardening via Offensive Replay on Emulated Topologies
Chen Frydman, Aviram Zilberman, Rubin Krief, Abed Showgan, Andres Murillo, Sekiya Motoyoshi, Asaf Shabtai, Yuval Elovici, Rami Puzis
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 97%
Last extracted: 7/5/2026, 4:01:19 AM
Summary
COHORT is an end-to-end framework designed to automate the generation and validation of deployable network mitigations. It uses a role-decomposed multi-agent LLM workflow (comprising a Suggester, Implementer, Critic, Judge, and Summarizer) to propose and refine real device configuration commands. Validation is performed on a high-fidelity GNS3 emulator running real vendor firmware (FortiGate, Cisco IOS, Open vSwitch) using 'offensive replay' via Caldera to ensure the mitigation disrupts the specific adversary without breaking legitimate connectivity (LAN/Internet). The framework demonstrates a 46.7% success rate in complex topologies, significantly outperforming single-agent baselines.
Entities (10)
Relation Signals (7)
LLM Agent (Implementer) → ispartof → COHORT
confidence 100% · An implementer translates a selected mitigation into device commands
LLM Agent (Suggester) → ispartof → COHORT
confidence 100% · The mitigation workflow is decomposed into role-specialized LLM agents.
LLM Agent (Critic) → ispartof → COHORT
confidence 100% · A critic reviews each implementation before replay
LLM Agent (Judge) → ispartof → COHORT
confidence 100% · a judge compares the per-step attack outcome to the unmitigated baseline.
COHORT → uses → GNS3
confidence 100% · all on a high-fidelity GNS3 emulator running real vendor firmware
COHORT → utilizes → Caldera
confidence 100% · with Caldera executing ATT&CK-aligned adversary actions.
FortiGate → runson → GNS3
confidence 90% · running real vendor firmware (firewall, switch, router) in GNS3
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Mitigating an observed adversary in an enterprise network typically takes weeks of expert work: an analyst derives a mitigation tailored to that adversary, validates it without breaking production, and verifies it disrupts the specific attack. The procedure relies on expert judgment and cannot safely be exercised against the production network. COHORT is the first end-to-end framework to automate this procedure for deployable mitigations. A role-decomposed multi-agent LLM workflow proposes candidates, implements them as real device commands, and refines them through a critique loop, all on a high-fidelity GNS3 emulator running real vendor firmware (firewall, switch, router). Each candidate is evaluated by offensive replay: re-executing the original adversary on the mitigated network for a paired comparison against the unmitigated baseline, rather than the reward-signal or expert-judgment proxies used in prior simulation, hybrid, and configuration-generation work. Two further checks complement replay: a connectivity-regression check (LAN ping and internet HTTP probe) rejects mitigations that disrupt legitimate LAN or internet connectivity, and a cumulative evaluation stacks approved mitigations onto a persistent state to surface compound effects. Across three topologies and four attack scenarios (ransomware, lateral movement, DNS exfiltration, data theft), 46.7% of generated mitigations both disrupt the attack and preserve connectivity under replay, 4.4 times the rate of a single-agent baseline using the same model and tool access. A demo video walking through the framework is available with our released artifacts.
Tags
Links
- Source: https://arxiv.org/abs/2606.30479v1
- Canonical: https://arxiv.org/abs/2606.30479v1
Trouble viewing inline? Open PDF directly →
Full Text
86,728 characters extracted from source content.
Expand or collapse full text
COHORT: Collaborative Orchestration for Hardening via Offensive Replay on Emulated Topologies Chen Frydman a , Aviram Zilberman b,a,∗ , Rubin Krief a , Abed Showgan a , Andres Murillo c , Sekiya Motoyoshi c , Asaf Shabtai a , Yuval Elovici a and Rami Puzis a a Ben-Gurion University of The Negev, Beer Sheva, Israel b Jerusalem College of Technology, Jerusalem, Israel c Fujitsu A R T I C L E I N F O Keywords: Network Emulation Automatic Network Mitigation Adversary Emulation Cumulative Mitigations Proactive Defense Multi Agent System A B S T R A C T Mitigating an observed adversary in an enterprise network typically takes weeks of expert work: an analyst derives a mitigation tailored to that adversary, validates it without breaking production, and verifies it disrupts the specific attack. The procedure relies on expert judgment and cannot safely be exercised against the production network. COHORT is the first end-to-end framework to automate this procedure for deployable mitigations. A role-decomposed multi-agent LLM workflow proposes candidates, implements them as real device commands, and refines them through a critique loop, all on a high-fidelity GNS3 emulator running real vendor firmware (firewall, switch, router). Each candidate is evaluated by offensive replay: re-executing the original adversary on the mitigated network for a paired comparison against the unmitigated baseline, rather than the reward-signal or expert-judgment proxies used in prior simulation, hybrid, and configuration-generation work. Two further checks complement replay: a connectivity-regression check (LAN ping and internet HTTP probe) rejects mitigations that disrupt legitimate LAN or internet connectivity, and a cumulative evaluation stacks approved mitigations onto a persistent state to surface compound effects. Across three topologies and four attack scenarios (ransomware, lateral movement, DNS exfiltration, data theft), 46.7% of generated mitigations both disrupt the attack and preserve connectivity under replay, 4.4× the rate of a single- agent baseline using the same model and tool access. A demo video walking through the framework is available with our released artifacts (Appendix 8.3). 1. Introduction Given an adversary profile reconstructed from forensic analysis, threat-intelligence feeds, or red-team exercises, the defender must derive a deployable mitigation tailored to that adversary, validate it without breaking production while verifying that it disrupts the specific attack. Two properties of this workflow drive its cost: it relies on expert knowledge, and it cannot safely be exercised on the production network, since experimentally adjusting firewall rules, routing policy, or host configuration risks breaking legitimate operation. Existing reactive tooling (IDS/IPS, SIEM, SOAR) executes curated or playbook-driven responses for known attack pat- terns; even AI-assisted SOAR operates within a maintained playbook set rather than synthesizing new controls Lyu, Gharakheili and Sivaraman (2024); Islam, Babar and Nepal (2019). Threat intelligence informs analysts on new attacks but does not generate per-incident defenses. Breach-and- attack simulation (BAS) replays curated adversary tech- niques against the unmodified network to validate detection coverage rather than to test newly proposed mitigations. The IBM Cost of a Data Breach report measures mean breach containment times of multiple weeks IBM Security (2025); ∗ Corresponding author ORCID(s): 0009-0006-1554-992X (C. Frydman); 0000-0002-8137-7314 (A. Zilberman); 0009-0000-9507-4368 (R. Krief); 0009-0006-6997-5043 (A. Showgan); 0000-0001-6965-2283 (A. Murillo); 0009-0008-6542-2681 (S. Motoyoshi); 0000-0003-0630-4059 (A. Shabtai); 0000-0002-9641-128X (Y. Elovici); 0000-0002-7229-3899 (R. Puzis) per-incident mitigation derivation is a cost driver inside that window. We present COHORT, the first end-to-end framework that autonomously generates and validates deployable mit- igations. To remove the expert-knowledge bottleneck, a multi-agent LLM workflow proposes, implements, and re- views each candidate mitigation as real device configu- ration commands. To avoid production exposure during evaluation, the workflow operates on a high-fidelity em- ulated enterprise network running real vendor firmware (firewall, switch, router) in GNS3, with Caldera Apple- baum, Miller, Strom, Korber and Wolf (2016); Chang, Cao, Teoh, Ear and Xu (2025) executing ATT&CK-aligned adversary actions. Validation rests on offensive replay: from an established foothold, the same Caldera adversary is re- executed against the network with the candidate mitigation in place, and a judge compares the per-step attack outcome to the unmitigated baseline. Three further mechanisms shape generation and evaluation: an iterative implementation– critique loop catches configuration-text errors before replay; a connectivity-regression check (a LAN ping and an internet HTTP probe) rejects mitigations that disrupt legitimate LAN or internet connectivity; a cumulative evaluation stacks approved mitigations on a persistent network state to surface compound effects. We target post-compromise enterprise networks against scripted, intelligence-equipped adversaries; adaptive adver- saries are out of scope by construction (Section 3). Frydman et al.: Preprint submitted to ElsevierPage 1 of 17 arXiv:2606.30479v1 [cs.NI] 29 Jun 2026 Prior work covers parts of this workflow but not all of it: simulation and hybrid platforms (CybORG/CAGE Kiely, Ahiskali, Borde, Bowman, Bowman, Van Bruggen, Cowan, Dasgupta, Devendorf, Edwards et al. (2025), CyberBat- tleSim Team. (2021), CSLE Hammar and Stadler (2022)) cannot test real configurations, configuration-generation sys- tems (Stackelberg planning Speicher, Steinmetz, Hoffmann, Backes and Künnemann (2019), NetConfEval Wang et al. (2024a)) never confront an active adversary, emulation platforms (CyGIL Li, Fayad and Taylor (2021)) target attacker training rather than defender mitigation, concurrent LLM-based defense agents Saroui, Marchioro and Olivereau (2025) skip replay-based re-validation, and commercial BAS replays curated rather than generated mitigations. None covers the full workflow on a high-fidelity emulator using real device configuration commands (Section 2). The mitigation workflow is decomposed into role-specialized LLM agents. A suggester proposes deployable defenses (firewall ACLs, routing-policy changes, host-hardening com- mands). An implementer translates a selected mitigation into device commands within a bounded per-cycle command budget. A critic reviews each implementation before replay; the implementation–critique loop runs to a fixed iteration limit before the mitigation is accepted or discarded. Two further agents support evaluation: a judge compares pre- and post-mitigation outcomes via Caldera replay, and a summarizer produces a human-readable artifact. The role- specialized design is contrasted against two ablations using the same model and tool access (Section 4.6, Section 5.1): a single-agent baseline that consolidates these roles into one conversation, and a multi-agent variant with the iterative critique loop disabled. We evaluate COHORT across small, medium, and large enterprise topologies emulated in GNS3, four attack sce- narios (Section 3), and 1782 mitigation attempts, all driven by GPT-5.4 mini. The multi-agent framework reaches an overall mitigation success rate (MSR) of 46.7%, where MSR is the rate of mitigations that both reduce attack progress (ME > 0) and pass both connectivity probes (Section 5.2). This is 4.4× the single-agent baseline of 10.7%, with the gap attributable to role specialization and the iterative cri- tique loop (Section 5.3). Stacking approved mitigations on a persistent network state plateaus at 52% cumulative attack- step reduction by the fourth defense (Section 5.3). We do not claim transfer to adaptive adversaries, Windows end- points, model families beyond GPT-5.4 mini, or vendor families beyond FortiGate, Cisco IOS, and Open vSwitch (Section 3.4); calibration against expert-derived mitigations is open (Section 6.5). All evaluation runs in an isolated em- ulator; production deployment of validated mitigations is a separate human-driven step that falls outside the framework. Contributions. • End-to-end framework that addresses both bottle- necks. The first framework that autonomously generates, deploys, and validates deployable mitigations against the specific observed adversary, removing the expert- knowledge dependency and confining experimentation to a high-fidelity emulator with real device commands rather than the production network. • Multi-agent decomposition. Lift from 10.7% to 46.7% MSR over a matched single-agent baseline (same model, inputs, tool access). An ablation decomposes the lift: role specialization accounts for roughly three-quarters, the iterative critique loop for the remaining quarter. • Cumulative evaluation. A non-regression acceptance rule accumulates approved mitigations on a persistent net- work state, exposing compound effects that per-mitigation evaluation cannot surface. 2. Background and Related Work We organize prior work along the six dimensions CO- HORT integrates: enterprise defense, automated configu- ration, multi-agent LLM reasoning, autonomous defense agents, the environments used to evaluate them, and com- mercial tooling. Each subsection ends by identifying the component of COHORT’s workflow that the corresponding line of work omits. 2.1. Enterprise Network Defense Modern enterprise defenses span firewalls, IDS/IPS, SIEM, SOAR, and EDR Lyu et al. (2024) under a funda- mental security–availability tradeoff: a control that disrupts an attack but breaks legitimate services is not deployable. Pre-deployment validation reasons over abstract infrastruc- ture models, while security orchestration and AI-driven incident response automate the reactive side with curated playbooks Islam et al. (2019); Hamadanian, Arzani, Fouladi, Kakarla, Fonseca, Billor, Cheema, Nkposong and Chandra (2023); neither synthesizes a new attack-specific mitigation, nor validates one by re-executing the observed adversary against the modified network. 2.2. Automated Configuration and Network Management Automating network configuration is challenging due to ambiguity in intent, configuration conflicts, and unintended side effects Bringhenti, Marchetto, Sisto and Valenza (2023). Recent LLM-based tools translate natural-language intents into device configurations: NetConfEval Wang et al. (2024a) benchmarks intent-to-config translation; GeNet Ifland, Krief, Zilberman, Duani, Ohana, Murillo, Manor, Lavi, Hikichi, Shabtai, Elovici and Puzis (2025) assists engineers with topology and configuration updates; LLM-NetCFG Lira, Caicedo and da Fonseca (2024) targets zero-touch service management; and recent multi-vendor intent-driven gen- erators Wang, He, Zhao, Xuan, Sun, Qi, Liang, Zhuang and Liao (2026) extend the paradigm. Recent surveys and validation tests across these systems assess intent fulfillment, syntactic correctness, and policy compliance; however, they do not evaluate how the generated configuration affects a re-executed adversary Hong, Tu and Hong (2025). 2 2.3. Large Language Models and Multi-Agent Reasoning in Cybersecurity large language models (LLMs) enable contextual reason- ing over heterogeneous security data Zhang, Bu, Wen, Liu, Fei, Xi, Li, Yang, Zhu and Meng (2025) and have been ap- plied to alert triage, threat explanation, and candidate reme- diation Wang, Xie, Zhang, Wang, Zhang and Cui (2024b); Kaheh, Kholgh and Kostakos (2023), but mostly in human- in-the-loop workflows and remain susceptible to halluci- nations that fabricate threats or miss real ones, motivating explicit validation Sood, Zeadally and Hong (2025). Multi- agent architectures decompose complex tasks into special- ized roles Li, Hammoud, Itani, Khizbullin and Ghanem (2023); Hong, Zhuge, Chen, Zheng, Cheng, Wang, Zhang, Wang, Yau, Lin et al. (2023); An, Yang, Lu, Li, Ren, Huang, Wang, Zhao, Kang, Ding et al. (2024); Luo, Sun, Liu, Niyato, Yu, Atiquzzaman and Dustdar (2026), with iterative refinement and coordinated reasoning across agents Du, Li, Torralba, Tenenbaum and Mordatch (2024). Prior work focused on analysis and recommendation, scoring defenses through RL reward signals, expert judgment, or averages over independent attack runs. Isolating a particular mitiga- tion’s effect on the targeted adversary requires re-executing the same adversary against the just-mitigated network, a step that these methods omit. 2.4. Autonomous Cyber Defense Autonomous defensive agents vary in their degree of autonomy and adaptability Stakhanova, Basu and Wong (2007): scripted agents execute fixed, predetermined ac- tion sequences with no runtime reasoning; planning-based agents dynamically construct action sequences by reasoning over a maintained model of current network state and goals; reinforcement learning (RL) agents learn policies through environmental interaction but remain opaque and difficult to transfer across environments Nguyen and Reddi (2021); and large language model (LLM)-based agents leverage pre-trained world knowledge and natural language reason- ing to generate, critique, and explain actions, addressing the explainability and transferability limitations of RL- only approaches Castro, Campbell, Lau, Villalobos, Duan and Cardenas (2025). RL-based agents learn mitigation policies in simulated enterprise environments Wang and Dechene (2024), while LLM-based defenders (e.g., Agent- NIRS Saroui et al. (2025), which generates firewall rules from NIDS alerts) and hybrid RL+LLM systems Loevenich, Adler, Huerten and Lopes (2025) drive containment in simulators, SOAR pipelines, and cyber ranges (Table 2). On the offensive side, Caldera has been used to evaluate cyber deception against AI-based attackers Kouremetis, Lawrence, Alford, Cheuvront, Davila, Geyer, Haigh, Michalak, Murphy and Russo (2024), and pre-trained LLMs themselves have been deployed as autonomous attackers in cybersecurity RL environments such as NetSecGame, matching or exceed- ing purpose-trained RL baselines without per-environment training Rigaki, Lukáš, Catania and Garcia (2023). Con- current work begins to standardize evaluation for intrusion- response systems on public NIDS datasets Marchioro, Saroui and Olivereau (2025), but defensive agents otherwise op- erate in abstract simulators or execute reactive playbooks, and adversary emulation platforms execute attacks without proposing or validating mitigations (Table 2). 2.5. Simulated and Emulated Network Environments for Adversarial Emulation Safe evaluation of autonomous cyber defense strategies requires isolated environments that can faithfully reproduce adversarial behavior without risk to production systems. The literature spans three approaches with a fidelity–speed tradeoff. Network simulation uses abstract graph-based or Markov-decision-process representations Yamin, Katt and Gkioulos (2020); attack and defense execute as program- matic state transitions, so simulators cannot surface the side effects of real configuration changes. Network emulation instantiates real operating systems, protocols, and devices in virtualized hardware, with backends ranging from containers and VMs to full hardware virtualization booting vendor firmware images (e.g., GNS3 Neumann (2015)); only the latter can evaluate deployable mitigations as they would behave on production hardware, building on the digital-twin paradigm of safe security testing on virtual replicas Eckhart and Ekelhart (2018). Hybrid platforms couple a fast simula- tion surrogate with an emulated backend Molina-Markham, Miniter, Powell and Ridley (2021); Janisch, Pevn ` y and Lis ` y (2023); Oesch, Chaulagain, Weber, Dixson, Sadovnik, Roberson, Watson and Austria (2024). Beyond fidelity, we evaluate environments along four validation properties. The first two of these properties are operationalized in Section 4: mitigation validation (replay- based attack disruption) and connectivity regression testing (verifying that legitimate LAN and internet reachability is preserved after a mitigation is deployed). Deployable mitiga- tions require that defenses be expressed as real device con- figuration commands rather than abstract state transitions, so that what is validated is what would deploy. Cumulative mitigations compose multiple independently proposed de- fenses on a persistent network state, capturing synergistic or interfering effects that per-mitigation evaluation misses. Across these environments, none provides all four val- idation properties: simulators cannot evaluate deployable mitigations, and even emulators that support offensive re- play Kouremetis et al. (2024); Hammar and Stadler (2022) do not pair it with connectivity regression testing or cumu- lative mitigation evaluation. Our framework addresses all four gaps by integrating Caldera adversary emulation with a GNS3-based emulated enterprise network that boots real vendor device images, enabling attack replay after mitigation deployment, automatic connectivity validation, and cumula- tive evaluation of stacked defenses (Table 1). 3 Table 1 Autonomous cyber defense environments compared on fidelity, attack/defense features, and the validation properties required to evaluate deployable mitigations.✓ = supported;× = not supported;~ = partial or unclear. Attack / Defence FeaturesValidation Properties Platform Network type Custom. topo. Vendor dev. imgs Custom. attacks Deployable Mitigation validation Connectivity regression Cumulative mit. Open source Emulation-based MIRAGE (2024)Emu., Sim. 1 ~×✓×✓× CyGIL † (2021)Emulation~×✓× Hybrid (simulation + emulation) Cyberwheel (2024)Hybrid✓~✓×~×✓ FARLAND (2021)Sim., Hybrid✓×~~✓× CSLE (2022)Hybrid✓×~×✓×✓ NASimEmu † (2023)Hybrid✓~~×✓ Simulation-based CybORG / CAGE (2021; 2025; 2024) Simulation 3 ~×~×~×✓ CybORG++ (2024)Simulation~×~×~×✓ PrimAITE (2024)Simulation✓×✓×~×✓ CyberBattleSim (2021)Simulation✓×~×~×✓ COHORT (this work)Emulation✓~ 2 Deployable = defenses expressed as real device configuration commands rather than abstract state transitions. † Attack-only platform; no defender side. 1 https://w.mitre.org/our-impact/intellectual-property/cyberlayer 2 See Section 8.3 for artifact and code release status. 3 CybORG also supports an AWS-based emulation backend; CAGE Challenges, the standard benchmarks built on it, run only the simulation mode. 2.6. Commercial Adversary Emulation and Continuous Validation Industry tooling spans three overlapping categories: Breach-and-Attack Simulation (AttackIQ AttackIQ, Cymu- late Cymulate, Picus Security Picus, SafeBreach SafeBreach, and SCYTHE SCYTHE); continuous attack-path and expo- sure management (XM Cyber XMCyber and Pentera Pen- tera); and high-fidelity cyber ranges (SimSpace SimSpace and Cyberbit Cyberbit). These platforms replay curated adversary playbooks against live or emulated infrastructure to surface control gaps; some now integrate AI assistants for triage and remediation orchestration SafeBreach; Pentera, but the autonomous generation of novel configuration miti- gations and their replay-based re-validation remain outside their scope. 3. Threat Model We consider the security of enterprise networks in the post-compromise phase. The defender’s goal is to harden the network so that an observed adversary, when re-executed against the modified topology, makes less progress with- out breaking legitimate operation. We evaluate four attack scenarios (Table 6): Thief (data theft), Ransomware, DNS Exfiltration, and Lateral Movement. Threat profile COHORT targets scripted, intelligence- equipped adversaries that have established a foothold and acquired targeted reconnaissance, replayable from forensic analysis, threat-intelligence feeds, or red-team exercises. We use Caldera as the replay platform; equivalent BAS or adversary-emulation platforms emitting a comparable operational report can substitute. Adaptive adversaries that re-plan in response to observed defenses are excluded (Sec- tion 3.4); fixing the adversary across baseline and post- mitigation runs is what makes the comparison attributable to the deployed mitigation rather than to attacker variation. 3.1. Adversary Model Knowledge The adversary has a foothold on an endpoint and is seeded with targeted intelligence: valid credentials for one LAN host, the IP of an exfiltration target, and sensitive file paths and extensions. All other facts (additional hosts, services, credentials) must be acquired through on-host re- connaissance and lateral movement during the attack. The adversary has no visibility into the mitigation framework. Capabilities The adversary operates with normal user privileges and executes MITRE ATT&CK techniques using tools staged on the foothold host. The operator control channel is delivered out of band from the production data path and lies outside the defender’s mitigation surface, so agent connectivity is never a defensive target; by contrast, exfiltration traffic traverses the production path and is in scope. Mitigations are credited for blocking exfiltration on the production path, not for severing the control channel, thereby preventing trivial wins and forcing engagement with the actual data flow. Implementation specifics (Caldera Sandcat agent, GNS3 interface naming) are deferred to Caldera Linux Agent. 4 Table 2 Comparison of autonomous cyber defense (ACD) systems and LLM-based network configuration systems.✓ = supported;× = not supported;~ = partial or limited. SystemAgentEnvironmentSuggest Deploy Config-level Open source LLM-based defensive systems AgentNIRS (2025)LLMDataset✓ CyberAlly (2025)LLM + KGLive cyber range✓× Castro et al. (2025)LLMSimulation✓~×✓ ShieldGPT (2024b)LLMAdvisory only✓×✓ Hybrid RL + LLM systems SecurityBot † (2024)RL + LLMSimulation~~× Loevenich et al. (2025) DRL + LLMSimulation~~× LLM-based network configuration systems (no adversary in the loop) NetConfEval (2024a)LLMBenchmark✓~✓ GeNet (2025)LLM (multimodal)Dataset✓×✓ LLM-NetCFG (2024)LLM (local)Use-case demo✓× ConfGen (2026)LLM + RAGDataset✓×✓× Emulation-based defensive systems MIRAGE (2024)Rule-basedEmulation~✓× COHORT (this work) LLM (multi-agent) Emulation✓~ 1 Environment refers to where mitigations are deployed/validated, not training. Config-level = mitigation expressed as device configuration commands rather than abstract state transitions. † SecurityBot operates as both attacker and defender; all other rows are defenders. 1 See Section 8.3 for artifact and code release status. 3.2. Defender Model The defender corresponds to a SOC operator with admin- istrative authority over their own infrastructure. The frame- work consumes a structured operational report from exe- cuting a known adversary profile (threat-intelligence feeds, the Caldera Stockpile, or red-team exercises) against an emulation of the production network; forensic-grade recon- struction of a real-world breach is not assumed. Emulation prerequisite A high-fidelity emulation of the production network is presupposed as input; automated production-to-emulation generation is an open problem (Section 6.5). The prerequisite is met by organizations that already maintain a staging environment or pre-production lab. Failure modes Perfect execution is not assumed. The agent pipeline can produce invalid configurations, disrupt legiti- mate connectivity, or generate operationally ineffective con- trols. The connectivity-preservation gate (Success Crite- rion, below) excludes mitigations that break connectivity. Connectivity-preserving but ineffective mitigations (rules at the wrong hop, evaluated in the wrong order, or targeting primitives the adversary does not use) surface as ME ≤ 0 in the replay; Section 5.3 characterizes the recurring patterns. 3.3. Success Criterion A mitigation succeeds on a run if both of the following hold: 1. it strictly reduces the Attack Step Success Rate (ASSR) compared to the unmitigated baseline (ME > 0); and 2. legitimate connectivity is preserved, verified by two inde- pendent probes from the compromised host (a LAN ping to an internal target and an HTTP request to an internet endpoint), both of which must succeed. Equivalently, letting Φ ∶ → 0,1 denote the connectiv- ity gate,Φ() = ퟏ[LAN ping passes∧ HTTP probe passes], the success indicator is ퟏ[ME(푚; 푡 ,) > 0 ∧ Φ( 푡+1 ) = 1], with 푡 , 푡+1 , and as introduced in Section 4.2. We report this binary outcome as the Mitigation Success Rate (MSR), the continuous reduction as the Mitigation Effectiveness (ME), and the compounding effect across stacked mitigations as the Cumulative ME (Section 5.2). The criterion is step-level by design. A step-level signal exposes which parts of the kill chain a mitigation disrupts, the granularity needed to drive the iterative critic loop; objective denial follows when the disrupted step lies on the only viable path to the objective (for example, blocking the final T1041 exfiltration step prevents data from leaving the network). Step-level disruption that contributes little to the aggregate effect remains visible through ME and Cumulative ME. 3.4. Scope and Generalization The reported numbers generalize to scripted adversaries with observable TTPs whose execution profile can be re- played deterministically. They do not generalize to: • Adaptive adversaries that observe deployed defenses and re-plan. Replay-based validation is non-adaptive by construction (Section 6.1); evaluation against adaptive 5 attackers requires a co-evolving measurement contract, left open (Section 6.5). • TTPs absent from MITRE ATT&CK or whose execu- tion Caldera cannot emulate. • Heterogeneous endpoint OSes. The current evaluation is Linux-only (Section 5.1); whether MSR transfers to Windows or ICS environments is open. • Network-device heterogeneity. The evaluation uses For- tiGate, Cisco IOS, and Open vSwitch images (Firewall, Router, Switch); transfer to other vendor families (e.g., Palo Alto, Juniper, Aruba) is open. • Operational regression beyond connectivity probes, including latency, application-layer disruption, and user- visible behavior beyond LAN/internet reachability. 3.5. Out of Scope • Attacks on the framework or its supporting infrastruc- ture. • Initial-access and pre-compromise phases of the attack lifecycle. • Insider threats and physical attacks. • Virtualization infrastructure failures. 4. COHORT 4.1. Framework Scope and Purpose COHORT focuses on post-compromise enterprise de- fense: it proposes network-level mitigations and validates them by replaying the original adversary in an emulated en- vironment, with no production exposure during evaluation. Fidelity derives from three design choices: (i) control and data planes run real vendor firmware on the firewall, switch, and router; (i) endpoints run as Ubuntu containers; (i) Caldera executes real commands, producing genuine traffic and host-level artifacts. We do not model enterprise- scale traffic load, configuration drift, Windows endpoints (Section 6.3), or live synchronization with production; the framework operates as a staging environment on static base- line snapshots. The fidelity claim is scoped to deployable mitigations (Section 3.4). 4.2. Problem Formalization Setting Let denote the space of network configuration states (device configurations, ACLs, routing policies, host- level configuration). Let denote a fixed adversary scenario executed by Caldera, and휌 the procedure that runs against a state and returns an operational report (ASSR is a function of , defined in Section 5.2). Let be the device configuration command space; a mitigation is a finite se- quence 푚 ∈ ∗ , applied via a state transition 휏 ∶ × ∗ → . Let 퐵 ∈ℕ be the per-refinement command budget and 퐾 ∈ℕ the implementer/critic refinement cap (Section 5.1). Mitigation Cycle A mitigation cycle starts from a baseline state 푡 and produces a candidate 푚 in three stages: (1) re- play: 푡 = 휌( 푡 ,); (2) suggest: the suggester reads 푡 and (per Section 4.3, Context Management) its prior suggestions, and proposes a candidate strategy that the implementer expands into commands; (3) refine: the implementer/critic alternation produces a sequence 푚 (0) ,푚 (1) ,...,푚 (푗) with 푗 ≤ 퐾 and|푚 (푖) | ≤ 퐵 for every 푖, halting on critic approval or at 푗 = 퐾, with 푚 ∶= 푚 (푗) . The post-mitigation state is 푡+1 = 휏( 푡 ,푚). The cycle’s binary outcome is the Success Criterion of Section 3 evaluated on ( 푡 , 푡+1 ,). Constraints (i) Per refinement iteration,|푚 (푖) | ≤ 퐵; the cycle therefore issues at most 퐾⋅퐵 commands across all it- erations (Section 5.1). (i) is held fixed across the baseline replay 휌( 푡 ,) and the post-mitigation replay 휌( 푡+1 ,) used to compute ME (Section 3). (i) The suggester is prompted to exclude indicator-based blocking, monitoring- only responses, and allowlists permitting the observed ex- filtration protocol; this is a design constraint on the policy class, enforced by prompting rather than by syntactic pro- jection. (iv) The single-mitigation environment is reset to a baseline snapshot between cycles; the cumulative environ- ment is not (Section 4.7). Cumulative Variant The cumulative environment main- tains a persistent state (0) , (1) ,... . After the 푘-th can- didate 푚 푘 is implemented and replayed on 휏( (푘−1) ,푚 푘 ), it is accepted, (푘) = 휏( (푘−1) ,푚 푘 ), only if (a) Cumu- lative ME does not decrease relative to (푘−1) , and (b) Φ(휏( (푘−1) ,푚 푘 )) = 1; otherwise (푘) = (푘−1) . 4.3. Agent Roles and Responsibilities The mitigation workflow is implemented using a team of role-specialized LLM-based agents (Figure 1) Liu, Li, Zhang, Wang, He, Hong, Liu, Zhang, Song, Zhu, Cheng, Wang, Wang, Luo, Jin, Zhang, Liu, Chen, Zhang, Yu, Shi, Li, Wu, Teng, Jia, Xu, Xiang, Lin, Liu, Liu, Su, Sun, Berseth, Nie, Foster, Ward, Wu, Gu, Zhuge, Liang, Tang, Wang, You, Wang, Pei, Yang, Qi and Wu (2025). Mitigation Suggester The mitigation suggester analyzes the adversary’s operational report, along with the network topology and configuration state, to propose configuration changes that disrupt the observed attack capabilities. Sug- gested mitigations must be reusable, generalizable, and ex- pressed at the level of enterprise controls (e.g., segmentation policies, routing constraints, firewall rules). Mitigation Implementer The mitigation implementer trans- lates a selected mitigation into concrete configuration changes within the emulated environment, modifying device config- urations or logical topology elements. Critic The critic reviews implemented mitigations prior to offensive replay, identifying configuration errors, policy vio- lations, or deviations from the intended mitigation objective. When issues are detected, it provides feedback to the mitiga- tion implementer, who revises the configuration accordingly. This review–revision loop halts on critic approval or at the iteration cap 퐾; the empirical lift over the no-critic ablation (Section 5.3) justifies running it. 6 Emulated Network Environment (GNS3) Send command to device Caldera Operation Report Emulated Network Topology Network Device Configuration Task Mitigation Implementer Agent Calculation of mitigation effectiveness Critic Agent Disapproves Approves / Reached mitigation iteration limit Judge Agent Mitigation Suggester Agent Next mitigation Caldera Adversary Emulation Selects mitigation Implements mitigation Evaluation Mitigations Summary Reached mitigations limit / No mitigations left Summarizer Agent Execution Logs Cumulative Hardened Emulated Network Topology Figure 1: Multi-agent automatic mitigation framework showing the overall architecture and agent interactions. Judge Whereas the critic reviews the implemented config- uration prior to attack execution, the judge evaluates the resulting attack outcome. It replays the attack scenario in the post-mitigation environment using Caldera and com- pares the resulting attack step success rate (ASSR) to the pre-mitigation baseline; a lower rate indicates a successful mitigation. The judge also categorizes each mitigation into strategic classes (e.g., Host Hardening, Network-Based Traf- fic Filtering). Summarizer After all mitigation cycles are complete, the summarizer reads the judge’s per-mitigation evaluations (rather than the full multi-round conversation) and pro- duces a compact retrospective covering the attack scenario, attempted mitigations and outcomes, cross-mitigation ob- servations, connectivity validation results, and the device commands executed by the implementer. 4.4. Context Management Each agent’s context scope is tailored to its role. The mitigation suggester receives the history of its own prior suggestions, allowing it to avoid proposing strategies it has already tried; outcomes from earlier mitigations are deliber- ately excluded both to prevent the suggester from exploiting variations of already-successful strategies and to keep its context (and therefore token cost) bounded across long ses- sions. In contrast, the implementer and critic agents receive only the current mitigation context, with no knowledge of prior mitigations, since each mitigation is evaluated indepen- dently. 4.5. Mitigation Cycle Each mitigation is evaluated through the following phases: Initialization. The single-mitigation environment is re- verted to the baseline snapshot. Attack Execution. The attack scenario is executed, produc- ing the operational report. Mitigation Suggesting. The suggester proposes a candi- date mitigation. Mitigation Implementation. The implementer translates the candidate into device commands. Critic Review. The critic reviews the implementation; re- finement runs up to 퐾 iterations. Evaluation. The judge replays the attack on the mitigated environment to determine whether the mitigation suc- ceeded. Connectivity Validation. Two Caldera probes (a LAN ping and an internet HTTP check) verify reachability; both must pass for the mitigation to count toward MSR (Section 5.2). Cumulative Replay. Successful mitigations are replayed onto the persistent cumulative project (Section 4.7). 4.6. Single-Agent Baseline To isolate the contribution of the multi-agent architec- ture, we evaluate a single-agent baseline that consolidates the Suggester, Implementer, and Critic roles into a single LLM; the Judge and Summarizer roles are unchanged. The single agent shares the same model (GPT-5.4 mini), inputs (operational report, network topology, configuration state), and device-configuration access as the multi-agent system. Figure 2 shows the simplified workflow. 4.7. Cumulative Mitigation Evaluation Per-mitigation evaluation reveals individual effective- ness but not how defenses interact when stacked. The cumu- lative evaluation (Figure 3) captures these compound effects: a mitigation that passed the per-mitigation Evaluation with connectivity preserved is replayed onto a persistent cumula- tive project, an emulated network instance that accumulates approved defenses. After each cumulative replay, the mitiga- tion is accepted into the project only if it (i) does not regress the Cumulative Mitigation Effectiveness (Cumulative ME, Section 5.2) and (i) preserves basic connectivity; otherwise 7 Emulated Network Environment (GNS3) Send command Caldera Operation Report Emulated Network Topology Network Device Configuration Task Calculation of mitigation effectiveness Finish Judge Agent Single Agent Next mitigation Caldera Adversary Emulation Evaluation Reached mitigations limit / No mitigations left Summarizer Agent Mitigations Summary Execution Logs Cumulative Hardened Emulated Network Topology Figure 2: Single-agent baseline workflow for mitigation sugges- tion, implementation, and self-validation. the project is rolled back to the previously accepted state. Be- cause acceptance requires non-regression rather than strict improvement, non-interfering mitigations are retained even when they add no measurable ASSR reduction, yielding a defense-in-depth posture. Failure modes Each role has a characteristic failure shape. The suggester proposes generalizable controls that do not bind the adversary to the actual primitive, for example, dis- abling a specific binary while equivalent alternatives remain available. The implementer produces syntactically valid con- figurations whose effective scope is wrong: rules placed at the wrong hop, evaluated in the wrong order, or scoped to a path that does not intersect the attacker’s read or write targets. The critic catches errors visible in the configuration text (syntax errors, policy violations, deviations from stated intent) but cannot distinguish a syntactically correct rule that fails in placement from one that succeeds, since placement effects only surface during replay. Section 5.3 reports the empirical breakdown by mitigation category. 5. Evaluation 5.1. Experimental Setup All agent roles (mitigation suggester, mitigation imple- menter, and critic) use GPT-5.4 mini (via Azure OpenAI) with the default sampling temperature of 1.0. The Single- Agent baseline (Section 4.6) uses a single GPT-5.4 mini instance to play all roles in a single conversation. Caldera replay is deterministic given a fixed scenario and network state, so the five repetitions per condition (below) capture only the variance attributable to LLM sampling at this temperature. Network Topologies Three representative enterprise net- work topologies of increasing complexity are used (Fig- ures 10–12). The small enterprise topology consists of one FortiGate firewall and a single layer-2 switch connecting workstations and servers, representing a low-complexity baseline environment. The medium enterprise topology introduces a two-segment network behind a centralized fire- wall, with two distribution routers and two layer-2 switches enabling inter-segment routing and realistic VLAN-based segmentation. The large enterprise topology adopts a hi- erarchical three-tier architecture: five core routers inter- connect four floor segments, each terminated by a ded- icated floor router, a floor-level firewall, and a floor L3 switch, with twelve departmental access switches distributed across the floors. These three topologies approximate small- office, mid-market-branch, and multi-department enterprise- campus deployments, respectively. Attack Scenarios We focus on adversary objectives that pose concrete, recurring risks to enterprise environments. All four attack scenarios are implemented as Caldera ad- versaries with MITRE ATT&CK-aligned abilities. Together, these adversaries span distinct tactical objectives across the attack lifecycle: • Thief, a multi-stage attack involving local file discovery, staging, archiving, and exfiltration over the C2 channel, stressing access controls and egress constraints (adapted from the Thief adversary pack in the MITRE Caldera Stockpile plugin Stockpile; Harrell, Chan, Han, Voss, Danke, Ji, Brobin and Esprit (2023)). • Ransomware, which performs system reconnaissance and file discovery on the compromised host, then pivots via SSH to a dedicated victim container (SSH Victim Container) to encrypt sensitive files with AES, deploy a ransom note, and establish cron-based persistence, before exfiltrating the encryption key to C2. • DNS Exfiltration, which base64-encodes sensitive file contents and exfiltrates them as DNS subdomain queries to bypass HTTP egress controls, then removes artifacts. • Lateral Movement, which scans the LAN for SSH ser- vices, brute-forces credentials against a dedicated victim container (SSH Victim Container), pivots in via SSH, enu- merates the remote host and collects files, then exfiltrates them to C2. All attacks include a dedicated C2 communication link (see Section 3.1) and end with the ability to validate LAN and internet connectivity. The C2 server orchestrates attacks against a compromised Caldera Linux Agent running the Caldera Sandcat agent; the Lateral_Movement and Ran- somware adversaries additionally target a dedicated SSH Victim Container that serves as the SSH lateral movement destination. Per-step breakdowns of each adversary, includ- ing tactic and MITRE ATT&CK technique identifiers, are provided in the Adversary Attack Steps appendix (Table 6). Experimental Conditions To disentangle the contribu- tion of role specialization from that of iterative critique, we evaluate three runtime conditions that share the same model (GPT-5.4 mini), the same Caldera adversaries, and the same empirical validation pipeline: • Single-Agent. A single GPT-5.4 mini instance plays the mitigation suggester, implementer, and critic roles within one conversation (Section 4.6). There is no iterative cri- tique loop: the agent proposes, implements, and self- checks in a single linear trajectory. 8 Apply mitigationRevert snapshotReplay attackCreate snapshot Initial state M1 Egress Block Initial state M2 Restrict write access Initial state M3 Restrict upload tools Initial state Apply M1 Initial state M1 mitigation Cumulative EN Original EN Original EN Initial vulnerability testing Mitigation success Mitigation failure Mitigation success Apply M3 Mitigation success Mitigation failure M1 Single Mitigations EN Cumulative Mitigations EN ✗ Single EN Figure 3: Evaluation workflow showing the parallel per-mitigation evaluations (independent, rolled-back) and the cumulative evaluation (persistent cumulative project with sequential mitigation replay). A demo video walking through this diagram can be found in Appendix 8.3. • Multi-Agent without Critic. The multi-agent graph is executed with role-specialized suggester and implementer agents, and the implementation–critique iteration is con- strained to a single pass, so the critic’s feedback is never fed back into a revision. The implementer’s initial output is taken as-is, without acting on the critic’s advice. • Multi-Agent. The full framework, including the critic and up to five implementation–critique iterations per mitiga- tion. Each of the 12 scenarios (4 attacks × 3 topologies) is repeated five times per condition, with a limit of 10 miti- gations per run, up to 5 implementation–critique iterations per mitigation, and a budget of 20 device commands per implementation cycle, instantiating 퐾 = 5 and 퐵 = 20 in the formalization of Section 4.2. Across the three conditions we collected 1 782 mitigation attempts across 179 runs: 600 mitigations (60 runs) Single-Agent, 599 (60 runs) Multi- Agent without Critic, and 583 (59 runs) Multi-Agent. Sub- cap mitigation counts reflect the multi-agent graph’s early- exit edge when the suggester proposes no further distinct candidates; one Multi-Agent replicate was excluded after its summary file failed to write. In all three conditions, the suggester operates under an additional evaluation-time restriction: candidate mitigations may not alter the network topology (no adding, removing, or substituting devices, links, or services). The restriction is motivated by deployability and by frequent agent failures observed when attempting topology-altering changes during early system testing. 5.2. Evaluation Criteria Our evaluation is grounded in operational outcomes: every mitigation is validated by replaying the attack on the mitigated network and comparing the resulting ASSR to the pre-mitigation baseline. Mitigation Effectiveness (ME) Mitigation effectiveness quantifies the reduction in the Attack Step Success Rate (ASSR) achieved by a deployed mitigation, with the symbols of Section 4.2: ME(푚; 푡 ,) = ASSR( 푡 ,)−ASSR(휏( 푡 ,푚),) (1) where ASSR(,) is the ratio of successfully executed Caldera abilities to attempted abilities when adversary is replayed against state , with each ability scored as success, failure, or timeout. ME = ASSR( 푡 ,) signifies complete mitigation (post-mitigation ASSR reduced to zero) and ME = 0 indicates no effect. ME can also be negative, where the attempt actively worsens defense, and the attack succeeds at a higher rate than at baseline. Mitigation Success Rate (MSR) MSR is the probability that a proposed mitigation works: the fraction of attempts that both disrupt the attack (ME > 0) and preserve legitimate connectivity (Φ = 1, Section 3). MSR = 푁 ME>0∧Φ=1 푁 total × 100%(2) where 푁 total is the total number of mitigation attempts and the numerator counts attempts whose 푖-th cycle satisfies ME(푚 푖 ; 푡 푖 ,) > 0 and Φ(휏( 푡 푖 ,푚 푖 )) = 1. Cumulative Mitigation Effectiveness (ME cumulative ) While ME and MSR measure each mitigation independently (with rollback between attempts), the cumulative evaluation mea- sures how defenses compound when deployed together. Cumulative ME after 푘 approved mitigations is ME cumulative,푘 = ASSR( (0) ,)−ASSR( (푘) ,) (3) where (0) ,..., (푘) is the sequence of cumulative-project states defined by the acceptance rule of Section 4.2 (Cumu- lative Variant). Cumulative ME accumulates as a step function: each ac- cepted mitigation either reduces ASSR by a discrete amount 9 Single-AgentMulti-Agent without Critic Multi-Agent 0 20 40 60 80 100 Mitigation Success Rate (%) 11% 15% 39% 41% 47% 55% MSR (regression-checked) MSR without regression check Figure 4: MSR by runtime condition, pooled across attacks and topologies. The hatched bar shows the rate without the connectivity-regression check. or has no incremental effect on top of the defenses already deployed on the cumulative project. Aggregate cumulative- ME behavior across all runs is reported in Figures 5 (by condition) and 6 (by topology). 5.3. Results MSR Results Figure 4 reports the headline MSR (per- attack split in Appendix 8.6). Across all attacks, MSR rises from 10.7% (Single-Agent) to 38.6% (Multi-Agent without Critic) to 46.7% (Multi-Agent). Every pairwise difference is statistically significant under Holm-Bonferroni correc- tion (full pairwise statistics in Appendix 8.5). Decomposed against the Single-Agent baseline, role specialization alone (Multi-Agent without Critic) accounts for roughly three- quarters of the total gain over Single-Agent, and the iterative critique loop contributes the remaining one-quarter. The same ranking holds at the run level: across the run’s up-to- ten sequential attempts, Multi-Agent produces at least one working mitigation in 98% of runs and Multi-Agent without Critic in 95%, against 78% for Single-Agent. Connectivity preservation moves non-monotonically: Multi-Agent with- out Critic improves it slightly over Single-Agent (94.3% vs. 92.0%), while the critic trades it down to 88.0% in exchange for higher MSR. The critic’s more aggressive controls occa- sionally break a probe while still blocking the attack. Multi-agent decomposition lifts MSR 4.4× over a Single-Agent baseline. Single-Agent 10.7%, Multi- Agent without Critic 38.6%, Multi-Agent 46.7%. Role specialization accounts for roughly three-quarters of the lift; the iterative critique loop accounts for the remain- ing quarter. Cost The lift carries a cost (Table 5, Per-Mitigation Cost). Per mitigation, Multi-Agent averages $0.47 and 124 s of wall-clock time, compared to $0.09 and 60 s for Single- Agent (≈5× cost and 2× wall-clock time). Multi-Agent with- out Critic falls in between at $0.22 and 96 s. Total experiment cost across all 1 782 mitigations was $464. 246810 Number of Sequential Defenses 0 20 40 60 80 100 Mean cumulative ME (%) Multi-Agent (n=60) Multi-Agent without Critic (n=60) Single-Agent (n=60) Figure 5: Cumulative ME by condition, pooled across attacks. 246810 Number of Sequential Defenses 0 20 40 60 80 100 Mean cumulative ME (%) Small (n=60) Medium (n=60) Large (n=60) Figure 6: Cumulative ME by topology, all conditions pooled. Cumulative Mitigation Effectiveness Figure 5 character- izes how defenses compound when stacked. Multi-Agent plateaus at ≈ 52% cumulative ME pooled across attacks, Multi-Agent without Critic at ≈ 47%, and Single-Agent at ≈28%. The topology view (Figure 6) shows that, once enough mitigations are stacked, the final cumulative ME is compa- rable across small, medium, and large enterprise networks. The topology slows the rate of ME accumulation more than it bounds the achievable ceiling. The cumulative track follows a greedy policy: each accepted mitigation is permanently retained on the cumulative project, and earlier choices are not revisited as later mitigations are added. MSR by Topology Figure 7 reports MSR by topology for the Multi-Agent condition. We do not detect a topology effect: small enterprise 48.9% (푛 = 188), medium 48.5% (푛 = 200), large 42.6% (푛 = 195); Welch’s ANOVA is non-significant (푝 = 0.37), with all pairwise comparisons non-significant under Holm correction (smallest 푝 = 0.21). The cumulative-ME view (Figure 6) is consistent: all three topologies plateau by the fourth sequential defense, at≈43% (small), ≈45% (medium), and ≈39% (large). 10 Small EnterpriseMedium EnterpriseLarge Enterprise 0 20 40 60 80 100 Mitigation Success Rate (%) 48.9% (n=188) 48.5% (n=200) 42.6% (n=195) Figure 7: MSR by enterprise topology (small/medium/large) for the Multi-Agent condition. Error bars indicate the standard error of the mean. 020406080100 Mitigation Success Rate (%) Network-Based Traffic Filtering Network-Based Traffic Inspection Network Isolation Host-Based Traffic Filtering Host-Based Intrusion Prevention Host Hardening Application Control 5.8% (n=139) 9.1% (n=11) 15.8% (n=19) 25.0% (n=20) 53.8% (n=13) 63.3% (n=341) 82.1% (n=39) Host-based Network-based Figure 8: MSR by mitigation category for the Multi-Agent condition, pooled across topologies. Category labels are as- signed for each mitigation and normalized to a shared reporting vocabulary (Table 3). Error bars indicate the standard error of the proportion. No detectable topology effect. Differences between small (48.9%), medium (48.5%), and large (42.6%) topologies are not statistically significant. Mitigation Categories To characterize which classes of defense the system produces, we group mitigations into high-level categories based on their defensive approach. These categories can be broadly grouped into network- centric controls (enforced at the network perimeter or within the infrastructure), host-centric controls (applied directly to endpoints), and reactive/operational measures. Figure 8 presents the per-category MSR for the Multi-Agent condi- tion, pooled across topologies; Table 3 gives definitions, examples, and counts. Two high-level patterns are visible. First, host-centric controls dominate in both volume and effectiveness: Host Hardening (the largest single bucket at 푛 = 341) achieves 63.3% MSR, Application Control 82.1% (푛 = 39), and Host-Based Intrusion Prevention 53.8% (푛 = 13, too few attempts to rank reliably against the larger buckets). Second, network-based filtering dominates in volume but not in effectiveness: Network-Based Traffic Filtering is the second- largest bucket (푛 = 139) but only 5.8% of its attempts reduce the attack. Inspection of the failed network-filtering mitigations shows a recurring failure mode: the control is syntactically correct and aligned with best-practice designs (VLAN segmentation, inter-VLAN blocks, router ACLs, egress firewall rules) but is not enforced at the correct point in the network hierarchy, is evaluated in the wrong rule order (e.g., a deny rule after an allow rule on FortiGate), or fails to account for hidden intermediaries on the routed path. The failed host-centric mitigations exhibit an analogous targeting error: the control is correctly applied but does not bind to the attacker’s actual primitive. The dominant patterns are disabling a specific binary (e.g., find, tar, python3) while the attacker’s tooling has equivalent alternates; tightening filesystem permissions on a path that does not intersect the attack’s actual read or write targets; and hardening SSH authentication when the validated attack path is already authenticated. Host-centric controls dominate; network filtering looks right on paper but rarely fires. Host Hardening, the largest bucket at 푛 = 341, reaches 63.3% MSR, while Network-Based Traffic Filtering, the second- largest by volume at 푛 = 139, reduces the attack only 5.8% of the time. Network filtering fails not from wrong intent but from rules placed at the wrong hop or evaluated in the wrong order. 6. Discussion 6.1. Attribution and the Optimization-Evaluation Gap Caldera replays each adversary scenario unchanged across pre- and post-mitigation runs (Section 3). Holding the attack fixed makes per-mitigation ASSR change attributable to the deployed mitigation rather than to concurrent attacker adaptation. The same property creates an optimization- evaluation gap: the suggester reads the operational report from the same replay that the judge later scores, so candi- dates can, in principle, be tuned to the evaluation steps. Several design properties bound this gap. The success criterion pairs ME > 0 with a connectivity check (Sec- tion 5.2), so trivial wins like severing all egress fail outright. Mitigations are credited against the attack’s production- path actions, not the operator’s C2 channel (Section 3.1), which forces engagement with the data flow rather than the attacker’s infrastructure. The suggester is restricted to reusable enterprise-level controls, with indicator-based blocking and monitoring-only responses excluded by prompt (Section 4.2). The framework’s outputs are reviewed by engineers before deployment, not auto-applied, so final acceptance rests on human review rather than on the metric. 6.2. Failure Analysis: Where Mitigations Miss Host Hardening reaches 63.3% MSR (푛 = 341) and Application Control 82.1% (푛 = 39), while Network-Based Traffic Filtering reaches only 5.8% (푛 = 139) (Section 5.3). Host-local controls succeed because the implementer has the resident context (file, process, user, binary) it acts on, so misses concentrate in cases where the chosen primitive does not bind the attacker’s: disabling find when tar suffices, hardening SSH when the validated path is already authenti- cated. Network controls fail because rules are reasoned about as syntactic objects rather than bound to a routed path: rules 11 Table 3 Mitigation categories, MITRE D3FEND mappings, and per-category MSR for the Multi-Agent condition, pooled across topologies. CategoryDefinitionExamplesD3FENDN MSR (%) Application Control Per-account execute restrictions on specific applications Per-user execute-denylist of attacker tooling (sshpass, nmap, dig, base64 + Python exfil staging); allowlists permitting only business-required network utilities D3-EAL D3-EDL 3982.1 Host Hardening Host-level reduction of attacker capability via permission and credential restriction chmod -x on attacker-used utilities (sshpass, curl, openssl, dig); SSH key-only authentication with password auth disabled; per-user write/permission tightening on staging paths D3-LFP D3-SPP D3-UAP 34163.3 Host-Based Intrusion Prevention Active host-runtime defenses that detect and respond to in-progress attack actions fail2ban SSH event-thresholding and account lockout on authentication-failure bursts; cron-based watchdog kills of reconnaissance binaries D3-ANET D3-AL D3-PT 1353.8 Host-Based Traffic Filtering Endpoint-local egress firewall enforcement Host iptables/nftables egress rules denying outbound HTTPS to C2 destinations while preserving LAN ICMP and HTTP validation D3-OTF h 2025.0 Network Isolation Network-segmentation controls limiting lateral reach FortiGate / Open vSwitch inter-VLAN ACLs; east-west SSH restrictions and floor-router segmentation between department LANs D3-BDI D3-NI 1915.8 Network-Based Traffic Inspection Application-layer inspection at network devices FortiGate DNS-security inspection blocking long / high-entropy subdomains and query-burst anomalies D3-DNSTA D3-DNSDL D3-PMAD 119.1 Network-Based Traffic Filtering Perimeter firewall egress and proxy controls FortiGate outbound HTTPS denies to C2 endpoints; OpenWrt UCI rules blocking DNS-tunnel ports (53, 8853); destination/IP allowlists D3-OTF D3-NTF D3-FRDDL 1395.8 h D3-OTF enforced on the endpoint rather than at a network device. at the wrong hop, allow before deny, or scoped past hidden intermediaries. The cumulative-ME plateau (52% pooled, 43%, 45%, and 39% across topology sizes; ANOVA 푝 = 0.37) is consistent with the same bound: stacked greedy defense saturates against the attack steps that no single host- or network-side control covers under our connectivity contract. 6.3. Linux-Only Endpoints GNS3 supports Windows endpoint images, but Windows guests are substantially more compute-intensive than Linux containers in emulation, so we restrict the evaluation to Linux endpoints. Whether the reported MSR transfers to Windows-hosted environments is open, and the exposure is uneven across mitigation categories. Host Hardening and Application Control depend on platform-specific binaries, policy primitives, and default services that differ substan- tially between Linux and Windows; Network-Based Traffic Filtering operates on the same TCP/IP path and inherits the same placement-bound constraints (Section 6.2). 6.4. Ability-Level Granularity MSR and ME are reported at the granularity of Caldera abilities. When a single ability bundles multiple discrete actions, intra-ability disruption is not resolved: a partially broken ability still counts as a successful step. This biases reported disruption downward rather than upward, since a mitigation that thwarts most but not all of a bundled ability fails the ME > 0 check. 6.5. Open Questions First, the cumulative project produces a persistent hard- ened network state at the end of each attack’s track. Reusing it as the starting topology for a different adversary would let the framework measure cross-attack transfer directly: whether mitigations stacked against one attack continue to disrupt a second, distinct adversary. Iterating this handoff across multiple adversaries extends the cumulative track into a sequential multi-attack evaluation, in which each attack inherits the hardened state left by the previous one. Second, this cumulative track currently uses a greedy stacking policy in which each accepted mitigation is re- tained on the cumulative project and never revisited. Non- greedy variants (lookahead over candidate mitigations, beam search across cumulative-project trajectories, or a revisit- and-replace operator that swaps earlier choices when later mitigations subsume them) would allow the framework to probe whether the observed cumulative-ME plateau is local to the greedy policy or near a global ceiling for stacked defense. Third, grounding the suggester in MITRE D3FEND or organization-specific incident-response playbooks would constrain proposals to a vetted defensive vocabulary, align autonomous mitigations with approved response actions, and produce mitigations whose primitives are immediately recognizable to security operations staff. Fourth, automated generation of emulated topologies from production network configurations would reduce the manual configuration burden associated with high-fidelity replicas. While LLMs can assist with such setup Ifland et al. 12 (2025); Wang et al. (2024a), fully automated production- to-emulation generation remains an open engineering chal- lenge, and is a precondition for scaling the framework to networks beyond hand-curated topologies. Fifth, all agents are backed by GPT-5.4 mini (Sec- tion 8.1.3); whether the reported MSR transfers to other model families is open. Sixth, the single-agent baseline ablates the multi-agent architecture but does not calibrate the framework against expert manual work. A controlled study with security en- gineers producing mitigations under comparable constraints would situate the reported MSR relative to a human refer- ence point. 7. Conclusion We presented COHORT, the first end-to-end framework to autonomously generate and validate deployable mitiga- tions against an observed adversary in emulated network. The two bottlenecks of expert dependency and production- exposure risk are addressed by a multi-agent LLM gen- eration pipeline and a high-fidelity emulator running real vendor firmware; each candidate is validated by offensive replay of the original Caldera adversary on the mitigated network. Across three topologies and four attack scenarios, COHORT outperforms a single-agent baseline (Section 5.3). The same autonomy opens new attack surfaces: a rou- tine hardening run surfaced a long-acknowledged container- escape weakness, making staging isolation and human re- view of accepted mitigations preconditions for safe opera- tion. Results are scoped to scripted Caldera adversaries on a Linux-only lab with FortiGate, Cisco IOS, and Open vSwitch under GPT-5.4 mini, and connectivity preservation is verified by two probes; cross-attack transfer, adaptive adversaries, other model families, calibration against expert- derived mitigations, and finer operational regression remain open (Section 6.5). LLM usage Grammar was edited using Grammarly and Claude. References An, K., Yang, F., Lu, J., Li, L., Ren, Z., Huang, H., Wang, L., Zhao, P., Kang, Y., Ding, H., et al., 2024. Nissist: An incident mitigation copilot based on troubleshooting guides. arXiv preprint arXiv:2402.17531 . Applebaum, A., Miller, D., Strom, B., Korber, C., Wolf, R., 2016. Intelli- gent, automated red team emulation, in: Proceedings of the 32nd Annual Conference on Computer Security Applications, p. 363–373. AttackIQ, . AttackIQ Security Optimization Platform. https://w.attack iq.com. Accessed: 2026-04. AutoGen, . AutoGen. https://github.com/microsoft/autogen. Autonomous Resilient Cyber Defence Programme, 2024. PrimAITE: Primary-AI training environment. https://github.com/Autonomous-R esilient-Cyber-Defence/PrimAITE. Bringhenti, D., Marchetto, G., Sisto, R., Valenza, F., 2023. Automation for network security configuration: State of the art and research trends. ACM Computing Surveys 56, 1–37. Castro, S.R., Campbell, R., Lau, N., Villalobos, O., Duan, J., Cardenas, A.A., 2025. Large language models are autonomous cyber defenders, in: 2025 IEEE Conference on Artificial Intelligence (CAI), IEEE. p. 1125–1132. Chang, C., Cao, M., Teoh, K., Ear, E., Xu, S., 2025. Characterizing caldera’s cyber attack emulation capabilities. Military Cyber Affairs 8, 5. Cyberbit, . Cyberbit Cyber Range. https://w.cyberbit.com. Accessed: 2026-04. Cymulate, . Cymulate Exposure Validation Platform. https://cymulate.com. Accessed: 2026-04. Du, Y., Li, S., Torralba, A., Tenenbaum, J.B., Mordatch, I., 2024. Improving factuality and reasoning in language models through multiagent debate, in: Forty-first international conference on machine learning. Eckhart, M., Ekelhart, A., 2018. Towards security-aware virtual environ- ments for digital twins, in: Proceedings of the 4th ACM Workshop on Cyber-Physical System Security, p. 61–72. Emerson, H., Bates, L., Hicks, C., Mavroudis, V., 2024. Cyborg++: An enhanced gym for the development of autonomous cyber agents. arXiv preprint arXiv:2410.16324 . Hamadanian, P., Arzani, B., Fouladi, S., Kakarla, S.K.R., Fonseca, R., Billor, D., Cheema, A., Nkposong, E., Chandra, R., 2023. A holistic view of ai-driven network incident management, in: Proceedings of the 22nd ACM Workshop on Hot Topics in Networks, p. 180–188. Hammar, K., Dhir, N., Stadler, R., 2024. Optimal defender strategies for cage-2 using causal modeling and tree search. arXiv preprint arXiv:2407.11070 . Hammar, K., Stadler, R., 2022. Intrusion prevention through optimal stopping. IEEE Transactions on Network and Service Management 19, 2333–2348. Harrell, B., Chan, M., Han, H., Voss, K., Danke, G., Ji, L., Brobin, O., Esprit, K., 2023. Testing Data Exfiltration with MITRE Caldera. https: //medium.com/@mitrecaldera/testing-data-exfiltration-with-mitre-c aldera-a7dfe3fc9bd. Hong, J., Tu, N.V., Hong, J.W.K., 2025. A comprehensive survey on llm- based network management and operations. International Journal of Network Management 35, e70029. Hong, S., Zhuge, M., Chen, J., Zheng, X., Cheng, Y., Wang, J., Zhang, C., Wang, Z., Yau, S.K.S., Lin, Z., et al., 2023. Metagpt: Meta programming for a multi-agent collaborative framework, in: The twelfth international conference on learning representations. IBM Security, 2025. Cost of a data breach report. URL: https://w.ibm. com/reports/data-breach. accessed: 2025-01-14. Ifland, B., Krief, R., Zilberman, A., Duani, E., Ohana, M., Murillo, A., Manor, O., Lavi, O., Hikichi, K., Shabtai, A., Elovici, Y., Puzis, R., 2025. GeNet: A Multimodal LLM-Based Co-Pilot for Network Topology and Configuration , in: 2025 IEEE 45th International Conference on Dis- tributed Computing Systems Workshops (ICDCSW), IEEE Computer Society, Los Alamitos, CA, USA. p. 117–122. URL: https://do i.ieeecomputersociety.org/10.1109/ICDCSW63273.2025.00026, doi:10.1109/ICDCSW63273.2025.00026. Islam, C., Babar, M.A., Nepal, S., 2019. A multi-vocal review of security orchestration. ACM Computing Surveys 52, 1–45. Janisch, J., Pevn ` y, T., Lis ` y, V., 2023. Nasimemu: Network attack simulator & emulator for training agents generalizing to novel scenarios, in: European Symposium on Research in Computer Security, Springer. p. 589–608. Kaheh, M., Kholgh, D.K., Kostakos, P., 2023. Cyber sentinel: Exploring conversational agents in streamlining security tasks with gpt-4. arXiv preprint arXiv:2309.16422 . Kiely, M., Ahiskali, M., Borde, E., Bowman, B., Bowman, D., Van Bruggen, D., Cowan, K., Dasgupta, P., Devendorf, E., Edwards, B., et al., 2025. Exploring the efficacy of multi-agent reinforcement learning for autonomous cyber defence: A cage challenge 4 perspective, in: Proceedings of the AAAI Conference on Artificial Intelligence, p. 28907–28913. Kim, M., Wang, J., Moore, K., Goel, D., Wang, D., Mohsin, A., Ibrahim, A., Doss, R., Camtepe, S., Janicke, H., 2025. Cyberally: Leveraging llms and knowledge graphs to empower cyber defenders, in: Companion 13 Proceedings of the ACM on Web Conference 2025, p. 2851–2854. Kouremetis, M., Lawrence, D., Alford, R., Cheuvront, Z., Davila, D., Geyer, B., Haigh, T., Michalak, E., Murphy, R., Russo, G., 2024. Mirage: cyber deception against autonomous cyber attacks in emulation and simulation. Annals of Telecommunications 79, 803–817. Li, G., Hammoud, H., Itani, H., Khizbullin, D., Ghanem, B., 2023. Camel: Communicative agents for" mind" exploration of large language model society. Advances in Neural Information Processing Systems 36, 51991– 52008. Li, L., Fayad, R., Taylor, A., 2021. Cygil: A cyber gym for training autonomous agents over emulated network systems. arXiv preprint arXiv:2109.03331 . Lira, O.G., Caicedo, O.M., da Fonseca, N.L., 2024. Large language models for zero touch network configuration management. IEEE Communica- tions Magazine 63, 146–153. Liu, B., Li, X., Zhang, J., Wang, J., He, T., Hong, S., Liu, H., Zhang, S., Song, K., Zhu, K., Cheng, Y., Wang, S., Wang, X., Luo, Y., Jin, H., Zhang, P., Liu, O., Chen, J., Zhang, H., Yu, Z., Shi, H., Li, B., Wu, D., Teng, F., Jia, X., Xu, J., Xiang, J., Lin, Y., Liu, T., Liu, T., Su, Y., Sun, H., Berseth, G., Nie, J., Foster, I., Ward, L., Wu, Q., Gu, Y., Zhuge, M., Liang, X., Tang, X., Wang, H., You, J., Wang, C., Pei, J., Yang, Q., Qi, X., Wu, C., 2025. Advances and challenges in foundation agents: From brain-inspired intelligence to evolutionary, collaborative, and safe systems. arXiv:2504.01990. see Section 15.2.2. Loevenich, J., Adler, E., Huerten, T., Lopes, R.R.F., 2025. Design and evaluation of an autonomous cyber defence agent using drl and an augmented llm. Computer Networks 262, 111162. Luo, H., Sun, G., Liu, Y., Niyato, D., Yu, H., Atiquzzaman, M., Dustdar, S., 2026. A trustworthy agentic multi-llm network: Challenges, solutions, and a use case. IEEE Wireless Communications . Lyu, M., Gharakheili, H.H., Sivaraman, V., 2024. A survey on enterprise network security: Asset behavioral monitoring and distributed attack detection. IEEE Access . Marchioro, T., Saroui, R., Olivereau, A., 2025. Network intrusion response systems: towards standardized evaluation of intrusion response, in: ESORICS 2025: 30th European Symposium on Research in computer security. Molina-Markham, A., Miniter, C., Powell, B., Ridley, A., 2021. Network environment design for autonomous cyberdefense. arXiv preprint arXiv:2103.07583 . Neumann, J.C., 2015. The book of GNS3: build virtual network labs using Cisco, Juniper, and more. No Starch Press. Nguyen, T.T., Reddi, V.J., 2021. Deep reinforcement learning for cyber security. IEEE Transactions on Neural Networks and Learning Systems 34, 3779–3795. Oesch, S., Chaulagain, A., Weber, B., Dixson, M., Sadovnik, A., Roberson, B., Watson, C., Austria, P., 2024. Towards a high fidelity training environment for autonomous cyber defense agents, in: Proceedings of the 17th Cyber Security Experimentation and Test Workshop, p. 91– 99. Pentera, . Pentera Automated Security Validation. https://pentera.io. Accessed: 2026-04. Picus, . Picus Security Validation Platform. https://w.picussecurity.com. Accessed: 2026-04. Rigaki, M., Lukáš, O., Catania, C.A., Garcia, S., 2023. Out of the cage: How stochastic parrots win in cyber security environments. arXiv preprint arXiv:2308.12086 . SafeBreach, . SafeBreach Breach and Attack Simulation. https://w.safe breach.com. Accessed: 2026-04. Saroui, R., Marchioro, T., Olivereau, A., 2025. Agentnirs: An llm-driven agent for network intrusion response, in: ACSAC 2025-Annual Com- puter Security Applications Conference Workshops. SCYTHE, . SCYTHE Adversary Emulation Platform. https://scythe.io. Accessed: 2026-04. SimSpace, . SimSpace Cyber Force Platform. https://w.simspace.com. Accessed: 2026-04. Sood, A.K., Zeadally, S., Hong, E., 2025. The paradigm of hallucinations in ai-driven cybersecurity systems: Understanding taxonomy, classification outcomes, and mitigations. Computers and Electrical Engineering 124, 110307. Speicher, P., Steinmetz, M., Hoffmann, J., Backes, M., Künnemann, R., 2019. Towards automated network mitigation analysis, in: Proceedings of the 34th ACM/SIGAPP symposium on applied computing, p. 1971– 1978. Stakhanova, N., Basu, S., Wong, J., 2007. A taxonomy of intrusion response systems. International Journal of Information and Computer Security 1, 169–184. Standen, M., Lucas, M., Bowman, D., Richer, T.J., Kim, J., Marriott, D., 2021. Cyborg: A gym for the development of autonomous cyber agents. arXiv preprint arXiv:2108.09118 . Stockpile, . MITRE Caldera plugin: Stockpile. https://github.com/mitre /stockpile. Team., M.D.R., 2021. Cyberbattlesim. https://github.com/microso ft/cyberbattlesim. Created by Christian Seifert, Michael Betser, William Blum, James Bono, Kate Farris, Emily Goren, Justin Grana, Kristian Holsheimer, Brandon Marken, Joshua Neil, Nicole Nichols, Jugal Parikh, Haoran Wei. Wang, C., et al., 2024a. Netconfeval: Can llms facilitate network configura- tion?, in: Proceedings of the 20th International Conference on emerging Networking EXperiments and Technologies (CoNEXT), ACM. p. 467– 481. Wang, J., He, B., Zhao, J., Xuan, Y., Sun, H., Qi, Q., Liang, J., Zhuang, Z., Liao, J., 2026. Llm-powered intent-driven configuration generation for multi-vendor networks. IEEE Transactions on Network and Service Management . Wang, M., Dechene, R., 2024. Multi-agent actor-critics in autonomous cyber defense. arXiv preprint arXiv:2410.09134 . Wang, T., Xie, X., Zhang, L., Wang, C., Zhang, L., Cui, Y., 2024b. Shieldgpt: An llm-based framework for ddos mitigation, in: Proceedings of the 8th Asia-Pacific Workshop on Networking, p. 108–114. Wu, Q., Bansal, G., Zhang, J., Wu, Y., Li, B., Zhu, E., Jiang, L., Zhang, X., Zhang, S., Liu, J., et al., 2024. Autogen: Enabling next-gen llm applications via multi-agent conversations, in: First conference on language modeling. XMCyber, . XM Cyber Continuous Exposure Management. https: //w.xmcyber.com. Accessed: 2026-04. Yamin, M.M., Katt, B., Gkioulos, V., 2020. Cyber ranges and security testbeds: Scenarios, functions, tools and architecture. Computers & Security 88, 101636. Yan, Y., Zhang, Y., Huang, K., 2024. Depending on yourself when you should: Mentoring llm with rl agents to become the master in cybersecurity games. arXiv preprint arXiv:2403.17674 . Zhang, J., Bu, H., Wen, H., Liu, Y., Fei, H., Xi, R., Li, L., Yang, Y., Zhu, H., Meng, D., 2025. When llms meet cybersecurity: A systematic literature review. Cybersecurity 8, 55. 8. Appendix 8.1. Experimental Setup 8.1.1. Emulation Platform GNS3 version 2.2.57 for our network emulation environ- ment. 8.1.2. Multi-Agent System AutoGen version 0.7.5 Wu, Bansal, Zhang, Wu, Li, Zhu, Jiang, Zhang, Zhang, Liu et al. (2024); AutoGen. 8.1.3. Language Model All agents were backed by OpenAI GPT-5.4 mini, snap- shot version 2026-03-17, served via an Azure OpenAI Global Standard deployment. 14 Table 4 Per-attack and overall pairwise condition comparisons on the strict joint MSR (attack disrupted and connectivity preserved). Each test is a Welch’s t-test on run-level mean MSR (one observation per run; 푛 = 59 / 푛 = 60 / 푛 = 60 for Multi-Agent / Multi-Agent without Critic / Single-Agent). Each cell shows the MSR difference (Δp, improvement) and 푝-value, with Holm-Bonferroni correction across the three pairs (훼 = 0.05). ComparisonDNS Exfiltration Lateral Movement Ransomware Thief Overall Single-Agent→ Multi-Agent without Critic +12.0 (푝=0.038) +44.0 (푝<10 −7 ) +24.0 (푝<10 −4 ) +31.6 (푝<10 −4 ) +27.9 (푝<10 −14 ) Multi-Agent without Critic→ Multi-Agent +1.3 (푝=0.83) +4.7 (푝=0.42) +14.4 (푝=0.022) +11.4 (푝=0.13) +7.9 (푝=0.036) Single-Agent→ Multi-Agent +13.3 (푝=0.0025) +48.7 (푝<10 −9 ) +38.4 (푝<10 −6 ) +43.0 (푝<10 −6 ) +35.8 (푝<10 −19 ) 8.1.4. Azure Server Specifications Experiments ran on a Microsoft Azure Standard D8s v3 instance (8 vCPUs, 32 GiB RAM) running Ubuntu 24.04.2 LTS. 8.1.5. Caldera MITRE Caldera version 5.3.0. 8.1.6. Caldera Linux Agent A custom Docker container on gns3/ubuntu:noble run- ning the Caldera Sandcat client. The agent reaches the Caldera C2 over HTTPS (port 8443) via a dedicated out- of-band management interface, so C2 connectivity survives mitigations applied to the topology interface. 8.1.7. SSH Victim Container A second container used as the lateral-movement target by the Lateral_Movement and Ransomware adversaries. It runs OpenSSH on port 22 with password authentication for a non-root user with a deliberately weak password, and pre- populates the user’s home directory with sensitive files for collection and encryption. The entry point emits periodic ICMP/HTTP background traffic to simulate an active host. No Caldera agent is installed. 8.1.8. Firewall FortiGate version 7.0.14. 8.1.9. Switch Open vSwitch 3.3.7. 8.1.10. Router Cisco 3640 running IOS 12.4(25d). 8.2. Adversary Attack Steps Table 6 enumerates, in execution order, the Caldera abilities that make up each adversary, together with their MITRE ATT&CK tactic and technique identifiers. The final two steps in every adversary (LAN and internet connectivity checks) are shared validation abilities used to confirm that baseline network reachability is preserved after mitigation. The full ability definitions and downloadable Caldera adver- sary YAMLs are released alongside this paper as described in Section 8.3. Table 5 Per-mitigation cost by runtime condition (mean ± std). Tokens are summed across all agent-role LLM calls within a miti- gation (which may include multiple implementation–critique iterations); duration is the wall-clock window from the first to the last context file in the mitigation directory; cost applies GPT-5.4-mini Azure pricing of $0.75 per 1M input tokens and $4.50 per 1M output tokens. ConditionNInput (k)Output (k)Duration (s)Cost/mit ($)Total ($) Multi-Agent591 609 ± 5463.8 ± 1.4124 ± 500.47 ± 0.41279.72 Multi-Agent without Critic 600 273 ± 1952.6 ± 0.496 ± 450.22 ± 0.15130.04 Single-Agent604109 ± 771.8 ± 0.360 ± 180.09 ± 0.0654.43 DNS ExfilLateral MovementRansomwareThief 0 20 40 60 80 100 Mitigation Success Rate (%) 27% 32% 60% 66% 51% 59% 49% 64% 25% 27% 55% 56% 37% 41% 37% 42% 13% 15% 11% 15% 13% 18% 5% 11% Multi-Agent Multi-Agent without Critic Single-Agent Failed regression check Figure 9: MSR by runtime condition and attack scenario. Hatched bar: rate without the connectivity-regression check. 8.3. Data and Code Availability In the spirit of open science, we release the following artifacts at https://github.com/user32133/cohort and https://cohort-experiments-app.streamlit.app/: • Agent prompts for all roles (Suggester, Implementer, Critic, Judge, Summarizer, and the single-agent baseline). • GNS3 topology files and device configurations for the small, medium, and large environments. • Docker images for the containers used in the experiments, including images prepared but not used in the final evalu- ation. • Caldera adversaries (Thief, Ransomware, DNS Exfiltra- tion, Lateral Movement) and the abilities that compose them, with ATT&CK mappings. • Conversation logs and post-mitigation operational reports from every run analyzed in this work. A demo video walking through the framework is avail- able at https://youtu.be/8Lj6rXAOwM8. The multi-agent runtime is AutoGen (Section 8.1.2). The agent code and the GNS3 automation code cannot be placed in the public domain; however, organizations involved in this research are open to reviewing disclosure requests submitted through the corresponding author. 8.4. Network Topologies The medium enterprise network topology was inspired by Simple Network Layout by Gilbert Nims (GNS3 Market- place). 15 Table 6 Caldera adversary steps for the four post-compromise scenarios. The final two rows in every adversary (LAN/Internet connectivity checks) are shared validation abilities. The Thief adversary is adapted from the Thief pack in the MITRE Caldera Stockpile plugin Stockpile; Harrell et al. (2023); the two final connectivity-validation steps are our addition. (a) Lateral_Movement adversary steps. # StepTacticTech. ID Description 1Scan target for open SSH port DiscoveryT1046nmap confirms port 22 is open on the lateral target 2SSH password brute force Credential Access T1110.003 Spray a short password list against SSH until a credential succeeds 3SSH lateral movement Lateral Movement T1021.004 Log into the remote host with the discovered credential 4Remote host enumeration DiscoveryT1082Enumerate users, processes, and network config on the remote host 5Collect files from remote host CollectionT1005Locate sensitive files on the remote host and copy them back via SCP 6Exfil remotely collected file to C2 Exfiltration T1041Upload the collected file to the Caldera C2 server 7LAN Connectivity Check (Ping) DiscoveryT1018Validate that LAN reachability is preserved 8Internet Connectivity Check (HTTP) DiscoveryT1016Validate that outbound internet reachability is preserved (b) Thief adversary steps. # StepTacticTech. ID Description 1Create staging directory CollectionT1074.001 Create a local directory to hold files for exfiltration 2Find filesCollectionT1005Locate files deemed sensitive on the compromised host 3Stage sensitive files CollectionT1074.001 Copy the discovered files into the staging directory 4Compress staged directory Exfiltration T1560.001 Archive the staging directory with a standard utility 5Exfil staged directory Exfiltration T1041Upload the archive to the C2 server over the agent channel 6LAN Connectivity Check (Ping) DiscoveryT1018Validate that LAN reachability is preserved 7Internet Connectivity Check (HTTP) DiscoveryT1016Validate that outbound internet reachability is preserved (c) DNS_Exfil adversary steps. # StepTacticTech. ID Description 1Find filesCollectionT1005Locate files deemed sensitive on the compromised host 2Encode sensitive data for DNS exfiltration Defense Evasion T1027Base64-encode the contents of a discovered file into DNS-safe chunks 3Exfiltrate data via DNS queries Exfiltration T1048.003 Send encoded chunks as subdomain lookups to bypass HTTP egress controls 4Remove DNS exfil artifacts Defense Evasion T1070.004 Delete local staging files and clear shell history to cover tracks 5LAN Connectivity Check (Ping) DiscoveryT1018Validate that LAN reachability is preserved 6Internet Connectivity Check (HTTP) DiscoveryT1016Validate that outbound internet reachability is preserved (d) Ransomware adversary steps. # StepTacticTech. ID Description 1System ReconDiscoveryT1082Gather basic system information before ransomware deployment 2Find filesCollectionT1005Locate files deemed sensitive for encryption 3Encrypt sensitive files ImpactT1486Generate a random AES key and encrypt the discovered files 4Write ransom note ImpactT1491.001 Drop a ransom note on the victim host 5Plant cron persistence Persistence T1053.003 Install a cron job that beacons back to the C2 server 6Exfil encryption key to C2 Exfiltration T1041Upload the AES encryption key to the C2 server over HTTP 7LAN Connectivity Check (Ping) DiscoveryT1018Validate that LAN reachability is preserved 8Internet Connectivity Check (HTTP) DiscoveryT1016Validate that outbound internet reachability is preserved 16 eth4 e0 eth5 e0 eth6 e0 CALDERA-MGMT Internet OpenvSwitch-1 PC1 PC2 PC3 PC4 UbuntuComputer UbuntuSSHVictim fortigate-firewall IP=192.168.1.3 DEFAULT_GATEWAY=192.168.1.1 WAN: 7.0.14.1 LAN: 192.168.1.1 Attacker IP: 203.0.113.1 Figure 10: Small enterprise network topology. The large enterprise network topology was inspired by Enterprise Network Lab: Bank Project by Kiki Oyewole (GNS3 Marketplace). 8.5. Pairwise Statistical Comparisons Table 4 reports per-attack and overall pairwise condition comparisons referenced in Section 5.3. Each test is a Welch’s t-test on run-level mean MSR (one observation per run; n=60/ 60/ 59for Single-Agent / Multi-Agent without Critic / Multi-Agent), with Holm-Bonferroni correction across the three pairs. 8.6. MSR by Attack Scenario Figure 9 disaggregates the headline MSR (Figure 4) by attack scenario. The hatched bar shows the rate without the connectivity-regression check. Sample sizes range from 138 to 150 mitigations per cell. Multi-Agent MSR ranges from 27% on DNS Exfiltration to60% on Lateral Movement; DNS Exfiltration’s lower rate is structural, reflecting that its egress channel (DNS) bypasses the network-based egress controls the suggester most often proposes. 8.7. Per-Mitigation Cost Table 5 reports per-mitigation token usage, duration, and dollar cost by runtime condition. Each mitigation may com- prise multiple implementation–critique iterations; the row aggregates across all such iterations within the mitigation. Cost is computed at GPT-5.4-mini Azure list pricing ($0.75 per 1M input tokens, $4.50 per 1M output tokens); the Total column sums across all mitigations in the condition. 17 CALDERA_MGMT f2/0f2/0 eth0 eth1 eth0 f0/0 e0 eth2 e0 eth3 e0 eth4 e0 eth5 e0 eth6 f0/0 eth0 eth2 e0 eth4 e0 eth5 e0 eth6 e0 caldera-br Port1 Port2 f1/0 Port3 f1/0 Cloud1 OpenvSwitch-1OpenvSwitch-2 PC10 PC11 PC2 PC3 PC4 PC5 PC6 PC7 PC8 PC9 UbuntuComputer UbuntuSSHVictim-1 cisco-router-R1 cisco-router-R2 fortigate-firewall Attack IP: 203.0.113.1 Port1: 7.0.14.1 Port2: 192.168.1.1 Port3: 192.168.4.1 IP:192.168.2.3 Gateway: 192.168.2.1 Port0: 192.168.2.1 Port1: 192.168.4.2 Port2: 10.0.0.2 Port0: 192.168.3.1 Port1: 192.168.1.3 Port2: 10.0.0.1 Figure 11: Medium enterprise network topology (inspired by Simple Network Layout by Gilbert Nims, GNS3 Marketplace). e0 eth1 e0 eth1 e0 eth2 e0 eth3 e0 eth2 e0 eth3 e0 eth2 e0 eth3 e3 e3 e2 e2 eth0 e1 e2 e2 docker0 e0 e1 e0 e5 e0 e6 e0 e2 e0 e4 e0 e6 e0 e4 e0 e6 e0 e4 e0 e5 e0 e6 e0 e4 e0 e5 e0 e6 e0 e4 e0 e5 e0 e6 e0 e1 e0 e4 e0 e2 e0 e4 e0 e6 e0 Accountant-PC-1Accountant-PC-2 Accountant-PC-3 Accountant-PC-4 Accountant-PC-5 Accountant-PC-6 Accountant-SW Admin-PC-2 Admin-PC-3 Admin-PC-4 Admin-PC-5 Admin-PC-6 Admin-SW CALDERA-MGMT CameraA Cloud1 CoreRouterA CoreRouterB CoreRouterC CoreRouterD CoreRouterE Costumer-PC-1 Costumer-PC-2 Costumer-PC-3 Costumer-PC-4 Costumer-PC-5 Costumer-PC-6 Customer-SW DVR FileServer Finance-PC-1 Finance-PC-2 Finance-PC-3 Finance-PC-4 Finance-PC-5 Finance-PC-6 Finance-SW Floor1-FW Floor1-L3SW Floor1-Router Floor2-FW Floor2-L3SW Floor2-Router Floor3-FW Floor3-L3SW Floor3-Router Floor4-FW Floor4-L3SW Floor4-Router HR-PC-1 HR-PC-2 HR-PC-4 HR-PC-5 HR-PC-6 HR-SW ICT-PC-1ICT-PC-2 ICT-PC-3ICT-PC-4 ICT-PC-5 ICT-PC-6 ICT-SW Logistics-PC-1 Logistics-PC-2 Logistics-PC-3 Logistics-PC-4 Logistics-PC-5 Logistics-PC-6 Logistics-SW Management-SW Manager-PC-1 Manager-PC-2 Manager-PC-3 Manager-PC-4 Manager-PC-5 Manager-PC-6 Marketing-PC-1 Marketing-PC-2 Marketing-PC-3 Marketing-PC-4 Marketing-PC-5 Marketing-PC-6 Marketing-SW Research-PC-1 Research-PC-2 Research-PC-3 Research-PC-4 Research-PC-5 Research-PC-6 Research-SW Server-Room-SW SwitchX(PoE) UbuntuComputer UbuntuSSHVictim-1 cameraB1 .2-.7 *Static IPs* Marketing - 192.168.10.0/24 Accountant - 192.168.20.0/24 Finance - 192.168.30.0/24 Management - 192.168.40.0/24 Research - 192.168.50.0/24 HR - 192.168.60.0/24 Customer - 192.168.70.0/24 Logistic - 192.168.80.0/24 ICT - 192.168.90.0/24 SwitchX - 192.168.100.0/24 ServerRoom - 192.168.110.0/24 Admin - 192.168.120.0/24 192.168.120.0/24 192.168.110.0/24 192.168.100.0/24 vLan 100 vLan 110 vLan 120 ip: 10.layer.floor.host/30 10.0.4.0/30 10.1.4.0/30 10.2.4.0/30 trunk port 192.168.70.0/24 192.168.80.0/24 Marketing - 192.168.10.0/24 Accountant - 192.168.20.0/24 Finance - 192.168.30.0/24 Management - 192.168.40.0/24 Research - 192.168.50.0/24 HR - 192.168.60.0/24 Customer - 192.168.70.0/24 Logistic - 192.168.80.0/24 ICT - 192.168.90.0/24 SwitchX - 192.168.100.0/24 ServerRoom - 192.168.110.0/24 Admin - 192.168.120.0/24 192.168.90.0/24 192.168.60.0/24 192.168.50.0/24 192.168.40.0/24 192.168.30.0/24 192.168.20.0/24 192.168.10.0/24 Upstream interfaces: WAN downstream interfaces: LAN 10.1.1.0/30 10.2.1.0/30 10.0.1.0/30 10.1.2.0/30 10.0.2.0/30 10.2.2.0/30 10.2.3.0/30 10.1.3.0/30 10.0.3.0/30 All VPCs have access to the core router. Floor routers perform NAT. Internal communication between floors is routed through the floor router interfaces eth2 and eth3. No access from the core router to the internet. 10.3.12.0/30 10.3.23.0/30 10.3.34.0/30 NAT NAT NAT NAT NAT Routers perform inter-LAN communication using Router-on-a-stick technics. IP: 192.168.120.3 Gateway: 192.168.120.1 Attacker IP: 203.0.113.1 Figure 12: Large enterprise network topology (inspired by Enterprise Network Lab: Bank Project by Kiki Oyewole, GNS3 Marketplace). 18