Paper deep dive
CSLE: A Reinforcement Learning Platform for Autonomous Security Management
Kim Hammar
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 98%
Last extracted: 4/27/2026, 6:21:24 PM
Summary
CSLE (Cyber Security Learning Environment) is a reinforcement learning platform designed for autonomous security management in networked systems. It bridges the gap between simulation and operational reality by using a two-part architecture: an Emulation System that creates a virtualized 'digital twin' of a target system to gather realistic data and a Simulation System that uses identified models (like Markov Decision Processes) to efficiently learn and optimize security strategies. The platform supports distributed deployment via Docker Swarm and has been validated through use cases such as flow control, replication control, segmentation control, and recovery control.
Entities (7)
Relation Signals (7)
CSLE → contains → Simulation System
confidence 100% · Second, it includes a simulation system where security strategies are efficiently learned...
CSLE → contains → Emulation System
confidence 100% · Conceptually, CSLE encompasses two systems. First, it includes an emulation system...
Emulation System → creates → Digital Twin
confidence 100% · the emulation system for creating a virtual replica (i.e., a digital twin) of the target system.
Emulation System → creates → Digital Twin
confidence 100% · the emulation system for creating a digital twin, i.e., a virtual replica of the target system.
CSLE → uses → Docker Swarm
confidence 100% · connected through a database (the metastore) and a virtualization layer provided by Docker Swarm.
Simulation System → uses → Markov Decision Process
confidence 90% · the simulation system where security strategies are efficiently learned through simulations of the system model [identified via MDP].
Simulation System → uses → Markov Decision Process
confidence 90% · identify a system model, such as a Markov decision process. Second, it includes a simulation system...
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Reinforcement learning is a promising approach to autonomous and adaptive security management in networked systems. However, current reinforcement learning solutions for security management are mostly limited to simulation environments and it is unclear how they generalize to operational systems. In this paper, we address this limitation by presenting CSLE: a reinforcement learning platform for autonomous security management that enables experimentation under realistic conditions. Conceptually, CSLE encompasses two systems. First, it includes an emulation system that replicates key components of the target system in a virtualized environment. We use this system to gather measurements and logs, based on which we identify a system model, such as a Markov decision process. Second, it includes a simulation system where security strategies are efficiently learned through simulations of the system model. The learned strategies are then evaluated and refined in the emulation system to close the gap between theoretical and operational performance. We demonstrate CSLE through four use cases: flow control, replication control, segmentation control, and recovery control. Through these use cases, we show that CSLE enables near-optimal security management in an environment that approximates an operational system.
Tags
Links
- Source: https://arxiv.org/abs/2604.15590v1
- Canonical: https://arxiv.org/abs/2604.15590v1
Trouble viewing inline? Open PDF directly →
Full Text
103,637 characters extracted from source content.
Expand or collapse full text
CSLE: A REINFORCEMENT LEARNING PLATFORM FOR AUTONOMOUS SECURITY MANAGEMENT Kim Hammar 1 ABSTRACT Reinforcement learning is a promising approach to autonomous and adaptive security management in networked systems. However, current reinforcement learning solutions for security management are mostly limited to simulation environments and it is unclear how they generalize to operational systems. In this paper, we address this limitation by presenting CSLE: a reinforcement learning platform for autonomous security management that enables experimentation under realistic conditions. Conceptually, CSLE encompasses two systems. First, it includes an emulation system that replicates key components of the target system in a virtualized environment. We use this system to gather measurements and logs, based on which we identify a system model, such as a Markov decision process. Second, it includes a simulation system where security strategies are efficiently learned through simulations of the system model. The learned strategies are then evaluated and refined in the emulation system to close the gap between theoretical and operational performance. We demonstrate CSLE through four use cases: flow control, replication control, segmentation control, and recovery control. Through these use cases, we show that CSLE enables near-optimal security management in an environment that approximates an operational system. 1INTRODUCTION Managing the security of networked systems alongside their service requirements and physical infrastructures is a major technical challenge that has grown exponentially with the rise of cloud computing, distributed networks, and IoT ser- vices. Examples of security management tasks include inci- dent response, risk analysis, strategy design, and threat hunt- ing. Today, many of these tasks remain manual processes carried out by security experts. Although this approach can be effective, it is labor-intensive and requires significant skills. For example, a recent study reports a global shortage of more than 4 million security experts (ISC2, 2024). A promising approach to address this challenge is to use reinforcement learning to automatically derive effective se- curity strategies. For example, Li et al. (2024) use rein- forcement learning to compute effective incident response strategies. Similarly, Kiely et al. (2025) use multi-agent reinforcement learning to derive effective defense strategies against advanced persistent threats. A comprehensive re- view of these developments is provided by Nguyen & Reddi (2023). While these works report encouraging results, key challenges remain. Chief among them is narrowing the 1 Division of Network and Systems Engineering, KTH Royal In- stitute of Technology, Sweden. Correspondence to: Kim Hammar <kimham@kth.se>. Proceedings of the9 th MLSys Conference, Bellevue, WA, USA, 2026. Copyright 2026 by the author(s). s 1,1 s 1,2 s 1,3 ... s 1,n s 2,1 s 2,2 s 2,3 ... s 2,n . . . . . . . . . . . . . . . EMULATION SYSTEM TARGET SYSTEM System identification Strategy mapping Selective Replication Strategy implementation SIMULATION SYSTEM Reinforcement learning Digital twin for data collection & strategy evaluation Automated security strategies Figure 1.Architectural overview of CSLE: a reinforcement learn- ing platform for autonomous security management. gap between the environment where strategies are evalu- ated and a scenario playing out in an operational system. Most of the results obtained so far are limited to simulation environments, leaving their practical utility unproven. In this paper, we address this limitation by presenting a platform that enables experimentation with reinforcement learning in realistic conditions. Conceptually, the platform consists of two systems, as illustrated in Fig. 1. First, we use an emulation system for creating a virtual replica (i.e., a digital twin) of the target system. This twin closely ap- proximates the functionality and timing behavior of the arXiv:2604.15590v1 [cs.CR] 16 Apr 2026 The Cyber Security Learning Environment (CSLE) target system, which allows us to run attack scenarios in a controlled environment. Such runs produce system mea- surements and logs, based on which we identify a system model, e.g., a Markov decision process. Second, we use a simulation system where near-optimal security strategies are incrementally learned through reinforcement learning. Learned strategies are extracted from the simulation system and evaluated in the digital twin. This process can be per- formed iteratively to provide progressively better security strategies that are adapted to changes in the target system, such as configuration changes and software updates. We refer to the platform as CSLE, which stands for the “Cyber Security Learning Environment.” CSLE includes an initial set of 15 digital twin configurations, more than 50 simulated security scenarios, 34 implemented reinforcement learning algorithms, and 4 implemented system identifica- tion algorithms, all of which can be extended. To evaluate CSLE experimentally, we use it to learn effective security strategies for four security management tasks: flow con- trol, replication control, segmentation control, and recovery control. Through these use cases, we show that CSLE en- ables autonomous security management in an environment that closely approximates an operational system. Moreover, we demonstrate the broad applicability of CSLE and its integration with various reinforcement learning techniques. In summary, the main contributions of this paper are: • We present CSLE, a reinforcement learning platform for autonomous security management that enables ex- perimentation under realistic operating conditions. •We evaluate CSLE on four different use cases: flow control, replication control, segmentation control, and recovery control. Our experimental results show that CSLE enables autonomous security management. Open sourceThe source code of CSLE is released under the C-BY-SA 4.0 license and available in the repository at (Hammar, 2023). In addition to the source code, this repos- itory also includes video demonstrations, Docker images, documentation, and datasets of system traces. 2AUTONOMOUS SECURITY MANAGEMENT THROUGH REINFORCEMENT LEARNING Before presenting our platform, we start by formulating security management as a reinforcement learning problem. To accomplish this formulation, we need a vocabulary in which to talk about the systems and actors involved. To this end, we refer to the operator of the target system as the defender, and we refer to an entity aiming to attack the system as the attacker. Both interact with the system by taking actions (e.g., attacks and responses), which affect the system’s state (e.g., the system’s security and service status). When selecting these actions, the defender and the attacker use measurements from the system (e.g., log files and security alerts), which we refer to as observations. A function that maps a sequence of observations to an action is called a strategy, and a strategy that is most advantageous according to some objective is optimal. NETWORKED SYSTEM SENSOR state observation management action attack Attacker Defender Figure 2.Autonomous and adaptive security management of a networked system as a reinforcement learning problem. In the context of reinforcement learning, we can view se- curity management as the problem of learning an effective defender strategy through repeated interaction with the sys- tem. In particular, by observing how the system responds to different actions, the defender can gradually improve its strategy to meet security objectives. However, many system-level challenges are encountered when applying this approach in practice. Chief among them are: 1. In an operational networked system, attacks and de- fender actions unfold over long time scales and can disrupt critical services. These factors make direct in- teraction with the system impractical for reinforcement learning, which typically requires executing thousands of actions to learn an effective strategy. 2.To enable safe and efficient learning, the learning pro- cess must, therefore, be executed in a simulation envi- ronment. However, the system’s behavior is often too complex to model, which means that the simulation dy- namics must be estimated from system measurements. 3.After learning a security strategy through simulation, it must be experimentally validated. In particular, the validity of the simulation must be verified by evaluating the learned strategy in an environment that closely approximates the target system. In the next section, we review existing platforms that attempt to address these challenges and explain their limitations. We then introduce our platform (CSLE), which is designed to overcome these limitations and enable reinforcement learning experimentation in realistic operating conditions. The Cyber Security Learning Environment (CSLE) PlatformSimulationEmulationOpen sourceRL LibraryManagementValidatedMaintainedDistributed CSLE (our platform)✓ CyberBattleSim✓✗✓✗✓✗ CyBorg✓✗✓✗ Yawning Titan✓✗✓✗ NaSim✓✗✓✗ ATMoS✗✓✗✓✗ Gym-FlipIt✓✗✓✗ Gym-IDSgame✓✗✓✗ MAB-Malware✗✓✗✓✗ Malware-RL✗✓✗✓✗ PenGym✓✗✓✗ CyGil✗✓✗?? NaSimEmu✓✗✓✗ Farland✓✗✓?? CyberWheel✓✗✓✗ CyberShield✓✗✓✗ Cyborg++✓✗✓✗✓✗ CyGym✓✗✓✗✓✗ C-CyberBattleSim✓✗✓✗✓✗ Table 1.Comparison between reinforcement learning platforms for autonomous security management based on key features: support for simulation-based optimization; support for emulation-based evaluation; open source code; whether the platform provides a library with implemented reinforcement learning algorithms to facilitate strategy optimization and system identification; whether the platform provides a management system for automating experiments and debugging strategies; whether the platform has been experimentally validated on practical use cases; whether the platform is actively maintained; and whether the platform supports distributed deployment. 3REINFORCEMENT LEARNING PLATFORMS FOR AUTONOMOUS SECURITY MANAGEMENT Over the past5years, several reinforcement learning envi- ronments for autonomous security management have been developed. They include CyberBattleSim by Microsoft (Blum, 2021), CyBorg by the Australian department of defense (Standen et al., 2021), NaSim by the University of Queensland (Schwartz et al., 2020), Yawning Titan by the UK defense science and technology laboratory (Andrew et al., 2022), CyGil by Canada’s department of defense (Li et al., 2021), NaSimEmu by the Czech Technical Univer- sity in Prague (Janisch et al., 2023), ATMoS by the Uni- versity of Waterloo (Akbari et al., 2020), Gym-FlipIt by Northeastern University (Oakley & Oprea, 2019), Gym- IDSgame by KTH Royal Institute of Technology (Hammar & Stadler, 2020), MAB-Malware by the University of Cali- fornia (Riverside) (Song et al., 2022), Malware-RL by the University of Virginia (Anderson et al., 2018), PenGym by Japan’s advanced institute of science and technology (Huynh Phuong Thanh et al., 2024), Farland by USA’s national security agency (Molina-Markham et al., 2021), CyberWheel by the Oak Ridge national laboratory (Oesch et al., 2024), CyberShield by the University of Malaga (Car- rasco et al., 2024), Cyborg++ by the Alan Turing Institute (Emerson et al., 2024), CyGym by Washington University (Lanier & Vorobeychik, 2025), and C-CyberBattleSim by the University of Lorraine (Terranova et al., 2025). Like CSLE, all of the referenced platforms include capa- bilities for learning security strategies using reinforcement learning. However, they differ from CSLE in several impor- tant ways, as highlighted in Table 1. First, most existing platforms are confined to simulations. By contrast, CSLE is centered around an emulation system based on virtual- ization. The benefit of our approach is that it narrows the gap between the environment where security strategies are evaluated and a scenario playing out in an operational sys- tem. Second, many of the referenced platforms are not open source and most of them are no longer maintained. By con- trast, CSLE is open source and has an active development community. Third, CSLE has been experimentally validated on a range of practical use cases, whereas most other plat- forms have only been evaluated on a single simulation use case. Fourth, unlike the other platforms, CSLE supports dis- tributed deployment, which improves scalability. Moreover, CSLE incorporates a novel management system that pro- vides infrastructure for automating reinforcement learning experiments and debugging the learned strategies. Lastly, we note that a few platforms for autonomous sys- tem operations based on large language models (LLMs) have recently emerged, most notably ITBench (Jha et al., 2025) and AIOpsLab (Chen et al., 2025). These platforms focus on using LLMs to automate general system opera- tion tasks. By contrast, CSLE is explicitly designed for automating security management tasks. Another difference is that the referenced platforms are designed for evaluating LLM-based agents, whereas CSLE is designed for develop- The Cyber Security Learning Environment (CSLE) ing reinforcement learning agents. This difference in scope leads to fundamental differences in platform architecture. In particular, LLMs do not require the same system-level support as reinforcement learning agents do. For example, CSLE supports identifying simulation models, optimizing strategies through reinforcement learning, and transferring strategies from simulation to emulation. None of these func- tions is provided by ITBench and AIOpsLab. 4ARCHITECTURE OF CSLE The architecture of CSLE is illustrated in Fig. 1 and is centered around an emulation system for creating a digital twin, i.e., a virtual replica of the target system. 1 We use this twin to run automated attack scenarios and defender responses. Such runs produce system measurements and logs, from which we estimate infrastructure statistics. These statistics allow us to instantiate a mathematical model of the target system through system identification. We then leverage this model to learn effective security strategies through simulation, whose performance is assessed using the digital twin. This closed-loop process can be executed iteratively to provide progressively better security strategies that are adapted to changes in the target system; see Fig. 3. SIMULATION s 1,1 s 1,2 s 1,3 ... s 1,4 s 2,1 s 2,2 s 2,3 ... s 2,4 DIGITAL TWIN ... Virtual network Virtual devices Emulated services Emulated actors TARGET SYSTEM Change events System identification Verified security strategy Optimized security strategy Figure 3.A digital twin in CSLE is a virtual replica of a target system that runs the same software and configuration, but on virtu- alized hardware. Moreover, the twin controls network delays and emulates actors to replicate operational workloads. The twin is used in CSLE for strategy evaluation and system identification. 4.1 Emulation System As described above, the emulation system in CSLE is used to create a digital twin of the target system. The concept 1 By target system, we mean the system where the learned security strategies are intended to be deployed. of a digital twin emerged in the 1960s when NASA used virtual environments to evaluate failure scenarios for lunar landers (Allen B. Danette, 2021). Since then, digital twin has emerged as a key technology in automation and has been adopted in several industries, including the manufacturing industry [see e.g., (Tao et al., 2019)], the automotive indus- try [see e.g., (Biesinger & Weyrich, 2019)], the healthcare industry [see e.g., (Liu et al., 2019)], and the technology industry [see e.g., (Wu et al., 2021)]. In CSLE, a digital twin is a virtual replica of a networked system that provides a controlled environment for virtual operations (e.g., cyberattacks and responses), the outcomes of which can be used to optimize operations in the target system. Such a twin enables us to systematically test secu- rity strategies under different conditions, including varying attacks, workloads, and network latencies. 4.2 Simulation System The simulation system in CSLE is used to run simulations and execute reinforcement learning algorithms. Although these algorithms could in principle be executed in the digital twin, this approach is not practical due to the long execu- tion times required for carrying out actions and collecting observations in the digital twin. For instance, executing a cyberattack or a defensive reconfiguration in a digital twin can take several minutes. In contrast, the simulation system abstracts these processes as actions in a Markov decision process, which reduces the execution time to milliseconds. With a simulation, we mean an execution of a discrete-time dynamical system of the form s t+1 ∼ f (s t ,a (D) t ,a (A) t ),(1) wheres t is the system state at timet,a (D) t is the defender action,a (A) t is the attacker action,fis the system dynam- ics, ands ∼ fmeans thatsis sampled fromf. For ex- ample, the dynamicsfmay represent a Markov decision process (MDP) or a Markov game. Each simulation path s 1 ,s 2 ,...,s t is associated with security consequences and costs. The goal of reinforcement learning is to identify the defender actions that control the simulation in an optimal manner according to a specified security objective. 4.3 Reinforcement Learning Methodology The emulation and simulation systems in CSLE enable a re- inforcement learning methodology with the following steps. Step 1 Defining the target system. •This is the system where the learned security strategies are intended to be deployed. In CSLE, the target system is defined through a configu- The Cyber Security Learning Environment (CSLE) ration file that specifies the system components, the network topology, the services, etc. Step 2 Creating a digital twin of the target system. • Given the target system specification, the cre- ation of a digital twin in CSLE is automated through the emulation system. Step 3 Collecting data from the digital twin. •After creating the digital twin, we use it to run attack scenarios. Such runs produce system traces (i.e., sequences of system metrics), which we collect through CSLE’s monitoring system. Step 4 Identifying a system model. •Having collected system measurements from the digital twin, we use the collected data to identify a model (e.g., through statistical learn- ing) that can be used for running simulations, such as a Markov decision process (MDP). Step 5 Learning an effective security strategy. • Given the identified system model (e.g., an MDP), we apply reinforcement learning tech- niques to learn an effective security strategy. Step 6 Evaluating the learned strategy in the digital twin. • After the learning process has converged, we evaluate the learned security strategy in the dig- ital twin. Such evaluation involves measuring system metrics from the digital twin in real time (e.g., security alerts), using them as input to the security strategy, and executing the action pre- scribed by the strategy in the digital twin. Step 7 Deploying the learned strategy in the target system. •If the evaluation is satisfactory, we deploy the learned strategy in the target system. Otherwise, we collect more data to update the simulation and then learn a new strategy. This procedure of updating the simulation and re-learning the strategy is repeated until a strategy with satis- factory performance is obtained. 5IMPLEMENTATION OF CSLE We have implemented CSLE in Python [≈ 275, 000lines of code], JavaScript [≈ 40, 000lines of code], and Bash [≈ 5, 000lines of code]. From an architectural point of view, the implementation can be divided into three systems: the emulation system, the simulation system, and the manage- ment system; see Fig. 4. Broadly speaking, the emulation system creates digital twins, the simulation system runs rein- forcement learning algorithms, and the management system orchestrates the platform. The rest of this section delves into the technical details of these three systems. Leader Metastore Python libraries Management API (GRPC) REST API (HTTP) Command-line interface Workers Docker Swarm Emulation system Management system Simulation system Figure 4.The architecture of CSLE. It is a distributed platform withNservers (N = 6in this example), which are connected through a database (the metastore) and a virtualization layer pro- vided by Docker Swarm. CSLE has four interfaces: a Python API, a GRPC API, a REST API, and a command-line interface. 5.1 Infrastructure CSLE runs on a distributed system withN ≥ 1servers connected through an IP network. Each server runs a virtu- alization layer provided by Docker Swarm (Merkel, 2014) and can be accessed through Python libraries, a web in- terface, a command-line interface, and a GRPC interface (Google, 2022). Platform metadata is stored in a distributed database referred to as the metastore, which is based on Citus (Cubukcu et al., 2021). This database consists ofN replicas, one per server. One replica is a designated leader and is responsible for coordination. The others are workers. A new leader is elected by a quorum whenever the current leader fails or becomes unresponsive. CSLE thus tolerates up to⌊ N−1 2 ⌋failing servers. This design enables horizontal scaling as the number of servers increases. Deployment of CSLE in both on-premise and cloud infras- tructures is automated using Ansible (Red Hat, 2024). This automation enables on-demand deployment, allowing CSLE to be launched dynamically for specific experiments or to run continuously as part of an operational environment. 5.2 The Emulation System The purpose of the emulation system in CSLE is to create a digital twin that replicates relevant components of the tar- get system. Creating such a twin involves three tasks: (i) emulating the target system’s physical infrastructure, such as processors, network interfaces, and network conditions; (i) emulating actors, i.e., attackers, defenders, and clients; and (i) instrumenting the twin with monitoring and man- agement capabilities. Each of these tasks is detailed below. The Cyber Security Learning Environment (CSLE) Emulating hosts and switches We emulate hosts and switches with Docker containers (Merkel, 2014), i.e., lightweight executable packages that include runtime systems, code, libraries, and configurations. This virtualization lets us quickly instantiate large digital twins; see Fig. 5. Resource allocation to containers, e.g., CPU and memory, is enforced using Cgroups. Contain- ers that emulate switches run OVS (Pfaff et al., 2015) and connect to controllers through OpenFlow (McKeown et al., 2008). Since the switches are programmed through flow tables, they can act as layer-two switches or as routers. 0 20 40 60 80 Infrastructure Configuration Management Deployment (min) 2481632641282565121024 5 10 Size of the digital twin (number of virtual containers) Cleanup (min) Figure 5.Time to deploy and cleanup a digital twin in CSLE. De- ploying the twin involves creating containers, attaching them to networks, configuring them, and starting management services. Cleanup involves stopping and deleting containers and networks. The time measurements were performed for a digital twin with a single network running on a server with a24-core Intel Xeon Gold 2.10GHz CPU and768GB RAM. Numbers and error bars indi- cate the mean and the standard deviation from 5 measurements. The hosts and switches of the digital twin are specified through a configuration file written in Python, which CSLE parses before deploying the twin. We provide a code snippet of the configuration file in Listing 1. from csle_common.dao.emulation_config. node_container_config import NodeContainerConfig from csle_common.dao.emulation_config. node_firewall_config import NodeFirewallConfig node_cfg = NodeContainerConfig(name="my-image", os=" Ubuntu22", ips=[..], subnets=[..], interfaces=[..], cpus=1, memory_gb=4) node_fw_config = NodeFirewallConfig(host="..", default_gw="",default_input="ACCEPT",default_output ="ACCEPT",default_forward="ACCEPT",fw_rules=[..]) Listing 1. Python code for configuring a container. Emulating network links We emulate network connectivity in digital twins through virtual links implemented by Linux bridges and network namespaces. If an emulated network spans multiple physical servers, we tunnel the traffic over the physical network using VXLAN (Mahalingam et al., 2014). In other words, the physical network of the servers provides a substrate, on top of which the emulated networks are overlaid. Network conditions of virtual links are created using the NetEm module in the Linux kernel (Hemminger, 2005). This module allows setting bit rates, packet delays, packet loss probabilities, and jitter. For example, the standard con- figuration in CSLE emulates connections between servers in an IT system with full-duplex, lossless connections of 1 Gbit/s capacity in both directions. Similarly, the default configuration for external communications is full-duplex connections of100Mbit/s capacity and0.1%packet loss with random bursts of1%packet loss. These numbers are based on measurements on enterprise and wide-area net- works; see e.g., (Kushida & Shibata, 2002; Paxson, 1997). The network conditions are configured in CSLE through Python objects. We provide an example in Listing 2. from csle_common.dao.emulation_config. node_network_config import NodeNetworkConfig NodeNetworkConfig(interface="eth0", packet_delay_ms=2, jitter_ms=0.5, delay_distribution="pareto", corrupt =0.02, duplicate=0.00001, correlation=25, reorder =2, rate_limit_mbit=100) Listing 2. Python code for configuring a network interface. Emulating actors All actors in CSLE are programmatically controlled through a management API based on GRPC, which allows changing configuration parameters, starting new actors, and stopping running ones. This automation enables attackers, clients, and defenders to operate in a fully autonomous environment. We emulate clients through processes in the digital twin that access services on emulated hosts. The client population is defined by (i) an arrival process (e.g., a Poisson process) that controls the rate at which new client processes are started; (i) a service time distribution (e.g., an exponential distribution) that controls how long a client will consume services before terminating; (i) a service configuration that specifies the services of the digital twin that clients will consume; and (iv) a Markov process that controls the sequence of service invocations that a client makes. All of these parameters are configured in CSLE through a Python file. We provide a code snippet of this file in Listing 3. from csle_collector.client_manager.dao.client import Client from csle_collector.client_manager.dao. constant_arrival_config import ConstantArrivalConfig clients=[Client(service_distribution=[0.5,0.2,0.3], arrival_config=ConstantArrivalConfig(lamb=20), mu =4, exponential_service_time=True)] Listing 3. Python code for configuring the client population. The Cyber Security Learning Environment (CSLE) Figure 6 shows the resource usage of two digital twins as a function of the client arrival rate. We observe, as expected, that the resource usage increases with the load imposed on the twins. In particular, higher client arrival rates lead to increased CPU utilization since the twins must process a larger number of service requests. In contrast, the memory usage remains stable when increasing the load. 5 10 15 31 containers 64 containers CPU usage (%) 20406080100120140160180200 30 40 50 60 31 containers 64 containers Memory usage (GB) Client arrival rate λ Figure 6.Resource usage of two digital twins in function of the client (Poisson) arrival rateλ. Numbers and error bars indicate the mean and the standard deviation from5evaluations. The CPU and memory usages are averaged over a monitoring period of30minutes. The blue curves relate to digital twins of an IT infrastructure with31and64hosts, respectively. The network topologies are shown in Figs. 9.a–b and the configurations are available in the supplementary material (Tables 5 and 6). Each client consumes a randomly selected service of the infrastructure for a time that is sampled from an exponential distribution with mean valueμ = 60seconds. We run the digital twins on a server with a24-core Intel Xeon Gold2.10GHz CPU and768GB RAM. Similar to how clients are emulated, attackers in CSLE are implemented as autonomous processes that execute actions from a pre-defined list, including reconnaissance commands, privilege escalation actions, and exploits. Table 2 lists some of the attacker actions that are automated in CSLE. The defender is emulated in a similar way, with actions imple- mented as system commands that can reconfigure network components, isolate hosts, or perform other mitigation steps. We provide several examples of defender actions in Table 3. 5.3 The Management System The role of the management system in CSLE is to sup- port the operation of digital twins and facilitate end-to-end reinforcement learning experiments. In particular, the man- agement system provides APIs for real-time monitoring and control of digital twins, as well as a web interface for man- aging reinforcement learning experiments and deployments. Each emulated device in a digital twin runs a management agent, which exposes a GRPC API (Google, 2022). This API is invoked to perform control actions, e.g., restarting services and updating configurations. The communication TypeActionsMITRE ATT&CK technique ReconnaissanceTCP SYN scan, UDP scanT1046 service scanning. TCP XMAS scanT1046 service scanning. VulscanT1595 active scanning. ping-scanT1018 system discovery. Brute-forceTelnet, SSHT1110 brute force. FTP, CassandraT1110 brute force. IRC, MongoDB, MySQLT1110 brute force. SMTP, PostgresT1110 brute force. ExploitCVE-2017-7494T1210 service exploitation. CVE-2015-3306T1210 service exploitation. CVE-2010-0426T1068 privilege escalation. CVE-2015-5602T1068 privilege escalation. CVE-2015-1427T1210 service exploitation. CVE-2014-6271T1210 service exploitation. CVE-2016-10033T1210 service exploitation. SQL injectionT1210 service exploitation. Table 2.Examples of attacker actions in CSLE; actions are iden- tified by identifiers in the common vulnerabilities and exposures (CVE) database (The MITRE Corporation, 2022); the actions are also linked to the corresponding attack techniques in the MITRE ATT&CK taxonomy (Strom et al., 2018). channels to the agents are provided by a management net- work. The reason for using a separate network to carry management traffic is to avoid interference and simplify control of the digital twin (Clemm & Cisco Systems, 2007). We provide an example of using the management system to execute control actions inside a digital twin in Listing 4. To complement the Python APIs, the management system also includes a web interface and a command-line interface, both of which provide the same functions as the Python API. A video demonstration of the management system is available at (Hammar, 2023) and screenshots of the web interface are provided in Appendix A in the supplementary material. from csle_common.metastore.metastore_facade import MetastoreFacade from csle_common.util.emulation_util import EmulationUtil twin=MetastoreFacade.get_twin(name="") EmulationUtil.execute_ssh(cmds=[cmd], ip="", twin=twin) Listing 4. Python code for executing a control action. To monitor processes and services running inside the dig- ital twin, we use a monitoring system based on a publish- subscribe architecture; see Fig. 7. Following this archi- tecture, each emulated device in a digital twin runs a mon- itoring agent, which reads local metrics of the host and pushes those metrics to an event bus implemented with Kafka (Kreps, 2011). The data in this bus is consumed by data pipelines, which process the data and write it to storage systems. In particular, the data is exported to an Elastic- search database that can be queried and visualized through Kibana dashboards for real-time monitoring. The Cyber Security Learning Environment (CSLE) DEVICES EVENT BUS STORAGE SYSTEMS Control actions Data pipelines π Control strategy Events Figure 7.Monitoring system of a digital twin in CSLE. Emulated devices run monitoring agents that periodically push metrics to an event bus, which is consumed by pipelines that process the data and write to storage systems; the processed data is also used as input to automated control strategies to decide on control actions. ActionMITRE D3FEND technique Revoke user certificatesD3-CBAN certificate revocation. Blacklist IPsD3-NTF network traffic filtering. Drop trafficD3-NTF network traffic filtering. Block gatewayD3-NI network isolation. Migrate servers between zonesD3-NI network isolation. Redirect trafficD3-NTF network traffic filtering. Isolate a serverD3-NI network isolation. Deploy new security functionsD3-NTPM network policy mapping. Shutdown a serverD3-HS host shutdown. Replicate a serviceD3-SVCDM service mapping. Start decoy servicesD3-D3 decoy environment. Table 3.Examples of defender actions in CSLE; the actions are linked to the corresponding defense techniques in the MITRE D3FEND taxonomy (Kaloroumakis & Smith, 2021). Figure 8 shows performance statistics related to the moni- toring system. In particular, Fig. 8.a shows that the CPU overhead introduced by the monitoring agents is around 6%, while the memory overhead is approximately 1%. Both values can be considered relatively low. Furthermore, Fig. 8.b shows that the number of monitoring events produced by the monitoring agents per monitoring interval increases with the size of the digital twin and also depends on the specific system configuration. Larger twins typically contain more monitored components, which naturally results in a higher number of generated events. In addition, the event rate is influenced by the types and number of monitoring mecha- nisms deployed. For example, configurations that include a larger number of intrusion detection systems (as is the case for the system in Fig. 9.b) generate more monitoring events. 5.4 The Simulation System The simulation system in CSLE is implemented in Python and consists of reinforcement learning environments and al- gorithms for learning security strategies. All environments follow the OpenAI Gym interface (Towers et al., 2024), which allows integration with standard reinforcement learn- ing frameworks. Each environment defines a Markov de- 0 2 4 6 8 6.2 1.2 CPUMEMORY 0 50 100 150 36 129 31 containers 64 containers a) Average. Overhead (%) b) # Monitoring events per monitoring interval Figure 8.Statistics of the monitoring system in CSLE. Plot a) shows the average overhead of a monitoring agent and plot b) shows the number of monitoring events per monitoring interval for two digital twins deployed with CSLE. The network topologies of the digital twins with31and64containers are shown in Fig. 9.a and Fig. 9.b, respectively. Numbers and error bars indicate the mean and the standard deviation from 5 evaluations. cision process or a game and can be configured through Python configuration files. CSLE includes an initial set of 34 reinforcement learning algorithms, over 50 simulation environments, and 4 identification algorithms. We provide an example of using the simulation system to run a rein- forcement learning algorithm in Listing 5. from csle_agents.agents.sarsa.sarsa_agent import SARSAAgent from csle_common.metastore.metastore_facade import MetastoreFacade from csle_common.dao.training.experiment_config import ExperimentConfig simulation = MetastoreFacade.get_simulation(..) experiment = ExperimentConfig(..) agent = SARSAAgent(simulation, experiment) execution = agent.train() MetastoreFacade.save_experiment_execution(execution) for strategy in execution.result.strategies.values(): MetastoreFacade.save_strategy(strategy) Listing 5. Python code for running the SARSA reinforcement learning algorithm in the simulation system. 6EXAMPLE USE CASES We demonstrate CSLE by applying it to four different secu- rity use cases. Each use case involves a target system and a system operator, which we refer to as the defender; see Fig. 9. (The detailed system configurations are available in Appendix B.) The use cases are described below. 6.1 Flow Control This use case involves an IT system that provides services to clients through a public gateway; see Fig. 9.a. While the gateway enables legitimate access for clients, it also exposes an entry point for potential attackers attempting to intrude on the system and compromise components. To protect the system against such intrusions, the defender continuously monitors network traffic and analyzes security The Cyber Security Learning Environment (CSLE) a) Target system for the flow control use case.b) Target system for the segmentation use case.c) Target system for the replication and recovery use cases. Service replicas . . . Clients Attacker R&D ZONE DMZ ADMIN ZONE service Defender ... AttackerClients 2 1 312 4 5 6 7 8 9 10 11 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 3031 32 33 34 35 3637 38 39 40 4142434445464748 49505152535455565758596061626364 AttackerClients ... Defender 1 789101165432 12 13141516 17 18 19 21 23 20 22 24 2526 2728293031 Figure 9. Target systems for the use cases in the experimental evaluation. The system configurations are available in Appendix B. alerts to identify suspicious or malicious activity. Based on these observations, the defender can control network flows to mitigate potential network intrusions. For example, the defender can block suspicious flows or redirect them to a honeypot. When making these decisions, the defender aims to balance the dual objectives of preserving service availability for clients and mitigating potential attacks. 6.2 Network Segmentation Control This use case involves a cloud infrastructure that is seg- mented into zones with virtual nodes that run network ser- vices; see Fig. 9.b. Services are realized by workflows that clients access through a cloud gateway, which is also open to an attacker. The attacker aims to intrude on the infras- tructure, compromise nodes, and disrupt workflows. To counter these threats, the defender continuously monitors the infrastructure by accessing and analyzing intrusion de- tection alerts and other statistics. Based on this information, the defender can respond to possible intrusions by changing the network segmentation. For example, the defender can migrate nodes between zones, change access controls, or shut down nodes. When deciding between these actions, the defender balances two conflicting objectives: maximizing workflow utility towards clients and minimizing the opera- tional cost of possible intrusions and defensive actions. 6.3 Recovery Control This use case involves a replicated system that provides a web service to a client population; see Fig. 9.c. Because multiple replicas deliver the same service, the system can continue operating even when some replicas are compro- mised. To track the evolving security status of the system, the defender analyzes security alerts that indicate potential compromises of service replicas. Based on these obser- vations, the defender decides when and which replicas to recover in order to maintain service availability. When making these recovery decisions, the goal is to ensure that compromised service replicas are recovered faster than new compromises occur while minimizing the recovery costs. 6.4 Replication Control In this use case, we study the problem of learning adaptive replication strategies for the system illustrated in Fig. 9.c. The goal is to enable the system to autonomously adjust the number of replicas in response to changing security and performance conditions. In particular, the defender observes indicators of replica failures or degradations (e.g., security alerts) and uses these signals to dynamically decide when to launch new replicas or retire existing ones. By adapting the replication factor, the system can maintain service availability even under fluctuating attack intensities or workload demands. The key challenge is to minimize the number of replicas (to reduce operational costs) while still satisfying service availability and reliability constraints. 7SYSTEM MODELS For each of the use cases described above, we consider two different reinforcement learning problems. First, we con- sider the problem of learning an optimal control strategy against an attacker that follows a fixed strategy. We model this problem as a Markov decision process (MDP) or a par- tially observed MDP (POMDP), depending on whether the system state is observable. Second, we consider the prob- lem of learning a control strategy that is effective against The Cyber Security Learning Environment (CSLE) 0102030405060 0 50 100 150 Cumulative discounted reward Training time (min) 0102030405060 −0.8 −0.6 −0.4 −0.2 0 Average reward Training time (min) 0102030405060 −40 −30 −20 Cumulative discounted reward Training time (min) 020406080100 1 2 3 Exploitability Training time (min) 01,0002,0003,0004,0005,0006,000 2 4 6 Exploitability Training time (min) 0102030405060 0 0.5 1 EVALUATION IN DIGITAL TWINEVALUATION IN SIMULATION Exploitability Training time (min) Flow control (POMDP) Replication control (MDP)Recovery control (POMDP) Flow control (Markov game)Network segmentation (Markov game)Replication control (Markov game) Figure 10.Convergence curves for the different use cases. The red curves relate to the performance in the simulations and the blue curves relate to the performance when evaluating the learned strategies in the digital twins. Curves show the mean values from evaluations with5 random seeds; shaded areas indicate standard deviations. The x-axes indicate the training times in the simulations. The top row relates to simulations of decision-theoretic models. The bottom row relates to simulations of game-theoretic models. a dynamic attacker that adapts its strategy to circumvent the defenses. We model this problem as a Markov game. In total, we consider six reinforcement learning problems, which are listed in Table 4. We address each problem using the general reinforcement learning methodology described in §4. Mathematical formulations are given in Appendix D. ModelAlgorithm Flow control POMDPSPSA (Spall, 1992) Replication control MDPPPO (Schulman et al., 2017) Recovery control POMDPRollout (Bertsekas, 2021) Flow control gameFictitious play with SPSA (Brown, 1951) Segmentation gameFictitious play with PPO (Brown, 1951) Replication control gamePPO (Schulman et al., 2017) Table 4.The reinforcement learning problems that we consider in the experimental evaluation and the algorithms that we use to ad- dress them. In the decision-theoretic problems (i.e., the MDP and POMDPs), the goal is to learn an optimal security strategy against a fixed attacker strategy, whereas in the game-theoretic problems, the goal is to learn an equilibrium strategy against an attacker that dynamically adapts its strategy to the defender’s strategy. 8EXPERIMENTAL EVALUATION OF CSLE In this section, we present our experimental results. Fol- lowing the methodology described in §4, we identify the parameters of each system model described in the preced- ing section (e.g., the MDP, POMDP, or game parameters) based on data collected from the digital twin. Then, given the identified model, we learn the security strategy through simulation, after which we evaluate the learned strategy in the digital twin. The reinforcement learning algorithms that we use are listed in Table 4. For details about the system identification and the collected data, see Appendix D. 8.1 Experimental Setup We run the reinforcement learning algorithms until conver- gence and evaluate the learned strategies periodically during training, both in the simulation and in the digital twin. The hyperparameters that we use to instantiate the algorithms are listed in Appendix C. The environment for training strate- gies and running simulations is a Tesla P100 GPU. The digital twins are deployed on a server with a 24-core Intel Xeon Gold 2.10 GHz CPU and 768 GB RAM. We consider two evaluation metrics: the reward and the exploitability. The reward measures the overall performance of the learned strategy in terms of its ability to achieve the defender’s objective in each use case. For the definitions of the reward functions, see Appendix D. The exploitability, on the other hand, measures the distance of the learned strategies from a Nash equilibrium, where an exploitability of 0 means that the strategies are in equilibrium. Baselines We compare the reinforcement learning algorithms listed in Table 4 against two baseline methods, namely phasic policy gradient (PPG) (Cobbe et al., 2021) and neural fictitious The Cyber Security Learning Environment (CSLE) 0102030405060 0 50 100 150 SPSAPPOThreshold Cumulative discounted reward Training time (min) 0102030405060 −0.8 −0.6 −0.4 −0.2 0 PPOPPG Average reward Training time (min) 0102030405060 −80 −60 −40 −20 RolloutPPO Snort Cumulative discounted reward Training time (min) 020406080100 2 4 FP. w. SPSA NFSP Exploitability Training time (min) 01,0002,0003,0004,0005,0006,000 2 4 6 8 FP. w. PPO NFSP Exploitability Training time (min) 0102030405060 0 0.5 1 PPO NFSP Exploitability Training time (min) Flow control (POMDP) Replication control (MDP)Recovery control (POMDP) Flow control (Markov game)Network segmentation (Markov game)Replication control (Markov game) Figure 11.Performance comparison between reinforcement learning methods and baseline strategies in the digital twin. Curves show the mean values from evaluations with5random seeds; shaded areas indicate standard deviations. The top row relates to the decision-theoretic models. The bottom row relates to the game-theoretic models. The acronym FP stands for fictitious play. self-play (NFSP) (Heinrich & Silver, 2016). Additionally, we compare the performance of the reinforcement learning methods with that of two static security strategies: a thresh- old flow control strategy and a recovery strategy based on the Snort intrusion detection and prevention system with community ruleset v2.9.17.1 (Roesch, 1999). The threshold strategy blocks network flows when the de- fender’s belief (probability) that the system is compromised exceeds a predefined thresholdα = 0.75. The belief is computed according to the POMDP model, which main- tains a probabilistic estimate of the underlying system state based on the sequence of observed system events. When the belief that the system is in a compromised state surpasses the threshold, the strategy proactively blocks network flows in order to limit potential attacker movement and prevent further propagation of the compromise. The Snort baseline follows a rule-based recovery strategy that represents a signature-based response mechanism. This strategy recovers a component of the target system (e.g., by redeploying it in a new virtual machine) when a Snort alert with priority medium or higher is generated. 8.2 Evaluation Results The evaluation results are summarized in Figs. 10–11. The red and blue curves in Fig. 10 represent the results from the simulator and the digital twin, respectively. An analysis of these curves leads us to the following conclusions. The learning curves converge to nearly constant mean values for all use cases and evaluation metrics. From this observation, we conclude that the learned strategies have also converged. Although the learned strategies, as expected, perform slightly better on the simulator than on the digital twin, we are encouraged by the fact that the curves of the digital twin are close to those of the simulator (cf. the blue and red curves). This small performance gap reflects inevitable discrepancies between the simulation model and the digital twin, such as differences in network latency, background processes, or unmodeled system dynamics. Figure 11 shows a comparison between different reinforce- ment learning methods and baseline security strategies. We observe that the performance varies substantially. Overall, the results show that the reinforcement learning strategies significantly outperform the static strategies. 8.3 Discussion The experimental results demonstrate that CSLE effectively enables the transfer of reinforcement learning-based secu- rity strategies from simulation to a digital twin that closely approximates an operational environment. The small perfor- mance gap observed between the simulator and the digital twin indicates that the identified simulation models capture the main dynamics of the target system. This transferability is a key step toward operational deployment, as it validates that security strategies learned in simulation remain effec- tive when tested under realistic operating conditions. For safety and operational reasons, we have not evaluated the learned strategies in a production environment. While the digital twin replicates the software, configuration, and timing behavior of a production environment, further study is needed to determine whether the learned strategies main- The Cyber Security Learning Environment (CSLE) tain comparable performance in a production environment. 8.4 Sensitivity to Model Misspecification The effective transfer of the learned strategies from simula- tion to the digital twin indicates that the identified simulation dynamics capture the main characteristics of the target sys- tem. However, in practice, model misspecification may still arise due to factors such as measurement noise or changes in system behavior over time (e.g., data drift). To better un- derstand the impact of such modeling inaccuracies, in this section, we analyze the sensitivity of the learned strategies to the misspecification of the system model. From a theoretical perspective, the performance loss of learned strategies due to deviations between the model and the system dynamics is upper-bounded as follows. Proposition 8.1 (Model misspecification error bound). Let ̃ f denote the dynamics of the system model [cf.(1)] and letfdenote the dynamics of the target system. Denote by J π and ̃ J π the value functions (i.e., the expected rewards) under a strategy pairπ = (π D ,π A )in the target system and in the simulation, respectively. If the dynamics satisfy X s ′ ∈S f (s ′ | s,a (D) ,a (A) )− ̃ f (s ′ | s,a (D) ,a (A) ) ≤ α, for all states and actions, and some constant α. Then ∥J π − ̃ J π ∥ ∞ ≤ αγβ (1− γ) 2 , where γ < 1 is the discount factor and β is defined by β =max s∈S,a (D) ∈A D ,a (A) ∈A A |r(s,a (D) ,a (A) )|, whereris the reward function,Sis the state space, and (A D ,A A ) are the action spaces. We present the proof of Prop. 8.1 in the supplementary material (Appendix E). This proposition states that the mis- specification error grows proportionally with the error of the discrepancy between the state transitions in the system model and the digital twin, as quantified by the parameter α. In practice, this parameter can be estimated by com- paring the simulated state trajectories of the model and the trajectories observed in the digital twin. While Prop. 8.1 provides a worst-case bound on the im- pact of model misspecification, it is conservative and does not necessarily reflect the sensitivity of a specific system in- stance. To complement the theoretical analysis, we therefore conduct an empirical sensitivity analysis on the flow control model used in our experiments. In this model, the probabil- ity that the attacker successfully compromises the system is governed by a parameterp∈ [0, 1]; see Appendix D.1 in the supplementary material for details. In the digital twin, we configure this parameter asp = 0.01. To introduce model misspecification, we varypin the simulation model and evaluate how the performance of the learned defender strategy changes as the discrepancy between the simulation model and the digital twin increases. The results of this sensitivity analysis are summarized in Fig. 12. 00.010.020.030.040.050.060.070.080.09 50 100 SimulationDigital twin Cumulative discounted reward Misspecification|p− ̃p| Figure 12.Analysis of the sensitivity to model misspecification in the flow control use case. Numbers and error bars indicate the mean and the standard deviation from 5 evaluations. Figure 12 shows that small modeling errors lead to notice- able differences between the performance predicted by the simulation and the performance observed in the digital twin. However, these discrepancies in the simulated performance translate only to small variations in the performance of the learned defender strategy when it is evaluated in the digital twin. This indicates that, although the model is sensitive to misspecification when estimating performance, the learned strategy itself is relatively robust to such errors. 9CONCLUSION In this paper, we present CSLE, a comprehensive research platform for autonomous security management through re- inforcement learning. This platform addresses the system- level challenges that arise in the operation and experimenta- tion with reinforcement learning in networked systems. In particular, it is based on a novel methodology for learning security strategies that combines a digital twin with system identification and simulation-based reinforcement learning. Our evaluation across four security use cases demonstrates that this methodology narrows the gap between simulated and practical performance of learned security strategies. Future Work Future work will focus on further developing the CSLE plat- form and expanding its open-source ecosystem. We plan to continue improving the platform’s usability by adding more learning resources, documentation, and example con- figurations to facilitate adoption by both researchers and practitioners. So far, our experimental validation of CSLE has focused primarily on IT systems. In future work, we aim to extend CSLE to cyberphysical systems. The Cyber Security Learning Environment (CSLE) REFERENCES Akbari, I., Tahoun, E., Salahuddin, M. A., Limam, N., and Boutaba, R. ATMoS: Autonomous threat mitigation in SDN using reinforcement learning. In NOMS IEEE/IFIP Network Operations and Management Symposium, p. 1–9, 2020. doi: 10.1109/NOMS47738.2020.9110426. Allen B. Danette. Digital twins and living models at NASA, 2021. Digital Twin Summit. Anderson, H. S., Kharkar, A., Filar, B., Evans, D., and Roth, P. Learning to evade static PE machine learning malware models via reinforcement learning, 2018. URL https://arxiv.org/abs/1801.08917. Andrew, A., Spillard, S., Collyer, J., and Dhir, N. De- veloping optimal causal cyber-defence agents via cyber security simulation. In Proceedings of the ML4Cyber workshop, ICML 2022, Baltimore, USA, July 17-23, 2022. PMLR, 2022. Bertsekas, D. P. Rollout, Policy Iteration, and Distributed Reinforcement Learning. Athena scientific optimization and computation series. Athena Scientific., 2021. ISBN 9781886529076. Biesinger, F. and Weyrich, M. The facets of digital twins in production and the automotive industry. In 2019 23rd International Conference on Mechatronics Technology (ICMT), p. 1–6, 2019. doi: 10.1109/ICMECT.2019. 8932101. Blum, W. Gamifying machine learning for stronger security and AI models, 2021. Brown, G. W. Iterative solution of games by fictitious play, 1951. Activity analysis of production and allocation. Carrasco, J. A. F., Pagola, I. A., Urrutia, R. O., and Ro- man, R. CYBERSHIELD: A competitive simulation environment for training AI in cybersecurity. In 2024 11th International Conference on Internet of Things: Sys- tems, Management and Security (IOTSMS), p. 11–18, 2024. doi: 10.1109/IOTSMS62296.2024.10710208. Chen, Y., Shetty, M., Somashekar, G., Ma, M., Simmhan, Y., Mace, J., Bansal, C., Wang, R., and Rajmohan, S. AIOpsLab: a holistic framework to evaluate AI agents for enabling autonomous clouds.https://mlsys.org/ virtual/2025/poster/3285, 2025. MLSys 2025 Poster. Clemm, A. and Cisco Systems, I. Network Management Fundamentals. Cisco Press fundamentals series. Cisco Press, 2007. ISBN 9781587201370. Cobbe, K. W., Hilton, J., Klimov, O., and Schulman, J. Phasic policy gradient. In Proceedings of the 38th In- ternational Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, p. 2020–2027. PMLR, 18–24 Jul 2021. Cubukcu, U. et al. Citus: Distributed PostgreSQL for data- intensive applications. In Proceedings of the 2021 Inter- national Conference on Management of Data, SIGMOD ’21, 2021. Dempster, A. P., Laird, N. M., and Rubin, D. B. Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society: Series B, 39: 1–38, 1977. URLhttp://web.mit.edu/6.435/ w/Dempster77.pdf. Emerson, H., Bates, L., Hicks, C., and Mavroudis, V. Cy- bORG++: An enhanced gym for the development of au- tonomous cyber agents, 2024. URLhttps://arxiv. org/abs/2410.16324. Glivenko, V. I. and Cantelli, F. P. Sulla determinazione empirica delle leggi di probabilità, 1933. Giorn. Ist. Ital. Attuari (in Italian). 4: 92–99. Google. Google remote procedure call, 2022. Hammar,K.Cybersecuritylearningenvi- ronment(CSLE),2023.URLhttps:// kim-hammar.github.io/csle/ . Documentation: https://kim-hammar.github.io/csle/, traces:https://github.com/Kim-Hammar/ csle/releases/tag/v0.4.0 ,sourcecode: https://github.com/Kim-Hammar/csle, video demonstration:https://w.youtube. com/watch?v=iE2KPmtIs2A&. Hammar, K. and Stadler, R. Finding effective security strate- gies through reinforcement learning and Self-Play. In International Conference on Network and Service Man- agement (CNSM 2020), Izmir, Turkey, 2020. Hammar, K. and Stadler, R. Intrusion prevention through optimal stopping. IEEE Transactions on Network and Service Management, 19(3):2333–2348, 2022. doi: 10. 1109/TNSM.2022.3176781. Hammar, K. and Stadler, R. Scalable learning of intrusion response through recursive decomposition. In Fu, J., Kroupa, T., and Hayel, Y. (eds.), Decision and Game Theory for Security, p. 172–192, Cham, 2023. Springer Nature Switzerland. ISBN 978-3-031-50670-3. Hammar, K. and Stadler, R. Intrusion tolerance for net- worked systems through two-level feedback control. In 2024 54th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN), p. 338– 352, 2024a. doi: 10.1109/DSN58291.2024.00042. The Cyber Security Learning Environment (CSLE) Hammar, K. and Stadler, R. Learning near-optimal intrusion responses against dynamic attackers. IEEE Transactions on Network and Service Management, 21(1):1158–1177, 2024b. doi: 10.1109/TNSM.2023.3293413. Hammar, K. and Stadler, R. Intrusion tolerance as a two- level game. In Sinha, A., Fu, J., Zhu, Q., and Zhang, T. (eds.), Decision and Game Theory for Security, p. 3–23, Cham, 2025. Springer Nature Switzerland. ISBN 978-3-031-74835-6. Hammar, K., Li, Y., Alpcan, T., Lupu, E. C., and Bertsekas, D. Adaptive network security policies via belief aggrega- tion and rollout, 2025. URLhttps://arxiv.org/ abs/2507.15163. Heinrich, J. and Silver, D. Deep reinforcement learning from self-play in imperfect-information games, 2016. URL https://arxiv.org/abs/1603.01121. Hemminger, S. Network emulation with NetEm. Linux Conf, 2005. Huynh Phuong Thanh, N., Chen, Z., Hasegawa, K., Fukushima, K., and Beuran, R. PenGym: Pentesting training framework for reinforcement learning agents. p. 498–509, 01 2024. doi: 10.5220/0012367300003648. ISC2.2024 Cybersecurity workforce study.Tech- nical report, ISC2, October 2024.URLhttps: //w.isc2.org/Insights/2024/10/ ISC2-2024-Cybersecurity-Workforce-Study. Based on an online survey of 15,852 cybersecurity pro- fessionals conducted in April–May 2024 in collaboration with Forrester Research Inc.; accessed 2025-09-07. Janisch, J., Pevný, T., and Lisý, V. NASimEmu: Network attack simulator & emulator for training agents generaliz- ing to novel scenarios, 2023. Jha, S., Arora, R. R., Watanabe, Y., Yanagawa, T., Chen, Y., Clark, J., Bhavya, B., Verma, M., Kumar, H., Kita- hara, H., Zheutlin, N., Takano, S., Pathak, D., George, F., Wu, X., Turkkan, B. O., Vanloo, G., Nidd, M., Dai, T., Chatterjee, O., Gupta, P., Samanta, S., Aggarwal, P., Lee, R., Ahn, J.-W., Kar, D., Paradkar, A., Deng, Y., Moogi, P., Mohapatra, P., Abe, N., Narayanaswami, C., Xu, T., Varshney, L. R., Mahindru, R., Sailer, A., Shwartz, L., Sow, D., Fuller, N. C. M., and Puri, R. ITBench: Evalu- ating AI agents across diverse real-world IT automation tasks. In Singh, A., Fazel, M., Hsu, D., Lacoste-Julien, S., Berkenkamp, F., Maharaj, T., Wagstaff, K., and Zhu, J. (eds.), Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, p. 27134–27197. PMLR, 13–19 Jul 2025. URLhttps://proceedings.mlr. press/v267/jha25a.html. Kaloroumakis, P. E. and Smith, M. J. Toward a knowledge graph of cybersecurity countermeasures. The MITRE Corporation, 11:2021, 2021. Kearns, M. and Singh, S. Near-optimal reinforcement learning in polynomial time. Machine Learning, 49 (2):209–232, Nov 2002. ISSN 1573-0565. doi: 10. 1023/A:1017984413808. URLhttps://doi.org/ 10.1023/A:1017984413808. Kiely, M., Ahiskali, M., Borde, E., Bowman, B., et al. CAGE Challenge 4: a scalable multi-agent reinforce- ment learning gym for autonomous cyber defence. AI Magazine, 46(e70021), 2025. doi: 10.1002/aaai.70021. Kreps, J. Kafka : a distributed messaging system for log processing. 2011. Kushida, T. and Shibata, Y. Empirical study of inter-arrival packet times and packet losses. In Proceedings of the 22nd International Conference on Distributed Computing Systems, p. 233–240, 2002. ISBN 0769515886. Lanier, M. and Vorobeychik, Y. CyGym: a simulation- based game-theoretic analysis framework for cybersecu- rity, 2025. URLhttps://arxiv.org/abs/2506. 21688. Li, L., Fayad, R., and Taylor, A. CyGIL: A cyber gym for training autonomous agents over emulated network systems, 2021. Li, T., Hammar, K., Stadler, R., and Zhu, Q. Conjectural on- line learning with first-order beliefs in asymmetric infor- mation stochastic games. In 2024 IEEE 63rd Conference on Decision and Control (CDC), p. 6780–6785, 2024. doi: 10.1109/CDC56724.2024.10886479. Liu, Y. et al. A novel cloud-based framework for the elderly healthcare services using digital twin. IEEE Access, 2019. Mahalingam, M. et al. Virtual eXtensible local area network (VXLAN): A framework for overlaying virtualized layer 2 networks over layer 3 networks, 2014. URLhttps: //w.rfc-editor.org/rfc/rfc7348. McKeown, N., Anderson, T., Balakrishnan, H., Parulkar, G., Peterson, L., Rexford, J., Shenker, S., and Turner, J.OpenFlow: Enabling innovation in campus net- works. SIGCOMM Comput. Commun. Rev., p. 69–74, mar 2008. ISSN 0146-4833. doi: 10.1145/1355734. 1355746.URLhttps://doi.org/10.1145/ 1355734.1355746. Merkel, D. Docker: lightweight linux containers for consis- tent development and deployment. Linux journal, 2014 (239):2, 2014. The Cyber Security Learning Environment (CSLE) Molina-Markham, A., Miniter, C., Powell, B., and Rid- ley, A. Network environment design for autonomous cyberdefense. 2021.https://arxiv.org/abs/ 2103.07583. Nguyen, T. T. and Reddi, V. J. Deep reinforcement learning for cyber security. IEEE Transactions on Neural Net- works and Learning Systems, 34(8):3779–3795, 2023. doi: 10.1109/TNNLS.2021.3121870. Oakley, L. and Oprea, A. QFlip: An adaptive reinforcement learning strategy for the FlipIt security game. In Alpcan, T., Vorobeychik, Y., Baras, J. S., and Dán, G. (eds.), Decision and Game Theory for Security, p. 364–384, Cham, 2019. Springer International Publishing. ISBN 978-3-030-32430-8. Oesch, S., Chaulagain, A., Weber, B., Dixson, M., Sadovnik, A., Roberson, B., Watson, C., and Austria, P. Towards a high fidelity training environment for autonomous cy- ber defense agents. In Proceedings of the 17th Cy- ber Security Experimentation and Test Workshop, CSET ’24, p. 91–99, New York, NY, USA, 2024. Associa- tion for Computing Machinery. ISBN 9798400709579. doi: 10.1145/3675741.3675752. URLhttps://doi. org/10.1145/3675741.3675752. Paxson, V.End-to-end internet packet dynamics.In IEEE/ACM Transactions on Networking, p. 277–292, 1997. Pfaff, B., Pettit, J., Koponen, T., Jackson, E., Zhou, A., Rajahalme, J., Gross, J., Wang, A., Stringer, J., Shelar, P., Amidon, K., and Casado, M.The de- sign and implementation of Open vSwitch. In 12th USENIX Symposium on Networked Systems Design and Implementation (NSDI 15), p. 117–130, Oakland, CA, May 2015. USENIX Association. ISBN 978-1- 931971-218.URLhttps://w.usenix.org/ conference/nsdi15/technical-sessions/ presentation/pfaff. Red Hat. Ansible, 2024. Roesch, M. Snort - lightweight intrusion detection for net- works. In Proceedings of the 13th USENIX Conference on System Administration, LISA ’99, p. 229–238, USA, 1999. USENIX Association. Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O.Proximal policy optimization algo- rithms.2017.URLhttp://arxiv.org/abs/ 1707.06347. Schwartz, J., Kurniawati, H., and El-Mahassni, E. POMDP + information-decay: Incorporating defender’s behaviour in autonomous penetration testing.Pro- ceedings of the International Conference on Auto- mated Planning and Scheduling, 30(1):235–243, Jun. 2020.URLhttps://ojs.aaai.org/index. php/ICAPS/article/view/6666. Song, W., Li, X., Afroz, S., Garg, D., Kuznetsov, D., and Yin, H. MAB-Malware: A reinforcement learning frame- work for blackbox generation of adversarial malware. In Proceedings of the 2022 ACM on Asia Conference on Computer and Communications Security, ASIA CCS ’22, p. 990–1003, New York, NY, USA, 2022. Associa- tion for Computing Machinery. ISBN 9781450391405. doi: 10.1145/3488932.3497768. URLhttps://doi. org/10.1145/3488932.3497768. Spall, J. C. Multivariate stochastic approximation using a simultaneous perturbation gradient approximation. IEEE TRANSACTIONS ON AUTOMATIC CONTROL, 37(3): 332–341, 1992. Standen, M., Lucas, M., Bowman, D., Richer, T. J., Kim, J., and Marriott, D. CybORG: A gym for the development of autonomous cyber agents. CoRR, 2021.https: //arxiv.org/abs/2108.09118. Strom, B. E., Applebaum, A., Miller, D. P., Nickels, K. C., Pennington, A. G., and Thomas, C. B. MITRE ATT&CK: Design and philosophy. In Technical report. The MITRE Corporation, 2018. Tao, F., Zhang, H., Liu, A., and Nee, A. Y. C. Digital twin in industry: State-of-the-art. IEEE Transactions on Industrial Informatics, 2019. Terranova, F., Lahmadi, A., and Chrisment, I. Scalable and Generalizable RL Agents for Attack Path Discov- ery via Continuous Invariant Spaces. In 2025 28th In- ternational Symposium on Research in Attacks, Intru- sions and Defenses (RAID), p. 18, Gold Coast, Aus- tralia, October 2025. URLhttps://hal.science/ hal-05182437. The MITRE Corporation. CVE database, 2022. URL https://cve.mitre.org/. Towers, M., Kwiatkowski, A., Terry, J., Balis, J. U., Cola, G. D., Deleu, T., Goulão, M., Kallinteris, A., Krimmel, M., KG, A., Perez-Vicente, R., Pierré, A., Schulhoff, S., Tai, J. J., Tan, H., and Younis, O. G. Gymnasium: A standard interface for reinforcement learning envi- ronments, 2024. URLhttps://arxiv.org/abs/ 2407.17032. Wu, Y., Zhang, K., and Zhang, Y. Digital twin networks: A survey. IEEE Internet of Things Journal, 8(18):13789– 13804, 2021. doi: 10.1109/JIOT.2021.3079510. The Cyber Security Learning Environment (CSLE) SUPPLEMENTARY MATERIAL This document provides supplementary material for the pa- per “CSLE: A Reinforcement Learning Platform for Au- tonomous Security Management”, published at the Machine Learning and Systems (MLSys) conference 2026. It con- tains the following appendices. • Appendix A: Management System Frontend. –This appendix provides example screenshots of the management system in CSLE. • Appendix B: Target System Configurations. –This appendix provides the detailed configura- tions of the target systems in Fig. 9. • Appendix C: Hyperparameters. –This appendix provides the hyperparameters that we use to instantiate the reinforcement learning algorithms in the experimental evaluation. • Appendix D: Simulation Models. –This appendix provides details about the simula- tion models used in the experimental evaluation. • Appendix E: Proof of Prop. 8.1. – This appendix provides the proof of Prop. 8.1. • Appendix F: Example: Observation Mapping. – This appendix provides an illustrative example of how we map log events (e.g., security alerts) to observations that are input to the reinforcement learning strategies. • Appendix G: Example: Digital twin configuration. – This appendix provides an example configuration of a digital twin in CSLE. • Appendix H: Artifacts appendix. –This appendix provides details about the artifacts associated with this paper. AMANAGEMENT SYSTEM FRONTEND Three screenshots of the web interface to the management system in CSLE are shown in Figs. 13–15. A video demon- stration is available at (Hammar, 2023). The backend is implemented as a REST API in Python based on the Flask framework. The frontend is implemented in JavaScript based on the React framework. BTARGET SYSTEM CONFIGURATIONS The configuration of the target system in Fig. 9.a is available in Table 5. (Note that each component in Fig. 9 is labeled with an identifierN i .) Similarly, the configuration of target systems in Fig. 9.b and Fig. 9.c are available in Table 6 and Table 7, respectively. CHYPERPARAMETERS The hyperparameters that we use to instantiate the reinforce- ment learning algorithms are listed in Table 8. Parameter(s)Value(s) Discount factor γ0.99 SPSA (Spall, 1992) c,ε,λ,A,a1, 0.101, 0.602, 100, 1 Rollout (Bertsekas, 2021) Rollout horizon20 Lookahead horizon1 Monte-Carlo samples20 PPO (Schulman et al., 2017) Learning rate, # hidden layers5.148· 10 −5 , 1 # Neurons/layer64 # Steps between updates2048 Batch size, discount factor γ16, 0.99 GAE λ, clip range, entropy coefficient0.95, 0.2, 2· 10 −4 Value coefficient, max gradient norm0.102, 0.5 Table 8. Hyperparameters used for the experimental evaluation. DSIMULATION MODELS In this appendix, we detail the models used to instantiate the simulations for the use cases described in §8. D.1 Flow Control POMDP We formulate the flow control use case when the attacker follows a static strategy as a POMDP, which is defined by the following nine-tuple ⟨S,A,f,r,γ,b 1 ,T,O,z⟩.(2) The POMDP evolves in time steps fromt = 1tot = T, which constitutes one episode. The constantγ ∈ [0, 1]is a discount factor,Sis the set of states, andAis the set of actions. The initial state is drawn fromb 1 ∈ ∆(S)and f (s t+1 | s t ,a t )is the probability of transitioning from state s t to states t+1 when taking actiona t . The set of obser- vations is denoted byOandz(o t | s t )is the observation function, where o t ∈O. The Cyber Security Learning Environment (CSLE) ID (s)Operating systemServicesVulnerabilities 1Ubuntu 20Snort (community ruleset v2.9.17.1), SSH- 2Ubuntu 20SSH, HTTP Erl-Pengine, DNSWeak password 4Ubuntu 20HTTP, Telnet, SSHWeak password 10Ubuntu 20FTP, MongoDB, SMTP, Tomcat, Teamspeak 3, SSHWeak password 12Debian JessieTeamspeak 3, Tomcat, SSHCVE-2010-0426, Weak password 17Debian WheezyApache 2, SNMP, SSHCVE-2014-6271 18Debian 9.2IRC, Apache2, SSHSQL injection 22Debian JessieProFTPd, SSH, Apache2, SNMPCVE-2015-3306 23Debian JessieApache2, SMTP, SSHCVE-2016-10033 24Debian JessieSSHCVE-2015-5602, Weak password 25Debian JessieElasticsearch, Apache 2, SSH, SNMPCVE-2015-1427 27Debian JessieSamba, NTP, SSHCVE-2017-7494 3,11,5-9Ubuntu 20SSH, SNMP, PostgreSQL, NTP- 13− 16,19− 21,26,28− 31Ubuntu 20NTP, IRC, SNMP, SSH, PostgreSQL- Table 5. Configuration of the target system in Fig. 9.a. ID(s)TypeOperating systemZoneServicesVulnerabilities 1GatewayUbuntu 20-Snort (ruleset v2.9.17.1), SSH, OpenFlow v1.3, Ryu SDN controller- 2 GatewayUbuntu 20DMZSnort (ruleset v2.9.17.1), SSH, OVS v2.16, OpenFlow v1.3- 28GatewayUbuntu 20R&DSnort (ruleset v2.9.17.1), SSH, OVS v2.16, OpenFlow v1.3- 3,12 SwitchUbuntu 22DMZSSH, OpenFlow v1.3 , OVS v2.16- 21, 22SwitchUbuntu 22-SSH, OpenFlow v1.3, OVS v2.16- 23SwitchUbuntu 22AdminSSH, OpenFlow v1.3, OVS v2.16- 29-48SwitchUbuntu 22R&DSSH, OpenFlow v1.3, OVS v2.16- 13-16HoneypotUbuntu 20DMZSSH, SNMP, PostgreSQL, NTP- 17-20HoneypotUbuntu 20DMZSSH, IRC, SNMP, SSH, PostgreSQL- 4 App nodeUbuntu 20DMZHTTP, DNS, SSHCWE-1391 5, 6App nodeUbuntu 20DMZSSH, SNMP, PostgreSQL, NTP- 7App nodeUbuntu 20DMZHTTP, Telnet, SSHCWE-1391 8App nodeDebian JessieDMZFTP, SSH, Apache 2, SNMPCVE-2015-3306 9,10App nodeUbuntu 20DMZNTP, IRC, SNMP, SSH, PostgreSQL- 11App nodeDebian JessieDMZApache 2, SMTP, SSHCVE-2016-10033 24 Admin systemUbuntu 20AdminHTTP, DNS, SSHCWE-1391 25Admin systemUbuntu 20AdminFTP, MongoDB, SMTP, Tomcat, Teamspeak 3, SSH- 26Admin systemUbuntu 20AdminSSH, SNMP, Postgres, NTP- 27Admin systemUbuntu 20AdminFTP, MongoDB, SMTP, Tomcat, Teamspeak 3, SSHCWE-1391 49-59Compute nodeUbuntu 20R&DSpark, HDFS- 60Compute nodeDebian WheezyR&DSpark, HDFS, Apache 2, SNMP, SSHCVE-2014-6271 61Compute nodeDebian 9.2R&DIRC, Apache 2, SSHCWE-89 62Compute nodeDebian JessieR&DSpark, HDFS, Teamspeak 3, Tomcat, SSHCVE-2010-0426 63Compute nodeDebian JessieR&DSSH, Spark, HDFSCVE-2015-5602 64Compute nodeDebian JessieR&DSamba, NTP, SSH, Spark, HDFSCVE-2017-7494 Table 6. Configuration of the target system in Fig. 9.b. ID(s)Operating systemBackground servicesVulnerabilities 1Debian 9.2Apache2CWE-89 2Debian JessieFTPCVE-2015-3306 3Ubuntu 20SSH, SparkCWE-1391 4Debian JessiePhpmailerCVE-2016-10033 5Debian WheezyNginxCVE-2014-6271 6Debian JessieSSH, GRPCCWE-1391, CVE-2010-0426 7Debian JessieSSH, Spring bootCVE-2015-5602, CWE-1391 8Debian JessiePostgreSQL, SambaCVE-2017-7494 Table 7.Configuration of the target system in Fig. 9.c. (The number of replicas is dynamically scalable. When starting a new replica, its configuration is selected randomly from the table.) The Cyber Security Learning Environment (CSLE) Figure 13.Screenshot of the web interface to the management system in CSLE. The figure shows the page for configuring emulations (i.e., digital twins). In addition to this page, the web interface allows viewing reinforcement learning experiments, debugging learned security strategies, real-time monitoring of digital twins, management of simulations, and integration with large language models. The Cyber Security Learning Environment (CSLE) Figure 14. Screenshot of the web interface to the management system in CSLE. The figure shows the list of available pages. ActionsAThe defender has two actions: (S)top and (C)ontinue. The action space is thusA = S, C. We encodeSwith1andCwith0to simplify the formal de- scription below. Each stop is associated with a flow control action, and the objective is to decide the optimal times for stopping. The number of stops the defender must execute to prevent an intrusion isL≥ 1, which is a predefined param- eter of our use case. The number of stop actions remaining is denoted by l∈1,...,L. StatesSThe states t is0if no intrusion occurs and1if an intrusion is ongoing. The terminal state∅is reached after the defender takes the final stop action. The state space is thusS = 0, 1,∅. The initial state iss 1 = 0. Hence, b 1 ∈ ∆(S) is the degenerate distribution b 1 (0) = 1. ObservationsOThe defender has a partial view of the system and observeso t = (∆x t , ∆y t , ∆z t ), where∆x t , ∆y t , and∆z t are bounded counters that denote the number of severe IDS alerts, warning IDS alerts, and login attempts generated during time step t, respectively. Transition functionf l (s ′ | s,a)We model the start of an intrusion by a Bernoulli process(Q t ) T t=1 , whereQ t ∼ Ber(p) is a Bernoulli random variable withp > 0. The first occurrence ofQ t = 1defines the intrusion start timeI, which thus is geometrically distributed. Consequently, we can define the transition function as f 1 (∅|·, 1) = f l (∅|∅,·) = 1,(3a) f l (0| 0,a) = 1− p,if l− a > 0,(3b) f l (1| 0,a) = p,if l− a > 0,(3c) f l (1| 1,a) = 1,if l− a > 0,(3d) wherea∈A. 23 All other state transitions occur with proba- bility0. Equation (3a) defines the transitions to the terminal state∅, which is reached when the final stop action is taken (i.e., whenl = 1anda = 1). If (3a) is not applicable, i.e., if the system does not reach the terminal state, then the tran- sitions are defined by (3b)-(3d). Equation (3b) captures the case where no intrusion occurs; (3c) specifies the case when the intrusion starts; and (3d) describes the case where an intrusion is in progress. Note that the intrusion states = 1 is absorbing until L stop actions have been taken. Observation functionz(o t | s t )We estimate the distri- butions of IDS alerts and login attempts using data from the digital twin. At the end of every30s interval on the twin, we collect the metrics∆x,∆y,∆z, which contain the alerts and login attempts that occurred during the interval. We useM = 21, 000i.i.d. samples to compute the empiri- cal distributionbz(· | s t )as an estimate ofz(· | s t ), where bz a.s. → z as M →∞ (Glivenko & Cantelli, 1933). Figure 16 2 Recall that we encode(S, C) = (1, 0); hencel t+1 = l t − a t . 3 We define p = 0.01 for the evaluation reported in the paper. The Cyber Security Learning Environment (CSLE) Figure 15.Screenshot of the web interface of the strategy debugger in CSLE, which allows the user to interactively step through a POMDP episode. The left panel shows the defender’s view, with infrastructure statistics updated in real time. The right panel shows the attacker’s view, which consists of partial knowledge of the system under attack. ‘ 5000100001500020000 ̂ z (∆ y | s ) ∆y # Warningidsalerts∆y 20406080100120140160180200220240260280300320 no intrusion (s= 0)intrusion (s= 1) ̂ z (∆ z | s ) ∆z # Login attempts∆z 2468101214161820 ̂ z (∆ x | s ) ∆x # Severeidsalerts∆x Figure 16.Estimated (smoothed) distributions of severe IDS alerts∆x(top row), warning IDS alerts∆y(middle row), and login attempts ∆z (bottom row) for the flow control POMDP. The distributions are estimated based on measurements from the digital twin. The Cyber Security Learning Environment (CSLE) shows some of the estimated (smoothed) distributions. The distributions during normal operation and intrusion overlap. However, the distributions during intrusions tend to have more probability mass at larger values of ∆x, ∆y, and ∆z. Reward functionr(s,a)The objective is to maintain service on the infrastructure while preventing a possible intrusion. Therefore, we define the reward function to give the maximum reward if the defender maintains service until the intrusion starts and then prevents the intrusion by taking Lstop actions. The reward per time stepr(s,a)is parame- terized by the reward that the defender receives for stopping an intrusion (R st > 0), the reward for maintaining service (R sla > 0), and the loss of being intruded (R int < 0): r(∅,·) = 0,(4a) r(s, C) = R sla + sR int L , s∈0, 1,(4b) r(s, S) = sR st L ,s∈0, 1.(4c) Equation (4a) states that the reward in the terminal state is zero. Equation (4b) states that the defender receives a positive reward (R sla ) for maintaining service and a loss ( R int L ) for each time step that it is under intrusion. Lastly, (4c) indicates that each stop incurs a cost by interrupting service (i.e., noR sla ) and possibly a reward ( R st L ) if it affects an ongoing intrusion. Time horizonTThe time horizonTis a random variable that indicates the timet > 1when the terminal state∅is reached. Theoretical analysis A detailed theoretical analysis of this POMDP can be found in (Hammar & Stadler, 2022). D.2 Flow Control Markov Game We formulate the flow control use case when the attacker follows a dynamic strategy as a partially observed zero-sum stochastic game (Markov game). The game follows similar dynamics as the POMDP defined above and has two players: the (D)efender and the (A)ttacker. In the following, we describe the components of the game, its evolution, and the players’ objectives. Time horizonTThe time horizonT > 1is a random variable representing the time when the attacker stops its intrusion or is prevented, depending on which event occurs first. State spaceSThe game has three states:s t = 0if no intrusion occurs,s t = 1if an intrusion is ongoing, and s T =∅if the game has ended. Hence,S =0, 1,∅. The initial state iss 1 = 0. Therefore, the initial state distribution b 1 ∈ ∆(S) is the degenerate distribution b 1 (0) = 1. Action spacesA k Each playerk ∈ Ncan invoke two actions: (S)top and (C)ontinue. The action spaces are thus A D =A A =S, C. Executing actionStriggers a change in the game, while actionCis a passive action. The attacker can invoke the stop action twice: the first to start the intru- sion and the second to terminate it. The defender can invoke the stop actionL≥ 1times. Each invocation corresponds to a defensive action against a possible intrusion. The number of stop actions remaining to the defender at timetis known to both players and is denoted byl t ∈ 1,...,L. Using the encoding(S, C) = (1, 0), we can writel t+1 = l t −a (D) t , wherea (D) t is the defender action at timet. At each time step, the attacker and the defender simultaneously choose their actions a t = (a (D) t ,a (A) t ), where a (k) t ∈A k . Observation spaceOThe attacker has complete observ- ability and knows the game state, the defender’s actions, and the defender’s observations. In contrast, the defender has a limited set of observationso t ∈O, whereOis a finite set. In our use case,o t relates to the weighted sum of IDS alerts triggered during time step t. Transition functionf l (s ′ | s,a (D) ,a (A) )At each time stept, a transition froms t tos t+1 occurs with probability f l (s t+1 | s t ,a (D) t ,a (A) t ), where f l is defined as f l>1 (0| 0, S, C) = f l (0| 0, C, C) = 1,(5a) f l>1 (1| 1,·, C) = f l (1| 1, C, C) = 1− φ l ,(5b) f l>1 (1| 0,·, S) = f l (1| 0, C, S) = 1,(5c) f l>1 (∅| 1,·, C) = f l (∅| 1, C, C) = φ l ,(5d) f l=1 (∅|·, S,·) = f l (∅|∅,·,·) = f l (∅| 1,·, S) = 1. (5e) All other state transitions have probability0. Equations (5a)–(5b) define the probabilities of the recurrent transitions 0→ 0and1→ 1. The game stays in state0with probability 1if the attacker selects actionCandl t −a (D) t > 0. Similarly, the game stays in state1with probability1 − φ l if the attacker chooses actionCandl t − a (D) t > 0. Here,φ l denotes the probability that the defender stops the intrusion, which is a parameter of the use case. The intrusion can be stopped at any time step, either because the attacker terminates the intrusion or as a consequence of previous stop actions by the defender, i.e., the effect of a defensive action is non-immediate. We assume thatφ l increases with each stop action that the defender takes. Equation (5c) captures the transition0→ 1, which occurs when the attacker chooses actionSandl t −a (D) t > 0. (5d)– (5e) define the probabilities of the transitions to the terminal state∅, which is reached in three cases: (i) whenl t = 1 The Cyber Security Learning Environment (CSLE) 5000100001500020000 ̂ z ( o t | s t = 0) o t Estimated distribution ofidsalerts weighted by priorityo t 5000100001500020000 ̂ z ( o t | s t = 1) o t Figure 17.Fitted Gaussian mixture models ofzwhen no intrusion occurs (s t = 0) and during intrusion (s t = 1) for the flow control Markov game. and the defender takes the final stop actionS(i.e., when l t − a (D) t = 0 ); (i) when the intrusion is stopped by the defender with probabilityφ l ; and (i) whens t = 1and the attacker terminates the intrusion (a (A) t = S = 1). Reward functionr l (s,a (D) ,a (A) )At time stept, the defender receives the rewardr t = r l (s t ,a (D) t ,a (A) t ) and the attacker receives the reward−r t . The reward func- tion is parameterized by the defender’s reward for stopping an intrusion (R st > 0), its cost of taking a defensive ac- tion (R cost < 0), and its cost while an intrusion occurs (R int < 0), as defined below. r l (∅,·) = 0,(6a) r l (1,·, S) = 0,(6b) r l (0, C,·) = 0,(6c) r l (0, S,·) = R cost l t , l t ∈1, 2,...,L,(6d) r l (1, S, C) = R st l t ,l t ∈1, 2,...,L,(6e) r l (1, C, C) = R int .(6f) Equations (6a)–(6b) state that the reward is zero in the ter- minal state and when the attacker terminates an intrusion. Equation (6c) states that the defender incurs no cost when no attack occurs and it does not take a defensive action. Equation (6d) indicates that the defender incurs a cost when taking a defensive action if no intrusion is ongoing. Equa- tion (6e) states that the defender receives a reward when taking a stop action while an intrusion occurs. Lastly, (6f) indicates that the defender incurs a cost for each time step during which an intrusion occurs. Observation functionzWe estimate the observation func- tionzbased on data from the digital twin. Specifically, at the end of every time step in the digital twin, i.e., at the end of each30s interval, we collect the number of IDS alerts with priorities1–4that occurred during the time step, where priorities1–4refer to the Snort priorities “very low”, “low”, “medium”, and “high”, respectively (Roesch, 1999) 4 . We do so for23, 000time steps, which provides us with a dataset to estimate the distribution of IDS alerts. Using this dataset, we apply expectation-maximization (Dempster et al., 1977) to fit Gaussian mixture distributionsbz(·| 0)andbz(·| 1)as estimates ofz(· | 0)andz(· | 1), which represent the true observation distributions in the target infrastructure. Figure 17 shows the fitted models over the discrete obser- vation spaceO = 0, 1,..., 22000. We note thatbz(·| 0) andbz(· | 1)are (discretized) Gaussian mixtures with one and three components, respectively. Both mixtures have the most probability mass within0–5000. The distribution bz(· | 1)also has substantial probability mass at larger val- ues. Theoretical analysis A detailed theoretical analysis of this game can be found in (Hammar & Stadler, 2024b). D.3 Network Segmentation Markov Game We formulate the network segmentation use case when the attacker follows a dynamic strategy as a partially observed zero-sum stochastic game (Markov game). The game has two players: the (D)efender and the (A)ttacker. The game is played on an IT infrastructure with application servers connected by a communication network that is seg- mented into zones. Overlaid on this physical infrastructure is a virtual infrastructure with a tree structure that includes nodes, which collectively offer services to clients. A service is modeled as a workflow, which comprises a set of interde- pendent nodes. A dependency between two nodes reflects information exchange through service invocations. 4 Note that according to Snort’s terminology (Roesch, 1999), 1is the highest priority. We invert the labeling in our model for convenience. The Cyber Security Learning Environment (CSLE) In the following, we describe the components of the game, its evolution, and the players’ objectives. Infrastructure We model the virtual infrastructure as a (finite) directed graphG = ⟨gw ∪ V,E⟩. The graph has a tree structure rooted at the gatewaygw. Each node i∈Vhas three state variables. Variablev (R) i,t represents the reconnaissance state. We havev (R) i,t = 1if the attacker has discovered the node,0otherwise. Variablev (I) i,t represents the intrusion state. We havev (I) i,t = 1if the attacker has compromised the node,0otherwise. Lastly, variablev (Z) i,t indicates the zone in which the node resides. We call a node active if it is functional as part of a workflow (denoted α i,t = 1). Due to a defender action (e.g., a shutdown), a nodei ∈ Vmay become inactive (α i,t = 0). The active state is determined by v (Z) i,t , i.e., α i,t is a function of v (Z) i,t . Workflows We model a workfloww ∈ Was a subtree G w =⟨gw∪V w ,E w ⟩of the infrastructure graph. Work- flows do not overlap except for the gateway, which belongs to all workflows. AttackerAt each timet, the attacker takes an actiona (A) t , which is defined as the composition of the local actions on all nodesa (A) t = (a (A) 1,t ,...,a (A) |V|,t ) ∈ A A , where the set A A is finite. A local action is either a null action (denoted with⊥) or an offensive action. An offensive action on a nodeimay change the reconnaissance statev (R) i,t or the intrusion statev (I) i,t . A nodeican only be compromised if it is discovered, i.e., ifv (R) i,t = 1. We express this constraint as a (A) t ∈A A (s (A) t ). The attacker state s (A) t = v (R) i,t ,v (I) i,t i∈V ∈S A evolves as s (A) t+1 ∼ f A ·| s (A) t ,a (A) t ,a (D) t ,(7) wherea (D) t represents the defender action at timet, as de- fined below. Defender At each timet, the defender takes actiona (D) t , which is defined as the composition of the local actions on all nodesa (D) t = (a (D) 1,t ,...,a (D) |V|,t ) ∈ A D , where the set A D is finite. A local action is either a defensive action or a null action⊥. Each defensive actiona (D) i,t ̸= ⊥leads to s (A) i,t+1 = (0, 0) and may affect v (Z) i,t+1 . The defender state s (D) t = v (Z) i,t i∈V ∈S D evolves as s (D) t+1 ∼ f D ·| s (D) t ,a (D) t .(8) ClientsClients consume infrastructure services by access- ing workflows. We model client behavior through stationary stochastic processes, which affect the observations available to the attacker and the defender. That is, the clients are implicitly modeled by the observation functionz, as defined next. Observability and strategiesAt each timet, the defender and the attacker both observeo t = o 1,t ,...,o |V|,t ∈ O , whereOis finite 5 . The observationo t is drawn from the random vectorO t = (O 1,t ,...,O |V|,t )whose marginal dis- tributionsz O 1 ,...,z O |V| are stationary and conditionally independent givens i,t = (s (D) i,t ,s (A) i,t ). (Note thatz O i de- pends on the traffic generated by clients.) As a consequence, the joint conditional distribution z(o| s) is given by z o| s = |V| Y i=1 z O i o i | s i ∀o∈O,s∈S A ×S D . (9) The sequence of observations and states at times1,...,t forms the histories h (D) t = (b (D) 1 ,s (D) 1 ,a (D) 1 ,o 2 ,...,a (D) t−1 ,s (D) t ,o t )∈H D , h (A) t = (b (A) 1 ,s (A) 1 ,a (A) 1 ,o 2 ,...,a (A) t−1 ,s (A) t ,o t )∈H A , wheres (D) ∼ b (D) 1 ands (A) ∼ b (A) 1 are the initial state distributions. Based on their respective histories, the defender and the attacker select actions according to their strategies. The de- fender’s behavior strategy is defined asπ D ∈ Π D =H D → ∆(A D ) and the attacker’s behavior strategy is defined as π A ∈ Π A =H A → ∆(A A ). Defender ObjectiveWhen selecting the strategyπ D , the defender must balance two conflicting objectives: maximiz- ing the workflow utility towards its clients and minimiz- ing the cost of intrusion. The weightη ≥ 0controls the trade-off between these two objectives, which results in the bi-objective J = ∞ X t=1 γ t−1 X w∈W X i∈V w ηu (W) i,t | z workflow utility − c (I) i,t |z intrusion cost , (10) whereγ ∈ [0, 1)is a discount factor,c (I) i,t = v (I) i,t +c (A) (a (D) i,t ) is the intrusion cost associated with nodeiat timet 6 , and u (W) i,t expresses the workflow utility associated with nodei at time t. 5 In our use case,o i,t relates to the number of intrusion alerts associated with node i. 6 c (A) is a non-negative function that represents the operational costs of defender actions. The Cyber Security Learning Environment (CSLE) Markov Game When the game starts att = 1,s (D) 1 and s (A) 1 are sampled fromb (D) 1 andb (A) 1 , respectively. A play of the game proceeds in time stepst = 1, 2,.... At each timet, the defender observesh (D) t and the attacker observes h (A) t . Based on these histories, both players select actions according to their respective strategies, i.e.,a (D) t ∼ π D (·| h (D) t ) anda (A) t ∼ π A (·| h (A) t ). As a result of these actions, five events occur at timet + 1: (i)o t+1 is sampled fromz; (i)s (D) t+1 is sampled fromf D ; (i)s (A) t+1 is sampled fromf A ; (iv) the defender receives the rewardr(s t ,a (D) t ); and (v) the attacker receives the reward−r(s t ,a (D) t ), where the reward functionris defined by the expression within brackets in (10). Theoretical analysis A detailed theoretical analysis of this game can be found in (Hammar & Stadler, 2023). D.4 Replication Control MDP We formulate the replication control use case when the attacker follows a static strategy as the following MDP. States We define the states t to represent the expected number of healthy replicas at timet. The state space is S =0, 1,...,s max and the initial state is s 1 = N 1 . Actions The actiona t = 1means that a new replica is added to the system;a t = 0otherwise. Hence, the action space is a t ∈0, 1 =A. Transition probabilities The state s t evolves as s t+1 ∼ f (·| s t ,a t ).(11) We estimate the conditional probability distributionfbased on measurements from the digital twin. A subset of the estimated conditional distributions is illustrated in Fig. 18. 2468101214161820 s = 0s = 10s = 20 0 s ′ f (s ′ | s, 0) Figure 18. Transition function for the replication control MDP. Objective Increasing the replication factors t improves service availabilityT (A) but increases cost. (T (A) is the fraction of time steps where service is available.) The goal of the controller is thus to find the optimal cost-redundancy trade-off, i.e., to minimize J = lim T→∞ " T X t=1 a t T # ,subject to T (A) ≥ ε A ,(12) whereε A is the chosen lower bound on service availability. For instance, ifε A = 0.999, then at most8.4hours of service disruption per year is allowed. Theoretical analysis A detailed theoretical analysis of this MDP can be found in (Hammar & Stadler, 2024a). D.5 Replication Control Markov Game The Markov game model of the replication control use case follows the same model as the MDP, except that the attacker’s strategy is dynamic. Specifically, the attacker can control which nodes in the system to attack, which causes them to be compromised with probabilityp A = 0.01. Hence, the attacker’s strategy influences the transition func- tion in(11). The objective of the attacker is diametrically opposed to the controller, i.e., it is a zero-sum game. Theoretical analysis A detailed theoretical analysis of this game can be found in (Hammar & Stadler, 2025). D.6 Recovery Control POMDP The recovery control use case involves a networked system withKservice replicas. We formulate this use case as the following POMDP. StatesEach replica has two states:1(compromised) or0 (safe), i.e.,s = (s 1 ,...,s K )wheres l ∈ 0, 1. Compro- mises occur randomly over time and incur operational costs. The transition probabilities are defined as follows. If replica lis compromised (s l = 1), then it remains so until recovery is applied (a l = 1), at which point the states l is set to0. Otherwise, the probability that it becomes compromised ismin0.2(1 +N l (s)), 1, whereN l (s)is the number of compromised neighbors of replica l in the network. Actions An action is defined as a vectora= (a 1 ,...,a K ) , where eacha l determines whether to recover componentl(a l = 1) or take no action (a l = 0). The goal is to determine an optimal recovery strategyπ ⋆ that balances security requirements against recovery costs. Observations Intrusion detection systems generate secu- rity alertso = (o 1 ,...,o K )that provide partial indications of the replicas’ states. We define the observation distribution as p(o| s,a) = K Y l=1 p(o l | s l ), for all o∈O,s∈S,a∈A, The Cyber Security Learning Environment (CSLE) where eachp(o l | s l )is estimated based on measurements from the digital twin. A subset of the estimated distributions is shown in Fig. 19. Rewards We define the reward function as r(s,a) =− K X l=1 intrusion cost z| 2s l (1− a l ) + recovery cost z | a l (1− s l ) ,(13) i.e., negative rewards are incurred for unmitigated intrusions (s l = 1) and unnecessary recovery actions (a l = 1and s l = 0). Theoretical analysis A detailed theoretical analysis of this POMDP can be found in (Hammar et al., 2025). EPROOF OF PROPOSITION 8.1 The proof follows the same chain of reasoning as the proof of the simulation lemma in (Kearns & Singh, 2002). For notational brevity, we useπ(s)as a shorthand for (π D (s),π A (s)).We start by expanding the difference | ̃ J π (s)− J π (s)| as | ̃ J π (s)− J π (s)| = r(s,π(s)) + γ X s ′ ∈S ̃ f (s ′ | s,π(s)) ̃ J π (s ′ )− r(s,π(s)) + γ X s ′ ∈S f (s ′ | s,π(s))J π (s ′ ) = γ X s ′ ∈S ̃ f (s ′ | s,π(s)) ̃ J π (s ′ )− γ X s ′ ∈S f (s ′ | s,π(s))J π (s ′ ) = γ X s ′ ∈S ̃ f (s ′ | s,π(s)) ̃ J π (s ′ )− γ X s ′ ∈S f (s ′ | s,π(s))J π (s ′ )+ γ X s ′ ∈S ̃ f (s ′ | s,π(s))J π (s ′ )− γ X s ′ ∈S ̃ f (s ′ | s,π(s))J π (s ′ ) = γ X s ′ ∈S ̃ f (s ′ | s,π(s)) ̃ J π (s ′ )− J π (s ′ ) + γ X s ′ ∈S ̃ f (s ′ | s,π(s))− f (s ′ | s,π(s)) J π (s ′ ) ≤ γ∥ ̃ J π − J π ∥ ∞ + γ X s ′ ∈S ̃ f (s ′ | s,π(s))− f (s ′ | s,π(s)) J π (s ′ ) (a) ≤ γ∥ ̃ J π − J π ∥ ∞ + γ X s ′ ∈S ̃ f (s ′ | s,π(s))− f (s ′ | s,π(s)) β 1− γ ≤ γ∥ ̃ J π − J π ∥ ∞ + γαβ 1− γ , where (a) follows because|J π (s)|≤ P ∞ t=0 γ t β = β 1−γ and the fact that|ab| =|a||b|(we use the triangle inequality to move the absolute value inside the sum). Since this upper bound holds for any state s, we have ∥ ̃ J π − J π ∥ ∞ ≤ γ∥ ̃ J π − J π ∥ ∞ + γαβ 1− γ =⇒ ∥ ̃ J π − J π ∥ ∞ − γ∥ ̃ J π − J π ∥ ∞ ≤ γαβ 1− γ =⇒ (1− γ)∥ ̃ J π − J π ∥ ∞ ≤ γαβ 1− γ =⇒ ∥ ̃ J π − J π ∥ ∞ ≤ γαβ (1− γ) 2 . FEXAMPLE: OBSERVATION MAPPING Consider the flow control use case (Fig. 9.a). Assume that the attacker attempts a brute-force SSH login against a server in the target system. The mapping from raw events in the digital twin to observations that are input to the security strategy proceeds in three stages. Stage 1 - raw log events The attacker launches an SSH brute-force attack against a server with IP 172.31.2.10. This produces raw log entries on the target host, e.g., in /var/log/auth.log: Mar 12 09:14:01 srv2 sshd[4821]: Failed password for root from 172.31.1.42 port 49822 ssh2 Mar 12 09:14:02 srv2 sshd[4821]: Failed password for root from 172.31.1.42 port 49822 ssh2 ... Simultaneously, the network traffic is captured by the moni- toring agent, which records packet-level statistics (e.g., flow byte counts and connection durations). Stage 2 - Detection alertsThe Snort IDS, running on the same network segment, matches the traffic against its ruleset and produces alerts: [ ** ] [1:2001219:20] ET SCAN Potential SSH Brute Force [ ** ] [Priority: 2] 03/12-09:14:05.003 172.31.1.42:49822 -> 172.31.2.10:22 The Cyber Security Learning Environment (CSLE) 02004006008001,000 Safe (s i = 0)Compromised (s i = 1) 02004006008001,00002004006008001,00002004006008001,000 Number of failure alerts o 1 Number of failure alerts o 2 Number of failure alerts o 3 Number of failure alerts o 4 ProbabilityProbabilityProbabilityProbability Figure 19. Empirical observation distributions for the recovery control use case based on measurements from the digital twin. [Classification: Attempted Information Leak] [ ** ] [1:2003068:7] ET SCAN Potential SSH Login Attempt [ ** ] [Priority: 2] 03/12-09:14:18.112 172.31.1.42:49830 -> 172.31.2.10:22 [Classification: Misc Attack] The monitoring agent on the host pushes both the raw met- rics and the IDS alerts to the Kafka event bus. A data pipeline then consumes these events, aggregates them over a fixed monitoring interval (e.g., 15 seconds), and writes the results to the metastore. Stage 3 - Mapping to observations At the end of each monitoring interval, the aggregated data is transformed into a numerical observation that serves as input to the security strategy learned through reinforcement learning. For the flow control POMDP, the observation at time step t is: o t = (number of severe alerts, number of warning alerts, number of login events). For the example above, supposing the monitoring interval covers the attack window, one realization might be: o t = (1, 9, 4). This observation is then used to update the defender’s belief state b t , which is input to the security strategy π D . GEXAMPLE: DIGITAL TWIN CONFIGURATION A complete configuration of a digital twin of the system in Fig. 9.a is available athttps: //github.com/Kim-Hammar/csle/blob/ master/emulation-system/envs/090/level_ 9/config.py. HARTIFACT APPENDIX H.1 Abstract All results presented in this paper are fully reproducible using open-source software and data. To enable independent verification of our results and encourage future research, we release a complete set of artifacts that allow the community to build upon our work without additional engineering effort. Specifically, we provide the following artifacts: •The source code of CSLE, our reinforcement learning platform for autonomous security management. Avail- able at:https://github.com/Kim-Hammar/ csle. •Pre-built Docker images for creating digital twins. Available at:https://hub.docker.com/u/ kimham. •A video that demonstrates how to install CSLE. Avail- able at:https://w.youtube.com/watch? v=l_g3sRJwwhc. •A video that demonstrates the web interface of CSLE. Available at:https://w.youtube. com/watch?v=iE2KPmtIs2A. • A dataset of attack statistics generated with CSLE. Available at:https://github.com/ Kim-Hammar/csle/releases/download/ v0.4.0/statistics_dataset_14_nov_ 22_json.zip. •Platformdocumentation.Availableat: https://github.com/Kim-Hammar/csle/ blob/master/releases(PDF version) and https://kim-hammar.github.io/csle/ (web version). H.2 Artifact check-list (meta-information) •Dataset:https://github.com/Kim-Hammar/ csle/releases/download/v0.4.0/ statistics_dataset_14_nov_22_json.zip. • Publicly available?:Yes,https://github.com/ Kim-Hammar/csleandhttps://hub.docker. com/u/kimham. • Code licenses (if publicly available)?: C-BY-SA 4.0. • Data licenses (if publicly available)?: C-BY-SA 4.0. •Archived:Yes,https://doi.org/10.5281/ zenodo.18869003. The Cyber Security Learning Environment (CSLE) H.3 Description H.3.1 How delivered The code, data, and documentation are available on GitHub:https://github.com/Kim-Hammar/ csle.The video demonstrations are available on YouTube:https://w.youtube.com/watch?v= l_g3sRJwwhcandhttps://w.youtube.com/ watch?v=iE2KPmtIs2A. The Docker images are avail- able on DockerHub:https://hub.docker.com/u/ kimham. H.3.2 Hardware dependencies Minimum hardware requirements to run CSLE are: 16GB memory (RAM), 1 CPU, and 50GB disk space. H.3.3 Software dependencies Docker, Python, PostgreSQL, Prometheus, cAdvisor, Grafana, Node, and NPM. A comprehensive list of soft- ware dependencies is available in the platform documenta- tion:https://github.com/Kim-Hammar/csle/ blob/master/releases. H.3.4 Datasets A dataset of attack statistics generated with CSLE is avail- able at:https://github.com/Kim-Hammar/ csle/releases/download/v0.4.0/ statistics_dataset_14_nov_22_json.zip. H.4 Installation The installation is automated through the Ansible playbooks available at:https://github.com/Kim-Hammar/ csle/tree/master/ansible. H.5 Evaluation and expected result To validate the availability of our artifacts,per- form the following steps:(i) access the code, doc- umentation, and data athttps://github.com/ Kim-Hammar/csle; (i) access the video demonstra- tions athttps://w.youtube.com/watch?v= l_g3sRJwwhcandhttps://w.youtube.com/ watch?v=iE2KPmtIs2A; and (i) access the Docker images at https://hub.docker.com/u/kimham.