Paper deep dive
CyberWorld: World Models for Sample-Efficient Autonomous Cyber Defense
Ryozo Masukawa, Sanggeon Yun, Raheeb Hassan, Hyunwoo Oh, SungHeon Jeong, Mohsen Imani
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 9/29/2026, 12:20:39 PM
Summary
The paper introduces CyberWorld, a Dreamer-style world modeling framework for autonomous cyber defense that learns latent cyber dynamics from multimodal representations (vector, graph, text, image) to enable sample-efficient policy optimization. Evaluated on the CyberWheel benchmark, the graph-based variant significantly outperforms model-free PPO in sample efficiency (requiring 3.6k-15.8k steps vs millions) and demonstrates robustness against topology-dependent attacks, while maintaining scalability across network sizes of 15 to 100 hosts.
Entities (7)
Relation Signals (5)
CyberWorld → uses → CyberWheel
confidence 95% · Using CyberWheel (Oesch et al., 2024), we evaluate three aspects of CyberWorld
CyberWorld → buildsupon → DreamerV3
confidence 92% · CyberWorld is a multimodal latent world model tailored for autonomous cyber defense, built upon the DreamerV3 framework
Graph Representation → outperforms → PPO
confidence 90% · the graph-based CyberWorld variant exceeds a strategy-agnostic control after 3.6k-15.8k environment steps, compared with millions of steps required by model-free PPO.
CyberWorld → utilizes → Recurrent State-Space Model
confidence 88% · a Recurrent State-Space Model (RSSM) ... turns the sequence of embeddings and defender actions into a latent state
CyberWheel → supports → MITRE ATT&CK
confidence 85% · CyberWheel ... scripted adversaries progress through MITRE ATT&CK- and Atomic Red Team-inspired attack stages
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Deep reinforcement learning has become a prominent approach to autonomous cyber defense. Existing methods are predominantly model-free and consequently require extensive environment interaction. World models provide an alternative by learning predictive dynamics and optimizing policies through imagined trajectories, yielding substantial gains in sample efficiency in robotics and embodied control. Extending this paradigm to cybersecurity raises a fundamental question: what should constitute the "world" in a cyber world model? We introduce CyberWorld, a Dreamer-style world modeling framework that learns latent cyber dynamics from vector, graph, textual, and multimodal representations of the defended network. Across all four scoreable CyberWheel attack strategies, the graph-based CyberWorld variant exceeds a strategy-agnostic control after 3.6k-15.8k environment steps, compared with millions of steps required by model-free PPO. Across representation choices, graph structure provides greater robustness under topology-dependent attacks, while simpler representations remain competitive in overall performance. Among successful runs, the number of episodes required to reach the control remains approximately constant as network size increases from 15 to 100 hosts. These results establish learned cyber dynamics as a sample-efficient and scalable basis for autonomous defense, and identify world representation as a central design axis for robustness and scalability.
Tags
Links
- Source: https://arxiv.org/abs/2609.31893v1
- Canonical: https://arxiv.org/abs/2609.31893v1
Trouble viewing inline? Open PDF directly →
Full Text
84,117 characters extracted from source content.
Expand or collapse full text
Preprint CYBERWORLD:WORLD MODELS FOR SAMPLE- EFFICIENT AUTONOMOUS CYBER DEFENSE Ryozo Masukawa 1 , Sanggeon Yun 1 , Raheeb Hassan 1 , Hyunwoo Oh 1 , SungHeon Jeong 1 , Mohsen Imani 1 1 University of California, Irvine rmasukaw@uci.edu ABSTRACT Deep reinforcement learning has become a prominent approach to autonomous cyber defense. Existing methods are predominantly model-free and consequently require extensive environment interaction. World models provide an alternative by learning predictive dynamics and optimizing policies through imagined trajectories, yielding substantial gains in sample efficiency in robotics and embodied control. Extending this paradigm to cybersecurity raises a fundamental question: what should constitute the “world” in a cyber world model? We introduce CyberWorld, a Dreamer-style world modeling framework that learns latent cyber dynamics from vector, graph, textual, and multimodal representations of the defended net- work. Across all four scoreable CyberWheel attack strategies, the graph-based CyberWorld variant exceeds a strategy-agnostic control after3.6k–15.8k environ- ment steps, compared with millions of steps required by model-free PPO. Across representation choices, graph structure provides greater robustness under topology- dependent attacks, while simpler representations remain competitive in overall performance. Among successful runs, the number of episodes required to reach the control remains approximately constant as network size increases from 15 to 100 hosts. These results establish learned cyber dynamics as a sample-efficient and scalable basis for autonomous defense, and identify world representation as a central design axis for robustness and scalability. code is available here 1 . 1INTRODUCTION Reinforcement learning (RL) (Sutton et al., 1998) has emerged as a prominent approach to au- tonomous cyber defense, where agents make sequential decisions to protect networked systems against evolving threats (Stakhanova et al., 2007; Kiely et al., 2025). In autonomous cyber defense, it is common to rely on model-free RL algorithms (Schulman et al., 2017; Dutta et al., 2023b; Singh et al., 2025; King et al., 2025; Kazeminajafabadi et al., 2026), directly optimizing defensive policies through repeated environment interaction. However, this prevailing paradigm is increasingly inadequate in the face of modern offensive capabilities. With frontier agentic models such as Claude Mythos (Anthropic, 2026), autonomous adversaries can increasingly reason over complex systems and exploit zero-day vulnerabilities (Zhu et al., 2026), intensifying the challenge faced by purely reactive DRL defenders. Recent evidence demonstrates this fragility. Masukawa et al. (2026) showed that even compact agentic language models (Belcak et al., 2025) can autonomously discover and ex- ploit blind spots in state-of-the-art DRL blue agents. Separately, McFadden et al. (2026) systematized the pitfalls of RL-based cyber defense, finding that existing defenses frequently fail to account for non-stationary adversaries, inadequately address partial observability, and lack demonstrated policy convergence. Beyond these robustness issues, model-free RL can also be sample-inefficient, requiring substantial environment interaction for learning and adaptation (Egorov & Shpilman, 2022). Meanwhile, world models have advanced rapidly in robotics, visual control, and embodied intelli- gence (Hafner et al., 2023; Zhang et al., 2023; Alonso et al., 2024). These methods learn predictive environment dynamics and optimize decisions through imagined trajectories (Hafner et al., 2023; 1 https://anonymous.4open.science/r/CyberWorld-768D/README.md 1 arXiv:2609.31893v1 [cs.LG] 25 Sep 2026 Preprint Burchi & Timofte, 2025b). This separation between dynamics modeling and policy learning is particularly attractive in cyber defense because the two components change at different rates. An attacker may change its strategy from one campaign to the next, but the network protocols, services, and detection mechanisms through which both attacker and defender act, that is, the world itself, do not (Oesch et al., 2024; Kiely et al., 2025). A model of these shared dynamics therefore remains valid when the adversary changes, whereas a reactive policy fitted to one adversary must be relearned. Server Host A Host B Host C Host D Firewall Imagine rollouts Learn dynamics ℎ 푡 =푓 휙 (ℎ 푡−1 ,푧 푡−1 ,푎 푡−1 퐵 ) 푧 푡 ~푞 휙 (푧 푡 |ℎ 푡 ,푒 푡 ) Ƹ푧 푡 ~푝 휙 (푧 푡 |ℎ 푡 ) ep 1ep 2ep 3 Learn with fewer real-world interactions How should we represent the CyberWorld? Text Logs & alerts Graph Network Topology & Relations Vector Simple embedding Image GUI rendering Need more Samples! Model-Free cyber defense Many real environment interactions are needed. CyberWorld A learnable world model of cyberspace capturing dynamics Real Cyber Environment Slow learning with many real interactions Higher Sample efficiency Figure 1: CyberWorld learns predictive cyber dynamics to enable more sample-efficient autonomous defense. Recent work further extends world models to visual, lan- guage, and multimodal repre- sentations (Zhou et al., 2025; Lin et al., 2024; Mazzaglia et al., 2024), which suggests that cyber dynamics, too, may be learnable from heteroge- neous observations and that a learned model may enable substantially more sample- efficient autonomous defense. Transferring world models to cybersecurity, however, first requires deciding what the cy- ber “world” is. Unlike visual control, where observations take standardized forms such as pixels or proprioceptive signals (Hafner et al., 2023; Zhou et al., 2025), a cyber defender observes heterogeneous information, including host attributes, network topology, alerts, logs, and their temporal histories, and the suitability of a given representation may change with the scale and structure of the network. The central question of this work is therefore: What should constitute the “world” in a world model for autonomous cyber defense? To investigate this question, we introduce CyberWorld, a Dreamer-style (Hafner et al., 2020; 2021; 2023) world model for autonomous cyber defense. CyberWorld learns latent cyber dynamics and trains defensive policies through imagined trajectories. Within a shared world-model framework, we study five representations of cyberspace: per-host vectors, host graphs, their combination, natural- language telemetry, and vector–text fusion with an added frame-reconstruction objective. Using CyberWheel (Oesch et al., 2024), we evaluate three aspects of CyberWorld: how cyber representations affect world-model-based control, how they scale as the network grows from 15 to 100 hosts, and whether world-model-based defense can substantially improve sample efficiency over model-free PPO (Schulman et al., 2017). We also examine end-of-training reward stability to distinguish fast initial learning from stable defensive behavior. Our experiments demonstrate a substantial reduction in environment interaction. Across all four scoreable attack strategies, the graph-based CyberWorld variant reaches the control within3.6k–15.8k steps, while PPO requires2.3M–3.1M steps or fails within its3.2M-step budget. Vector and graph representations perform comparably on most attackers, and the graph representation is the most reliable on the topology-following attacker. On larger networks, successful runs require a similar number of episodes to reach the control, although the fixed interaction budget becomes restrictive at 100 hosts. End-of-training reward stability varies across representations, indicating that rapid acquisition of a defensive behavior does not by itself imply convergence. Our contributions are threefold: •We introduce CyberWorld, to the best of our knowledge, the first Dreamer-style world- model framework for autonomous cyber defense, learning predictive cyber dynamics and training defensive policies through imagined trajectories. •We investigate which representation of the cyber world is appropriate for world modeling by comparing various representations, and further study the vector and graph representations across network scales; the results highlight the benefit of explicit graph structure when attack progression depends on network topology. 2 Preprint •We demonstrate that CyberWorld can achieve effective cyber defense with orders of magni- tude fewer environment interactions than model-free PPO. 2BACKGROUND & RELATED WORKS 2.1FROM MODEL-FREE REINFORCEMENT LEARNING TO MULTIMODAL WORLD MODELS RL formulates sequential decision making through interactions between an agent and its environment (Sutton et al., 1998). Modern deep RL has been dominated by model-free approaches such as PPO (Schulman et al., 2017), which optimize policies directly from collected experience without explicitly learning environment dynamics. While effective, this paradigm can require substantial environment interaction, motivating renewed interest in model-based learning for improved sample efficiency. PlaNet (Hafner et al., 2019) showed that latent dynamics learned from high-dimensional observations can support planning. Building on this, the Dreamer family further established latent imagination as a practical framework for learning policies primarily inside a predictive world model (Hafner et al., 2020; 2021; 2023). Recent work has expanded this framework with Transformer-based dynamics and more accurate latent prediction (Micheli et al., 2023; Robine et al., 2023; Burchi & Timofte, 2025a). At the same time, world models have begun to move beyond primarily visual observations: Lin et al. (2024) jointly model language and vision for imagined policy learning, while Mazzaglia et al. (2024) connect multimodal foundation representations with latent environment dynamics. Structured representations have also emerged as an alternative to monolithic latent states, including causal graph formulations for explicit state reasoning (Zhou et al., 2026). These developments motivate CyberWorld’s central question of how heterogeneous cyber observations, including vectors, network topology, text, and visual interfaces, should be represented within a predictive world model. 2.2MODEL-BASED AUTONOMOUS CYBER DEFENSE Model-based cyber defense has traditionally relied on environment dynamics that are manually specified or explicitly estimated rather than learned as general predictive latent models. Prior work has used transition, observation, or causal models to support belief tracking, planning, and defensive decision making (Le & Stadler, 2025; Miehling et al., 2018; Andrew et al., 2022), including domain- specific Partially Observable Markov Decision Process (POMDP) dynamics updated online from observations (Dutta et al., 2023a). In contrast, most recent DRL approaches learn defensive policies directly from interaction without explicitly modeling how the cyber environment evolves (Dutta et al., 2023b). Recent work has begun to bridge these two paradigms by learning selected aspects of the environment. Kazeminajafabadi et al. (2026), for example, infer latent attacker context to condition defensive policies while leaving the underlying environment dynamics implicit; latent dynamics have also been used for MCTS-based cyber-defense planning (Li et al., 2026). Closest to CyberWorld, Karacelebi et al. (2026) learn a graph-structured world model for networked systems with recurrent latent dynamics and imagined rollouts. Their setting, however, models non-adversarial network evolution. To the best of our knowledge, CyberWorld is the first Dreamer-style world model for autonomous cyber defense, learning predictive latent dynamics that capture the interaction between attacker behavior, defender interventions, and evolving network state. 3CYBERWORLD CyberWorld is a multimodal latent world model tailored for autonomous cyber defense, built upon the DreamerV3 framework (Hafner et al., 2023) and adapted to the discrete, sparse-reward, and relational topology of enterprise network defense. The agent constructs a predictive model of the environment from replayed interactions and trains its policy entirely inside the model’s imagination, decoupling the sample complexity of behavior learning from the high computational overhead of interacting with live network simulators. Its cyber-specific components are an environment-independent intermediate representation of the defender’s view, encoders for heterogeneous cyber observations (host vectors, the host graph, and telemetry text) that are fused into a single latent embedding, and modality-specific reconstruction objectives that determine what the latent state must preserve (Figure 2). 3.1PROBLEM FORMULATION We instantiate our formulation in CyberWheel (Oesch et al., 2024), a configurable simulation environment for autonomous cyber defense. CyberWheel models enterprise networks with explicit hosts, subnets, routing, services, and firewall constraints, while scripted adversaries progress through MITRE ATT&CK- and Atomic Red Team-inspired attack stages (MITRE, 2026; Red Canary, 2026). The defender receives only detector alerts and the state of its own assets, and responds through cyber 3 Preprint deception by deploying or removing decoy hosts along the adversary’s likely path. Compared with other cyber RL environments (Microsoft Defender Research Team., 2021; Janisch et al., 2023; Kiely et al., 2025), CyberWheel is particularly well suited to controlled world-model evaluation because it combines a centralized single-defender setting with configurable attacker strategies, detector models, and network topologies. This design avoids additional confounds from multi-agent coordination and credit assignment (Lyu et al., 2021; Egorov & Shpilman, 2022; Zhang et al., 2025; Xue et al., 2026; Zhao et al., 2026), while allowing the same simulation substrate to support our representation and scalability studies. We formalize blue-agent network defense as a POMDP(S,A B ,O,T , Ω,R,γ), whereSrepresents the underlying global environment states (network topology, host compromise status, decoy configura- tions, and latent adversarial intent),A B denotes the defender’s action set,Ois the observation space, Tspecifies the global transition dynamics,Ω(o | s)maps states to partial defender observations, R(s,a B ) is the reward function, and γ ∈ [0, 1) is the temporal discount factor. In enterprise network defense, the transition dynamicsTgovern the coupled execution of defender countermeasures and adversarial tactics. Leta B t ∈A B denote the blue defender action,a R t ∈A R denote the concurrent red adversary action, andc ∈ Crepresent an adversary strategy. From the defender’s perspective, the effective state transitionT c (s t+1 | s t ,a B t ) factors into: T c (s t+1 | s t ,a B t ) = X a R t ∈A R T mech (s t+1 | s t ,a B t ,a R t )π R c (a R t | b R t ),(1) whereT mech captures the underlying cyber mechanisms (protocol semantics, service configura- tions, vulnerability exploits, and deception capture constraints), which constrain the attacker as much as the defender, since an adversary can exploit a host only through the protocols and services the network exposes, whileπ R c reflects the adversary’s stochastic policy conditioned on their internal beliefb R t . When an adversary shifts tactics (c → c ′ ),π R c changes, induc- ing an apparent shift in overall dynamicsT c → T c ′ . However,T mech is shared across strate- gies under this formulation. A model-free policy does not explicitly preserve this shared tran- sition factor, whereas a latent world model separates dynamics prediction from policy learn- ing and can, in principle, retain mechanism information across changes in adversary strategy. Input modalities / encoders 1 Vector encoder Embedding Node Features 풙 (풗) ∈ℝ |푉|×푁 푓푒푎푡 Graph encoder Graph Neural Networks (GNNs) 퐺=(푉,퐸) Host graphs Text encoder (optional) Sentence Encoder (e.g. BERT) 풙 (풕) Telemetry sentences GUI frame encoder (optional) Vision Encoder (e.g. ViT) 풙 (풇) GUI rendered frame 푻 풕 (푣) ∈ℝ 푁 푣 ×푑 푻 풕 (푔) ∈ℝ 푁 푣 ×푑 푻 풕 (푡) ∈ℝ 푁 푡 ×푑 푻 풕 (푓) ∈ℝ 푁 푓 ×푑 Cross Modal Fusion 2 Decoder / prediction heads 4 Per-stream tokens Per modality Self attention ෩ 푼 풕 = ෩ 푼 풕 풗 ; ෩ 푼 풕 푔 ; ෩ 푼 풕 풕 ; ෩ 푼 풕 풇 ∈ℝ 푁×푑 ,푁= 푁 푣 +푁 푔 +푁 푡 +푁 푓 Concatenate Fused Streams 푻 풕 (푣) Self-Attention 푻 풕 (푔) Self-Attention 푻 풕 (푡) Self-Attention 푻 풕 (푓) Self-Attention Cross Attention 푄 (푣) Cross Attention 푄 (푔) Cross Attention 푄 (푡) Cross Attention 푄 (푓) Latent Cross Attention (풆 풕 : ) Fixed Query (풒) Recurrent State Space Model 3 ℎ 푡−1 Prior 푝 휙 (푧 푡−1 |ℎ 푡−1 ) Posterior 푞 휙 (푧 푡−1 |ℎ 푡−1 ,푒 푡−1 ) 푎 푡−1 푒 푡−1 ℎ 푡 Prior 푝 휙 (푧 푡 |ℎ 푡 ) Posterior 푞 휙 (푧 푡 |ℎ 푡 ,푒 푡 ) 푎 푡 푒 푡 ℎ 푡+1 Prior 푝 휙 (푧 푡+1 |ℎ 푡+1 ) Posterior 푞 휙 (푧 푡+1 ℎ 푡+1 ,푒 푡+1 ) 푎 푡+1 푒 푡+1 z 푡−1 z 푡 z 푡+1 Decoder World-model state representation 푠 푡 =[ℎ 푡 ;푧 푡 ]∈ℝ 푑 State head 푝 휙 (푦 푡 |푠 푡 ) Reconstruct node bits Reward head 푝(푟 푡 |푠 푡 ) Two-hot Encoding 푝(푐 푡 |푠 푡 ) Continue head Binary Cross Entropy 푝(푥 푡 (푓) |푠 푡 ) Frame head Reconstruct Frame Text head 푝(푥 푡 (푡) |푠 푡 ) Frozen 3B LM decoder Figure 2: Architecture of CyberWorld. Multimodal telemetry and structural observations are mapped into a unified latent space via cross-attention pooling, driving an RSSM that tracks recurrent belief dynamics and trains defensive behavior purely within imagined rollouts. We provide a formal analysis of this fac- torization in Appendix A, showing how adversary-strategy shifts affect the effec- tive transition kernel and how model error decomposes into shared-mechanism and strategy-dependent components. The defender’s observationo t is the stream of detector alerts and asset states described above, which we refer to as telemetry. Telemetry is incomplete and delayed and never exposes the adver- sary’s internal state, so the observation sequence is non-Markovian. Optimal de- fense therefore requires a temporal belief stateb t (s) = p(s t = s| o ≤t ,a B <t ), which CyberWorld approximates by a recurrent latent states t = [h t ; z t ]≈ b t (s). Under this formulation, the choice of observa- tion representationψ(o t )is a dynamics- modeling decision: a representation is ad- equate when the sequenceψ(o ≤t )carries sufficient statistics to identifyT mech and to trackb R t . Because attacks manifest both as localized changes in host status and as non-local traversals of the network topol- ogy, a representation must capture both local state and topological information for the transitions underT c to be predictable. 4 Preprint 3.2WORLD MODEL The world model follows the data flow of Figure 2: each observation channel is encoded into a token sequence, the token sequences are fused into one observation embeddinge t , a Recurrent State-Space Model (RSSM) (Hafner et al., 2019; 2023) turns the sequence of embeddings and defender actions into a latent states t , and modality-specific heads reconstruct the observation, the reward, and the continuation flag froms t . LetMdenote the set of channels an arm encodes andKthe set of heads it trains; both are configuration, and every arm shares the same fusion, latent dynamics, head architectures, and behavior learning. Modality Encoders.Each channelm∈Mproduces a token sequenceT m t ∈ R L m ×d of common widthd. Vector observations are encoded independently per host, while the graph encoder augments the same host features with graph-attention message passing overG(Veli ˇ ckovi ́ c et al., 2018). Text telemetryw t is encoded by a frozen pretrained language model (Team, 2024) and projected into the shared token space through a trainable adapter. Visual observationsI t are similarly mapped into patch-level tokens by a visual encoder and projected into the shared token space; when enabled as a prediction target, they are reconstructed by the corresponding frame head. Visual encoding is optional and unused in the reported configurations. Observation Representations. Every channel is rendered from one environment-agnostic inter- mediate representation (IR) of the defender’s view, which is the only object an encoder or renderer receives; ground-truth compromise labels and the adversary’s position are kept in a separate anno- tation record used for analysis only. The IR fixes the node set before an episode starts: then host physical hosts occupy the indices given by the network configuration, andn decoy reserved per-subnet slots receive decoys as they are deployed, each decoy binding to the lowest free slot of its subnet and releasing it on removal, so that theN = n host + n decoy node indices refer to the same hosts throughout an episode and never reveal decoys that have not yet been deployed. The vector channel is the defender-visible host state x t ∈0, 1 N×5 ,x t [i, :] = present, is_decoy, isolated, alert_now, alert_ever i , (2) whose alert fields originate from the defender’s own detectors (Stakhanova et al., 2007) and are keyed on the source host; the IR carries neither a destination host nor a technique label, since either would reveal the attacker’s next target. For reconstruction, the five fields are mapped to four role bits y t [i, :] = max(present, isolated), is_decoy, alert_now, alert_ever i ∈0, 1 4 .(3) The encoder usesx t , while the state head reconstructsy t . The graph channel adds the sparse adjacencyG = (V,E)with|V| = N, taken from the network configuration: hosts on the same subnet are adjacent, cross-subnet edges exist where an interface is declared, and the topology is static within an episode. The text channelw t renders the same record as prose, namely an alert summary, the defender’s own action, the source host of any detected activity, the changes since the previous step, and the present hosts grouped by alert state with decoys named individually. The frameI t draws the nodes at positions determined by the topology, adds the edges ofG, and colours each present node byy t , so a frame is exactly invertible to the role bits. All channels are deterministic functions of the same IR record; every arm therefore receives the same information, and the arms differ in representation alone. Benchmark-specific constants are given in subsection 4.1. Cross-Modal Fusion. Fusion keeps the channels as separate token streams until a final pooling step. WritingAttn(Q,K,V ) = softmax QK ⊤ / √ d Vfor multi-head attention, each stream is first tagged with a learned channel embedding and refined by a stack of self-attention blocks in which the stream attends to itself, U m t = Attn T m t , T m t , T m t ,(4) and, when more than one channel is active, each refined stream then attends to the concatenation of the other streams, ̃ U m t = Attn U m t , [U m ′ t ] m ′ ̸=m , [U m ′ t ] m ′ ̸=m ,(5) so that, for instance, a text token can attend to the node token it describes. A single learned query q ∈ R d finally pools the concatenated fused streams into a fixed-size observation embedding (Jaegle et al., 2021), e t = Attn q, [ ̃ U m t ] m∈M , [ ̃ U m t ] m∈M ∈ R d ,(6) whose size is independent of the number of hosts or text tokens. With a single active channel, the cross-attention stage is omitted and q pools U m t directly. 5 Preprint Latent Dynamics.The RSSM maintains a deterministic recurrent stateh t and a discrete stochastic state z t ; with parameters φ, Recurrent model:h t = f φ (h t−1 ,z t−1 ,a B t−1 ) Posterior representation: z t ∼ q φ (z t | h t ,e t ) Prior transition dynamics: ˆz t ∼ p φ (ˆz t | h t ), (7) where the posterior incorporates the current fused observatione t , while the prior predicts the stochastic state from history alone and is used for imagined rollouts. The remaining stochastic parameterization follows DreamerV3 (Hafner et al., 2023). The concatenated latent vector s t = [h t ; z t ](8) constitutes the model state supplied to all decoding heads, policy actors, and value critics;s t is the learned counterpart of the defender belief b t and only the defender’s own action conditions it. Prediction and Reconstruction Heads.Every head reads the posterior states t and reconstructs a quantity of the same stept: the model is trained to explain the present from its belief, and the future enters only through the prior in Equation 7. All configurations reconstruct the four defender-visible node-state rolesy t with a shared per-node decoder, and predict reward and episode continuation. The state decoder conditions ons t and a learned node query, so its parameters are independent of network size. The reward head uses distributional symlog two-hot parameterization (Hafner et al., 2023). Text-enabled configurations additionally reconstructw t through the frozen language model conditioned on a learned latent prefix, while the multimodal configuration reconstructsI t with a convolutional decoder. Cyber-Specific Loss Weighting.Two properties of cyber telemetry make the standard reconstruc- tion objectives degenerate, and CyberWorld modifies them accordingly; both are departures from DreamerV3. First, most node attributes persist between consecutive steps, allowing an unweighted objective to succeed largely by predicting no change. We therefore up-weight state elements that change: L state = 1 4N N X i=1 4 X j=1 λ t,i,j BCE ˆy t,i,j , y t,i,j , λ t,i,j = ρ x if y t,i,j ̸= y t−1,i,j 1otherwise. (9) Second, the reward is dominated by rare events, namely the decoy captures and decoy-budget violations, which occur on a small fraction of steps and therefore contribute little to a uniformly weighted reward loss. CyberWorld balances the reward-bearing stepsR = t : |r t | ≥ τand the remaining stepsR c within each batch: L reward = X t ω t ℓ t , ω t = ( 1 2|R| if t∈R 1 2|R c | otherwise, (10) where ℓ t is the two-hot reward loss. When a batch contains one group, the mean is used. World-Model Objective. Over replayed trajectory segments, the world model parametersφare trained to minimise L(φ) = E q φ h X t X k∈K β k L k + β dyn L dyn + β rep L rep i ,(11) whereL k denotes the active prediction and reconstruction losses, andL dyn andL rep are DreamerV3’s dynamics and representation KL objectives (Hafner et al., 2023). 3.3BEHAVIOR LEARNING AND ONLINE TRAINING The actor and the critic are trained entirely on trajectories imagined by the world model, and both follow DreamerV3 unchanged (Hafner et al., 2023): imagined trajectories start from posterior states of replayed segments and are rolled forward through the RSSM prior, and the actor and the distributional 6 Preprint Table 1: Environment steps until the trailing-10 return first exceeds the control. All methods start from scratch. CyberWorld and GNN-PPO share the same interaction budget at eachN; PPO uses its original, longer training horizon. Mean±sd over 3 seeds;(k/n): seeds that crossed. Bold: fastest of GNN-PPO, vector, and graph with all seeds crossed. PPO: 3 seeds atN ≤ 50, none atN = 100; GNN-PPO: 3 seeds, two BFSServerDowntime seeds unfinished at N = 100. attackerNPPOGNN-PPOours: vectorours: graph BFSServerDowntime 152,310,400± 640,000 (2/3)4,017± 8214,933± 6335,525± 575 (2/3) 252,549,547± 410,31012,368± 5,398 (2/3)10,257± 7,1235,100± 524 50NEVER (0/3)2,777± 28937,570± 3,400 (2/3)32,725± 8,245 (2/3) 100–NEVER (0/1)NEVER (0/3)NEVER (0/3) ServerDowntime 153,078,400± 0 (1/3)25,875± 4,025 (2/3)3,633± 623,600± 41 253,274,880± 1,579,16221,845± 0 (1/3)5,922± 1,7697,027± 2,668 50456,960± 03,457± 92424,593± 12,03829,580± 12,300 100– 35,175± 13,400 (2/3)47,570± 4,690 (2/3)16,080± 0 (1/3) Exfiltration 152,630,400± 448,000 (2/3)11,367± 9,1823,900± 2834,600± 41 25373,547± 102,57814,110± 7,0304,222± 4246,290± 367 50602,027± 205,1558,330± 5,141 33,745± 17,255 (2/3)25,103± 8,951 100–9,715± 3,316 28,308± 17,252 (2/3) 34,840± 10,720 (2/3) BFSExfiltration 15NEVER (0/3)9,117± 4,128 31,650± 12,450 (2/3)15,750± 12,960 252,839,680± 615,46623,913± 10,0319,633± 6,3515,468± 1,627 50NEVER (0/3)14,705± 5,525 (2/3)NEVER (0/3)15,810± 0 (1/3) 100–8,208± 838 (2/3)NEVER (0/3)17,085± 0 (1/3) critic are optimized with bootstrappedλ-returns, REINFORCE, entropy regularization, and percentile- based return normalization. Training likewise follows DreamerV3’s concurrent scheme. The agent collects one episode with its current policy, appends it to a growing replay buffer, and performs a fixed number of world-model and actor–critic updates on sampled segments before collecting the next episode. Training starts from randomly initialized parameters with no offline data, and every environment interaction, including the random prefill of the buffer, counts toward the reported budget; the prefill size, update ratio, and budget are given in subsection 4.1. The one cyber-specific consideration is that every declared defender action remains available at every step, so the policy can avoid the decoy-budget penalty only if the world model represents it, which is what the balanced reward objective of Equation 10 provides. 4EXPERIMENTS 4.1EXPERIMENTAL SETUP Our evaluation answers three Research Questions (RQs): (RQ1) Can a world model substantially improve the sample efficiency of autonomous cyber defense over model-free DRL? (RQ2) Which representation of the cyber environment best supports world-model-based control? (RQ3) Does the answer survive changes in network scale? RQ1 establishes whether learning the dynamics pays off in the resource that is scarce when a defender is trained against a live or emulated network, the number of environment interactions. RQ2 determines which of the representations of section 3.2 lets the world model capture the dynamics that matter for control, and RQ3 whether that answer and the interaction saving hold as the defended network grows. Environment. All experiments run in CyberWheel (Oesch et al., 2024) against seven scripted ART attacker strategies, four of which are scoreable (the other three have a control return of 0). RQ1 and RQ2 use the 15-host, three-subnet network; RQ3 scales the same network toN ∈ 25, 50, 100 hosts with the decoy budget scaled withN. Each CyberWorld and GNN-PPO run uses the same 52,500-step interaction budget, corresponding to 1,050 episodes atN =15(50 random prefill and 1,000 training episodes). Representation arms. Five arms share the latent dynamics, behavior learning, and hyperparameters and differ only in encoders and reconstruction heads: vector (per-node MLP + self-attention), graph (GAT over the host graph), vec+graph, text (frozen Qwen2.5-3B over telemetry sentences), and all (vector + text fusion with an added frame-reconstruction head). All share one training configuration: DreamerV3’s return normalisation, the balanced reward loss of Equation 10, a 1,000-step actor–critic warm-up, and cosine learning-rate decay. Baselines. PPO is CyberWheel’s own PPO trained from scratch under the same environment and observation but with its original substantially longer training budget, reaching up to 3.2M environment steps atN =15. GNN-PPO is a representation-matched baseline: the graph arm’s observation and GAT encoder trained with PPO under the same online interaction budget and seeds as CyberWorld, without a world model. The strategy-agnosticdeploy_then_stoppolicy serves as a fixed rule- 7 Preprint 10 3 10 4 10 5 10 6 environment steps −10 5 −10 4 −10 3 −10 2 0 10 2 10 3 episode return (symlog) BFSServerDowntime 5.5k (2/3) 4.0k 4.9k 2.3M (2/3) 10 3 10 4 10 5 10 6 environment steps ServerDowntime 3.6k 25.9k (2/3) 3.6k 3.1M (1/3) 10 3 10 4 10 5 10 6 environment steps Exfiltration 4.6k 11.4k 3.9k 2.6M (2/3) 10 3 10 4 10 5 10 6 environment steps BFSExfiltration 15.8k 9.1k 31.6k (2/3) never CyberWorld (vector), trailing-50CyberWorld (graph), trailing-50GNN-PPO, trailing-50trailing-10first crossingPPO checkpoint evaluationcontrol Figure 3: Return vs. environment steps for CyberWorld (vector and graph), GNN-PPO, and PPO on the four scoreable attackers atN =15(seed mean, band = min/max over three seeds; PPO: 30-episode checkpoint evaluations). Dashed: thedeploy_then_stopcontrol; markers: mean first crossing, with the seed count where not all seeds crossed. 10 4 10 5 10 6 env steps to reach the control PPO GNN-PPO vector graph vec+graph text multimodal budget (a) sample efficiency 02040 60 80 % of post-crossing episodes above the control (b) reward after crossing 02040 60 80 decoy-limit penalties per 100 late episodes (c) stability 10 1 10 2 GPU-hours per cell CPU (d) compute BFSServerDowntimeServerDowntimeExfiltrationBFSExfiltrationmean over attackersdid not reach the control Figure 4: Representation comparison atN =15: (a) steps to control, (b) post-crossing success rate, (c) late decoy-limit penalties, and (d) GPU-hours. Small markers denote attackers, large markers their mean;×: no crossing. based control, defining the minimum return threshold that a learned defender must exceed; the environment reward is unchanged throughout. Metrics. The primary metric is the number of environment steps until the trailing-10 episode return first exceeds the control, reported as mean±sd over 3 seeds; four text/multimodal cells use 2 seeds because the third run did not complete. Runs that never cross are reported as such. Curves show the trailing-10 return and a trailing-50 trend. PPO’s crossing uses its 30-episode checkpoint evaluations, ours the online training return; no accuracy-based metric is used anywhere. Implementation details.The world model uses a GRU with 1,024 deterministic units, a 32×32categorical latent, encoders of width 512, and two-layer MLP heads of width 512. We optimize the world model with AdamW (learning rate3×10 −4 , weight decay10 −4 , and gradient clipping at 1.0) using batches of 16 windows of 16 steps.We set the dynam- ics and representation KL weights to 0.5 and 0.1, respectively, use 1 free nat and 1% uni- form mixing, assign a change weight of 5 to node bits, weight the text and frame recon- struction heads by 0.3 and 20, respectively, and define reward-bearing steps by|r t | ≥ 50. 10 −2 10 −1 state-bit loss (BCE) BFSServerDowntime 10 −1 4 × 10 −2 6 × 10 −2 2 × 10 −1 3 × 10 −1 ServerDowntime 10 −1 4 × 10 −2 6 × 10 −2 2 × 10 −1 Exfiltration 10 −3 10 −2 10 −1 BFSExfiltration 0200400 600 8001000 online iteration (episodes) 10 −1 10 0 reward-head loss (two-hot CE) 0200400 600 8001000 online iteration (episodes) 10 −1 10 0 0200400 600 8001000 online iteration (episodes) 10 −1 10 0 0200400 600 8001000 online iteration (episodes) 10 −2 10 −1 10 0 CyberWorld (vector)CyberWorld (graph)CyberWorld (vec+graph)CyberWorld (text)CyberWorld (multimodal) Figure 5: World-model fidelity across representations at N =15: replay-batch state-bit loss (top) and reward-head loss (bottom). The actor and critic use learning rate 3×10 −5 with a 1,000-step warm-up and cosine decay, imagination horizon 15,γ = 0.997,λ = 0.95, and entropy weight 3×10 −4 . Each run collects a 50-episode random prefill, then one episode per iter- ation with 32 world-model and 32 actor– critic updates for 1,000 iterations, on a single NVIDIA A100 shared with other jobs; PPO trains on CPU with 128 parallel environments. 4.2RESULTS Sample efficiency (RQ1). Figure 3 summarizes the primary sample-efficiency result for the vector and graph arms. On three of the four scoreable attackers, both arms reach the control after3.6k– 5.5k environment steps, whereas PPO requires2.3M–3.1M steps, two to three orders of magnitude more; on BFSExfiltration, where no PPO seed crosses within its3.2M-step budget, the graph arm crosses on every seed at15,750steps and the vector arm on two of three seeds at31,650, while the text arm, not shown, crosses on every seed at7,850(exact values in Table 2; pooled profile in Figure 8). GNN-PPO, which shares the graph arm’s observation and encoder, exhibits brief early crossings under the trailing-10 criterion, but these are not sustained in the longer-term trend and are 8 Preprint 10 3 10 4 environment steps −10 4 −10 3 −10 2 0 10 2 10 3 episode return (symlog) 31.6k (2/3) 15.8k 9.1k BFSExfiltration, N= 15 hosts 10 3 10 4 environment steps −10 4 −10 3 −10 2 0 10 2 10 3 9.6k 5.5k 23.9k BFSExfiltration, N= 25 hosts 10 4 environment steps −10 5 −10 4 −10 3 −10 2 0 10 2 10 3 10 4 vector: never (0/3) 15.8k (1/3) 14.7k (2/3) BFSExfiltration, N= 50 hosts 10 4 environment steps −10 5 −10 4 −10 3 −10 2 0 10 2 10 3 10 4 vector: never (0/3) 17.1k (1/3) 8.2k (2/3) BFSExfiltration, N= 100 hosts CyberWorld (vector), trailing-50CyberWorld (graph), trailing-50GNN-PPO, trailing-50trailing-10first crossing (seeds)control Figure 6: Scaling with network size for the vector, graph, and GNN-PPO arms. Curves show trailing- 50 means with min/max bands; thin lines show trailing-10. followed by severe return degradation under decoy-limit penalties (Figure 4b,c). In contrast, the world-model variants maintain ServerDowntime performance with substantially fewer such penalties. These results indicate two distinct advantages: relative to PPO, world-model learning reduces the required environment interaction, while relative to a model-free learner with the same encoder, it yields greater stability after first reaching the control. These results concern environment interactions rather than computational cost, which is analyzed separately below. Appendix D shows that the main sample-efficiency conclusion is robust to the smoothing convention, while the BFSExfiltration vector crossing is sensitive to this choice. Representation comparison (RQ2). On crossing speed, the three numeric arms are indistinguishable within seed variation on three of the four attackers (Figure 4(a)), despite clear differences in world- model fidelity. The graph arm achieves the lowest state-bit and reward-head losses on the attackers that separate the representations, whereas elevated vector reward loss coincides with higher decoy- limit penalties (Figure 4(c), Figure 5). Richer representations provide selective benefits: the text arm crosses on every scoreable attacker, on all but one of its seeds, and is fastest on BFSExfiltration, whereas the multimodal arm crosses on every seed of the two non-BFS attackers but on a single seed of each BFS attacker. BFSExfiltration provides the clearest structural distinction among the three numeric representations: because its target order follows the network topology, only the graph arm crosses on every seed at N =15 (Figure 6). Zero-control attackers are omitted (Appendix E). 152550 100 hosts N 10 4 10 5 10 6 env steps to reach the control 1581381313 budget (hollow = did not reach) (a) interaction cost grows with N CyberWorld (vector): ∝N 1.09 CyberWorld (graph): ∝N 1.00 GNN-PPO: ∝N −0.04 PPO (mean of cells that reached it) 152550 100 hosts N 10 2 10 3 episodes to reach the control (b) episodes stay near 100 CyberWorld (vector): ∝N 0.09 CyberWorld (graph): ∝N −0.00 GNN-PPO: ∝N −1.04 Figure 7: (a) Environment steps and (b) episodes to the control vs.Nfor vector, graph and GNN-PPO: small markers are runs that crossed, lines are power-law fits over them, hollow markers at the budget line count runs that did not cross, and×is PPO’s mean where it crossed. Scalability (RQ3). Among successful runs, the number of episodes required to reach the control remains approximately constant as the network grows, while en- vironment steps increase with episode length (Figure 7). Scalability becomes representation-dependent under BFSEx- filtration: vector and graph both reach the control atN ≤ 25, whereas among the world-model arms only the graph arm does so atN =50andN =100(Figure 6). AtN =100, the fixed interaction budget covers relatively few episodes, making crossing frequency increasingly budget-limited. GNN-PPO’s trailing-10 return also crosses at these scales, but its trailing-50 trend does not remain above the control once decoy-limit penalties accumu- late. Thus, simple representations suffice for most attackers, while explicit graph structure becomes increasingly important when attack progression depends on network topology. See Appendix F. Discussion and Limitations. CyberWorld reaches effective defense with two to three orders of magnitude fewer environment interactions than PPO, exhibits greater post-crossing stability than a representation-matched model-free baseline, and, among successful runs, requires an approximately constant number of episodes to reach the control as the network grows from 15 to 100 hosts. Among the evaluated representations, the host graph provides the most consistent performance under topology- dependent attacks. These results are subject to three main limitations. First, interaction efficiency does not imply equivalent wall-clock efficiency, particularly for text and multimodal variants that incur substantially higher inference cost. Second, our evaluation considers an RSSM-based latent dynamics model and does not compare alternative world-model backbones such as transformers or diffusion models. Third, all experiments are conducted in CyberWheel with scripted adversaries, so broader validation across environments and adversary models remains necessary. 9 Preprint 5CONCLUSION We introduced CyberWorld, a world-model-based framework for autonomous cyber defense that learns predictive latent dynamics of the defended network and optimizes defensive policies through imagined trajectories. Our results show that explicit dynamics modeling can substantially reduce environment interaction while maintaining effective defensive behavior across adversary strategies and network scales, and that representation choice materially affects this capability when attack progression depends on network topology. Future work should extend CyberWorld to alternative latent dynamics architectures, more efficient semantic representations, additional cyber-defense environments, emulated networks, and learned adversaries, with the broader goal of developing transferable cyber world models that capture reusable network dynamics across heterogeneous operational settings. REFERENCES Eloi Alonso, Adam Jelley, Vincent Micheli, Anssi Kanervisto, Amos Storkey, Tim Pearce, and François Fleuret. Diffusion for world modeling: Visual details matter in atari. Advances in Neural Information Processing Systems, 37:58757–58791, 2024. Alex Andrew, Sam Spillard, Joshua Collyer, and Neil Dhir. Developing optimal causal cyber-defence agents via cyber security simulation. arXiv preprint arXiv:2207.12355, 2022. Anthropic.Claude mythos preview.https://red.anthropic.com/2026/ mythos-preview/, 2026. [Accessed 18-09-2026]. Karl Johan Åström. Optimal control of markov processes with incomplete state information i. Journal of mathematical analysis and applications, 10:174–205, 1965. Peter Belcak, Greg Heinrich, Shizhe Diao, Yonggan Fu, Xin Dong, Saurav Muralidharan, Yingyan Ce- line Lin, and Pavlo Molchanov. Small language models are the future of agentic ai. arXiv preprint arXiv:2506.02153, 2025. Maxime Burchi and Radu Timofte. Accurate and efficient world modeling with masked latent transformers. In Aarti Singh, Maryam Fazel, Daniel Hsu, Simon Lacoste-Julien, Felix Berkenkamp, Tegan Maharaj, Kiri Wagstaff, and Jerry Zhu (eds.), Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, p. 5894–5912. PMLR, 13–19 Jul 2025a. URLhttps://proceedings.mlr.press/v267/ burchi25a.html. Maxime Burchi and Radu Timofte. Learning transformer-based world models with contrastive predictive coding. In International Conference on Learning Representations, volume 2025, p. 6949–6970, 2025b. Ashutosh Dutta, Ehab Al-Shaer, Samrat Chatterjee, and Qi Duan. Autonomous cyber defense against dynamic multi-strategy infrastructural ddos attacks. In 2023 IEEE Conference on Communications and Network Security (CNS), p. 1–9, 2023a. doi: 10.1109/CNS59707.2023.10288937. Ashutosh Dutta, Samrat Chatterjee, Arnab Bhattacharya, and Mahantesh Halappanavar. Deep reinforcement learning for cyber system defense under dynamic adversarial uncertainties. arXiv preprint arXiv:2302.01595, 2023b. Vladimir Egorov and Alexei Shpilman. Scalable multi-agent model-based reinforcement learning. In Proceedings of the 21st International Conference on Autonomous Agents and Multiagent Systems, AAMAS ’22, p. 381–390, Richland, SC, 2022. International Foundation for Autonomous Agents and Multiagent Systems. ISBN 9781450392136. Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In International conference on machine learning, p. 2555–2565. PMLR, 2019. Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. In International Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=S1lOTC4tDS. 10 Preprint Danijar Hafner, Timothy P Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with discrete world models. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=0oabwyZbOu. Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models. arXiv preprint arXiv:2301.04104, 2023. Andrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and Joao Carreira. Perceiver: General perception with iterative attention. In International conference on machine learning, p. 4651–4664. PMLR, 2021. Jaromír Janisch, Tomáš Pevný, and Viliam Lisý. Nasimemu: Network attack simulator & emulator for training agents generalizing to novel scenarios. In Computer Security. ESORICS 2023 International Workshops: CPS4CIP, ADIoT, SecAssure, WASP, TAURIN, PriST-AI, and SECAI, The Hague, The Netherlands, September 25–29, 2023, Revised Selected Papers, Part I, p. 589–608, Berlin, Heidelberg, 2023. Springer-Verlag. ISBN 978-3-031-54128-5. doi: 10.1007/978-3-031-54129-2_ 35. URL https://doi.org/10.1007/978-3-031-54129-2_35. Leslie Pack Kaelbling, Michael L Littman, and Anthony R Cassandra. Planning and acting in partially observable stochastic domains. Artificial intelligence, 101(1-2):99–134, 1998. Can Karacelebi, Yusuf Talha Sahin, Elif Surer, and Ertan Onur. Learning ad hoc network dynamics via graph-structured world models. arXiv preprint arXiv:2604.14811, 2026. Armita Kazeminajafabadi, Amirhossein Ravari, Nathaniel D Bastian, Tian Lan, and Mahdi Imani. Posterior-conditioned multi-agent reinforcement learning for adaptive cyber defense. Artificial Intelligence, p. 104606, 2026. Michael Kearns and Satinder Singh. Near-optimal reinforcement learning in polynomial time. Machine learning, 49(2):209–232, 2002. Mitchell Kiely, Metin Ahiskali, Etienne Borde, Benjamin Bowman, David Bowman, Dirk Van Bruggen, KC Cowan, Prithviraj Dasgupta, Erich Devendorf, Ben Edwards, et al. Exploring the efficacy of multi-agent reinforcement learning for autonomous cyber defence: A cage challenge 4 perspective. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, p. 28907–28913, 2025. Isaiah J King, Benjamin Bowman, and H Howie Huang. Automated cyber defense with generalizable graph-based reinforcement learning agents. arXiv preprint arXiv:2509.16151, 2025. Duc Huy Le and Rolf Stadler. Learning optimal defender strategies for cage-2 using a pomdp model. In 2025 21st International Conference on Network and Service Management (CNSM), p. 1–9. IEEE, 2025. Yu Li, Sizhe Tang, Rongqian Chen, Fei Xu Yu, Guangyu Jiang, Mahdi Imani, Nathaniel D Bastian, and Tian Lan. Acdzero: Mcts agent for mastering automated cyber defense. arXiv preprint arXiv:2601.02196, 2026. Jessy Lin, Yuqing Du, Olivia Watkins, Danijar Hafner, Pieter Abbeel, Dan Klein, and Anca Dragan. Learning to model the world with language. In Forty-first International Conference on Machine Learning, 2024. URL https://openreview.net/forum?id=7dP6Yq9Uwv. Xueguang Lyu, Yuchen Xiao, Brett Daley, and Christopher Amato. Contrasting centralized and decentralized critics in multi-agent reinforcement learning. In Proceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems, AAMAS ’21, p. 844–852, Richland, SC, 2021. International Foundation for Autonomous Agents and Multiagent Systems. ISBN 9781450383073. Ryozo Masukawa, Ian Bryant, Armita Kazeminajafabadi, Sanggeon Yun, Hyunwoo Oh, SungHeon Jeong, Nathaniel D Bastian, Mahdi Imani, and Mohsen Imani. Trident: How to break deep reinforcement learning cyber defenses (agentic). arXiv preprint arXiv:2608.04317, 2026. 11 Preprint Pietro Mazzaglia, Tim Verbelen, Bart Dhoedt, Aaron Courville, and Sai Rajeswar. GenRL: Multimodal-foundation world models for generalization in embodied agents. In The Thirty- eighth Annual Conference on Neural Information Processing Systems, 2024. URLhttps: //openreview.net/forum?id=za9Jx8yqUA. Shae McFadden, Myles Foley, Elizabeth Bates, Ilias Tsingenopoulos, Sanyam Vyas, Vasilios Mavroudis, Chris Hicks, and Fabio Pierazzi. SoK: The pitfalls of deep reinforcement learn- ing for cybersecurity. In 35th USENIX Security Symposium (USENIX Security 26), Baltimore, MD, August 2026. USENIX Association. URLhttps://w.usenix.org/conference/ usenixsecurity26/presentation/mcfadden. Vincent Micheli, Eloi Alonso, and François Fleuret. Transformers are sample-efficient world models. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=vhFu1Acb0xb. Microsoft Defender Research Team.Cyberbattlesim, 2021.Created by Christian Seifert, Michael Betser, William Blum, James Bono, Kate Farris, Emily Goren, Justin Grana, Kristian Hol- sheimer, Brandon Marken, Joshua Neil, Nicole Nichols, Jugal Parikh, Haoran Wei. Erik Miehling, Mohammad Rasouli, and Demosthenis Teneketzis. A pomdp approach to the dynamic defense of large-scale cyber networks. IEEE Transactions on Information Forensics and Security, 13(10):2490–2505, 2018. doi: 10.1109/TIFS.2018.2819967. MITRE. Mitre att&ck. https://attack.mitre.org/, 2026. [Accessed 20-04-2026]. Sean Oesch, Amul Chaulagain, Brian Weber, Matthew Dixson, Amir Sadovnik, Benjamin Rober- son, Cory Watson, and Phillipe Austria. Towards a high fidelity training environment for autonomous cyber defense agents. In Proceedings of the 17th Cyber Security Experimen- tation and Test Workshop, CSET ’24, p. 91–99, New York, NY, USA, 2024. Association for Computing Machinery. ISBN 9798400709579. doi: 10.1145/3675741.3675752. URL https://doi.org/10.1145/3675741.3675752. Red Canary. Atomic red team.https://w.atomicredteam.io/, 2026. [Accessed 20-04- 2026]. Jan Robine, Marc Höftmann, Tobias Uelwer, and Stefan Harmeling. Transformer-based world models are happy with 100k interactions. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=TdBaDGCpjly. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. Aditya Vikram Singh, Ethan Rathbun, Emma Graham, Lisa Oakley, Simona Boboila, Peter Chin, and Alina Oprea. Hierarchical multi-agent reinforcement learning for cyber network defense. Reinforcement Learning Journal, 6:790–810, 2025. Natalia Stakhanova, Samik Basu, and Johnny Wong. A taxonomy of intrusion response systems. Int. J. Inf. Comput. Secur., 1(1/2):169–184, January 2007. ISSN 1744-1765. doi: 10.1504/IJICS.2007. 012248. URL https://doi.org/10.1504/IJICS.2007.012248. Richard S Sutton, Andrew G Barto, and Andrew Barto. Reinforcement learning: An introduction, volume 1. MIT press Cambridge, 1998. Qwen Team. Qwen2.5: A party of foundation models, September 2024. URLhttps://qwenlm. github.io/blog/qwen2.5/. Petar Veli ˇ ckovi ́ c, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph attention networks. In International Conference on Learning Representations, 2018. URL https://openreview.net/forum?id=rJXMpikCZ. Di Xue, Jing Jiang, Shaowei Zhang, Wenhao Guo, Lei Yuan, Zongzhang Zhang, and Yang Yu. Learn- ing disentangled multi-agent world model for decentralized control. In Forty-third International Conference on Machine Learning, 2026. URLhttps://openreview.net/forum?id= nYyfpPubnW. 12 Preprint Hongxin Zhang, Zeyuan Wang, Qiushi Lyu, Zheyuan Zhang, Sunli Chen, Tianmin Shu, Behzad Dariush, Kwonjoon Lee, Yilun Du, and Chuang Gan. COMBO: Compositional world models for embodied multi-agent cooperation. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=YXRyYkb1im. Weipu Zhang, Gang Wang, Jian Sun, Yetian Yuan, and Gao Huang. Storm: Efficient stochastic transformer based world models for reinforcement learning. Advances in Neural Information Processing Systems, 36:27147–27166, 2023. Zijie Zhao, Honglei Guo, Shengqian Chen, Kaixuan Xu, Bo Jiang, Yuanheng Zhu, and Dongbin Zhao. Empowering multi-robot cooperation via sequential world models. In The Fourteenth International Conference on Learning Representations, 2026. URLhttps://openreview.net/forum? id=IvUM6UwYCJ. Gaoyue Zhou, Hengkai Pan, Yann LeCun, and Lerrel Pinto. DINO-WM: World models on pre-trained visual features enable zero-shot planning. In Forty-second International Conference on Machine Learning, 2025. URL https://openreview.net/forum?id=D5RNACOZEI. Xuanru Zhou, Jiachen Lian, Henry Hong, Xinyi Yang, and Gopala Anumanchipalli. Speech world model: Causal state–action planning with explicit reasoning for speech. In The Fourteenth International Conference on Learning Representations, 2026. URLhttps://openreview. net/forum?id=YGUKPGO182. Yuxuan Zhu, Antony Kellermann, Akul Gupta, Philip Li, Richard Fang, Rohan Bindu, and Daniel Kang. Teams of LLM agents can exploit zero-day vulnerabilities. In Vera Demberg, Kentaro Inui, and Lluís Marquez (eds.), Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), p. 23–35, Rabat, Morocco, March 2026. Association for Computational Linguistics. ISBN 979-8-89176-380-7. doi: 10.18653/ v1/2026.eacl-long.2. URL https://aclanthology.org/2026.eacl-long.2/. APPENDIX ATHEORETICAL ANALYSIS OF THE PROBLEM FORMULATION A.1MECHANISM INVARIANCE AND TRANSFER ACROSS ADVERSARY STRATEGIES Equation (1) decomposes the defender-facing transition law into a strategy-dependent adversary policy π R c and a shared mechanism kernelT mech . We formalize three consequences of this factorization. First, changes in the effective transition law are bounded by changes in the adversary strategy. Second, under partial observability, the filtering posterior provides a sufficient statistic for defender control, motivating the use of a recurrent latent state. Third, model error under a new adversary can be decomposed into mechanism-model error and strategy-estimation error. Proofs are provided in Appendix B. Setup. Consider a family of POMDPsM c c∈C , M c = (S,A B ,O,T c , Ω,R,γ), that share all components except the effective transition lawT c , which factors according to Eq. (1). Rewards are bounded as|R(s,a B )| ≤ R max , withR max = 5,000in CyberWheel, andγ ∈ [0, 1). We writeTV(μ,ν)for total-variation distance andV π M for the value of policyπin modelM, with the initial-state distribution suppressed from the notation. Assumption 1 (Closed information state). The global states∈Sincludes the adversary’s internal beliefb R (“latent adversarial intent” in Section 3.1), such thatπ R c (a R | b R )is a measurable function of s and Eq. (1) defines a Markov kernel onS for every c. Assumption 2 (Shared mechanisms).T mech ,Ω, andRare identical acrossc ∈ C; adversary strategies differ only through π R c . 13 Preprint Assumption 2 formalizes the intended separation between strategy and environment mechanisms: protocol semantics, action effects, exploit preconditions, and decoy behavior are assumed to remain unchanged when the adversary strategy changes. Assumption 1 is a state augmentation used to express the coupled defender–adversary process as a Markov process. Sensitivity of the effective dynamics to strategy shift. Lemma 1 (Strategy-shift bound). Under Assumptions 1–2, for everys∈S,a B ∈A B , andc,c ′ ∈C, TV T c (·| s,a B ),T c ′ (·| s,a B ) ≤ TV π R c (·| b R (s)),π R c ′ (·| b R (s)) . Lemma 1 shows that, under the shared-mechanism factorization, a change in adversary strategy cannot induce a larger total-variation change in the effective transition kernel than the corresponding change in the adversary action distribution. Thus, strategy variation affects the defender-facing dynamics through a constrained component of the transition model, whileT mech remains shared by assumption. Belief-state sufficiency under partial observability. Proposition 1 (Belief-state sufficiency). For every c, the filtering posterior b t . = p(s t | o ≤t ,a B <t ) is a sufficient statistic of the interaction history. In particular, there exists an optimal defender policy of the form π ⋆ (a B t | b t ), and the belief state evolves according to a Bayes operator b t+1 = U (b t ,a B t ,o t+1 ), whose dependence oncenters throughT c , and hence throughπ R c under Assumption 2 (Åström, 1965; Kaelbling et al., 1998). Proposition 1 provides a control-theoretic interpretation of the recurrent latent states t = [h t ;z t ]. An exact belief state is generally unavailable, and CyberWorld does not require the learned latent state to recoverb t exactly. Rather, the RSSM can be viewed as learning a history-dependent predictive statistic from which the actor and value functions are computed. Its priorp φ (z t | h t )and posterior q φ (z t | h t ,e t )are respectively analogous to the prediction and observation-update stages of Bayesian filtering. This interpretation motivates RQ2: different observation representations may preserve different amounts of information relevant to constructing such a predictive latent state. Error decomposition under adversary shift.The following decomposition is an analytical abstrac- tion of the transition factorization in Equation (1); CyberWorld does not explicitly parameterize b T mech andbπ R c as separate learned modules. To characterize the potential benefit of shared mechanisms, suppose an idealized factored predictive model has obtained a mechanism model b T mech from data generated under one or more adversary strategies and, for a possibly new adversaryc ′ , an estimate bπ R c ′ of its strategy. Define the corresponding imagined dynamics as b T c ′ (s ′ | s,a B ) = X a R b T mech (s ′ | s,a B ,a R )bπ R c ′ (a R | b R (s)), with error radii ε mech . = sup s,a B ,a R TV T mech (·| s,a B ,a R ), b T mech (·| s,a B ,a R ) , and ε strat (c ′ ) . = sup s TV π R c ′ (·| b R (s)),bπ R c ′ (·| b R (s)) . Theorem 1 (Decomposed simulation bound). Under Assumptions 1–2, for every stationary defender policy π and every c ′ ∈C, V π M c ′ − V π c M c ′ ≤ 2γR max (1− γ) 2 (ε mech + ε strat (c ′ )). Consequently, if bπ ∈ arg max π V π c M c ′ , then V π ⋆ M c ′ − V bπ M c ′ ≤ 4γR max (1− γ) 2 (ε mech + ε strat (c ′ )). 14 Preprint Theorem 1 separates two sources of error. The termε mech measures error in the component assumed to be shared across adversary strategies, whereasε strat (c ′ )measures error associated with the particular adversaryc ′ . Consequently, improvements to b T mech can, under Assumption 2, benefit predictions under multiple adversary strategies. Adaptation toc ′ nevertheless remains dependent on how accurately its strategy can be represented or inferred. Corollary 1 (Residual error under a previously learned mechanism). Fix a target suboptimalityε and suppose ε mech ≤ (1− γ) 2 8γR max ε. Then the bound in Theorem 1 guarantees suboptimality at most ε whenever ε strat (c ′ )≤ (1− γ) 2 8γR max ε. Thus, once the mechanism error is below the stated threshold, the bound requires no further reduction in ε mech for this target accuracy; the remaining contribution is ε strat (c ′ ). Notably, the bound contains no explicit dependence on the number of hostsN. This does not imply that adaptation is independent of network size: larger networks may make state estimation, belief tracking, function approximation, or data collection more difficult. Rather, under the stated factorization, network size does not introduce an additional term into the error decomposition itself. The approximately stable episodes-to-control observed from15to100hosts in RQ3 are empirically consistent with this interpretation, while not constituting a direct consequence of the bound. Remark 1 (Sensitivity of a fixed policy to strategy shift). For a fixed policyπ c , the same simulation argument gives V π c M c − V π c M c ′ ≤ 2γR max (1− γ) 2 sup s TV π R c (·| b R (s)),π R c ′ (·| b R (s)) . This bound quantifies the sensitivity of a fixed policy to an adversary shift. It does not, by itself, establish the interaction complexity required to adapt a model-free agent toc ′ . The distinction relevant to Theorem 1 is therefore analytical: under the factorization of Equation (1), mechanism information constitutes a shared component that can in principle remain useful when the strategy-dependent component changes. BPROOFS FOR SECTION A B.1PROOF OF LEMMA 1 Fix s,a B and abbreviate b = b R (s) and K a R (·) =T mech (·| s,a B ,a R ). By Eq. (1), for any measurable E ⊆S , T c (E | s,a B )−T c ′ (E | s,a B ) = X a R K a R (E) π R c (a R | b)− π R c ′ (a R | b) . Since 0≤ K a R (E)≤ 1, taking the supremum over E and using TV(μ,ν) = sup E |μ(E)− ν(E)| together with sup E X i w i K i (E) ≤ 1 2 X i |w i | for signed weights satisfying P i w i = 0 yields TV T c (·| s,a B ),T c ′ (·| s,a B ) ≤ 1 2 X a R π R c (a R | b)− π R c ′ (a R | b) . The right-hand side is TV(π R c (·| b),π R c ′ (·| b)), which proves the result. 15 Preprint B.2PROOF OF PROPOSITION 1 Sufficiency of the filtering posterior and the existence of an optimal belief-measurable policy are standard results for POMDPs. The belief update is b t+1 (s ′ )∝ Ω(o t+1 | s ′ ) X s T c (s ′ | s,a B t )b t (s). Under Assumption 2, the dependence oncenters only throughT c , which by Eq. (1) depends on the adversary strategyπ R c . Henceb t is a sufficient statistic for optimal control and evolves according to a Bayes operator of the stated form. The proposition concerns the exact filtering posterior of the POMDP. The RSSM interpretation in Appendix A is approximate: CyberWorld does not assume thats t = [h t ;z t ]recoversb t , nor do the RSSM losses establish such recovery. Rather,p φ (z t | h t )andq φ (z t | h t ,e t )play roles analogous to predictive and observation-conditioned latent distributions, respectively, providing a learned recurrent representation on which the policy is conditioned. B.3PROOF OF THEOREM 1 Step 1: One-step model error. Introduce the hybrid kernel e T (s ′ | s,a B ) = X a R b T mech (s ′ | s,a B ,a R )π R c ′ (a R | b R (s)), which combines the learned mechanism model with the true adversary strategy. By the triangle inequality, TV T c ′ , b T c ′ ≤ TV T c ′ , e T + TV e T , b T c ′ . The first term is bounded byε mech because it averages mechanism-model deviations underπ R c ′ (·| b R (s)). The second term is bounded byε strat (c ′ )by the same mixture-contraction argument as Lemma 1. Therefore, TV T c ′ (·| s,a B ), b T c ′ (·| s,a B ) ≤ ε mech + ε strat (c ′ ). Step 2: Simulation bound. Let ε . = ε mech + ε strat (c ′ ). For any stationary policyπ,∥V π ∥ ∞ ≤ R max /(1− γ). A standard simulation-lemma argument (Kearns & Singh, 2002) gives V π M c ′ − V π c M c ′ ≤ X t≥0 γ t+1 2ε R max 1− γ . Summing the geometric series yields V π M c ′ − V π c M c ′ ≤ 2γR max (1− γ) 2 ε. Step 3: Policy suboptimality. Let bπ ∈ arg max π V π c M c ′ . Then V π ⋆ M c ′ − V bπ M c ′ = V π ⋆ M c ′ − V π ⋆ c M c ′ + V π ⋆ c M c ′ − V bπ c M c ′ + V bπ c M c ′ − V bπ M c ′ . The middle term is non-positive by optimality ofbπin c M c ′ , and the first and third terms are each bounded by Step 2. Hence V π ⋆ M c ′ − V bπ M c ′ ≤ 4γR max (1− γ) 2 (ε mech + ε strat (c ′ )). 16 Preprint B.4PROOF OF REMARK 1 Fix a policyπ c and compare its value underM c andM c ′ . Applying the simulation argument from Theorem 1 together with Lemma 1 gives V π c M c − V π c M c ′ ≤ 2γR max (1− γ) 2 sup s TV π R c (·| b R (s)),π R c ′ (·| b R (s)) . This establishes sensitivity of a fixed policy to a change in adversary strategy. The inequality does not imply a particular sample complexity for retraining or adaptation underc ′ . Its role here is to contrast the fixed-policy sensitivity bound with the explicit mechanism–strategy decomposition available in Theorem 1. CCROSSING STATISTICS AND PERFORMANCE PROFILES AT N=15 This appendix provides the measurements behind RQ1 in Section 4.2, in particular the two-to-three- orders-of-magnitude gap to PPO and the first-crossing comparison with GNN-PPO. Figure 3 reports, for each attacker, the environment step at which each method first exceeds the strategy-agnostic control. Table 2 lists the exact values with seed counts, and Figure 8 aggregates them across attackers as a performance profile: for each method, the fraction of all (attacker, seed) runs that have reached the control by environment stepτ. A profile that rises earlier and higher denotes a more sample-efficient defender, and a run that never crosses remains censored at its budget. Table 2: Environment steps to first reach thedeploy_then_stopcontrol, 15 hosts, all methods from scratch. Mean±sd over 3 seeds; four text/multimodal cells use 2 seeds because the third run did not complete;(k/n): seeds that crossed when not all did. Bold: lowest mean in the column. PPO: raw 30-episode checkpoint evaluations, 3 seeds. GNN-PPO: the graph arm’s observation and GAT encoder trained with PPO instead of the world model, online training return, 3 seeds. DFSImpact, BruteForce and Inactive are omitted: their control is 0, so no crossing is defined (all arms end at a final return of 0 there). BFSServerDowntimeServerDowntimeExfiltrationBFSExfiltration PPO (from scratch)2,310,400± 640,000 (2/3)3,078,400 (1/3)2,630,400± 448,000 (2/3)NEVER (0/3) GNN-PPO (from scratch)4,017± 82125,875± 4,025 (2/3)11,367± 9,1829,117± 4,128 ours: vector4,933± 6333,633± 623,900± 28331,650± 12,450 (2/3) ours: graph5,525± 575 (2/3)3,600± 414,600± 4115,750± 12,960 ours: vec+graph6,433± 1,4624,183± 5545,533± 1,89722,325± 12,525 (2/3) ours: text4,900± 750 (2/3)5,283± 1,1094,000± 4007,850± 1,203 ours: multimodal10,100± 0 (1/3)4,600± 1005,525± 1,12512,200± 0 (1/2) 10 3 10 4 10 5 10 6 environment steps τ 0.0 0.2 0.4 0.6 0.8 1.0 fraction of runs at the control by τ CyberWorld budget CyberWorld (vector) (n=12) CyberWorld (graph) (n=12) CyberWorld (vec+graph) (n=12) CyberWorld (text) (n=11) CyberWorld (multimodal) (n=9) GNN-PPO (n=12) PPO (n=12) Figure 8: Performance profile atN =15: fraction of (attacker, seed) runs that have reached the control by environment stepτ, pooled over the four scoreable attackers. Runs that never cross are censored at their budget. Dashed vertical line: the 52.5k-step budget of the world-model runs. 17 Preprint The three numeric representations reach the control in 92% of their runs within the52.5k-step budget, and the text representation does so in all but one of its runs. In contrast, PPO’s profile remains at zero until approximately2M steps and reaches only 5 of 12 runs (42%) by the end of its3.2M-step budget. The multimodal representation plateaus lowest among the world-model arms because only one seed crosses on each of the two BFS attackers. GNN-PPO, the representation-matched baseline trained in the same loop, also reaches 92%, and its profile rises at similar steps to those of the numeric world-model arms; relative to GNN-PPO, the clearest advantage of the world model therefore lies in post-crossing retention rather than initial crossing speed, as Table 3 shows. PPO is particularly seed-sensitive on ServerDowntime: only one of its three seeds ever exceeds the control, whereas every vector and graph seed does so within 4k steps. DSENSITIVITY OF THE CROSSING CRITERION TO THE SMOOTHING WINDOW This appendix examines how the primary trailing-10 crossing metric changes under alternative smoothing windows and tests whether the main sample-efficiency conclusion depends on this conven- tion. The first crossing is defined on the trailing-10 episode return. Because the environment contains a rare−5,000penalty, a short window makes the return curves volatile and could in principle shift the crossing. Figure 9a shows one trajectory (vector, BFSServerDowntime, seed 1) as raw returns and at trailing-10, trailing-50, and trailing-100 resolution; Figure 9b recomputes the first-crossing step of the vector arm on every attacker under windowsW ∈10, 25, 50, 100, as mean±sd over seeds, annotated with the number of seeds that still cross whenever it falls below three. 10 2 10 3 10 4 environment steps −10 5 −10 4 −10 3 −10 2 0 10 2 10 3 episode return (symlog) (a) one trajectory 10 2550 100 window W (episodes) 10 4 2 × 10 3 3 × 10 3 4 × 10 3 6 × 10 3 2 × 10 4 3 × 10 4 4 × 10 4 env steps to reach the control 2/3 2/3 (b) first crossing vs window raw episode return trailing-10 trailing-50 trailing-100 control BFSServerDowntime ServerDowntime Exfiltration BFSExfiltration Figure 9: Smoothing-window sensitivity. (a) One trajectory at four temporal resolutions. (b) First- crossing step of the vector arm vs. the windowWused to define it (mean±sd over 3 seeds;k/3 where fewer seeds cross). In these runs, widening the window generally delays the crossing, but by at most a factor of two on ServerDowntime, Exfiltration, and BFSServerDowntime betweenW =10andW =50, against a10 2 –10 3 gap to PPO. On BFSExfiltration, where the vector arm settles close to the control after crossing, onlyW =10registers a crossing, and only for two of three seeds; wider windows do not, which indicates that the behavior is acquired but not consistently held. The headline conclusion is therefore independent of the smoothing convention, whereas the BFSExfiltration result depends on it, and the main text reports that cell together with its seed count. 18 Preprint EPER-REPRESENTATION LEARNING DYNAMICS AND POST-CROSSING RETENTION This appendix expands the representation comparison of RQ2 in Section 4.2: it provides the complete learning curves summarized in Figure 4 and Figure 5, and the post-crossing statistics on which the retention argument of RQ1 (the contrast between the world-model arms and GNN-PPO) rests. Figure 4 summarizes each representation by its crossing step per attacker. This section provides the underlying evidence: Figure 10 shows the complete learning curve of every representation on every scoreable attacker, Figure 11 presents the same crossing steps as a lookup grid, and Table 3 reports what happens after the crossing, namely the share of post-crossing episodes above the control, the median return of the last 200 training episodes, and the rate of decoy-limit penalties in those episodes, for every method including PPO and GNN-PPO. 10 3 10 4 10 5 10 6 −10 4 −10 3 −10 2 0 10 2 10 3 BFSServerDowntime return (symlog) CyberWorld (vector) (n=3) 10 3 10 4 10 5 10 6 CyberWorld (graph) (n=3) 10 3 10 4 10 5 10 6 CyberWorld (vec+graph) (n=3) 10 3 10 4 10 5 10 6 CyberWorld (text) (n=3) 10 3 10 4 10 5 10 6 CyberWorld (multimodal) (n=3) 10 3 10 4 10 5 10 6 GNN-PPO (n=3) 10 3 10 4 10 5 10 6 −10 5 −10 4 −10 3 −10 2 0 10 2 10 3 ServerDowntime return (symlog) 10 3 10 4 10 5 10 6 10 3 10 4 10 5 10 6 10 3 10 4 10 5 10 6 10 3 10 4 10 5 10 6 10 3 10 4 10 5 10 6 10 3 10 4 10 5 10 6 −10 5 −10 4 −10 3 −10 2 0 10 2 10 3 Ex fi ltration return (symlog) 10 3 10 4 10 5 10 6 10 3 10 4 10 5 10 6 10 3 10 4 10 5 10 6 10 3 10 4 10 5 10 6 10 3 10 4 10 5 10 6 10 3 10 4 10 5 10 6 environment steps −10 4 −10 3 −10 2 0 10 2 10 3 BFSEx fi ltration return (symlog) 10 3 10 4 10 5 10 6 environment steps 10 3 10 4 10 5 10 6 environment steps 10 3 10 4 10 5 10 6 environment steps 10 3 10 4 10 5 10 6 environment steps 10 3 10 4 10 5 10 6 environment steps trailing-50 (seed mean, band = min/max)trailing-10PPO checkpoint evaluationcontrol Figure 10: Learning curves of the five representations and GNN-PPO (columns) on the four scoreable attackers (rows) atN =15. Thick: trailing-50 seed mean with min/max band; thin: trailing-10; black ×: PPO checkpoint evaluations; dashed: control. BFSServerDowntimeServerDowntimeExfiltrationBFSExfiltration vector graph vec+graph text multimodal 4.9k 3/3 3.6k 3/3 3.9k 3/3 31.6k 2/3 5.5k 2/3 3.6k 3/3 4.6k 3/3 15.8k 3/3 6.4k 3/3 4.2k 3/3 5.5k 3/3 22.3k 2/3 4.9k 2/3 5.3k 3/3 4.0k 2/2 7.9k 3/3 10.1k 1/3 4.6k 2/2 5.5k 2/2 12.2k 1/2 Figure 11: Environment steps to first reach the control per representation and attacker atN =15 (darker = fewer;k/n= seeds that reached it; grey = no seed reached it within the52.5k-step budget). The crossing is the same event for every arm: a transition from the random-policy return of about −10 4 to the level of the control within a few hundred episodes. The arms differ in retention, where 19 Preprint Table 3: Retention after the first crossing, 15 hosts. Each cell: share of post-crossing episodes above the control / median return of the last 200 training episodes / decoy-limit penalties per 100 of those episodes. CyberWorld statistics are taken at the end of its52.5k-step budget with the stochastic training policy; PPO’s are its 30-episode checkpoint evaluations after its own crossing and its last three checkpoints, i.e. after up to3.2M steps. The two columns therefore measure retention under very different budgets and are not bolded. Seeds are pooled; four text and multimodal cells have two seeds. BFSServerDowntimeServerDowntimeExfiltration BFSExfiltration PPO56% / 899 / 2.653% / 886 / 8.939% / 290 / 5.6– /−1 / 0.0 GNN-PPO23% /−3,700 / 237.2 28% /−3,700 / 220.812% / 0 / 167.817% / 0 / 66.0 CyberWorld (vector)33% / 888 / 23.751% / 1,192 / 0.8 30% / 285 / 55.021% / 0 / 0.5 CyberWorld (graph)30% / 388 / 14.749% / 1,181 / 1.732% / 190 / 2.737% /−4 / 0.0 CyberWorld (vec+graph)43% / 1,092 / 15.751% / 1,281 / 35.5 38% / 582 / 14.88% /−3 / 0.0 CyberWorld (text)38% / 886 / 19.735% / 900 / 57.0 26% / 192 / 72.226% /−2 / 0.0 CyberWorld (multimodal)14% / 490 / 23.038% / 1,186 / 0.528% / 89 / 3.217% /−8 / 0.0 the comparison with PPO is mixed; Table 3 accordingly carries no bold entries, since the two budgets differ by a factor of60. On ServerDowntime the vector and graph arms hold the control in about half of their post-crossing episodes, with a late median above it and fewer than two penalties per 100 episodes, which is on par with PPO after3M steps. On Exfiltration the vector and text arms incur the decoy-limit penalty in more than half of their late episodes, whereas the graph and multimodal arms do not. On BFSExfiltration the vec+graph, text, and multimodal arms settle at the do-nothing return of 0 after a brief excursion above it, which corresponds to a collapsed rather than a stable policy. The multimodal arm holds the control with the fewest penalties on ServerDowntime and Exfiltration, at the cost of crossing later or on fewer seeds than the numeric arms. GNN-PPO is the clearest instance of a crossing that does not constitute a defence: its late median return is−3,700 on both ServerDowntime attackers and 0 on both Exfiltration attackers, with 66 to 237 decoy-limit penalties per 100 late episodes, that is, the policy that crossed continues to deploy beyond the cap. FSCALING ANALYSIS: CROSSING STATISTICS, LEARNING CURVES, AND COMPUTATIONAL COST FOR N ∈25, 50, 100 This appendix provides the evidence behind RQ3 in Section 4.2: the exact values summarized in Figure 7 and Figure 6, every learning curve atN ∈25, 50, 100, and the computational cost that the limitations in the Summary paragraph and in Section 4.2 refer to. Figure 7 fits the cost of adaptation againstNover the runs that crossed. Table 1 in the main text gives the crossing step per attacker and size with seed counts, including PPO at every size at which it was trained. Figure 12 reports the cost of each representation asNgrows, in terms of peak GPU memory, training seconds per environment step, and the size of the intermediate representation, and Figure 13 shows every learning curve for every attacker and size, including the three attackers whose control is 0 and for which no crossing is defined. AtN =25the curves resemble those atN =15with a longer prefill. AtN =50the crossing moves to 16–38k steps and the seed spread around the control widens in absolute return, because the control itself is5–6×larger. AtN =100the52.5k-step budget covers only 157 episodes, the curves are still rising when the budget ends, and the wide seed bands indicate that crossing becomes highly seed-dependent; the power-law fit in Figure 7 uses only the runs that did cross and is therefore a statement about the cost of the successful runs rather than a guarantee of success. The graph representation’s advantage on the topology-following attacker is obtained at the cost of memory, 43 GB atN =100against 10 GB for the vector representation, at similar seconds per environment step. GNN-PPO’s curves in Figure 13 exhibit the pattern of Table 1 at every size, an early excursion above the control followed by a return that settles far below it, reaching−9,000to−11,000on the ServerDowntime attackers at N =50. 20 Preprint 152550 100 hosts N 0 10 20 30 40 GPU memory (GB) (a) memory 152550 100 hosts N 0.0 0.5 1.0 1.5 2.0 2.5 train s / env step (b) compute 152550 100 hosts N 10 2 10 3 10 4 count (c) IR size CyberWorld (vector)CyberWorld (graph)IR nodesedges Figure 12: Compute scaling of the two numeric representations. (a) Peak GPU memory and (b) training seconds per environment step (median over runs, band = min/max); (c) nodes and edges of the intermediate representation vs. N . GOPEN-LOOP IMAGINATION OF THE LEARNED WORLD MODEL Figure 14 visualizes open-loop trajectories imagined by the learned world model. The model consumes the host representation and adjacency defined in Section 3.2. The192×256semantic canvas shown here is an exactly invertible rendering of the four reconstruction-role bits: hosts are drawn as circles, decoys as squares, and node colour encodes the observable role state. Because the rendering is invertible, decoded node states can be repainted onto the same canvas as the observed states, making the two rows directly comparable. The top row shows a held-out BFSExfiltration episode. The graph arm receives observations for the first four displayed steps. After the fourth observed step, the latent state is rolled forward open-loop using the true defender actions but without further environment observations, and each imagined latent state is decoded and repainted. Hosts whose imagined state differs from the observed state are marked with red rings, while steps on which the adversary touched a decoy are marked with yellow borders. The topology is fixed by construction throughout the visualization, while the model approximately preserves the observable host-state layer during the open-loop rollout. Its errors are concentrated in two aspects of the decoded state. First, the model can misplace deployed decoys among reserved slots within the same subnet. These slots are implementation-level bindings that are only indirectly reflected in the observation, so an imagined rollout may assign a deployed decoy to a neighbouring slot while preserving the broader subnet-level structure. Second, the model can fail to anticipate new alert activity. Once the rollout becomes fully open-loop, the prior receives no new observation indicating that adversarial activity has reached another host, and the decoded state can therefore continue to predict that host as quiet after the observed trajectory begins to alert there. The reward predictions expose the same limitation. Decoy-touch events are sparse and depend on latent adversarial progress that is only partially revealed through subsequent alerts. During the open-loop rollout, the predicted reward often remains near zero at these events, and large negative rewards can be temporally misaligned with the observed decoy-limit penalties. Thus, the figure should not be interpreted as demonstrating pixel-perfect or event-perfect long-horizon prediction. Instead, it shows that the learned latent dynamics preserve substantial coarse host-state structure while becoming less reliable for events that depend on unobserved adversarial progress and precise reward timing. 21 Preprint 10 3 10 4 10 5 10 6 −10 4 −10 3 −10 2 0 10 2 10 3 BFSServerDowntime return (symlog) N= 15 hosts 10 3 10 4 10 5 10 6 −10 5 −10 4 −10 3 −10 2 0 10 2 10 3 N= 25 hosts 10 4 10 5 10 6 10 7 −10 5 −10 4 −10 3 −10 2 0 10 2 10 3 10 4 N= 50 hosts 10 4 −10 5 −10 4 −10 3 −10 2 0 10 2 10 3 10 4 N= 100 hosts 10 3 10 4 10 5 10 6 −10 5 −10 4 −10 3 −10 2 0 10 2 10 3 ServerDowntime return (symlog) 10 3 10 4 10 5 10 6 −10 5 −10 4 −10 3 −10 2 0 10 2 10 3 10 4 10 5 10 6 10 7 −10 5 −10 4 −10 3 −10 2 0 10 2 10 3 10 4 10 4 −10 5 −10 4 −10 3 −10 2 0 10 2 10 3 10 4 10 3 10 4 10 5 10 6 −10 5 −10 4 −10 3 −10 2 0 10 2 10 3 Ex fi ltration return (symlog) 10 3 10 4 10 5 10 6 −10 5 −10 4 −10 3 −10 2 0 10 2 10 3 10 4 10 5 10 6 10 7 −10 5 −10 4 −10 3 −10 2 0 10 2 10 3 10 4 10 4 −10 5 −10 4 −10 3 −10 2 0 10 2 10 3 10 4 10 3 10 4 10 5 10 6 −10 4 −10 3 −10 2 0 10 2 10 3 BFSEx fi ltration return (symlog) 10 3 10 4 10 5 10 6 −10 5 −10 4 −10 3 −10 2 0 10 2 10 3 10 4 10 5 10 6 10 7 −10 5 −10 4 −10 3 −10 2 0 10 2 10 3 10 4 −10 6 −10 5 −10 4 −10 3 −10 2 0 10 2 10 3 10 4 10 3 10 4 10 5 10 6 −10 4 −10 3 −10 2 0 DFSImpact return (symlog) 10 3 10 4 10 5 10 6 −10 4 −10 3 −10 2 0 10 4 10 5 10 6 10 7 −10 5 −10 4 −10 3 −10 2 0 10 4 −10 5 −10 4 −10 3 −10 2 0 10 3 10 4 10 5 10 6 −10 4 −10 3 −10 2 0 BruteForce return (symlog) 10 3 10 4 10 5 10 6 −10 4 −10 3 −10 2 0 10 4 10 5 10 6 10 7 −10 5 −10 4 −10 3 −10 2 0 10 4 −10 5 −10 4 −10 3 −10 2 0 10 3 10 4 environment steps −10 4 −10 3 −10 2 0 Inactive return (symlog) 10 3 10 4 environment steps −10 4 −10 3 −10 2 0 10 4 environment steps −10 5 −10 4 −10 3 −10 2 0 10 4 environment steps −10 5 −10 4 −10 3 −10 2 0 CyberWorld (vector) CyberWorld (graph) GNN-PPOPPO checkpoint evaluationcontrol Figure 13: Return vs. environment steps for every attacker (rows) and network size (columns): CyberWorld vector, graph, and GNN-PPO (trailing-50 seed mean with min/max band, thin = trailing- 10), PPO checkpoint evaluations, and the control. 22 Preprint ground truth t=12 deploy 2 r=-1 t=13 remove 5 r=+0 t=14 deploy 3 r=-5000 t=15 deploy 3 r=-5000 t=16 deploy 3 r=-1 t=17 remove 4 r=+0 t=18 deploy 1 r=-5000 t=19 deploy 3 r=+99 CyberWorld (graph) ̂ r=-4637 ̂ r=-0 ̂ r=-0 ̂ r=-0 ̂ r=-4205 ̂ r=+0 ̂ r=-13 ̂ r=-3835 ground truth t=20 remove 6 r=+99 t=21 remove 5 r=+100 t=22 deploy 3 r=+99 t=23 remove 4 r=+99 t=24 remove 6 r=+100 t=25 deploy 1 r=+100 t=26 deploy 1 r=+0 t=27 deploy 2 r=-5000 CyberWorld (graph) ̂ r=-0 ̂ r=+0 ̂ r=-0 ̂ r=+0 ̂ r=+0 ̂ r=+0 ̂ r=-4883 ̂ r=-1 host, quiet host, alerted earlier host, alerting now host, alerting now + earlier decoy, quiet decoy touched by red host predicted wrongly context (observed) decoy touched, r= +100 Figure 14: Open-loop imagination of the graph arm on a held-out BFSExfiltration episode. Top: the observed canvas at each step, annotated with the defender action that led to the step and the true reward. Bottom: the decoded model state, obtained from the posterior during the four observed steps (green border) and from the RSSM prior under the true defender actions thereafter, with the predicted reward shown below each canvas. Red rings mark hosts whose imagined state differs from the observed state; yellow borders mark steps on which the adversary touched a decoy. Despite these open-loop errors, the learned policy reaches the control in the corresponding experi- ments. This suggests that precise long-horizon reconstruction of every latent event is not necessary for acquiring the defensive behavior observed in our evaluation, although improving prediction of adversary-dependent events remains an important direction for future work. 23