Paper deep dive
Towards Hierarchical Cyber Defense with Large Language Models: From Planning to Execution
Harshith Doppalapudi, Nathaniel D. Bastian, Ankit Shah
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 10/2/2026, 9:18:33 AM
Summary
This paper investigates the use of frozen, zero-shot Large Language Models (LLMs) in a hierarchical cyber defense framework to eliminate the need for retraining across different network scales. Using the Cyberwheel environment, the authors compare Reinforcement Learning (RL) and LLM-based controllers in a planner-executor hierarchy. Results indicate that while LLM-only planning offers limited gains, extending LLM control to tactical execution (LLM+LLM) allows capable models (e.g., Llama-3.3-70B) to maintain strong defensive performance across small, medium, and large networks without retraining, significantly outperforming RL baselines which require scale-specific retraining.
Entities (10)
Relation Signals (7)
RL+RL → requires → Retraining
confidence 95% · the RL baseline is retrained for each scale
Cyberwheel → usedby → LLM+LLM
confidence 95% · Using the high fidelity Cyberwheel environment... we compare RL+RL, LLM+RL, and LLM+LLM configurations
LLM+LLM → eliminates → Retraining
confidence 93% · frozen LLMs can maintain strong defensive performance across the evaluated network scales without task-specific retraining
Llama-3.3-70B → achievesperformancein → LLM+LLM
confidence 92% · For instance, a frozen general purpose 70B model holds successful lateral movement to approximately 1% of steps... using the same model weights
Cyberwheel → developedby → Oak Ridge National Laboratory
confidence 90% · Cyberwheel... developed by Oak Ridge National Laboratory
LLM+LLM → outperforms → RL+RL
confidence 90% · extending LLM control to execution produces notable improvements... while the RL baseline is retrained for each scale
Red Team Agent → mappedto → MITRE ATT&CK
confidence 88% · built-in automated red team agent mapped to the MITRE ATT&CK framework
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:An autonomous cyber defender trained with reinforcement learning (RL) is typically tied to the network on which it was trained, limiting its ability to generalize as network scale changes. Hierarchical RL reduces decision complexity by separating strategic targeting from tactical execution, but it does not eliminate this retraining dependence. We investigate whether frozen, zero-shot large language models (LLMs) can provide retraining-free control in hierarchical cyber defense and how performance changes as LLM control is extended from planning to execution. We formulate a controller-agnostic planner-executor hierarchy in which the planner selects a subnet to defend over a fixed horizon and the executor selects defensive actions within that subnet. Using the high fidelity Cyberwheel environment, with its built-in automated red team agent mapped to the MITRE ATT&CK framework, we compare RL+RL, LLM+RL, and LLM+LLM configurations using six models ranging from 3B to 70B parameters, including two cybersecurity-specialized models, across small, medium, and large networks. Replacing only the planner with an LLM yields limited gains as network size increases. In contrast, extending LLM control to execution produces notable improvements for sufficiently capable models. For instance, a frozen general purpose 70B model holds successful lateral movement to approximately 1% of steps and attacker impact near zero across all three network scales using the same model weights, while the RL baseline is retrained for each scale. Our results show that sufficiently capable frozen LLMs can maintain strong defensive performance across the evaluated network scales without task-specific retraining, while also indicating that strong tactical execution is important to realizing the benefits of LLM-based control.
Tags
Links
- Source: https://arxiv.org/abs/2610.00590v1
- Canonical: https://arxiv.org/abs/2610.00590v1
Trouble viewing inline? Open PDF directly →
Full Text
35,048 characters extracted from source content.
Expand or collapse full text
Towards Hierarchical Cyber Defense with Large Language Models: From Planning to Execution Harshith Doppalapudi Affiliation: Indiana University Affiliation: Bloomington, IN, USA Email: hdoppala@iu.edu Nathaniel D. Bastian Affiliation: Johns Hopkins University Affiliation: Baltimore, MD, USA Email: ndbastian@jhu.edu Ankit Shah Note: Affiliation: Indiana University Affiliation: Bloomington, IN, USA Email: ankit@iu.edu Abstract An autonomous cyber defender trained with reinforcement learning (RL) is typically tied to the network on which it was trained, limiting its ability to generalize as network scale changes. Hierarchical RL reduces decision complexity by separating strategic targeting from tactical execution, but it does not eliminate this retraining dependence. We investigate whether frozen, zero-shot large language models (LLMs) can provide retraining-free control in hierarchical cyber defense and how performance changes as LLM control is extended from planning to execution. We formulate a controller-agnostic planner-executor hierarchy in which the planner selects a subnet to defend over a fixed horizon and the executor selects defensive actions within that subnet. Using the high-fidelity Cyberwheel environment, with its built-in automated red team agent mapped to the MITRE ATT&CK framework, we compare RL+RL, LLM+RL, and LLM+LLM configurations using six models ranging from 3B to 70B parameters, including two cybersecurity-specialized models, across small, medium, and large networks. Replacing only the planner with an LLM yields limited gains as network size increases. In contrast, extending LLM control to execution produces notable improvements for sufficiently capable models. For instance, a frozen general purpose 70B model holds successful lateral movement to approximately 1% of steps and attacker impact near zero across all three network scales using the same model weights, while the RL baseline is retrained for each scale. Our results show that sufficiently capable frozen LLMs can maintain strong defensive performance across the evaluated network scales without task-specific retraining, while also indicating that strong tactical execution is important to realizing the benefits of LLM-based control. A Preprint †footnotetext: Send correspondence to ankit@iu.edu © 2026 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works. Keywords Autonomous cyber defense, Network security, Hierarchical control, Reinforcement learning, Large language models, Agentic AI 1 Introduction Autonomous cyber defense requires agents to make sequential decisions as an attacker moves through a network toward critical assets. As enterprise networks grow, this problem becomes increasingly difficult as the defender must determine both where to intervene and which countermeasure to execute, while operating from incomplete and noisy observations. Reinforcement learning (RL) provides a natural framework for learning such policies, but learned defenders are typically coupled to the network configurations on which they are trained. Changes in network size or topology can therefore require costly retraining, limiting their ability to transfer across operational environments. Hierarchical reinforcement learning (HRL) addresses part of this challenge by decomposing defense into decisions at different levels of abstraction. A high level policy can select the region of the network requiring attention, while a lower-level policy determines the defensive action within that region [2]. This decomposition reduces the complexity of the joint decision space and has shown promise for scalable autonomous cyber defense. However, hierarchy alone does not eliminate the dependence of learned policies on their training environments. A defender that must be retrained whenever the underlying network changes remains difficult to deploy across heterogeneous or evolving networks. Pretrained large language models (LLMs) offer a fundamentally different form of generalization. A frozen LLM can exploit knowledge acquired during pretraining without task-specific parameter updates and can reason over structured representations of previously unseen situations. Recent studies have explored LLMs as autonomous cyber defenders [9] and as mechanisms for designing rewards for RL-based defenders [10]. However, prior work has not examined frozen LLM control within a two-level planner-executor hierarchy across vastly different network scales, nor how performance changes as LLM control is extended from strategic planning to tactical execution. The planner appears to be a natural fit. In a hierarchical network defender, the planner selects among a relatively small set of network regions, whereas the executor must translate that strategic choice into host-level defensive actions. Because the planner’s decision space changes little as the underlying network grows, one might expect that replacing a learned planner with a frozen LLM would capture much of the generalization benefit while retaining an efficient RL executor. Yet this assumption has not been systematically tested. Also, it is unknown whether the principal limitation of a hierarchical defender lies in strategic targeting, i.e., deciding where to act, or in tactical execution, i.e., deciding which action to take once the target region has been selected. We investigate these questions through a controller-agnostic planner-executor hierarchy in which either level can be instantiated by a frozen, zero-shot LLM. This design enables a controlled comparison of three architectures under identical observations, actions, rewards, attacker behavior, and hierarchical interfaces. These architectures include a fully trained RL hierarchy (RL+RL), an LLM planner coupled with a trained RL executor (LLM+RL), and an all-LLM hierarchy (LLM+LLM). We evaluate six general purpose and cybersecurity-specialized models ranging from 3B to 70B parameters in Cyberwheel [1] across three network sizes (small, medium, and large). Our experiments suggest a result that is more nuanced than simply replacing RL with an LLM. Introducing an LLM only at the planning level provides limited improvement as network size grows, and increasingly capable planners produce relatively small gains when paired with the same trained executor. In contrast, when a sufficiently capable frozen LLM controls both planning and execution, performance changes dramatically. For instance, we found that Llama-3.3-70B reduces and maintains compromise rate to approximately 1% across all three network sizes and drives attacker impact to nearly zero, using the same model weights without retraining. This effect is not universal as smaller models and some cybersecurity-specialized models fail to maintain this robustness, indicating that pretrained control generalizes only when sufficient capability extends to the tactical layer. These patterns are directly reflected in our cross-scale RL+RL, LLM+RL, and LLM+LLM experimental results. We therefore ask the following research questions (RQs): • RQ1: Can a frozen, zero-shot LLM-based hierarchical defender maintain effective cyber defense across network scales without task-specific retraining? • RQ2: How does extending LLM control from strategic planning to tactical execution affect hierarchical cyber defense across network scales? • RQ3: How do model scale and cybersecurity specialization affect the robustness of frozen LLM controllers as network size increases? Our study makes three contributions: • Controller-agnostic hierarchical framework: We develop a planner-executor architecture that permits frozen LLMs to be interchanged at the planner and executor levels while preserving the same environment, reward, observation, and action interfaces. • Controlled cross-scale evaluation: We compare RL+RL, LLM+RL, and LLM+LLM defenders using six models from 3B to 70B parameters across a 67-fold increase in network size, explicitly testing whether pretrained controllers can avoid environment-specific retraining. • Characterization of LLM control across the hierarchy: We show that stronger planning alone provides limited benefit when tactical execution remains fixed, whereas extending sufficiently capable LLM control to execution yields substantially stronger cross-scale performance. We further find that cybersecurity specialization can improve particular models but does not, by itself, guarantee robustness as the network scales. The rest of the paper is organized as follows. We first review the related work, then describe the hierarchical control architecture and experimental setup, and finally present the results, conclusions, and directions for future research. 2 Related Work Autonomous cyber defense environments. Cyber defense gyms formulate blue-versus-red interaction as a sequential decision problem. CybORG and the associated CAGE challenges support increasingly complex autonomous defense scenarios, from earlier single agent settings [8] to CAGE Challenge 4, a scalable multi-agent environment in which defenders protect different network subnets [7]. Cyberwheel, used in this study, is a configurable simulation and emulation platform developed by Oak Ridge National Laboratory for evaluating autonomous cyber defense under controllable network, attacker, observation, action, and reward settings [1]. Its configurability enables our controlled evaluation across network scales. Hierarchical reinforcement learning for cyber defense. Hierarchical RL reduces control complexity by decomposing high-level coordination from lower-level action selection. A hierarchical proximal policy optimization (PPO) architecture was recently proposed for cyber network defense in which a master policy coordinates PPO-trained sub-policies, addressing large policy spaces and improving performance and convergence across multiple adversaries [2]. Their results also demonstrate that learned sub-policies can be transferred and fine-tuned when network or adversary conditions change. Building on this hierarchical control paradigm, we use PPO for the learned controllers and investigate whether frozen LLMs can provide effective control across network scales without task-specific retraining. Large language models for cyber defense. On the defensive side, recent work has evaluated LLM-based defenders in the multi-agent CybORG CAGE 4 environment, including mixed teams of LLM- and RL-based agents [9], while other studies have used LLM-generated reward functions to train RL-based defenders in Cyberwheel [10]. These studies establish complementary roles for LLMs in autonomous cyber defense, but do not study frozen LLM control within a two-level planner-executor hierarchy across network scales. To our knowledge, prior work has not systematically compared planner-only LLM integration with extending LLM control to tactical execution under common hierarchical interfaces, nor evaluated the same frozen controllers across substantially different network sizes. We address these gaps through a controlled comparison of RL+RL, LLM+RL, and LLM+LLM while varying model scale and cybersecurity specialization. 3 A Controller-Agnostic Hierarchical Control Architecture To separate the effect of controller type from its role in the hierarchy, we formulate autonomous cyber defense as a controller-agnostic planner-executor architecture. The planner determines where defensive effort should be focused, while the executor determines which action should be taken within the selected region. As illustrated in Figure 1, this formulation supports a controlled progression from a fully learned hierarchy (RL+RL) to LLM-based planning (LLM+RL) and finally, to LLM control at both planning and execution levels (LLM+LLM). This design allows us to examine whether LLM-based planning alone is sufficient and what changes when LLM control is extended to tactical execution. The details about the environment, attacker, controller settings, and hyperparameters are described in Section 4. Figure 1: Controller-agnostic planner-executor architecture. LLM+RL introduces frozen LLM control at strategic planning, while LLM+LLM extends it to tactical execution; RL+RL serves as the trained hierarchical baseline. 3.1 Defense as a Partially Observable Decision Process We model the defender as a partially observable Markov decision process (POMDP) ℳ=(,,P,R,Ω,O,γ),M= (S,A,P,R, ,O,γ ), (1) where S is the hidden environment state, A is the defender action set, P(s′∣s,a)P(s s,a) is the transition kernel induced jointly by the defender and the scripted attacker, R:×→ℝR:S×A×S is the reward, Ω is the observation space, O(o∣s)O(o s) is the stochastic observation kernel, and γ∈(0,1)γ∈(0,1) is the discount factor. The hidden state captures network topology, per-host compromise, attacker kill-chain progress, and defender modifications. The defender observes only noisy detector-derived evidence, ot∼O(⋅∣st)o_t O(· s_t), rather than the true state. Because the observation includes decayed alert-history features (Section 4.1), we use reactive policies conditioned on the current observation as a practical approximation to full history- or belief-state policies. 3.2 Planner-Executor Decomposition Let the network be partitioned into B subnets, defining the goal set =1,…,BG=\1,…,B\, where each goal identifies the subnet on which the defender should focus. A primitive defender action is a=(τ,u),u∈,a=(τ,u), u , (2) where u is a defensive primitive and τ is its target when required. The target granularity depends on the primitive, which includes host-level for isolation, subnet-level for decoy placement, and none for a no-op. The hierarchy separates two qualitatively different decisions. The planner πhi _hi performs strategic targeting by selecting a subnet, whereas the executor πlo _lo performs tactical execution by selecting the defensive action within that subnet. This distinction is central to our experimental design as the planner operates over the comparatively compact goal set G, while the executor translates that goal into local host- and action-level decisions. The planner acts every k environment steps, gj∼πhi(⋅∣ojkhi),gt=g⌊t/k⌋,g_j _hi\! (· o^hi_jk ), g_t=g_ t/k , (3) where ohio^hi is a subnet-level representation of the observation. The selected goal remains fixed for the following k steps. Conditioned on this goal, the executor acts at every environment step: at∼πlo(⋅∣ot,gt).a_t _lo\! (· o_t,g_t ). (4) In practical terms, the hierarchy first maps a noisy observation to a subnet-level defensive goal and then maps that goal, together with the current observation, to a concrete defensive action. This fixed-duration hierarchy induces a temporal abstraction in which planner decisions are separated by k primitive transitions, with discount γkγ^k between successive planner decisions. Defense quality is defined by the episodic discounted return J=[∑t=0T−1γtrt],rt=R(st,at,st+1),J=E\! [ _t=0^T-1γ^tr_t ], r_t=R(s_t,a_t,s_t+1), (5) over an episode of length T. RL controllers are trained to maximize (5), whereas frozen LLM controllers receive no task-specific parameter updates and are evaluated under the same environment reward. 3.3 Reward Structure All controller configurations operate under the same defensive objective. The reward is additively decomposed into four terms: rt=−∑c∈tσ(c)⏟attacker progress+∑b∈ℬtβ(b)⏟defensive benefit−κ(at)⏟action cost−ϕ(at,st)⏟collateral disruption,r_t= -\!\! _c _tσ(c)_attacker progress+ _b _tβ(b)_defensive benefit- κ(a_t)_action cost- φ(a_t,s_t)_collateral disruption, (6) where tC_t contains the successful attacker sub-actions realized during the t→t+1t\!→\!t+1 transition and σ weights them according to kill-chain severity, increasing from reconnaissance to the terminal impact action. ℬtB_t contains productive defensive outcomes, such as diverting the attacker into a decoy, detecting or containing it, or blocking lateral movement, with corresponding benefit β. The term κ(at)κ(a_t) captures the cost of defensive actions, including immediate and capped recurring costs, while ϕ(at,st)φ(a_t,s_t) penalizes unnecessary disruption, such as isolating an uncompromised host. For clarity, we omit from (6) a zone-matching scale factor used by the implementation when the defender acts outside the attacker’s subnet. The values of σ, β, κ, and ϕφ are provided in Section 4.1. The same reward is used across RL+RL, LLM+RL, and LLM+LLM configurations, so all controllers are evaluated against a common operational objective. 3.4 Controller Realizations The planner and executor in (3)-(4) expose common decision interfaces but do not prescribe how those decisions are produced. We instantiate each role using either a trained RL policy or a frozen, zero-shot LLM. This separation allows controller type to be varied without changing the hierarchical decomposition, environment dynamics, or reward. 3.5 Experimental Instantiations The controller-agnostic formulation allows us to vary how LLM capability is introduced into the hierarchy while keeping the decision process and interfaces fixed. • RL+RL is the hierarchical RL baseline, in which both planner and executor are trained with PPO. This configuration represents the conventional learned hierarchy and provides the reference against which retraining-free LLM control is evaluated. • LLM+RL replaces only the planner with a frozen, zero-shot LLM while retaining the trained PPO executor. This configuration tests the natural planner-first hypothesis motivated by the hierarchy, as described in the Introduction section. • LLM+LLM extends frozen LLM control from planning to execution, replacing the PPO executor with an LLM controller as well. Comparing LLM+RL with LLM+LLM therefore measures what changes when LLM control is extended from strategic targeting to tactical action selection while retaining LLM-based planning. All configurations share the same underlying decision process ℳM, reward (6), goal set G, planner horizon k, and environment interfaces. Thus, the configurations hold the decision process and environment interfaces fixed while varying controller composition under a common experimental protocol. 4 Experimental Setup 4.1 Cyberwheel Environment and Threat Model We instantiate the hierarchical defender in Cyberwheel [1], a high-fidelity cyber defense environment developed by the U.S. Department of Energy’s Oak Ridge National Laboratory. Cyberwheel represents an enterprise network as a graph of hosts organized into subnets and connected through routers, routing rules, and firewall policies. We evaluate three network scales: small (15 hosts), medium (100 hosts), and large (1,010 hosts). Attacker. The red agent is scripted and executes an Atomic Red Team-based attack sequence mapped to MITRE ATT&CK techniques [6, 5]. Under the server downtime objective, the attacker begins from a randomly selected entry host and progresses toward a server through the ordered sequence pingsweep → portscan → discovery → lateral movement → privilege escalation → impact. The terminal impact action disrupts the target service and represents the attacker’s primary objective. Consistent with the common reward defined in Section 3.3, successful attacker actions are weighted according to kill-chain severity: σ=1, 2, 5, 10, 20, 50,σ=\1,\,2,\,5,\,10,\,20,\,50\, (7) corresponding respectively to pingsweep, portscan, discovery, lateral movement, privilege escalation, and impact. Observations. The defender receives detector-derived observations rather than the underlying network state. For each host, the observation encodes the current alert, decayed alert history, defensive state indicators such as host isolation and decoy deployment, and an inferred attack stage index ranging from 0 (no observed activity) to 6 (impact). These host-level features are augmented with subnet-level aggregates describing current alert volume, historical alert activity, and trend, together with the number of deployed decoys. Consistent with the planner-executor decomposition in Section 3, the planner receives a subnet-level view ohio^hi for strategic targeting. The executor acts at every environment step using the current observation together with a one-hot representation of the selected goal gtg_t. Thus, the planner determines where defensive attention should be directed, while the executor determines which action and target should implement that decision. Actions. The primitive defensive action set is =Deploy Decoy,Isolate Host,Nothing.U=\ Deploy Decoy,\, Isolate Host,\, Nothing\. (8) Deploy Decoy places a decoy within the selected subnet to detect or divert the attacker, Isolate Host removes a host from normal network connectivity to contain lateral movement, and Nothing leaves the current configuration unchanged. Host isolation incurs an immediate action cost of −1-1 and a recurring cost of −0.1-0.1, whereas decoy deployment incurs no immediate cost but a small recurring cost of −0.1-0.1, with recurring costs capped at −1.5-1.5 per step. Successful defensive outcomes receive bonuses of +15+15 for diverting the attacker into a decoy or blocking lateral movement, +10+10 for catching the attacker, and +25+25 for protecting a critical asset, while unnecessary disruption, such as isolating an uncompromised host, incurs a penalty of −10-10. These terms instantiate the common reward defined in Section 3.3. The planner horizon is fixed at k=5k=5, so each selected subnet remains the active goal for five environment steps. 4.2 Controller Instantiation and LLM Models The three configurations introduced in Section 3.5 differ only in how the planner and executor controllers are instantiated. For each network scale, the RL+RL baseline uses PPO policies trained specifically for that network under the same reward, observation, action, and hierarchical interfaces used by the LLM-based variants. Accordingly, the RL baseline is retrained as network scale changes rather than transferred unchanged across the three environments. The RL and LLM controllers therefore represent different training regimes. Our comparison is intended to evaluate their deployment behavior under common defense interfaces, rather than to provide a matched comparison of training data, compute, or sample efficiency. We evaluate six LLMs selected to vary two dimensions, model scale and cybersecurity specialization, relevant to our research questions. The general purpose models are Llama-3.2-3B, Qwen3-8B, Qwen3-32B, and Llama-3.3-70B. We additionally evaluate Foundation-Sec-8B [3] and Trendyol-Cybersecurity-70B [4] as cybersecurity-specialized models, providing specialized comparators at the 8B and 70B scales. For Qwen3-8B and Qwen3-32B, the models’ explicit reasoning/thinking mode is enabled at the planning level. Each LLM receives a textual rendering of the observation appropriate to its role and returns a structured JSON decision that is parsed into the corresponding planner goal or executor action. All models are served using Q4_K_M quantization through Ollama on a single NVIDIA H100 GPU with low-temperature decoding. No gradients or task-specific parameter updates are applied to any LLM. 4.3 Evaluation Protocol To preserve comparability across controller configurations, all evaluations use a common global seed so that environment stochasticity and scripted attacker behavior are matched across conditions. Each configuration is evaluated over 20 episodes of 100 environment steps. The common seed supports controlled paired comparisons across configurations, but the present evaluation does not characterize variability across independent random seeds. The executor uses deterministic (argmax) decoding during evaluation. Across all LLM experiments, every generated decision is successfully parsed into a valid planner goal or executor action, with no heuristic fallback required. Thus, the reported outcomes reflect the decisions produced directly by the evaluated controllers rather than recovery rules for malformed outputs. The protocol is designed around two complementary comparisons. First, performance across the small, medium, and large networks tests whether a frozen LLM can retain defensive effectiveness as network scale changes without model retraining. Second, the comparison between LLM+RL and LLM+LLM tests how performance changes when frozen LLM control is extended from strategic planning to tactical execution. RL+RL provides the trained hierarchical reference at each network scale. 4.4 Evaluation Metrics We evaluate defensive performance using three step normalized rates. Let ctlatc_t^lat indicate a successful red lateral-movement action at step t, ctimpc_t^imp indicate a successful terminal impact action, and dtd_t indicate that the red action either fails or is diverted into a decoy. For an episode of length T, we define the following metrics: • Compromise rate measures the frequency with which the attacker successfully establishes a new foothold through lateral movement. It is interpreted as a step normalized attack progression rate. Compromise=100T∑t=0T−1ctlatCompromise= 100T _t=0^T-1c_t^lat (9) • Impact rate measures how frequently the attacker reaches and successfully executes its terminal kill-chain objective. Impact=100T∑t=0T−1ctimpImpact= 100T _t=0^T-1c_t^imp (10) Because impact represents the attacker’s most severe outcome and ultimate objective, we treat it as the primary security outcome metric. Lower compromise and impact rates indicate stronger defense. • Defense rate is a passive success measure that counts steps in which the red action fails for any reason or is diverted into a decoy. Because this measure can include intrinsic attacker failure rather than an explicit defender block, we use it as a supporting indicator rather than as the primary measure of defender effectiveness; higher values are better. Defense=100T∑t=0T−1dtDefense= 100T _t=0^T-1d_t (11) In Equation 11, dt=1d_t=1 when the red action fails or is diverted into a decoy, and dt=0d_t=0 otherwise. 5 Results Table 1 reports compromise, impact, and defense rates for the various configurations across network scales. 5.1 RL Baseline Performance Across Network Scale The trained RL+RL baseline becomes less effective as network size increases despite being trained separately for each scale. Compromise rises from 5.32% on the small network to 14.88% and 17.95% on the medium and large networks, respectively. Impact follows the same pattern, increasing from 1.22% to 5.10% and 15.20%. These results establish the scale dependent behavior of the trained RL reference against which the frozen LLM configurations are evaluated. 5.2 LLM Planning Alone Provides Limited Gains Replacing only the planner with a frozen LLM improves performance in several settings but does not remove the degradation at larger scales. On the large network, the LLM+RL configurations remain clustered between 10.75% and 13.05% compromise for all models except the 3B model, while impact remains between 8.40% and 10.85%. Increasing planner model size therefore produces relatively limited additional benefit when tactical execution remains with the fixed PPO executor. 5.3 Extending LLM Control to Execution Enables Strong Cross-Scale Performance The largest gains emerge when capable LLMs control both levels of the hierarchy. Llama-3.3-70B achieves compromise rates of 1.15%, 1.40%, and 1.00% across the small, medium and large networks, respectively, while impact remains at 0.10%, 0.05%, and 0.00%. Qwen3-32B shows a similar pattern, with compromise at or below 2.25% and impact at or below 0.60% across all three scales. Operationally, the same frozen LLM weights achieve these results across all network sizes without task-specific retraining, while the RL reference is trained separately at each scale. Comparing LLM+RL with LLM+LLM further shows the effect of extending LLM control to tactical execution. For Llama-3.3-70B on the large network, compromise decreases from 11.75% to 1.00% and impact from 9.25% to 0.00%. For Qwen3-32B, compromise similarly decreases from 11.75% to 1.15%. Thus, planner-only integration does not capture the full benefit observed when a capable LLM also performs tactical action selection. Notably, LLM control does not improve performance universally. Llama-3.2-3B underperforms the RL baseline at several scales, while Trendyol-Cybersecurity-70B performs well on the small network but degrades on the large network. Conversely, Foundation-Sec-8B outperforms general-purpose Qwen3-8B at the largest scale. These mixed results suggest that cybersecurity specialization can help at matched scale but does not guarantee cross-scale robustness. Table 1: Defense performance across network scales. C = Compromise (%), I = Impact (%), and D = Defense (%). Bold indicates the best result within each controller configuration. † denotes a cybersecurity-specialized model. LLM+RL LLM+LLM Model C I D C I D Small network (with RL+RL baseline C=5.32, I=1.22, D=75.52) Llama-3.2-3B 5.45 10.25 64.40 10.60 4.45 68.25 Qwen3-8B 6.00 3.65 70.45 7.10 3.80 75.55 Foundation-Sec-8B† 4.35 0.25 78.00 6.35 0.15 72.30 Qwen3-32B 4.80 3.50 73.60 2.10 0.60 90.70 Trendyol-70B† 6.90 13.65 56.10 1.30 0.05 92.45 Llama-3.3-70B 7.15 16.45 53.10 1.15 0.10 90.90 Medium network (with RL+RL baseline C=14.88, I=5.10, D=68.38) Llama-3.2-3B 14.75 5.15 30.05 17.85 5.75 22.10 Qwen3-8B 13.35 3.85 37.30 6.95 3.90 65.20 Foundation-Sec-8B† 11.35 2.70 47.05 10.20 3.20 50.60 Qwen3-32B 10.45 2.25 48.45 2.25 0.25 84.70 Trendyol-70B† 9.20 2.40 51.55 8.70 2.80 57.80 Llama-3.3-70B 11.95 3.20 41.50 1.40 0.05 87.00 Large network (with RL+RL baseline C=17.95, I=15.20, D=50.01) Llama-3.2-3B 11.15 9.10 28.40 17.00 14.10 2.20 Qwen3-8B 11.75 9.25 25.00 4.60 2.95 65.90 Foundation-Sec-8B† 10.75 8.40 32.30 1.85 0.85 82.30 Qwen3-32B 11.75 9.25 25.00 1.15 0.10 83.10 Trendyol-70B† 13.05 10.85 20.05 14.45 11.60 12.95 Llama-3.3-70B 11.75 9.25 25.00 1.00 0.00 87.95 6 Conclusion and Future Work We investigated whether frozen, zero-shot LLMs can provide effective hierarchical cyber defense across network scales without task-specific retraining and how performance changes as LLM control extends from strategic planning to tactical execution. Across Cyberwheel networks, planner-only LLM integration provides limited gains at larger scales, whereas capable LLMs controlling both levels achieve strong cross-scale performance. Smaller and some cybersecurity-specialized models degrade as networks grow, indicating that domain specialization does not guarantee scale generalization. Adding pretrained reasoning only at the top of a hierarchical defender may be insufficient; effective tactical execution is important for translating strategic decisions into defensive performance. Several questions remain. First, evaluating the complementary RL+LLM configuration would better isolate the contributions of planning and execution. Second, analyzing planner and executor decision traces could explain why some frozen LLMs generalize across scales while others degrade. Finally, future work should extend evaluation across multiple independent seeds and test generalization to unseen topologies, attacker strategies, and multi-agent enterprise environments. ACKNOWLEDGMENTS This work was supported in part by the U.S. Military Academy (USMA) under Cooperative Agreement No. W911NF-25-2-0008 and the Defense Advanced Research Projects Agency (DARPA) under Cooperative Agreement No. HR0011-24-2-0004. The views and conclusions expressed in this paper are those of the authors and do not reflect the official policy or position of USMA, U.S. Army, U.S. Department of War, or U.S. Government. References [1] S. Oesch, A. Chaulagain, B. Weber, M. Dixson, A. Sadovnik, B. Roberson, C. L. Watson, and P. Austria, “Towards a high fidelity training environment for autonomous cyber defense agents,” in Proc. 17th Cyber Security Experimentation and Test Workshop (CSET), 2024, p. 91–99, doi: 10.1145/3675741.3675752. [2] A. V. Singh, E. Rathbun, E. Graham, L. Oakley, S. Boboila, P. Chin, and A. Oprea, “Hierarchical multi-agent reinforcement learning for cyber network defense,” Reinforcement Learning Journal, vol. 6, p. 790–810, 2025. [3] P. Kassianik et al., “Llama-3.1-FoundationAI-SecurityLLM-Base-8B technical report,” arXiv:2504.21039, 2025. [4] Trendyol Group Security Team, “Trendyol-Cybersecurity-LLM-v2-70B-Q4_K_M,” Hugging Face model repository, 2025. [Online]. Available: https://huggingface.co/Trendyol/Trendyol-Cybersecurity-LLM-v2-70B-Q4_K_M [5] B. E. Strom, A. Applebaum, D. P. Miller, K. C. Nickels, A. G. Pennington, and C. B. Thomas, “MITRE ATT&CK: Design and philosophy,” MITRE Technical Report MP180360R1, 2020. [6] Red Canary, “Atomic Red Team: A library of tests mapped to MITRE ATT&CK,” GitHub repository. [Online]. Available: https://github.com/redcanaryco/atomic-red-team [7] M. Kiely et al., “CAGE challenge 4: A scalable multi-agent reinforcement learning gym for autonomous cyber defence,” AI Magazine, vol. 46, no. 3, Art. no. e70021, 2025, doi: 10.1002/aaai.70021. [8] M. Kiely, D. Bowman, M. Standen, and C. Moir, “On autonomous agents in a cyber defence environment,” in Proc. 2nd Int. Workshop on Adaptive Cyber Defense (ACD), 2023, arXiv:2309.07388. [9] S. R. Castro, R. Campbell, N. Lau, O. Villalobos, J. Duan, and A. A. Cárdenas, “Large language models are autonomous cyber defenders,” in 2025 IEEE Conference on Artificial Intelligence (CAI), 2025, p. 1125–1132, doi: 10.1109/CAI64502.2025.00195. [10] S. Mukherjee, S. Chatterjee, E. Purvine, T. Fujimoto, and T. Emerson, “Large language model-based reward design for deep reinforcement learning-driven autonomous cyber defense,” in AAAI-26 Workshop on Artificial Intelligence for Cyber Security (AICS), Singapore, 2026.