Paper deep dive

Strategic Evaluation of Planning Strategies for LLM Agents in Cyber-Physical Systems

J. de Curtò, I. de Zarzà

Year: 2026Venue: arXiv preprintArea: cs.MAType: PreprintEmbeddings: 50

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 89%

Last extracted: 8/9/2026, 3:14:07 AM

Summary

This paper introduces a physics-grounded benchmark to evaluate Large Language Model (LLM) planning agents in cyber-physical systems, specifically smart grid demand response. It compares four planning architectures (Predefined, Sequential, Hierarchical, Search) using a controlled protocol with 40 heterogeneous prosumers. Key findings indicate that planning architecture materially affects outcomes, with forced search acting as the oracle. The study highlights that execution fidelity requires more than mode agreement, as objective substitution can maintain agreement while worsening physical metrics. Additionally, incorporating deadline feasibility constraints significantly reduces regret compared to fixed sequential strategies.

Entities (13)

LLM Planning Agents · system-component · 98%Search Executor · planning-strategy · 95%Smart Grid Demand Response · application-domain · 95%Hierarchical Executor · planning-strategy · 92%Predefined Executor · planning-strategy · 92%Sequential Executor · planning-strategy · 92%Llama 3.3 70B Instruct · model · 90%Prosumers · agent · 90%Deadline Feasibility · constraint · 88%Radial Feeder · physical-infrastructure · 88%DistFlow · simulation-model · 85%Regret · metric · 85%Voltage Shortfall · metric · 85%

Relation Signals (6)

Search Executor → isoraclefor → Smart Grid Demand Response

confidence 95% · forced search is the oracle in all five baseline seeds.

Objective Substitution → increases → Voltage Shortfall

confidence 90% · objective substitution holds agreement at 1.0 while increasing voltage shortfall by 2.68x.

Deadline Feasibility → reduces → Regret

confidence 90% · applying known deadline feasibility before quality prediction cuts regret to 29.0

LLM Planning Agents → uses → DistFlow

confidence 85% · DistFlow independently evaluates the physical consequences.

Predefined Executor → comparesto → Sequential Executor

confidence 80% · It implements predefined, sequential, hierarchical, and search executors

Hierarchical Executor → comparesto → Search Executor

confidence 80% · It implements predefined, sequential, hierarchical, and search executors

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Evaluations of LLM planning agents largely ask whether a task succeeds or a declared plan is followed. In strategic cyber-physical systems, a stronger question is whether the planning architecture remains appropriate after autonomous participants respond and physics constrains the outcome. We introduce a controlled, physics-grounded benchmark built around planning-induced control trajectories: the ordered planning operations and directives through which an execution architecture acts on other agents and the physical process. It implements predefined, sequential, hierarchical, and search executors in a smart-grid demand-response system with 40 heterogeneous prosumers and an independently simulated radial feeder. The LLM is bounded to typed policy declaration and short operator messages, while schedule construction, prosumer dynamics, and power flow remain explicit code. The protocol uses paired forced-mode counterfactuals, common random response draws, and event-level deadline feasibility. Three properties follow. Architecture materially changes outcomes: forced search is the oracle in all five baseline seeds. Execution fidelity needs more than mode agreement: objective substitution holds agreement at 1.0 while increasing voltage shortfall by 2.68x. A 144-scenario, 576-episode bank has feasible oracles from three of the four architectures. A prespecified stress-held-out ridge has mean regret 90.7 (95% interval [73.8, 108.6]) and no detectable value over fixed sequential; applying known deadline feasibility before quality prediction cuts regret to 29.0 and improves over fixed sequential by 61.1. An all-feasible ablation does not beat fixed search, localising the remaining challenge to within-feasible quality selection. A five-model extension separates stress-conditioned, state-blind, and invariant declarers; latency tails show that live feasibility should be treated probabilistically.

PDF

Open source PDF →

PDF not stored locally. Use the link above to view on the source site.

Full Text

46,977 characters extracted from source content.

Expand or collapse full text

Strategic Evaluation of Planning Strategies for LLM Agents in Cyber-Physical Systems J. de Curt ` o ∗† and I. de Zarz ` a †‡ ∗ Department of Computer Applications in Science and Engineering, BARCELONA Supercomputing Center, Barcelona, Spain † Escuela T ́ ecnica Superior de Ingenier ́ ıa (ICAI), Universidad Pontificia Comillas, Madrid, Spain Email: jdecurto@icai.comillas.edu ‡ Human-Centered AI, Data and Software, LUXEMBOURG Institute of Science and Technology, Esch-sur-Alzette, Luxembourg Email: irene.zarza@list.lu Abstract—Existing evaluations of LLM planning agents largely ask whether a task succeeds or a declared plan is followed. In strategic cyber-physical systems, a stronger question is whether the planning architecture remains appropriate after autonomous participants respond and physics constrains the outcome. We introduce a controlled, physics-grounded evaluation methodology and benchmark built around planning-induced control trajectories: the ordered planning operations and time-indexed directives through which an execution architecture acts on other agents and the physical process. The benchmark implements predefined, sequential, hierarchical, and search executors in a demand- response system with 40 heterogeneous prosumers in a smart grid and an independently simulated radial feeder. The LLM is deliberately bounded to structured policy declaration and communication: it selects or advises a typed dispatch policy and generates or evaluates short operator messages, while schedule construction, base prosumer dynamics, stochastic action, and power flow remain explicit code. Its protocol uses paired forced- mode counterfactuals, exact-prompt caching, independent random streams with common prosumer-response draws, critic isolation, and event-level deadline feasibility. The experiments establish three separable properties. First, architecture materially changes outcomes: forced search is the oracle in all five baseline seeds. Second, execution fidelity requires more than mode agreement: objective substitution preserves mode agreement at 1.0 while increasing cumulative voltage shortfall by2.68×. Third, the 144- scenario, 576-episode factorial bank contains feasible oracles from predefined, sequential, and search. The prespecified stress- held-out ridge has mean regret 90.7 (bootstrap 95% interval [73.8, 108.6]) and no detectable value over fixed sequential. A secondary constraint-aware analysis of the same bank applies known deadline feasibility before quality prediction, reducing regret to 29.0 [19.3, 40.4] and improving over fixed sequential by 61.1 [39.6, 82.2]. An all-feasible ablation does not improve over fixed search, localising the remaining challenge to within-feasible quality selection. A five-model, 300-declaration extension further separates stress-conditioned, concentrated state-blind, and fully invariant declarers; observed shared-endpoint latency tails show that live feasibility should be treated probabilistically rather than as a deterministic mode constant. Index Terms—LLM agents, planning strategies, multi-agent systems, cyber-physical systems, smart grids I. INTRODUCTION Planning is becoming an explicit systems component of LLM agents. Rather than relying on one undifferentiated reasoning loop, an agent may construct a plan once, revise it sequentially, decompose it hierarchically, or search over candidate trajectories [1]–[4]. Making this choice explicit improves auditability: a declaration module can select a strategy, a router can dispatch its executor, and a verifier can record which execution pattern actually ran. This architecture answers an execution question, but not the more consequential evaluation question. Agreement between declared and executed modes does not show that the selected strategy was appropriate, that its subgoals were grounded in the available state, that autonomous participants would comply, or that the resulting physical trajectory was safe. Conventional agent benchmarks compress different decompositions, revisions, retries, and recovery paths into terminal scores [5]–[8]. Trace- level analysis recovers some of this structure, but an evaluator expressed in the same language representations as the agent remains partly self-referential. We therefore separate three evaluation dimensions. Planning- strategy heterogeneity asks whether operationally distinct architectures induce different trajectories and outcomes un- der matched conditions. Execution fidelity asks whether the realised trajectory follows both the declared architecture and the intended objective. Adaptive selection asks whether pre- decision state can identify a low-cost, feasible architecture. These distinctions are acute in multi-agent cyber-physical systems: planning decisions alter autonomous participants, whose realised actions then drive physics. We call the ordered commitments, decompositions, revisions, candidate evaluations, and directives a planning-induced control trajectory. It is the decision-layer cause, not the electrical or mechanical state path that emerges after response. Demand response in smart grids provides a rigorous testbed for this distinction. A distribution operator requests flexibility from heterogeneous prosumers while respecting feeder-head import and voltage constraints. The system combines strategic private costs, a natural spatial hierarchy for decomposition, and an external power-flow model that can determine whether a plausible-looking directive was physically useful. The domain is therefore an instrument for studying planning, not merely an application of an LLM to a grid. The operational task is concrete. Whenever forecast feeder import exceeds the 1.1 MW cap or voltage approaches the arXiv:2608.04265v1 [cs.MA] 4 Aug 2026 0.95 pu floor, the operator must decide how to organise a curtailment campaign: which execution architecture to use, which nodes to target, how strongly to request flexibility, and how to frame the request. The objective in Eq.(4) rewards removal of cap and voltage violations while penalising unnecessary curtailment. In the uncontrolled calibration trace, for example, hour 17 reaches 1.644 MW at the feeder head and a minimum voltage of 0.9344 pu. The controller must therefore coordinate actual flexible demand, not merely produce a plausible verbal plan. The LLM’s control boundary is deliberately narrow. It does not solve power flow, construct the numerical node allocation, or decide a household’s action. It operates at two interfaces: high-level policy declaration or bounded mode advice, and language-mediated communication. Typed strategy executors convert policy fields into schedules; a game-theoretic response model, augmented by a bounded LLM persuasion shift, determines prosumer behaviour; and DistFlow independently evaluates the physical consequences. We use LLM agent for this composite controller rather than for an unconstrained language model acting directly on the feeder. Accordingly, we study three questions. RQ1: do distinct planning architectures induce measurably different strategic and physical outcomes under matched operating conditions? RQ2: does faithful execution of a declared mode also preserve the intended physical objective? RQ3: can a selector exploit scenario-dependent strategy heterogeneity under observability and real-time constraints? We answer through a four-layer pipeline comprising strategy selection, mode-specific execution, strategic response, and independent physical verification. Every forced comparison keeps the objective, agents, forecast, latent response draws, and network dynamics fixed; paired per- scenario regret then measures selection quality directly. Contributions. (1) We introduce a controlled, physics- grounded benchmark that isolates planning architecture from strategic adaptation and physical evolution. (2) We define planning-induced control trajectories as a common abstraction for comparing predefined, sequential, hierarchical, and search execution independently of the application domain. (3) We develop a counterfactual protocol combining paired forced execution, exact-prompt caching, common random numbers, event-level feasibility, response-aware metrics, and external physical verification. (4) We show that scenario-dependent oracle diversity defines a nontrivial selection problem, and that known deterministic feasibility is more effective as a routing constraint than as a regression penalty. (5) A five- model interface extension shows that declaration collapse is model-dependent and that serving latency can be heavy- tailed, motivating risk-aware rather than purely deterministic deployment gates. Code and data have been released in the GitHub repository. 1 I. RELATED WORK Planning strategies for LLM agents. ReAct interleaves reasoning and action [1]; Plan-and-Solve makes decomposition 1 https://github.com/drdezarza/LLMstrategicplanning explicit [2]; Tree of Thoughts searches over candidate reasoning paths [3]; and Reflexion introduces verbal feedback across attempts [4]. These approaches differ in execution structure, compute, observability, and recovery, not merely in prompt wording. Agent benchmarks make these differences visible at the task level [5]–[8], while broader evaluation work argues for multidimensional rather than scalar assessment [9]. We provide a controlled setting for comparing and selecting among available execution architectures. Faithfulness and trajectory evaluation. Chain-of-thought studies show that stated rationales can diverge from the effective basis of an answer [10], [11]. The analogous plan-level gap is that a verifier may certify the named executor while remaining blind to target substitution, unsupported state assumptions, or physically ineffective actions. We therefore combine trace- derived quantities with an external physical referent. Strategic multi-agent systems. LLM-agent populations have been studied as social simulations and strategic systems [12]–[14]. Controlled experiments examine finitely repeated games [15] and common-pool-resource dilemmas [16]. Our response model builds on cooperation theory [17], [18] and prior work on coevolutionary adaptation and bounded LLM influence [19]. The LLM does not directly set prosumer actions; it provides a bounded modulation of a heterogeneous game- theoretic base probability. Cyber-physical evaluation and smart grids. Demand response coordinates flexible consumption under network and market constraints [20], [21]. AI methods are increasingly used in demand response [22], and recent work examines LLM integration in smart grids [23], [24]. Here the grid is methodological: linearised DistFlow [25], a standard radial model and basis of later branch-flow relaxations [26], provides an evaluator independent of the planner. The demand-response environment used here extends the microgrid coordination setting of [27], which modelled pro- sumer compliance as a repeated Prisoner’s Dilemma on a social network driven by an LLM influence compiler; the present work replaces that influence compiler with a plan-declaring planner and adds network physics as an external referent for plan quality. I. STRATEGIC EVALUATION FRAMEWORK Lets t denote the observed system state andg t the control objective. The strategy set isM= PREDEFINED, SEQUENTIAL, HIERARCHICAL, SEARCH. A selectorρchoosesm t ∈M, and the corresponding executor E m t emits a structured policyp t . Strategic agentoresponds according to B o , and the physical system evolves through F : m t = ρ(s t ,g t ),p t = E m t (s t ,g t ),(1) a o t ∼ B o (p t ,h o t ),s t+1 = F (s t ,a t ),(2) whereh i t is local interaction history. Over an episode, strategy m generates ξ m = (z t ,u t ) T−1 t=0 ,(3) Fixed scenario and objective state, task, agents Policy interface LLM, fixed, or learned Typed executor schedule + revisions Communication interface LLM message + shift Strategic response heterogeneous agents Physical trajectory voltage, import, cost LLM-capableLLM call External evaluator mode agreement | faithfulness | groundedness | minimality realisation | physical cost | counterfactual regret Evaluation dimensions strategy heterogeneity execution fidelity adaptive selection Fig. 1. Strategic evaluation and isolation logic. Orange boxes mark the two bounded LLM interfaces: policy declaration/advice and language-mediated communication. In forced counterfactuals, the policy interface is replaced by a fixed typed policy. Strategy executors, the prosumer base game, stochastic actions, and power flow remain explicit code; the external evaluator resolves planning-strategy heterogeneity, execution fidelity, and adaptive selection. wherez t records planning operations andu t contains the issued control directives. Figure 1 separates this decision- layer trajectory from the strategic and physical trajectories it induces. Matched forced modes measure planning-strategy heterogeneity; trace and physical metrics measure execution fidelity; and paired oracle regret measures adaptive selection. LLM interfaces and control boundary. Table I shows every language-model call site in the reported system. The policy interfaces receive a compact, serialised state rather than raw feeder arrays or power-flow equations. Their outputs are parsed as JSON, clipped to admissible ranges, and statically type-checked before an executor can act. No free-form text is translated directly into kW commands. The communication interfaces are also bounded: the LLM writes the operator message and estimates only its incremental persuasive effect; the prosumer’s base utility, memory, resistance, random action, and realised curtailment remain outside the model. Illustrative dispatch. At hour 17 of the uncontrolled calibration trace, feeder import is 544 kW above the cap and minimum voltage is 0.9344 pu. The planner is not asked to produce appliance commands. It receives the state summary and may emit a type-valid policy such asmode=SEARCH, target=HIGH_LOAD,branching=4,replan=NONE. The search executor then constructs targeting–intensity sched- ules, evaluates each through forecast DistFlow, and retains the lowest predicted-cost schedule. For the selected households, the message prompt receives the framing, hour, and kW exceedance; a second prompt receives each sampled prosumer’s personality, requested reduction, neighbour behaviour, and recent history, and returns only a bounded willingness shift. The game-theoretic probability plus that shift determines stochastic compliance, after which the physical simulator, not the LLM, computes import, voltage, and cost. This end-to-end example is the intended meaning of an LLM planning agent in the benchmark. Typed policy and execution semantics. A policy records mode, targeting rule, hierarchy depth, search branching, re- planning trigger, intensity, message theme, and decision budget. Static checks detect, for example, a hierarchical policy without decomposition or search with fewer than two candidates. All four executors receive the same implemented objective: remove feeder-head excess above 1.1 MW and voltage shortfall below 0.95 pu while limiting unnecessary curtailment, as formalised in Eq.(4). PREDEFINED compiles the remaining-horizon allocation once and commits. SEQUENTIAL dispatches a step, observes realised response, and reconstructs the unexecuted suffix. HIERARCHICAL allocates the required reduction first across laterals and, at depth three, feeder segments. SEARCH generates complete targeting–intensity schedules, rolls them out through forecast power flow, and retains the lowest predicted- cost candidate. Thus the experiment varies when the controller commits, whether it revises or decomposes, and whether alternatives are explicitly evaluated. Strategic response. Each of 40 prosumers in a smart grid has an archetype, resistance, flexible load, neighbourhood, and memory. Compliance starts from a cooperation-based prob- ability combining reciprocity, exploitation memory, comfort fatigue, and archetype bias. For 40% of targeted agents, an LLM evaluates the operator narrative and returns a bounded shift δ ∈ [−0.30, 0.30], attenuated by resistance. Compliance and realised curtailment remain stochastic agent decisions rather than direct LLM outputs. External evaluation. The executed-mode verifier checks architecture agreement and policy type. The external evaluator additionally measures: faithfulness (Jaccard overlap of declared and issued targets), groundedness (share of directives within 105% of true flexible load), minimality (oracle divided by realised curtailment on successful hours), coherence (one minus normalised revision churn), and directive realisation (realised divided by requested power). The physical objective is J = kWh overcap + 4000 X ∆v + 0.25 kWh curtailed ,(4) where lower is better. For scenario q, paired routing regret is R ρ (q) = J (ρ(q),q)− min m∈M J (m,q).(5) IV. EXPERIMENTAL METHODOLOGY Environment and model. The 24-hour feeder has four laterals of ten prosumer nodes (N = 40),S base = 1MVA, TABLE I LLM INTERFACES IN THE LIVE PROTOCOL. STRATEGY EXECUTORS, THE BASE PROSUMER GAME, STOCHASTIC ACTIONS, AND FEEDER PHYSICS ARE EXPLICIT CODE. InterfacePrompt payloadConstrained outputUse in the reported experiments Natural policy decla- ration Hour; stress and burden; remaining hori- zon; volatility and spatial spread; feeder cap and forecast head import; previous minimum voltage; recent compliance. Typed JSON containing mode, target rule, message theme, hierarchy depth, search branching, replanning trigger, in- tensity, confidence, and a one-sentence rationale. E1 natural declaration and the natural routed arm of E2. The interface is replaced by fixed policies in forced counterfactual banks. Hybrid mode adviceThe same state plus deterministic feature- derived base scores for all four modes. A per-mode shift in[−0.30, 0.30]and a short reason. Code applies the shifts, samples the mode, and constructs the remaining policy fields. E1/E2 bounded hybrid and E4 before the deterministic deadline gate. The LLM does not directly choose the final mode. Operatormessage generation Message framing, dispatch hour, and forecast kW above the import cap. A specific two–three sentence customer- facing curtailment request. Every live episode with a non-empty dispatch; one message is reused for all targets in that hour. Prosumer message evaluation Archetype description, requested kW, neighbourcompliance,exploitation count, recent response history, and the operator message. A willingness shift in[−0.30, 0.30], a persuasion flag, and one short reason. The shift is resistance-attenuated and cannot itself choose the action. Applied to a common-random-number sample of 40% of targeted prosumers; E6 attributes the resulting bounded shifts and actions. Optional plan critic Proposed typed policy, state summary, decision deadline, and stipulated execu- tor cost. Approval, risk level, and adjusted inten- sity. Implemented for ablation, but the LLM critic is disabled in the reported causal banks. E4 uses only the deterministic, type-valid deadline gate. per-segmentr = 0.022andx = 0.014pu, a 1.1 MW import cap, and a 0.95 pu voltage floor. Cooling-dominated load and declining afternoon PV produce eight baseline violation hours. The reported live run uses Llama-3.3-70B-Instruct [28], a 70B instruction-tuned member of the Llama 3 family [29], with temperature 0.2 and at most 400 completion tokens. E1, E2, and E5 use seeds7, 13, 42, 101, 202; E3, E4, E6, and each E7 factor combination use 7, 13, 42. Isolation controls. In paired forced-mode comparisons, all four strategies receive the same feeder, load–PV realisation, observed forecast, prosumer population, objective, targeting rule, narrative theme, and stress-calibrated intensity. Forecast, planning, tool-failure, adversarial, and response randomness use separate deterministic streams; latent narrative sampling, com- pliance, and realisation draws are shared across strategies by (seed,t, node). Every cache entry is keyed by endpoint/model, system message, and complete prompt, with a collision guard. The last scheduled step is executed rather than replaced by a final-hour redeclaration. The LLM critic is disabled in causal forced banks; E4 activates only a deterministic, type-valid deadline gate. Every declaration or revision incurs a stipulated nominal cost, and feasibility requires that no individual planning event exceed the decision deadline. LLM role by experiment. E1 and the routed arms of E2 exercise the policy interfaces in Table I: natural declaration asks the LLM for the full typed policy, whereas the hybrid asks only for bounded score shifts. The forced arms of E2, all of E3, E5, E6, and the E7 counterfactual bank fix the mode and shared policy fields to isolate execution semantics; the LLM continues only in message generation and bounded prosumer interpretation. E4 uses hybrid declaration followed by the deterministic deadline gate, never the LLM critic. The E7 ridge and constraint-aware diagnostic are offline statistical selectors over committed counterfactual data and make no additional LLM calls. Multimodel interface extension. To distinguish framework effects from properties of the Llama backbone used in E1–E7, M0–M5 replay the same typed declaration, bounded mode- advice, and prosumer-interpretation interfaces with Llama-3.3- 70B-Instruct, DeepSeek-V4-Pro [30], Gemma-3-27B-IT [31], GLM-5.2 [32], and MiniMax-M3 [33]. M1–M2 use 20 de- terministic planner-visible states (four stress multipliers by five decision hours) and three declarations per state: 60 per model and 300 total. M3 records 40 bounded score-shift calls per model; M4 records 48 archetype–message evaluations per model. M2 retains a conservative permutation test over all 20 states and adds a secondary, power-oriented three-band association and logistic stress trend. M5 reports declaration- only wall time observed through one shared endpoint, separately from the other interfaces. The extension makes no feeder rollouts and therefore tests interface behaviour and deployment assumptions, not cross-model replication of the E2–E7 physical rankings. Experiments. E1–E4 characterize strategy use, paired physical performance, observability, and deadline feasibil- ity. Specifically, E1 compares natural and bounded feature- conditioned declaration; E2 forces all four strategies and both selectors on the same five seeds; E3 varies forecast noiseσ ∈0,.05,.10,.20,.30,.40; and E4 crosses deadlines 2, 4, 8, 30s with the deadline gate. E5–E6 evaluate execution fidelity and strategic response: E5 varies aggregator honesty from 1 to 0 with the sequential executor fixed, while E6 retains every directive for attribution by prosumer archetype and electrical position. E7 asks whether strategy usefulness can be learned beyond the baseline. It crosses four feeder-stress groups, three noise levels, two deadlines, two mean resistance levels, and three seeds: 144 scenarios and 576 forced episodes. A mode is PRESEQHIERSRCH 0.0 0.2 0.4 0.6 0.8 1.0 declaration share natural (H=0.00) PRESEQHIERSRCH hybrid (H=0.00) Fig. 2. E1 declaration shares. Natural declaration selects SEQUENTIAL in 5/5 episodes; the bounded hybrid selects HIERARCHICAL in 5/5. Both have normalised entropy H = 0. feasible iff it has no event-level deadline miss, and the prespecified selection cost is C m (q) = J (m,q) + 10 4 1[m infeasible in q ].(6) One ridge model per mode (fixedα = 10) predictsC m from cap tightness, line resistance, forecast noise, deadline, and mean resistance. Cross-fitting holds out an entire stress group; uncertainty uses 5000 paired scenario-level bootstrap draws. A secondary post-hoc analysis, using the same bank, features, folds, regularisation, and bootstrap, first removes modes known to miss the event deadline and predictsJon feasible rows. A 72-scenario long-deadline subset tests quality selection when all modes are feasible. No new episodes or model calls are made. Inferential quantities remain controlled, descriptive comparisons rather than deployment-level estimates. M0–M5 provide the complementary multimodel interface study: capability probing, declaration diversity, state dependence, bounded mode advice, persuasion structure, and observed declaration latency. V. RESULTS: THREE EVALUATION DIMENSIONS A. Declaration concentration motivates counterfactual evalua- tion Natural declaration selects SEQUENTIAL in every episode, whereas the bounded hybrid selects HIERARCHICAL in every episode (Figure 2); both haveH = 0. The feature prior changes which mode is exposed, but three architectures remain unobserved under each selector. This concentration motivates forced counterfactual evaluation of the full strategy set. The multimodel extension below shows that this collapse is a property of particular declarers rather than a universal consequence of the interface. B. Model choice separates three declaration regimes Across 300 declarations (60/model), every output parses, but the distributions separate sharply. MiniMax uses all four modes (H = 0.712[0.550, 0.815]); from the zero- to high-stress band, its expensive-mode share rises from 0.222 to 0.727 and its search share from 0.111 to 0.667 (V = 0.442,p = 0.0023; trendβ = 2.803,p = 0.0129; Figure 3a). GLM also uses all four modes (H = 0.544) and shows band association (V = 0.345,p = 0.0343), although its trend is marginal 0.00.20.40.60.81.0 declaration share MiniMax M3 GLM 5.2 DeepSeek V4 Gemma 3 27B Llama 3.3 70B (a) Mode use PRESEQHIERSRCH none moderate high 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 P(search or hierarchy) (b) Stress conditioning MiniMax M3 GLM 5.2 DeepSeek V4 Gemma 3 27B Llama 3.3 70B 5101520 declaration latency (s) MiniMax M3 GLM 5.2 DeepSeek V4 Gemma 3 27B Llama 3.3 70B 4 s (c) Shared-endpoint latency p50 p95 Fig. 3. Declaration behaviour across five models on identical planner inputs (60 declarations per model). (a) Probability of selecting an expensive strategy (search or hierarchy) by feeder-stress band: MiniMax rises from 0.222 to 0.727 across bands, GLM rises weakly, and DeepSeek, Gemma and Llama stay flat. (b) Declaration latency on a shared endpoint,p50(circle) top95(square), against the 4 s event budget of Sec. V-E; the tail, not the median, determines feasibility. Panel (a) separates stress-conditioned from state-blind declarers; panel (b) shows that the deadline constraint binds at the serving layer as well as the strategy layer. PRE SEQ HIER SRCH natural hybrid 0 250 500 750 1000 1250 paired oracle regret forcedrouted Fig. 4. E2 mean paired oracle regret with 95% Student-tintervals over five seeds. Search is the per-seed oracle; routed policies provide the selection baseline. (p = 0.062). The conservative 20-state permutation tests give p = 0.076and0.111, respectively, so the banded analysis is secondary. DeepSeek uses only two modes (96.7% sequential) without measurable state dependence; Gemma and Llama are fully invariant. Declaration collapse is therefore model- dependent, separating stress-conditioned, concentrated state- blind, and invariant regimes. The other bounded interfaces show partial qualitative stability. DeepSeek returns an all-zero score shift in 39/40 M3 calls, effectively reverting the hybrid to its deterministic prior, whereas the other models use meanℓ 1 budgets of 0.19–0.38 without saturation. In M4, four of five models rank idealists first and pragmatists second; all place pragmatists above opportunists, and four show a positive descriptive response to neighbour compliance. These are interface-level results, not cross-model replications of E2–E7 feeder outcomes. Observed declaration latency also challenges deterministic deployment costs (Figure 3b). Llama has shared-endpoint p50 = 2.46s butp95 = 23.68s, a9.6×tail; GLM and MiniMax also exceed 4 s atp95, while Gemma and DeepSeek remain below it in this run. Because these measurements mix generation, serving load, and queueing, they are not intrinsic model speeds; they motivatePr(L m ≤ d | x)or a latency- quantile margin for live routing. E2 establishes that executor choice is consequential: forced TABLE I E2 COMPARISON OVER FIVE PAIRED SEEDS. REGRET IS COMPUTED AGAINST THE BEST FORCED STRATEGY FOR THE SAME SEED. PolicyJRegretFaith.Real. Forced PREDEFINED15712941.000.540 Forced SEQUENTIAL15502721.000.542 Forced HIERARCHICAL238811110.890.548 Forced SEARCH127700.790.533 Routed, natural20127341.000.486 Routed, bounded hybrid238811110.890.548 0.000.25 state noise 0.8 0.9 1.0 plan groundedness (a) Predefined Sequential Hierarchical Search 0.000.25 state noise 1500 2000 objective (b) 0.000.25 state noise 0.0 0.5 1.0 STATE_INFER. share (c) Fig. 5. E3 forecast-noise sweep: (a) groundedness, (b) physical objective, and (c) share assigned the deterministic STATEINFERENCE failure label. search is the oracle in all five baseline seeds (Table I). The bounded selector is an additional 376.36 objective units relative to natural declaration (pairedt = 5.514, two-sided p = 0.0053) because it selects the highest-cost forced strategy. This reinforces that the target is calibrated strategy selection rather than declaration diversity alone. Natural routing exposes a second error channel. It always declares SEQUENTIAL, yet incurs 462.05 additional objective units relative to forced sequential (p = 0.0019). Since the executor architecture is the same, the gap is caused by other LLM-set fields such as target rule, intensity, and narrative theme. Mode selection and within-mode policy parameterisation must therefore be evaluated separately. Search’s lower target- set faithfulness (0.79) is not evidence of failure: legitimate candidate exploration changes the final target set, while the physical objective verifies that the resulting trajectory is superior. Groundedness declines with noise at slopes−0.404(prede- fined),−0.404(sequential),−0.660(hierarchical), and−0.709 (search). Objective slopes are respectively+821,+926,+225, and+1380. Search is therefore most sensitive on both axes, while hierarchical loses groundedness more quickly than the flat modes without the same objective slope. The failure taxonomy assigns STATEINFERENCE to 66.7% of episodes atσ = 0.10 and 91.7% atσ = 0.20(Figure 5). At larger noise, other higher-priority labels may fire, so categorical shares need not be monotone even when forecast error increases. C. Deadline-aware gating restores event-level feasibility The deadline gate provides a positive feasibility control. Ungated hybrid declaration executes hierarchical planning; at 2 and 4 s it produces nine late planning events per episode 0.00.51.0 honesty h 0.7 0.8 0.9 1.0 targeting fidelity mode_match (a) 0.00.51.0 honesty h 0.6 0.8 1.0 leverage ratio (b) 0.00.51.0 honesty h 0.0 0.1 0.2 voltage shortfall (pu) (c) Fig. 6. E5 objective substitution. The sequential architecture continues to match its declaration while targeting fidelity, physical leverage, and voltage shortfall deteriorate. Error bars are 95% Student-t intervals. 0.00.20.40.60.8 compliance rate opportunist skeptic strategist conformist pragmatist idealist (a) 0.000.050.10 mean LLM shift (b) Fig. 7. E6 attribution over 2831 directives: (a) compliance and (b) bounded LLM shift by prosumer archetype. and accumulates 51.03 s of nominal decision cost. At 2 s the gate substitutes a type-valid sequential policy, leaving zero misses, 13.33 s accumulated cost, and meanJ = 1484.99. At 4 s it selects a two-candidate search policy, again with zero misses, 2.75 s accumulated cost, andJ = 1257.76. At 8 and 30 s the gate is inert. These are mechanism and feasibility results under stipulated per-invocation costs, not claims about measured serving latency or the optimality of the fallback ranking. D. External physics exposes hidden objective substitution The E5 aggregator always executes the declared sequen- tial architecture. With probability1 − hat each planning decision, however, it substitutes low-voltage-leverage nodes and issues only 60% of the declared depth. Consequently, mode_match=1.0 for every row. Ath = 0, targeting faithfulness falls to 0.804 (95% CI [0.722, 0.885]), and physical leverage falls from 0.975 to 0.664 (95% CI [0.606, 0.722]; pairedp = 0.00020). Cumulative voltage shortfall increases from 0.0493 to 0.1322 pu, a2.68×increase (p = 0.0041), while mean objective rises from 1550 to 2467. Mode verifi- cation certifies the machinery but not the goal it pursues; the external physical evaluator exposes the substitution directly. E. Response attribution resolves strategic heterogeneity Across 2831 unique directives, idealists have the highest compliance rate (0.739) and largest mean positive LLM shift (+0.104). Opportunists have the lowest compliance (0.477) and the only negative mean shift (−0.041). The remaining archetypes fall between these endpoints. Directive-level attribu- tion therefore resolves which subpopulations support or resist a 050100 scenarios oracle one-stage constraint (a) PRESEQHIERSRCH fixed SEQ one-stageconstraint- aware 0 20 40 60 80 100 mean oracle regret (b) Fig. 8. E7 routing diagnostics. (a) Oracle membership (ties split), one-stage selections, and secondary constraint-aware selections. (b) Mean oracle regret with 95% paired-bootstrap intervals; dotted line: empirical feature-information floor (18.1). plan instead of hiding them in aggregate realisation. Electrical position is potentially confounded by targeting frequency and fatigue, so the experiment supports archetype attribution but not an unconditional feeder-depth effect. F. Oracle heterogeneity creates an adaptive-selection opportu- nity E7 establishes a nontrivial adaptive-selection target. Search belongs to the minimum-cost oracle set in 59 scenarios, predefined in 44, and sequential in 43; two scenarios tie between search and predefined, while hierarchical is never oracle. The prespecified ridge selects sequential in 102/144 scenarios and predefined in 42 (Figure 8); exact-oracle selection is 31.9%, mean regret is 90.74 [73.76, 108.60], and value versus fixed sequential is −0.68 [−4.99, 3.45]. The secondary constraint-aware analysis filters known deadline-infeasible modes before predicting physical quality. It selects search in 72 scenarios, sequential in 51, and predefined in 21; exact-oracle selection rises to 67.4%, mean regret falls to 29.00 [19.30, 40.42], and value versus fixed sequential is+61.06[39.61, 82.24]. It closes 84.8% of the gap from fixed sequential to an optimistic feature-cell reference. On the 72 all-feasible scenarios, however, value versus fixed search is−18.50[−34.40,−3.88]. The gain is therefore feasibility- aware; within-feasible quality ranking remains unresolved by these features and this linear estimator. VI. DISCUSSION AND IMPLICATIONS The benchmark resolves three dimensions often collapsed into terminal success. Planning-strategy heterogeneity: E2– E4 show architecture-specific performance, robustness, and feasibility, while E7 contains predefined-, sequential-, and search-optimal scenarios. Execution fidelity: E5 preserves perfect mode agreement under physically harmful objective substitution, and E6 identifies which prosumer groups realise the directives. Adaptive selection: E1 reveals concentrated natural use, whereas E7 supplies a paired, stress-held-out target for learned selectors. The multimodel extension scopes those claims. Declara- tion collapse is model-dependent: two declarers use all four strategies and respond to coarse stress bands, one remains concentrated without measurable state dependence, and two are invariant. The communication interface preserves broad archetype order for most models, but bounded mode advice can degenerate to a fixed prior. Measured serving tails also refine the E7 design principle: deterministic structural feasibility can still filter impossible executor types, but live deployment requires a separate probabilistic latency margin. This decomposition is constructive. The counterfactual bank establishes genuine value to identify, while the secondary diagnostic supplies a concrete design principle: deterministic feasibility known at dispatch should constrain the candidate set rather than be learned through a large penalty. The all-feasible ablation then isolates physical-quality ranking as the remaining challenge. The protocol is deliberately conservative about causal comparisons. Complete prompts define cache identity; planning and response randomness are separated; paired strategies share latent prosumer draws; the final scheduled step is not redeclared; forced banks disable the LLM critic; and feasibility is evaluated per planning event. These controls matter because search and hierarchy otherwise consume different random draws, stale cached responses can cross experimental conditions, and accumulated episode latency can be mistaken for a per-decision deadline. The study remains a controlled mechanism analysis. The physical E1–E7 bank uses one Llama backbone, one radial topology, a parametric prosumer population, linearised power flow, and stipulated executor costs. The five-model extension repeats the LLM interfaces but not the complete strategic and physical rollouts; it therefore scopes interface claims without establishing that E2–E7 rankings transfer across backbones. E1–E6 have three to five seeds; E7 varies scenario parameters on the same topology, and its bootstrap resamples factor combinations even though seeds recur. The response cache statistics are preserved, but the in-memory response cache itself was not serialised. Stronger external claims require full multi-backbone feeder reruns, controlled repeated latency mea- surements, compute–quality frontiers, standard balanced and unbalanced feeders, larger independent scenario families, and nonlinear or hardware-in-the-loop validation. The constraint- aware result is post-hoc, and the observed shared-endpoint latency tails require confirmation under controlled load before deployment-level feasibility claims. Although demand response in a smart grid is the evaluation domain, the decomposition in Figure 1 applies whenever an LLM selects a planning architecture, autonomous entities react, and physics decides whether the induced trajectory is acceptable, including robotics, traffic control, warehouse coordination, and industrial automation. VII. CONCLUSION We introduced a controlled, physics-grounded benchmark for evaluating planning strategies used by LLM agents in multi-agent cyber-physical systems. Its central object is the planning-induced control trajectory, and its paired counter- factuals, response-aware metrics, event-level feasibility, and external physical referent make planning-strategy heterogeneity, execution fidelity, and adaptive selection separately measurable. The demand-response study shows that architecture matters: forced search is the baseline oracle, strategies have distinct robustness profiles, deadline-aware gating restores feasible execution, and the factorial bank contains three oracle modes. It also shows that perfect mode agreement can coexist with a2.68×increase in voltage shortfall. The prespecified ridge provides a transparent reference; the secondary constraint-aware analysis reduces regret from 90.7 to 29.0 and outperforms fixed sequential, while its all-feasible ablation localises the remaining challenge to physical-quality ranking. The multimodel exten- sion further shows that declaration collapse is not universal and that live feasibility depends on model- and serving- stack latency tails. Together, these results motivate selectors that combine structural constraints, state-conditioned quality prediction, and probabilistic latency margins. The methodology extends naturally to robotics, transportation, logistics, and industrial automation. Code and data are publicly available in: https://github.com/drdezarza/LLMstrategicplanning ACKNOWLEDGEMENTS This research was supported by the LUXEMBOURG Insti- tute of Science and Technology through the projects “ADIALab- MAST” and “LLMs4EU” (Grant Agreement No 101198470) and the BARCELONA Supercomputing Center through the project “TIFON” (File number MIG-20232039). REFERENCES [1]S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao, “ReAct: Synergizing reasoning and acting in language models,” in International Conference on Learning Representations (ICLR), 2023. [2]L. Wang, W. Xu, Y. Lan, Z. Hu, Y. Lan, R. K.-W. Lee, and E.-P. Lim, “Plan-and-solve prompting: Improving zero-shot chain-of-thought reasoning by large language models,” in Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).Toronto, Canada: Association for Computational Linguistics, 2023, p. 2609–2634. [3] S. Yao, D. Yu, J. Zhao, I. Shafran, T. L. Griffiths, Y. Cao, and K. Narasimhan, “Tree of thoughts: Deliberate problem solving with large language models,” in Advances in Neural Information Processing Systems, vol. 36. Curran Associates, Inc., 2023, p. 11 809–11 822. [4] N. Shinn, F. Cassano, A. Gopinath, K. Narasimhan, and S. Yao, “Reflexion: Language agents with verbal reinforcement learning,” in Advances in Neural Information Processing Systems, vol. 36. Curran Associates, Inc., 2023, p. 8634–8652. [5]X. Deng, Y. Gu, B. Zheng, S. Chen, S. Stevens, B. Wang, H. Sun, and Y. Su, “Mind2Web: Towards a generalist agent for the web,” in Advances in Neural Information Processing Systems, vol. 36.Curran Associates, Inc., 2023, p. 28 091–28 114. [6] S. Zhou, F. F. Xu, H. Zhu, X. Zhou, R. Lo, A. Sridhar, X. Cheng, T. Ou, Y. Bisk, D. Fried, U. Alon, and G. Neubig, “WebArena: A realistic web environment for building autonomous agents,” in International Conference on Learning Representations (ICLR), 2024. [7]C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. Narasimhan, “SWE-bench: Can language models resolve real-world github issues?” in International Conference on Learning Representations (ICLR), 2024. [8]M. Shridhar, X. Yuan, M.-A. C ˆ ot ́ e, Y. Bisk, A. Trischler, and M. Hausknecht, “ALFWorld: Aligning text and embodied environments for interactive learning,” in International Conference on Learning Representations (ICLR), 2021. [9]P. Liang, R. Bommasani, T. Lee, D. Tsipras, D. Soylu, M. Yasunaga, Y. Zhang, D. Narayanan, Y. Wu, A. Kumar et al., “Holistic evaluation of language models,” Transactions on Machine Learning Research, 2023. [10] T. Lanham, A. Chen, A. Radhakrishnan, B. Steiner, C. Denison, D. Hernandez, D. Li, E. Durmus, E. Hubinger, J. Kernion et al., “Measuring faithfulness in chain-of-thought reasoning,” arXiv preprint arXiv:2307.13702, 2023. [11]M. Turpin, J. Michael, E. Perez, and S. R. Bowman, “Language models don’t always say what they think: Unfaithful explanations in chain- of-thought prompting,” in Advances in Neural Information Processing Systems, vol. 36, 2023, p. 74 952–74 965. [12]J. S. Park, J. C. O’Brien, C. J. Cai, M. R. Morris, P. Liang, and M. S. Bernstein, “Generative agents: Interactive simulacra of human behavior,” in Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology, 2023, p. 1–22. [13]T. Guo, X. Chen, Y. Wang, R. Chang, S. Pei, N. V. Chawla, O. Wiest, and X. Zhang, “Large language model based multi-agents: A survey of progress and challenges,” arXiv preprint arXiv:2402.01680, 2024. [14] Z. Xi, W. Chen, X. Guo, W. He, Y. Ding, B. Hong, M. Zhang, J. Wang, S. Jin, E. Zhou et al., “The rise and potential of large language model based agents: A survey,” arXiv preprint arXiv:2309.07864, 2023. [15] E. Akata, L. Schulz, J. Coda-Forno, S. J. Oh, M. Bethge, and E. Schulz, “Playing repeated games with large language models,” Nature Human Behaviour, vol. 9, p. 1380–1390, 2025. [16] G. Piatti, Z. Jin, M. Kleiman-Weiner, B. Sch ̈ olkopf, M. Sachan, and R. Mi- halcea, “Cooperate or collapse: Emergence of sustainable cooperation in a society of LLM agents,” in Advances in Neural Information Processing Systems, vol. 37. Curran Associates, Inc., 2024, p. 111 715–111 759. [17] R. Axelrod, The Evolution of Cooperation. New York: Basic Books, 1984. [18]M. A. Nowak, “Five rules for the evolution of cooperation,” Science, vol. 314, no. 5805, p. 1560–1563, 2006. [19]J. de Curt ` o and I. de Zarz ` a, “LLM-driven social influence for cooperative behavior in multi-agent systems,” IEEE Access, vol. 13, p. 44 330– 44 342, 2025. [20]P. Siano, “Demand response and smart grids—a survey,” Renewable and Sustainable Energy Reviews, vol. 30, p. 461–478, 2014. [21]Y. Parag and B. K. Sovacool, “Electricity market design for the prosumer era,” Nature Energy, vol. 1, no. 4, p. 16032, 2016. [22]M. A. Khan, A. M. Saleh, M. Waseem, and I. A. Sajjad, “Artificial intelligence enabled demand response: Prospects and challenges in smart grid environment,” IEEE Access, vol. 11, p. 1477–1505, 2022. [23]S. Madani, A. Tavasoli, Z. K. Astaneh, and P.-O. Pineau, “Large language models integration in smart grids,” Energy Reports, vol. 14, p. 1562– 1577, 2025. [24]H. Shi, L. Fang, X. Chen, C. Gu, K. Ma, X. Zhang, Z. Zhang, J. Gu, and E. G. Lim, “Review of the opportunities and challenges to accelerate mass-scale application of smart grids with large-language models,” IET Smart Grid, vol. 7, no. 6, p. 737–759, 2024. [25]M. E. Baran and F. F. Wu, “Network reconfiguration in distribution systems for loss reduction and load balancing,” IEEE Transactions on Power Delivery, vol. 4, no. 2, p. 1401–1407, 1989. [26]M. Farivar and S. H. Low, “Branch flow model: Relaxations and convexification—part i,” IEEE Transactions on Power Systems, vol. 28, no. 3, p. 2554–2564, 2013. [27] J. de Curt ` o and I. de Zarz ` a, “LLM-mediated demand response co- ordination in smart microgrids,” in 18th International Conference on Sustainability in Energy and Buildings (SEB-26), Chania, Crete, Greece, 2026, arXiv:2606.11050. [28] Meta, “Llama 3.3 70B Instruct model card,” Meta Llama modeldocumentation,2024,releasedDecember6,2024. [Online]. Available: https://github.com/meta-llama/llama-models/blob/ main/models/llama33/MODELCARD.md [29]A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan et al., “The llama 3 herd of models,” arXiv preprint arXiv:2407.21783, 2024. [30]DeepSeek-AI, “Deepseek-v4: Towards highly efficient million-token context intelligence,” arXiv preprint arXiv:2606.19348, 2026. [Online]. Available: https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro [31]Gemma Team, “Gemma 3 technical report,” arXiv preprint arXiv:2503.19786, 2025. [Online]. Available: https://arxiv.org/abs/ 2503.19786 [32] Z.ai, “GLM-5.2 model card,” Hugging Face model documentation, 2026. [Online]. Available: https://huggingface.co/zai-org/GLM-5.2 [33]MiniMaxAI, “MiniMax-M3 model card,” Hugging Face model documentation, 2026. [Online]. Available: https://huggingface.co/ MiniMaxAI/MiniMax-M3

Instant research discovery