Paper deep dive

PowerMarketJax: A JAX Benchmark Suite for Multi-Agent Reinforcement Learning in Power Markets

Zhanhua Pan, Xin Qin, Xiao Liu, Zhilong Cao, Jianhong Wang, Dawei Qiu

Year: 2026Venue: arXiv preprintArea: cs.MAType: PreprintEmbeddings: 213

Intelligence

Status: not_run | Model: - | Prompt: - | Confidence: 0%

Entities (0)

No extracted entities yet.

Relation Signals (0)

No relation signals yet.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Power markets are a natural testbed for multi-agent reinforcement learning (MARL), where multiple self-interested participants repeatedly submit bids. A market-clearing mechanism then determines dispatch and prices subject to power grid constraints and market settlement rules. However, existing MARL environments typically focus on a single market setting, implement simplified clearing mechanisms, or rely on CPU-based optimization solvers that slow large-scale training and limit the systematic study of bidding strategies and market behavior. We introduce PowerMarketJax, a benchmark suite for MARL across five power markets: day-ahead wholesale, real-time balancing, ancillary services, peer-to-peer double auctions, and local flexibility. Each environment implements its own clearing, pricing, and settlement rules while providing a common framework for learning and evaluation. We find that learned bidding behavior depends strongly on the market design: independent learners can miss better strategies when gains require many agents to change together, when more profitable strategies lie beyond a region of lower profit, or when profits disappear as more agents adopt the same strategy. PowerMarketJax implements both market simulation and policy training in JAX, allowing the entire pipeline to run on the GPU with 1,024 X 1,200 parallelisms across both environments and market participants, achieving up to 33X speedup over CPU-based baselines. Our open-source benchmark is available at: this https URL.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

209,538 characters extracted from source content.

Expand or collapse full text

PowerMarketJax: A JAX Benchmark Suite for Multi-Agent Reinforcement Learning in Power Markets Zhanhua Pan, Xin Qin, Xiao Liu, Zhilong Cao, Jianhong Wang, Dawei Qiu Nanyang Technological University, Singapore. Cornell University, USA. University of Bristol, UK. †thanks: Corresponding author: dawei.qiu@ntu.edu.sg. †These authors contributed equally. ‡Co-supervisors. Abstract Power markets are a natural testbed for multi-agent reinforcement learning (MARL), where multiple self-interested participants repeatedly submit bids. A market-clearing mechanism then determines dispatch and prices subject to power grid constraints and market settlement rules. However, existing MARL environments typically focus on a single market setting, implement simplified clearing mechanisms, or rely on CPU-based optimization solvers that slow large-scale training and limit the systematic study of bidding strategies and market behavior. We introduce PowerMarketJax, a benchmark suite for MARL across five power markets: day-ahead wholesale, real-time balancing, ancillary services, peer-to-peer double auctions, and local flexibility. Each environment implements its own clearing, pricing, and settlement rules while providing a common framework for learning and evaluation. We find that learned bidding behavior depends strongly on the market design: independent learners can miss better strategies when gains require many agents to change together, when more profitable strategies lie beyond a region of lower profit, or when profits disappear as more agents adopt the same strategy. PowerMarketJax implements both market simulation and policy training in JAX, allowing the entire pipeline to run on the GPU with 1 024 × 1 200 parallelisms across both environments and market participants, achieving up to 33× speedup over CPU-based baselines. Our open-source benchmark is available at: https://github.com/powermarketjax/PowerMarketJax. 1 Introduction Unlike most commodities, electricity cannot be stored efficiently at scale, so power markets are needed to continuously match supply and demand through mechanisms that determine which resources produce and at what price. Historically, these markets involved a relatively small set of large generators and mainly traded energy, making bidding strategies easier to characterize (David and Wen, 2000). This structure is now changing. Renewable generation, flexible demand, and energy storage have expanded both who participates and what is traded (Matamala and Strbac, 2025): alongside energy, markets now trade reserves, balancing, and ancillary services across multiple timescales and grid levels, each with its own clearing, pricing, and settlement rules (Zhu et al., 2025). This growing diversity of participants, products, and market mechanisms makes bidding behavior increasingly difficult to understand, creating an urgent need to study how participants bid and interact across different market settings. The repeated interactions among many market participants make power markets a natural setting for multi-agent reinforcement learning (MARL) (Liang et al., 2020). In a MARL formulation, each market participant can be modeled as an agent that observes market information, submits bids, receives clearing outcomes, and updates its bidding strategy over time. Since a participant’s profit depends on both its own bid actions and the actions of others through market clearing, bidding in power markets is inherently a multi-agent decision problem. MARL enables participants to learn such bidding strategies through repeated market interactions without requiring explicit knowledge of competitors’ exact strategies (Du et al., 2021). A number of environments have been developed for learning-based bidding in power markets, but important gaps remain for systematic MARL study. Most focus on a single market study, limiting their use as general benchmarks for MARL-based bidding (Yeh et al., 2023; Harder et al., 2025; El Helou et al., 2023; Salazar-Peña et al., 2026). A second gap is computational burden. An environment must solve a market-clearing optimization subject to grid physical constraints after every round of bids. Existing implementations typically rely on CPU solvers, making the millions of clearing problems required during MARL training a major computational bottleneck, especially with many agents and parallel environments (Wolgast and Nieße, 2024). To reduce this computational cost, some environments simplify pricing or settlement rules, but these simplifications can remove important market features that shape bidding incentives (Wolgast and Nieße, 2024). As a result, it remains difficult to systematically study how bidding strategies emerge and differ across power markets. Here, we introduce PowerMarketJax, a power market MARL benchmark implemented entirely in JAX (Bradbury et al., 2018). It parallelizes both optimization-based market environments and multi-agent policy computation on the GPU. By compiling clearing and learning into a single computation graph, PowerMarketJax avoids repeated CPU solver calls and host-device synchronization at each market step, enabling large-scale exploration of bidding strategies across different markets. This design achieves up to 33× higher throughput than CPU-based implementations and runs 1 024 parallel 1 200-participant market environments at over 27 000 environment steps per second (Appendix B). Our contributions are summarized as follows. First, we present a unified MARL benchmark covering five power markets: day-ahead wholesale, real-time balancing, ancillary services, peer-to-peer energy trading, and local flexibility. The benchmark spans different clearing mechanisms, timescales, and grid levels while preserving each market’s clearing, pricing, and settlement rules under a common interface. Second, we implement both market-clearing optimization and MARL policy training entirely in JAX, enabling GPU-based parallel computation across environments and agents for large-scale training and evaluation. Third, using this benchmark, we empirically show that market design strongly shapes what independent learners can discover from their own profit signals: profitable outcomes may require coordinated changes by many agents, lie beyond locally unfavorable regions, or disappear as more agents adopt the same strategy. These findings reveal learning challenges that arise from the interaction between strategic incentives and physically constrained power markets. 2 Related Work Power Market RL Benchmarks. A growing number of environments support RL for bidding in power markets. At the transmission level, SustainGym (Yeh et al., 2023) studies storage bidding in a network-constrained real-time market. ASSUME (Harder et al., 2025) supports RL-based bidding in European power markets. At the distribution level, OpenGridGym (El Helou et al., 2023) studies competing bidding agents under distribution network constraints. MARLEM (Salazar-Peña et al., 2026) considers MARL-based storage participation in local energy markets. Collectively, these works demonstrate the potential of (MA)RL for learning bidding strategies under different market mechanisms and physical constraints. However, existing environments typically focus on a single market, limiting systematic study of bidding strategies across diverse power markets. PowerMarketJax instead covers five representative markets spanning transmission and distribution levels. It further provides a unified framework for agent–market interaction while preserving each market’s clearing, pricing, settlement, and physical constraints. Finally, existing environments commonly separate CPU-based market simulation from GPU-based policy learning, creating repeated CPU–GPU data transfers and making market clearing a major bottleneck for MARL training. PowerMarketJax implements both market clearing and agent policy computation in JAX, enabling parallel execution on the GPU. JAX-based RL Benchmarks. A growing number of RL benchmarks have been implemented in JAX, motivated by the need for fast and highly parallel simulation (Rutherford et al., 2024). Isaac Gym (Makoviychuk et al., 2021) showed that moving environment simulation onto the GPU can reduce CPU–GPU communication overhead. JAX (Bradbury et al., 2018) further supports accelerator-native RL through compilation and vectorization. These advances have enabled high-throughput JAX environments across video games (Koyamada et al., 2023; Radji et al., 2026), physics (Freeman et al., 2021), robot learning (Zakka et al., 2025), and autonomous driving (Gulino et al., 2023). Recent work has also brought JAX-based acceleration to economic and trading environments. EconoJax (Ponse et al., 2025) simulates agent-based markets with order matching, JAX-LOB (Frey et al., 2023) accelerates limit-order-book simulation. These environments primarily rely on predefined if–else rule-based matching logic rather than constrained optimization. In contrast, power markets provide a distinctive setting where strategic economic decisions are tightly coupled to a real-world physical grid, requiring market outcomes to satisfy both market rules and engineering constraints. PowerMarketJax targets this setting by implementing optimization-based market clearing and strategic bidding entirely in JAX across five representative power markets. 3 Background Power systems maintain supply-demand balance through a range of markets for energy, reserves, and flexibility operating at different timescales and grid levels. Following the market structures and flexibility mechanisms discussed in Kirschen and Strbac (2026), this benchmark considers five representative settings: day-ahead wholesale, real-time balancing, ancillary services, peer-to-peer (P2P) energy trading, and local flexibility, as illustrated in Figure 1. They cover dispatch scheduling against day-ahead forecasts, adjustment to real-time imbalances, procurement of system reserves, local energy exchanges among prosumers, and distribution network support from flexible resources. These markets share a common decision structure but differ in their mechanisms. In each market, self-interested participants such as generators, storages, and consumers submit bids based on their asset states and public market signals, without knowing competitors’ private information. Their bids jointly determine market clearing, prices, and profits. However, the five markets differ in their traded products, information structures, physical constraints, and clearing and settlement rules. These differences shape participants’ incentives and bidding behavior, ultimately affecting market outcomes. Figure 1: Five power markets in PowerMarketJax across different timescales and grid levels. M1: Day-ahead wholesale (Day d−1d-1, transmission level). Generation must be scheduled before delivery day d because thermal units require time to start up and are subject to time-coupling constraints such as minimum up/down times. The day-ahead market clears supply–demand bids using forecasts for day d, determining unit commitment, generation schedules, and locational marginal prices (LMPs) that reflect transmission constraints. Cleared schedules then establish the baseline for subsequent real-time balancing. M2: Real-time balancing (Day d, transmission level). Actual demand and renewable generation may deviate from day-ahead forecasts because weather and consumption patterns cannot be predicted exactly one day in advance. The real-time market redispatches available resources over short intervals, typically every 15–30 minutes on day d, to maintain supply-demand balance under updated system conditions. Deviations from day-ahead schedules are settled at real-time prices, which can be more volatile and therefore expose participants to greater price risk. M3: Ancillary services (Day d, transmission level). Scheduled energy alone cannot ensure reliable operation because system disturbances such as a sudden generator outage can unbalance the system within seconds, far faster than the 15–30-minute energy dispatch interval. The ancillary services market procures capabilities such as frequency regulation and operating reserves, with requirements on availability and response speed. In this market, service providers are paid for reserved capacity, so their revenue can depend on maintaining available capacity rather than solely on energy production. M4: Peer-to-peer energy trading (Day d, distribution level). Prosumers with rooftop solar and storage can trade energy directly with nearby consumers instead of relying solely on upstream retailers. The P2P market matches buy bids and sell offers within a local energy community. The market-clearing price is determined by the intersection of the aggregated supply and demand curves, while matched quantities are determined from the submitted buy and sell quantities. Unmatched demand is supplied by the upstream grid at the higher retail price, while unmatched supply is exported to the grid at the lower feed-in tariff, creating an economic incentive for local peer-to-peer trading. M5: Local flexibility (Day d, distribution level) Even when sufficient electricity is available from the transmission system, distribution networks may become overloaded or local voltages may violate operating limits. To address these issues, the distribution system operator (DSO) procures flexibility from local resources, paying them to increase or decrease generation or consumption at specified locations, times, and durations. Under the pay-as-bid rule considered here, accepted flexibility providers are paid their submitted bid prices rather than a common clearing price. 4 PowerMarketJax: Power Market Environments in JAX PowerMarketJax is a suite of five power market environments implemented in JAX, with each market formulated as a MARL environment, as illustrated in Figure 2. The environments span day-ahead wholesale (M1), real-time balancing (M2), ancillary services (M3), peer-to-peer energy trading (M4), and local flexibility (M5). The markets preserve their distinct clearing, pricing, settlement, and physical constraints while following a common agent–market interaction interface. Despite their different market rules, all five environments follow the same step sequence. Agents first observe the information available to them and submit their market actions. The market then clears according to its specific mechanism, the resulting dispatches and prices are settled, and the environment returns the next states and rewards. We define the reward as the participant profit under the corresponding settlement rule. Together, these operations define one transition of the corresponding POSG and are implemented through a common step interface. Further details of the five environments and their accelerator-based execution are provided below. Figure 2: Execution paradigm of PowerMarketJax. 4.1 Five market environments as POSGs We formulate each market environment as a general-sum partially observable stochastic game (POSG) (Hansen et al., 2004). For each environment, we describe the market step, agent observations and actions, the clearing mechanism, reward, and state transition. Detailed market models and complete POSG formulations are provided in Appendix A.1–A.5. M1: Day-ahead wholesale. One step corresponds to one day-ahead market clearing, in which all 24 hours of the following day are cleared jointly. Each generator agent observes its initial commitment status, initial power output, and residual minimum up- and down-time requirements, together with the public day-ahead demand forecast. Its action consists of the prices submitted across all bid segments and delivery periods. Given the joint bids, the market-clearing mechanism determines unit commitment, dispatches, and LMPs, and each generator receives its daily profit as the reward. M2: Real-time balancing. One step corresponds to one 30-minute real-time market clearing during the delivery day. Each generator agent observes its previous realized dispatch, the day-ahead commitment and schedule for the corresponding hour, the associated day-ahead LMP, and the public realized net-demand conditions. Its action consists of the prices submitted across all bid segments for the current real-time interval. Given the joint bids from all generators, the market-clearing mechanism redispatches the committed generators subject to network and ramping constraints and determines the real-time LMPs. Each generator receives its realized interval profit under the two-settlement rule as the reward, where the day-ahead schedule remains settled at the day-ahead LMP and only deviations are settled at the real-time LMP. The realized dispatch of the current interval is carried forward as the previous dispatch for the next interval. M3: Ancillary services. One step corresponds to one 30-minute joint energy–reserve market clearing during the delivery day. Each generator agent observes its previous realized dispatch, the day-ahead commitment and schedule for the corresponding hour, the associated day-ahead LMP, the public realized net-demand conditions, and the published reserve requirements. Its action consists of the energy bid prices across all bid segments together with one reserve bid price for each reserve product. Given the joint bids of both energy and reserve, the market-clearing mechanism co-optimizes energy dispatch and reserve procurement subject to network, ramping, response time, and reserve capacity constraints, and determines both real-time LMPs and reserve prices. Each generator then receives its realized interval profit as the reward, including day-ahead and real-time energy settlements and reserve capacity payments. The realized dispatch of the current interval is carried forward as the previous dispatch for the next interval. M4: Peer-to-peer energy trading. One step corresponds to one 15-minute P2P trading interval. Each prosumer agent observes its own battery state of charge, net energy position, and the public retail and export prices, where a positive net position corresponds to a buy bid and a negative net position to a sell offer. It does not observe the private states or contemporaneous bids and asks of other prosumers. Its trading quantity is fixed before the auction based on its local generation, demand, and battery operation, so its action is the price submitted as an ask when selling or a bid when buying. Given the joint submissions, the double-auction mechanism sorts sellers and buyers, determines the matched quantities and uniform local clearing price, and settles any unmatched surplus or deficit with the upstream grid. Each prosumer then receives its realized interval profit as the reward, accounting for both P2P and residual grid transactions. The battery state of charge evolves to the next interval, while generation and demand update the prosumer’s next net energy position. M5: Local flexibility. One step corresponds to one 1-hour local flexibility procurement interval. Each aggregator agent observes its own battery state of charge, photovoltaic generation, local demand, and available flexibility, together with the public voltage and line-congestion requirements published by the DSO. It does not observe the private operating states, delivery costs, or contemporaneous bids of competing aggregators. Its action is a price–quantity bid specifying the requested payment and the maximum flexibility quantity it is willing to provide. Given the joint offers, the DSO clears the market by minimizing procurement cost subject to network voltage, line flow, and flexibility delivery constraints, and may partially accept a bid. Each aggregator then receives its realized interval profit as the reward under the pay-as-bid rule, with accepted flexibility paid at the submitted bid price. The accepted flexibility changes the battery state of charge and therefore determines the flexibility available in the next interval, while photovoltaic generation, local demand, and network conditions evolve to form the next market state. 4.2 Clearing and learning in a single computation graph Despite their different clearing and settlement mechanisms, all five environments share the same accelerator-based execution framework, with market simulation and policy learning running within a single compiled computation graph. Every operation within a training iteration, including market clearing, runs inside a single compiled computation graph on the device. reset and step are pure functions: each step sequentially applies action submission, market clearing, settlement, and observation construction, while the environment state is represented as a fixed-structure pytree that is passed as input and returned as output. Each step is compiled with jit, vectorized across parallel environments with vmap, and repeated over a fixed rollout horizon using lax.scan. The policy is evaluated within the same graph: during rollout, it maps observations to actions that are passed directly to the environment step. Policy updates are implemented with a second lax.scan over epochs and minibatches. A complete training iteration, consisting of rollout and update, is compiled as a single function. Further details of the compiled execution graph are provided in Appendix C.8. Market clearing remains inside the compiled graph because each clearing mechanism is expressed with fixed shape operations and iteration counts. The optimization-based markets use a common primal–dual interior-point solver for linear programs, with a fixed number of Newton iterations. The day-ahead market additionally includes binary commitment decisions. Rather than invoking a mixed-integer solver inside the compiled graph, we use a three-stage procedure. We first relax the commitment problem to a linear program, round the resulting commitment decisions, and then re-solve the continuous dispatch problem with the rounded commitment fixed (Appendix A.1.1). It is noted that the P2P market does not require an optimization solver. Its sorting, matching, and settlement operations are implemented directly using JAX Primitives (Bradbury et al., 2018). We validate these market-clearing implementations in three ways (Appendix B.4). First, clearing and settlement results are compared against a NumPy reference implementation. Second, dual prices produced by the JAX linear-program solver are compared with those obtained from HiGHS (Huangfu and Hall, 2018), an independent high-performance optimization solver. Third, the three-stage procedure is compared with the exact mixed-integer solution from HiGHS, with the commitment decisions kept binary. The resulting optimality gap is reported in Appendix B.4. We parallelize computation across both environments and agents. Each environment has its own market state and random key, and vmap batches multiple environments for parallel execution. Within each environment, agents remain coupled through the market-clearing mechanism, since their submitted actions jointly determine the market outcome. For policy evaluation, agents are also vectorized, so the actions of all agents across all parallel environments are computed simultaneously on the GPU in a single batched policy evaluation. 5 Results Systems and data. The three transmission-level markets (M1–M3) use the British case29gb system11 1 https://webhomes.maths.ed.ac.uk/OptEnergy/NetworkData/reducedGB/ with 29 buses and 66 generators as the main test case (Appendix E.1.1). They use one year of Great Britain demand and day-ahead forecast data, with 329 days for training and 36 held-out days for evaluation. M2 and M3 inherit the commitment and schedules obtained from the day-ahead market M1. Additional experiments on the 73-unit RTS-GMLC system22 2 https://github.com/GridMod/RTS-GMLC and the 151-unit Australian NEM system33 3 https://github.com/akxen/egrimod-nem-dataset are reported in Appendices E.1 to E.3. At the distribution level, M4 uses 1,200 Belgian households with PV from the Fluvius smart meter dataset44 4 https://opendata.fluvius.be/. M5 uses the 129-bus SwissDN feeder (Zapparoli et al., 2025) with battery fleets of 24 and 34 aggregators under its 2040 and 2050 scenarios. Detailed market formulations, system parameters, and data descriptions are provided in Appendices A and E. MARL algorithms. We evaluate two independent MARL algorithms, IPPO (de Witt et al., 2020) and SAC (Haarnoja et al., 2018) (the independent version), each with parameter sharing (PS) and agent-specific parameters (NoPS), giving four learners: IPPO-PS, IPPO-NoPS, SAC-PS, and SAC-NoPS. Each agent observes only its own available information and maximizes its own market profit, without a centralized critic or inter-agent communication. We use common default hyperparameters across markets and parallel environment rollouts on the GPU. Full training configurations and market-specific hyperparameters are provided in Appendix D. Baselines and evaluation. We compare the learned policies against truthful bidding and the corresponding untrained networks. Truthful bidding provides a competitive reference, while the untrained network is used to measure the effect of policy training. We further compare the learned policies with market-specific predefined non-learning strategies, including tests over different generator bidding levels in M1–M3, a fixed battery arbitrage schedule in M4, and unilateral flexibility-price deviations in M5. M1–M4 use 3 random seeds per experiment, while M5 uses 3–10 depending on the configuration. Evaluation uses held-out data and averages results across seeds. Full evaluation protocols and additional results are reported in Appendix E. All experiments run on a workstation with RTX 4500 Ada GPUs, described in Appendix B. The market models and data, the evaluation protocol and further results are in Appendices A to E. 5.1 Day-ahead wholesale market We start with the day-ahead market (M1) on case29gb. Each generator i bids at αi _i times its marginal generation cost, where αi≥1 _i≥ 1. Here, αi=1 _i=1 corresponds to truthful bidding, while αi>1 _i>1 represents bidding above marginal cost. Figure 3(a) shows that all four learners end near a bidding strategy around αi=1.5 _i=1.5. Over the 36 evaluation days, IPPO-PS and IPPO-NoPS learn to raise market prices with αi=1.50 _i=1.50 and αi=1.56 _i=1.56, respectively, and thereby increase the total generator profits by £643 and £917 million. The other two test systems show the same effect on a smaller scale, with αi _i reaching 1.07–1.29 (Appendix E.1.3). Figure 3: M1 day-ahead wholesale market (a) learning curves, (b) profit evaluation under joint and unilateral bidding strategies. However, the learned policies still leave substantial joint profit unrealized. As shown in Figure 3(b), when all 66 generators use the same α, total profit reaches the learned policy level near α=1.5α=1.5 and rises to £697 million at α=2α=2. In contrast, when one generator’s α is increased at a time while the others keep their learned IPPO-PS bids fixed, the resulting profit never exceeds the learned-policy level and is highest around α=1.5α=1.5–1.71.7. The learners therefore settle near a level where raising the bid further does not benefit an individual generator, even though all generators could gain substantially by raising their bids together. This gap arises for two reasons. First, the joint gain is not visible in each agent’s individual learning signal. An independent learner observes only how its own bidding strategy affects its own profit. For example, cheap nuclear units almost never set the market price (Appendix E.1.4), so raising their bids alone has little effect on price. Second, independent exploration rarely reaches the high-profit joint bidding region. The sampled α values remain concentrated around 1.5, making it unlikely that all units simultaneously explore the higher-bid region where the joint gain occurs. 5.2 Real-time balancing and ancillary services markets Figure 4: M2 real-time balancing market (a) bidding strategy, (b) learning curves, and M3 ancillary services market (c) profit under joint and unilateral bidding strategies. We next consider the real-time balancing (M2) and ancillary services (M3) markets. In M2, raising a bid alone is rarely profitable because a generator can lose output to its competitors. Accordingly, αi _i of the 30 day-ahead committed generators decreases during training under all four learners, reaching 1.04 under IPPO-PS and 1.24–1.40 under the other three methods (Figure 4(a,b)). This contrasts with the day-ahead market, where bidding above cost can increase an individual generator’s profit. The difference between individual and joint incentives is even clearer in M3 (Figure 4(c)). Raising the reserve bid of a single generator produces little gain: across the 29 committed generators, the largest profit increase is only £0.08 million per day. However, when all 66 generators raise their reserve bids together to £150/MWh, total profit increases by £9.05 million per day. However, the learned policies remain far below this joint gain, earning only £0.56–£3.69 million per day more than truthful bidding, and none outperforms its corresponding non-learning strategies. These results show how markets shape the learning signal available to each independent learner. In M2, raising a bid can reduce a generator’s own profit, pushing the learned bid toward cost. In M3, the individual profit signal is nearly zero, making the much larger joint gain difficult to discover. 5.3 Peer-to-peer energy market We next consider the P2P energy market (M4), where 1,200 households with PV and a battery trade in a double auction every 15 minutes. The learned policies improve household economics relative to truthful bidding. IPPO-NoPS increases household profit on all evaluation days, while IPPO-PS achieves the highest average profit, reducing the daily net cost per household from €1.20 under truthful bidding to €0.69–0.77 (Appendix E.4.3). The learners, however, adopt different battery strategies (Figure 5(a)). IPPO-NoPS and SAC-NoPS charge during the midday PV surplus and discharge during the evening demand peak. IPPO-PS and SAC-PS keep their batteries close to the minimum state of charge after selling their initial energy. To understand these differences, we let an increasing number of households follow a fixed schedule that charges from 11:00–15:00 and discharges from 18:00–22:00. A single household gains about €1.13 per day, but this gain decreases as more households adopt the same strategy and becomes negative at around 210–240 households (black line in Figure 5(b)), detailed in Appendix E.4.4. The decline is caused by the resulting change in market prices (Figure 5(c)). Simultaneous charging raises the midday clearing price, while simultaneous discharging lowers the evening price, reducing the price differentials on which the arbitrage relies. Once enough households participate, the remaining differential no longer covers efficiency losses and degradation costs. Thus, a strategy that is profitable for one household may become unprofitable when widely adopted. Figure 5(b) further suggests that PS can make market outcomes requiring heterogeneous behavior harder to learn: IPPO-NoPS leads only a subset of households to arbitrage, while IPPO-PS leads none to do so. Figure 5: P2P energy market (a) battery state of charge, (b) household gain, (c) local clearing price. 5.4 Local flexibility market Finally, we consider the local flexibility market (M5), where the DSO procures flexibility from battery aggregators to relieve network congestion. The market is pay-as-bid with a price floor. All four learners remain close to the floor. Yet a single aggregator can earn more by raising its offer sufficiently far above the floor; in the two 3.9 MW configurations, its return exceeds the floor-level return again at prices around 3.2 times the floor (Figure 6(a,b)). The reason is the shape of the individual return. A small bidding price increase from an aggregator causes the operator to procure from other cheaper aggregators instead, sharply reducing the deviating aggregator’s award and profit. Only at much higher prices does its return recover. Gradient-based learners therefore encounter a local barrier: small moves away from the floor reduce profit, while the more profitable region lies much farther away. The price parameterization further weakens exploration near the floor because the softplus mapping becomes nearly flat. The learners instead increase profit through their planned charging decisions. Planned charging raises the baseline flow on the congested line and therefore increases the flexibility requirement that the operator subsequently procures. With SAC-PS learners, the share of periods with procurement rises from 5.0% to 18.7% with 24 aggregators and from 4.9% to 15.4% with 34 aggregators (Figure 6(c)). A baseline monitor that excludes this learner-induced charging largely removes the effect, causing both procurement and learner returns to fall sharply. Thus, M5 exposes two distinct challenges for MARL in markets: profitable deviations may lie beyond a local learning barrier, and agents can change the demand for the service they are paid to provide. Figure 6: M5 local flexibility market (a) schematic return versus own offer price, (b) measured return in four system configurations, (c) procurement frequency versus the number of SAC-PS learners (seed mean with 95% bootstrap CI). 6 Conclusion PowerMarketJax shows that MARL benchmarks for power markets can preserve market-design fidelity while remaining computationally scalable. By integrating market clearing, settlement, policy evaluation, and learning into a GPU-accelerated JAX pipeline, the benchmark supports fast multi-agent rollouts while retaining the mechanisms and physical constraints that shape strategic bidding. The five markets provide a common POSG framework for evaluating learned bidding behavior and market outcomes across diverse market designs. Discussion. The experiments show that learned bidding behavior depends strongly on market design and the profit signal available to each agent. Across the five markets, learners may miss profitable joint outcomes, move toward truthful bidding when individual deviations are unprofitable, lose arbitrage gains when many agents adopt the same strategy, or fail to reach more profitable strategies beyond lower-profit regions. More broadly, these results show that market mechanisms shape the learning landscape itself: they determine which incentives are applied to individual agents and how the actions of many learners feed back into prices, dispatch, and procurement. This reflects challenges for MARL beyond maximizing return, including coordinated exploration, endogenous market responses, and heterogeneous agent behavior. Such learning-based simulations can also help stress-test market designs and reveal strategic incentives before new mechanisms are deployed. Limitations. PowerMarketJax is a research benchmark rather than a deployment-ready market simulator or bidding system, and the five environments simplify aspects of real market operation, network modeling, participant constraints, and market rules. Our experiments cover only the considered systems and independent PPO and SAC variants, so the learned policies should not be interpreted as predictions of real participant behavior or as equilibrium strategies. Future work could extend the benchmark to richer market designs, participant models, uncertainty, and broader classes of MARL algorithms. Acknowledgement Dawei Qiu is supported by the Nanyang Technological University (NTU) Start-Up Grant project #026749-00001 “Market Design for Low-Carbon Power Systems: Towards a Reliable, Affordable, and Resilient Transition”. Jianhong Wang is supported by the Engineering and Physical Sciences Research Council (EPSRC) [Grant Ref: EP/Y028732/1]. References Agrawal et al. (2018) A. Agrawal, R. Verschueren, S. Diamond, and S. Boyd A rewriting system for convex optimization problems. Journal of Control and Decision 5 (1), p. 42–60. Cited by: §B.3. Bradbury et al. (2018) J. Bradbury, R. Frostig, P. Hawkins, M. J. Johnson, Y. Katariya, C. Leary, D. Maclaurin, G. Necula, A. Paszke, J. VanderPlas, S. Wanderman-Milne, and Q. Zhang JAX: composable transformations of Python+NumPy programs. External Links: Link Cited by: §1, §2, §4.2. Brown et al. (2018) T. Brown, J. Hörsch, and D. Schlachtberger PyPSA: python for power system analysis. Journal of Open Research Software 6 (1), p. 4–4. Cited by: §B.3. David and Wen (2000) A.K. David and F. Wen Strategic bidding in competitive electricity markets: a literature survey. In 2000 Power Engineering Society Summer Meeting (Cat. No.00CH37134), Vol. 4, p. 2168–2173 vol. 4. Cited by: §1. de Witt et al. (2020) C. S. de Witt, T. Gupta, D. Makoviichuk, V. Makoviychuk, P. H. S. Torr, M. Sun, and S. Whiteson Is independent learning all you need in the StarCraft Multi-Agent Challenge?. Note: arXiv:2011.09533 External Links: 2011.09533 Cited by: §5. Diamond and Boyd (2016) S. Diamond and S. Boyd CVXPY: a Python-embedded modeling language for convex optimization. Journal of Machine Learning Research 17 (83), p. 1–5. Cited by: §B.3. Du et al. (2021) Y. Du, F. Li, H. Zandi, and Y. Xue Approximating Nash equilibrium in day-ahead electricity market bidding with multi-agent deep reinforcement learning. Journal of Modern Power Systems and Clean Energy 9 (3), p. 534–544. Cited by: §1. El Helou et al. (2023) R. El Helou, K. Lee, D. Wu, L. Xie, S. Shakkottai, and V. Subramanian OpenGridGym: an open-source AI-friendly toolkit for distribution market simulation. IEEE Transactions on Smart Grid 14 (2), p. 1555–1565. Cited by: §1, §2. Freeman et al. (2021) D. Freeman, E. Frey, A. Raichuk, S. Girgin, I. Mordatch, and O. Bachem Brax - a differentiable physics engine for large scale rigid body simulation. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, Vol. 1, p. . Cited by: §2. Frey et al. (2023) S. Y. Frey, K. Li, P. Nagy, S. Sapora, C. Lu, S. Zohren, J. Foerster, and A. Calinescu JAX-LOB: a gpu-accelerated limit order book simulator to unlock large scale reinforcement learning for trading. In Proceedings of the Fourth ACM International Conference on AI in Finance, p. 583–591. Cited by: §2. Gulino et al. (2023) C. Gulino, J. Fu, W. Luo, G. Tucker, E. Bronstein, Y. Lu, J. Harb, X. Pan, Y. Wang, X. Chen, et al. Waymax: an accelerated, data-driven simulator for large-scale autonomous driving research. Advances in Neural Information Processing Systems 36, p. 7730–7742. Cited by: §2. Haarnoja et al. (2018) T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine Soft actor-critic: off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Proceedings of the 35th International Conference on Machine Learning, J. Dy and A. Krause (Eds.), Proceedings of Machine Learning Research, Vol. 80, p. 1861–1870. Cited by: §5. Hansen et al. (2004) E. A. Hansen, D. S. Bernstein, and S. Zilberstein Dynamic programming for partially observable stochastic games. In Proceedings of the Nineteenth National Conference on Artificial Intelligence, Sixteenth Conference on Innovative Applications of Artificial Intelligence, July 25-29, 2004, San Jose, California, USA, p. 709–715. Cited by: §4.1. Harder et al. (2025) N. Harder, K. K. Miskiw, M. Khanra, F. Maurer, P. Patil, R. Qussous, C. Weinhardt, M. Klobasa, M. Ragwitz, and A. Weidlich ASSUME: an agent-based simulation framework for exploring electricity market dynamics with reinforcement learning. SoftwareX 30, p. 102176. Cited by: §1, §2. Huangfu and Hall (2018) Q. Huangfu and J. A. J. Hall Parallelizing the dual revised simplex method. Mathematical Programming Computation 10 (1), p. 119–142. Cited by: §B.4, §4.2. Kirschen and Strbac (2026) D. S. Kirschen and G. Strbac Fundamentals of power system economics. 3rd edition, Wiley. Cited by: §3. Koyamada et al. (2023) S. Koyamada, S. Okano, S. Nishimori, Y. Murata, K. Habara, H. Kita, and S. Ishii Pgx: hardware-accelerated parallel game simulators for reinforcement learning. Advances in Neural Information Processing Systems 36, p. 45716–45743. Cited by: §2. Liang et al. (2020) Y. Liang, C. Guo, Z. Ding, and H. Hua Agent-Based modeling in electricity market using deep deterministic policy gradient algorithm. IEEE Transactions on Power Systems 35 (6), p. 4180–4192. Cited by: §1. Makoviychuk et al. (2021) V. Makoviychuk, L. Wawrzyniak, Y. Guo, M. Lu, K. Storey, M. Macklin, D. Hoeller, N. Rudin, A. Allshire, A. Handa, and G. State Isaac Gym: high performance GPU-Based physics simulation for robot learning. Note: arXiv:2108.10470 External Links: 2108.10470 Cited by: §2. Matamala and Strbac (2025) C. Matamala and G. Strbac Strategic bidding in the frequency-containment ancillary services market. Applied Energy 401, p. 126811. Cited by: §1. Ponse et al. (2025) K. Ponse, A. Plaat, N. van Stein, and T. M. Moerland EconoJax: a fast & scalable economic simulation in jax. In AAMAS ’25: Proceedings of the 24th International Conference on Autonomous Agents and Multiagent Systems, Richland, SC, p. 1679–1687. Cited by: §2. Radji et al. (2026) W. Radji, T. Michel, and H. Piteau Octax: accelerated chip-8 arcade environments for reinforcement learning in jax. In International Conference on Learning Representations, Vol. 2026, p. 72220–72256. Cited by: §2. Raffin et al. (2021) A. Raffin, A. Hill, A. Gleave, A. Kanervisto, M. Ernestus, and N. Dormann Stable-Baselines3: reliable reinforcement learning implementations. Journal of Machine Learning Research 22 (268), p. 1–8. Cited by: §B.3. Rutherford et al. (2024) A. Rutherford, B. Ellis, M. Gallici, J. Cook, A. Lupu, G. Ingvarsson, T. Willi, R. Hammond, A. Khan, C. S. de Witt, A. Souly, S. Bandyopadhyay, M. Samvelyan, M. Jiang, R. Lange, S. Whiteson, B. Lacerda, N. Hawes, T. Rocktäschel, C. Lu, and J. Foerster JaxMARL: Multi-Agent RL environments and algorithms in JAX. In Advances in Neural Information Processing Systems 37 (NeurIPS 2024), Datasets and Benchmarks Track, p. 50925–50951. Cited by: §2. Salazar-Peña et al. (2026) N. Salazar-Peña, A. Tabares, and A. González-Mancera MARLEM: a multi-agent reinforcement learning simulation framework for implicit cooperation in decentralized local energy markets. Applied Energy 412, p. 127546. Cited by: §1, §2. Wolgast and Nieße (2024) T. Wolgast and A. Nieße Approximating energy market clearing and bidding with Model-Based reinforcement learning. IEEE Access 12, p. 145106–145117. Cited by: §1. Yeh et al. (2023) C. Yeh, V. Li, R. Datta, J. Arroyo, N. Christianson, C. Zhang, Y. Chen, M. M. Hosseini, A. Golmohammadi, Y. Shi, Y. Yue, and A. Wierman SustainGym: reinforcement learning environments for sustainable energy systems. In Advances in Neural Information Processing Systems 36, p. 59464–59476. Cited by: §1, §2. Zakka et al. (2025) K. Zakka, B. Tabanpour, Q. Liao, M. Haiderbhai, S. Holt, J. Luo, A. Allshire, E. Frey, K. Sreenath, L. Kahrs, C. Sferrazza, Y. Tassa, and P. Abbeel Demonstrating MuJoCo Playground. In Robotics: Science and Systems XXI, External Links: 2502.08844 Cited by: §2. Zapparoli et al. (2025) L. Zapparoli, A. Oneto, M. Parajeles Herrera, B. Gjorgiev, G. Hug, and G. Sansavini Future deployment and flexibility of distributed energy resources in the distribution grids of Switzerland. Scientific Data 12, p. 1491. Cited by: §5. Zhu et al. (2025) Z. Zhu, S. Bu, K. W. Chan, F. Li, Y. Ye, C. Y. Chung, and G. Strbac Designing the future electricity spot market with high renewables via reliable simulations. Nature Reviews Electrical Engineering 2 (5), p. 320–337. Cited by: §1. Appendix Contents 1 Introduction 2 Related Work 3 Background 4 PowerMarketJax: Power Market Environments in JAX 4.1 Five market environments as POSGs 4.2 Clearing and learning in a single computation graph 5 Results 5.1 Day-ahead wholesale market 5.2 Real-time balancing and ancillary services markets 5.3 Peer-to-peer energy market 5.4 Local flexibility market 6 Conclusion References A Market Models and Partially Observable Stochastic Game A.1 M1: Day-ahead Wholesale Market A.1.1 Market model A.1.2 Pricing and settlement A.1.3 Partially Observable Stochastic Game A.2 M2: Real-Time Balancing Market A.2.1 Market model A.2.2 Pricing and settlement A.2.3 Partially Observable Stochastic Game A.3 M3: Ancillary Services Market A.3.1 Market model A.3.2 Pricing and settlement A.3.3 Partially Observable Stochastic Game A.4 M4: Peer-to-Peer Local Energy Market A.4.1 Market model A.4.2 Pricing and settlement A.4.3 Partially Observable Stochastic Game A.5 M5: Local Flexibility Market A.5.1 Market model A.5.2 Payment and settlement A.5.3 Partially Observable Stochastic Game B Computation Performance B.1 Three layouts on the peer-to-peer market B.2 Across the five markets B.3 Against open-source tools B.4 Implementation validation C Quickstart and API C.1 Installation C.2 Minimal rollout C.3 The environment interface C.4 Constructing the five markets C.5 Scenario and learner configuration C.6 Command-line usage C.7 Cases and data C.8 Clearing and learning in a single computation graph D Training Setup and Hyperparameters D.1 Four learners D.2 Hyperparameters D.3 Training scale D.4 Evaluation protocol E Additional Market Results E.1 M1: Day-ahead Wholesale Market E.1.1 Market setting E.1.2 Agent training E.1.3 Discussion of results E.1.4 Details of supply curves E.1.5 Markups of different learners E.2 M2: Real-Time Balancing Market E.2.1 Market setting E.2.2 Agent training E.2.3 Discussion of results E.2.4 Unilateral and joint markups E.3 M3: Ancillary Services Market E.3.1 Market setting E.3.2 Agent training E.3.3 Discussion of results E.3.4 Reserve offer analysis E.4 M4: Peer-to-Peer Local Energy Market E.4.1 Market setting E.4.2 Agent training E.4.3 Discussion of results E.4.4 Arbitrage gain and participation E.5 M5: Local Flexibility Market E.5.1 Market setting E.5.2 Agent training E.5.3 Analysis of the procurement requirement E.5.4 Analysis of a unilateral markup Appendix A Market Models and Partially Observable Stochastic Game A.1 M1: Day-ahead Wholesale Market The day-ahead market is a centralized auction held on day d−1d-1 that clears all T hourly delivery periods of day d simultaneously, with Δda=1hour ^da=1~hour. Market participants submit their strategic bids, which the market operator clears through a security-constrained unit commitment (SCUC) problem. The resulting schedules are priced and settled using locational marginal prices (LMPs). Section A.1.1 presents the market clearing model, section A.1.2 introduces the pricing and settlement rules, and section A.1.3 formulates the partially observable stochastic game (POSG) for strategic bidding in day-ahead wholesale market. A.1.1 Market model Each generator agent i submits, for each delivery period t, a stepwise supply curve consisting of K segments. Each segment k has quantity Gi,k=p¯i−p¯iKG_i,k= p_i- p_iK (1) and bid price πi,k,t _i,k,t, where p¯i p_i and p¯i p_i are the minimum and maximum outputs of the unit. Bid prices are required to be non-decreasing across segments, i.e., πi,k,t≤πi,k+1,t _i,k,t≤ _i,k+1,t. Technical parameters, including ramp rates, minimum up- and down-times, start-up costs, and no-load costs, are registered with the market operator and are not part of the per-period bid. Demand is inelastic and treated as exogenous. The day-ahead market jointly determines unit commitment and dispatch over all T hourly delivery periods by solving min⁡∑t=1Tg,u,v,w,s⁡[∑i,kπi,k,tgi,k,t+∑iΔdaNLiui,t+VOLL∑nsn,t]+∑i,tSivi,t, _\g,\,u,\,v,\,w,\,s\\; _t=1^T [ _i,k _i,k,t\,g_i,k,t+ _i ^da\,NL_i\,u_i,t+VOLL _ns_n,t ]+ _i,tS_i\,v_i,t, (2a) subject to pi,t=p¯iui,t+∑kgi,k,t, p_i,t= p_i\,u_i,t+ _kg_i,k,t, (2b) 0≤gi,k,t≤Gi,kui,t, 0≤ g_i,k,t≤ G_i,k\,u_i,t, (2c) Pn,t=∑i:bus⁡(i)=npi,t+sn,t−dn,t, P_n,t= _i\,:\,bus(i)=np_i,t+s_n,t-d_n,t, (2d) ∑nPn,t=0(:λt), _nP_n,t=0 (: _t), (2e) −F≤PTDF⋅Pt≤F(:μt+,μt−), -F · P_t≤ F (:μ^+_t,\,μ^-_t), (2f) 0≤sn,t≤dn,t(:ρn,t), 0≤ s_n,t≤ d_n,t (: _n,t), (2g) −RidnΔda−p¯iwi,t≤pi,t−pi,t−1≤RiupΔda+p¯ivi,t, -R^dn_i ^da- p_i\,w_i,t\;≤\;p_i,t-p_i,t-1\;≤\;R^up_i ^da+ p_i\,v_i,t, (2h) ui,t−ui,t−1=vi,t−wi,t, u_i,t-u_i,t-1=v_i,t-w_i,t, (2i) vi,t+wi,t≤1, v_i,t+w_i,t≤ 1, (2j) ∑τ=t−UTi+1tvi,τ≤ui,t, _τ=t-UT_i+1^tv_i,τ≤ u_i,t, (2k) ∑τ=t−DTi+1twi,τ≤1−ui,t, _τ=t-DT_i+1^tw_i,τ≤ 1-u_i,t, (2l) ui,t,vi,t,wi,t∈0,1. u_i,t,\,v_i,t,\,w_i,t∈\0,1\. (2m) The objective (2a) minimizes the total offered operating cost over the delivery day. The first term represents the cost of accepted energy offers, the second accounts for the no-load cost of committed generators, and the third penalizes involuntary load shedding. The start-up cost SiS_i is incurred whenever generator i transitions from off to on. Here, Δda=1h ^da=1~h is the duration of each day-ahead delivery period, NLiNL_i is the no-load cost, and VOLLVOLL is the value of lost load, which is set sufficiently high so that load shedding is used only as a last resort. Constraints (2b) and (2c) define the output of each generator. The binary variable ui,tu_i,t indicates whether generator i is committed. When ui,t=1u_i,t=1, its output consists of the minimum generation level p¯i p_i plus the accepted quantities gi,k,tg_i,k,t from its offer segments. Each accepted segment is bounded by Gi,kG_i,k, so the total output remains within [p¯i,p¯i][ p_i,\, p_i]. When ui,t=0u_i,t=0, all segment quantities and the generator output are zero. Constraints (2d)–(2f) impose the network constraints. Equation (2d) defines the net injection Pn,tP_n,t at bus n as local generation plus load shedding minus exogenous demand dn,td_n,t. Constraint (2e) enforces system-wide supply–demand balance. Under the DC power-flow approximation, constraint (2f) maps the nodal injections to transmission-line flows through the power transfer distribution factor matrix PTDF∈ℝNline×NbusPTDF ^N^line× N^bus and limits each line flow by its rating F. Network losses are not modeled. Constraint (2g) bounds load shedding at each bus by the local demand. The variable sn,ts_n,t provides emergency feasibility when available generation and transmission capacity are insufficient to serve all demand. Because load shedding carries the large penalty VOLLVOLL in the objective, it is used only when lower-cost feasible alternatives are unavailable. Constraint (2h) limits the change in generator output between consecutive delivery periods according to the upward and downward ramp rates RiupR_i^up and RidnR_i^dn. The start-up term p¯ivi,t p_iv_i,t allows a unit that starts in period t to move from zero output to at least its minimum generation level, while the shut-down term p¯iwi,t p_iw_i,t allows a unit that shuts down to reduce its output to zero. Constraints (2i) and (2j) describe commitment transitions. The binary variables vi,tv_i,t and wi,tw_i,t denote start-up and shut-down events, respectively. Equation (2i) links these events to changes in commitment status, while equation (2j) prevents a generator from starting up and shutting down in the same period. Constraints (2k) and (2l) enforce the minimum up- and down-times UTiUT_i and DTiDT_i. After a start-up, the unit must remain committed for at least UTiUT_i periods; after a shut-down, it must remain offline for at least DTiDT_i periods. The initial commitment status ui,0u_i,0, initial output pi,0p_i,0, and any residual minimum up- or down-time requirements from the previous day are treated as exogenous initial conditions. Solution method. Optimization (2) is a security-constrained unit commitment (SCUC) problem formulated as a mixed-integer linear program (MILP). Exact MILP solvers rely on dynamic branch-and-bound procedures that are difficult to batch and compile efficiently on accelerators. We therefore use a relax-round-resolve procedure. First, the binary commitment variables are relaxed to ui,t∈[0,1]u_i,t∈[0,1], and the resulting linear program is solved. Second, the relaxed commitment is rounded by setting ui,tint=1u^int_i,t=1 if ui,t>εu_i,t> and ui,tint=0u^int_i,t=0 otherwise, from which the start-up and shut-down indicators vi,tintv^int_i,t and wi,tintw^int_i,t are determined. Third, these commitment decisions are fixed and the continuous dispatch problem is re-solved as a security-constrained economic dispatch (SCED). Dispatch schedules are obtained from this final solve, while LMPs are computed from the dual variables of its power balance and transmission constraint. This separation follows the common practice of determining commitment first and obtaining dispatch and prices with commitment fixed. The optimality gap relative to the exact solution is evaluated offline and reported in Appendix B.4. A.1.2 Pricing and settlement Given the fixed commitment obtained from the solution procedure above, LMPs are computed from the dual variables of the final SCED problem. The LMP at bus n at period t is the marginal cost of serving one additional MWh at that bus and is given by LMPn,t=λt−∑l(μl,t+−μl,t−)PTDFl,n−ρn,t,LMP_n,t= _t- _l (μ^+_l,t-μ^-_l,t )\,PTDF_l,n- _n,t, (3a) where the first term is the system marginal energy price, which is common across the transmission network; the second term is the congestion component, which is zero when no transmission constraint is binding and causes nodal prices to differ under congestion; and the third term can be non-zero only when the upper bound on load curtailment is binding. There is no loss component because transmission losses are not modeled. Cleared quantities pi,tp_i,t are settled at the LMP of the bus where unit i is located. The revenue, cost, and profit of unit i over the market day are revenuei _i =∑tΔda⋅LMPbus⁡(i),tpi,t, = _t ^da·LMP_bus(i),t\;p_i,t, (3b) costi _i =∑t[Δda⋅TCi(pi,t)+Δda⋅NLiui,t]+∑tSivi,t, = _t [ ^da·TC_i(p_i,t)+ ^da·NL_i\,u_i,t ]+ _tS_i\,v_i,t, (3c) profiti _i =revenuei−costi, =revenue_i-cost_i, (3d) where TCi(p)=∫0pMCi(x)xTC_i(p)= _0^pMC_i(x)\,dx is the true variable generation cost rate of unit i, and MCi(p)=aip2+bip+ciMC_i(p)=a_ip^2+b_ip+c_i is its true marginal cost. Setting ai=0a_i=0 recovers a linear marginal cost curve. The true variable generation cost TCi(⋅)TC_i(·) is used only to evaluate participant profit and does not enter market clearing; the clearing problem instead uses submitted energy offer prices together with the registered no-load and start-up costs. Settlement does not include uplift payment, so a committed unit may earn negative profit when its energy market revenue does not recover its start-up and no-load costs. A.1.3 Partially Observable Stochastic Game We formulate repeated participation in the day-ahead wholesale market as a general-sum POSG DA=⟨ℐ,,ii∈ℐ,ii∈ℐ,P,rii∈ℐ,γ⟩,G^DA= \ I,\,S,\O_i\_i ,\A_i\_i ,\,P,\,\r_i\_i ,\,γ\ , (4) where ℐI is the set of strategic generator agents, S is the market state space, iO_i and iA_i are the observation and action spaces of agent i, P is the state transition kernel, rir_i is the individual profit-based reward, and γ∈[0,1]γ∈[0,1] is the discount factor. One episodic market step corresponds to one day-ahead market clearing, in which all T delivery periods of the following day are cleared jointly. State. At market day d, the underlying state contains the system information required to clear the following delivery day, sd=(xdsys,xi,di∈ℐ)∈,s_d= (\,x^sys_d,\x_i,d\_i \, ) , (5) where xdsysx^sys_d contains the day-ahead forecast of system demand dn,t,d_n,t,d and other exogenous system conditions such as renewable generation. The generator state is xi,d=(ui,d0,pi,d0,τi,dup,τi,ddn),x_i,d= (u^0_i,d,p^0_i,d,τ^up_i,d,τ^dn_i,d ), (6) where ui,d0u^0_i,d and pi,d0p^0_i,d are the commitment status and power output of generator i entering the delivery day, and τi,dupτ^up_i,d and τi,ddnτ^dn_i,d denote any residual minimum up- and downtime requirements carried over from the previous day. Other system and generator parameters, such as the PTDF matrix PTDFPTDF, transmission limits F, generation capacity limits (p¯i,Gi,k)( p_i,G_i,k), and ramp rates (Riup,Ridn)(R_i^up,R_i^dn), are fixed environment parameters known to the market operator. Observation. Each generator i observes its own private information and the public information available before bidding, oi,d∈i(sd)=(xi,d,xdpub),o_i,d _i(s_d)= (x_i,d,x^pub_d ), (7) where xdpubx^pub_d includes the public day-ahead system information of demand forecast dn,t,d_n,t,d. An agent does not observe the private generation costs or contemporaneous offers of competing generators before market clearing. Hence, oi,do_i,d does not fully reveal sds_d, making the bidding problem partially observable. Action. The action of generator i is a single markup αi,d _i,d on its truthful offer πi,10≤⋯≤πi,K0π^0_i,1≤·s≤π^0_i,K, in which each segment is priced at the unit’s true marginal cost MCiMC_i, giving its complete day-ahead price bid over all segments and delivery periods, ai,d=αi,d∈i=[1,α¯],πi,k,t,d=αi,dπi,k0,k=1,…,K;t=1,…,T,a_i,d= _i,d _i=[1, α], _i,k,t,d= _i,d\,π^0_i,k, k=1,…,K;\;t=1,…,T, (8) where α¯ α is the largest admissible markup. The segment quantities Gi,kG_i,k are fixed as defined in Section A.1.1, while the agent chooses the markup. Admissible bids satisfy the market price bounds and the monotonicity rule π¯≤πi,1,t,d≤πi,2,t,d≤⋯≤πi,K,t,d≤π¯,∀t. π≤ _i,1,t,d≤ _i,2,t,d≤·s≤ _i,K,t,d≤ π, ∀ t. (9) The joint action ad=(a1,d,…,a|ℐ|,d)a_d=(a_1,d,…,a_|I|,d) therefore represents the set of supply bids submitted by all strategic generators for the following delivery day. Market clearing and outcome. Given state sds_d and joint action ada_d, the market operator applies the day-ahead clearing model and pricing rules described in Sections A.1.1–A.1.2. We denote this mapping by zd=DA(sd,ad),z_d=C^DA(s_d,a_d), (10) where the market outcome zdz_d contains the commitment decisions ui,1…T,du_i,1...T,d, accepted generation quantities gi,k,1…T,dg_i,k,1...T,d, dispatch schedules pi,1…T,dp_i,1...T,d, load curtailment sn,1…T,ds_n,1...T,d, and nodal LMPs LMPn,1…T,dLMP_n,1...T,d for all T delivery periods. Through this clearing mechanism, the payoff of one agent depends not only on its own offer but also on the bids submitted by the other agents. Reward. Each agent i receives its realized market profit as reward. Using the settlement defined in Section A.1.2, ri,d r_i,d =profiti,d=revenuei−costi =profit_i,d=revenue_i-cost_i =∑t=1TΔdaLMPbus⁡(i),t,dpi,t,d = _t=1^T ^da\,LMP_bus(i),t,d\ p_i,t,d −∑t=1T[ΔdaTCi(pi,t,d)+ΔdaNLiui,t,d]−∑t=1TSivi,t,d. - _t=1^T [\, ^da\,TC_i(p_i,t,d)+ ^da\,NL_iu_i,t,d\, ]- _t=1^TS_i\,v_i,t,d. (11) Thus, submitted bidding prices affect the reward only through the resulting market-clearing outcomes; the true variable generation cost TCi(⋅)TC_i(·) is used only to evaluate the realized cost and then profit. Since each generator maximizes its own profit rather than a common system-wide reward, the resulting POSG is general-sum. State transition. After the delivery periods cleared at market step d, the terminal operating condition of each generator is carried into the next market step. Specifically, the initial commitment status and power output for the next day are ui,d+10=ui,T,d,pi,d+10=pi,T,d,u^0_i,d+1=u_i,T,d, p^0_i,d+1=p_i,T,d, (12) while the residual minimum up- or downtime requirements τi,d+1upτ^up_i,d+1 and τi,d+1dnτ^dn_i,d+1 are updated according to the terminal commitment trajectory. Hence, the next generator state xi,d+1=(ui,d+10,pi,d+10,τi,d+1up,τi,d+1dn)x_i,d+1= (u^0_i,d+1,p^0_i,d+1,τ^up_i,d+1,τ^dn_i,d+1 ) (13) depends on the commitment and dispatch outcomes produced by the joint bids at market step d. The exogenous system state xd+1sysx^sys_d+1, including the next day-ahead forecasts of demand, evolves according to the underlying data process. The overall state transition is therefore sd+1∼P(⋅∣sd,ad),s_d+1 P(\,· s_d,a_d), (14) where the dependence on ada_d arises through the cleared commitment and dispatch decisions carried into the next market step. Agent objective. Each generator uses a policy σi(ai,d∣oi,d) _i(a_i,d o_i,d) to choose its day-ahead bid and seeks to maximize its expected discounted profit over an episode of H market days, Ji(σi,σ−i)=⁡[∑d=1Hγd−1ri,d].J_i( _i, _-i)=E [ _d=1^Hγ^d-1r_i,d ]. (15) where σ−i _-i denotes the joint policies of all other generators. Because market clearing depends on the joint bids of all generators, the expected return of agent i depends on both its own policy σi _i and the policies of the other agents σ−i _-i. Each agent therefore learns its bidding policy through repeated market interactions without observing competitors’ private information or policies. A.2 M2: Real-Time Balancing Market The real-time balancing market consists of a sequence of auctions during delivery day D, with one auction for each real-time interval Δrt=30mins ^rt=30~mins. It takes the commitment and schedules established by the day-ahead wholesale market as fixed and redispatches available resources against realized system demand. Only deviations from the day-ahead schedules are settled at real-time LMPs, while the day-ahead schedules remain settled at day-ahead LMPs. This section presents the real-time balancing market clearing model in subsection A.2.1, the two-settlement rule in subsection A.2.2, and the corresponding POSG in subsection A.2.3. A.2.1 Market model Let τ index the 30-minute real-time intervals and let h⁡(τ)h(τ) denote the corresponding hourly period in the day-ahead market. The day-ahead outcome enters the real-time market as fixed data through the commitment ui,h⁡(τ)dau^da_i,h(τ), scheduled generation qi,h⁡(τ)daq^da_i,h(τ), and day-ahead price LMPn,h⁡(τ)daLMP^da_n,h(τ). The realized net demand at bus n is d~n,τ d_n,τ. Generators submit price–quantity bids with the same structure as in the day-ahead market, and one market clearing problem is solved for each real-time interval: min⁡∑i,kg,s⁡πi,k,τgi,k,τ+VOLL∑nsn,τ, _\g,\,s\\; _i,k _i,k,τ\,g_i,k,τ+VOLL _ns_n,τ, (16a) subject to pi,τ=p¯iui,h⁡(τ)da+∑kgi,k,τ, p_i,τ= p_i\,u^da_i,h(τ)+ _kg_i,k,τ, (16b) 0≤gi,k,τ≤Gi,kui,h⁡(τ)da, 0≤ g_i,k,τ≤ G_i,k\,u^da_i,h(τ), (16c) Pn,τ=∑i:bus⁡(i)=npi,τ+sn,τ−d~n,τ, P_n,τ= _i:bus(i)=np_i,τ+s_n,τ- d_n,τ, (16d) ∑nPn,τ=0(:λτ), _nP_n,τ=0 (: _τ), (16e) −F≤PTDF⋅Pτ≤F(:μτ+,μτ−), -F · P_τ≤ F (: _τ^+, _τ^-), (16f) 0≤sn,τ≤max(d~n,τ,0)(:ρn,τ), 0≤ s_n,τ≤ ( d_n,τ,0) (: _n,τ), (16g) −RidnΔrt−p¯iwi,τda≤pi,τ−pi,τ−1≤RiupΔrt+p¯ivi,τda. -R_i^dn ^rt- p_iw^da_i,τ≤ p_i,τ-p_i,τ-1≤ R_i^up ^rt+ p_iv^da_i,τ. (16h) The objective minimizes the as-bid redispatch cost and load-shedding penalty for each real-time interval Δrt ^rt. Unlike the day-ahead market, unit commitment is not re-optimized: ui,h⁡(τ)dau^da_i,h(τ) is fixed by the day-ahead outcome, and the real-time market only adjusts the dispatch of committed units. The nodal balance and transmission constraints ensure that the resulting dispatch remains physically feasible under realized net demand. Consecutive real-time intervals are coupled through the ramping constraint, where pi,τ−1p_i,τ-1 is the realized dispatch from the previous interval. The start-up and shut-down indicators vi,τdav^da_i,τ and wi,τdaw^da_i,τ are determined by changes in the mapped day-ahead commitment and are non-zero only when the commitment status changes between consecutive real-time intervals. Constraint (16b) defines the real-time output of generator i. Because the commitment ui,h⁡(τ)dau^da_i,h(τ) is fixed by the day-ahead market, only generators committed in the corresponding day-ahead period can produce in real time. Their output equals the minimum generation level plus the accepted quantities from the submitted bid segments. Constraint (16c) limits each accepted segment gi,k,τg_i,k,τ to its available quantity Gi,kG_i,k. Constraint (16d) defines the net injection Pn,τP_n,τ at bus n as local generation plus load shedding minus realized net demand. Constraint (16e) enforces system-wide supply–demand balance, while constraint (16f) keeps transmission flows within the line ratings. The associated dual variables λτ _τ, μτ+ _τ^+, and μτ− _τ^- are used to determine the real-time LMPs. Constraint (16g) allows emergency load curtailment only up to the positive net demand at each bus. Hence, sn,τ=0s_n,τ=0 when renewable generation exceeds local demand. The dual variable ρn,τ _n,τ is associated with the upper bound on load shedding. Constraint (16h) limits changes in generator output between consecutive real-time intervals according to the upward and downward ramp rates. Here, pi,τ−1p_i,τ-1 is the realized dispatch from the previous real-time interval. The fixed indicators vi,τdav^da_i,τ and wi,τdaw^da_i,τ account for start-up and shut-down transitions implied by the day-ahead commitment. Unlike the day-ahead market, the real-time market does not re-optimize unit commitment. The day-ahead commitment is fixed, and the real-time clearing adjusts only the dispatch quantities in response to realized system conditions. Consecutive real-time intervals are coupled through generator ramping. A.2.2 Pricing and settlement Real-time LMPs are computed from the dual variables of the real-time clearing problem using the same energy–congestion decomposition as in equation (3a): LMPn,τrt=λτ−∑l(μl,τ+−μl,τ−)PTDFl,n−ρn,τ.LMP^rt_n,τ= _τ- _l (μ^+_l,τ-μ^-_l,τ )\,PTDF_l,n- _n,τ. (17) Unlike day-ahead LMPs, the real-time LMPs reflect realized demand and short-term operating constraints. They can therefore be more volatile and may become negative under local oversupply or binding downward-flexibility constraints. The market follows a two-settlement rule. The day-ahead schedule remains settled at the day-ahead LMP, while only the deviation from that schedule is settled at the real-time LMP. For generator i in real-time interval τ, revenuei,τda ^da_i,τ =ΔrtLMPbus⁡(i),h⁡(τ)daqi,h⁡(τ)da, = ^rt\,LMP^da_bus(i),h(τ)\,q^da_i,h(τ), (18a) revenuei,τrt ^rt_i,τ =ΔrtLMPbus⁡(i),τrt(pi,τ−qi,h⁡(τ)da), = ^rt\,LMP^rt_bus(i),τ (p_i,τ-q^da_i,h(τ) ), (18b) costi,τ _i,τ =ΔrtTCi(pi,τ)+ΔrtNLiui,h⁡(τ)da+Sivi,τda, = ^rt\,TC_i(p_i,τ)+ ^rt\,NL_iu^da_i,h(τ)+S_iv^da_i,τ, (18c) profiti,τ _i,τ =revenuei,τda+revenuei,τrt−costi,τ. =revenue^da_i,τ+revenue^rt_i,τ-cost_i,τ. (18d) In this context, a generator producing above its day-ahead schedule sells the positive deviation at the real-time LMP, while a generator producing below its schedule effectively buys back the shortfall at the real-time LMP. The day-ahead revenue, commitment status, and associated no-load and start-up costs are fixed by the day-ahead outcome and do not depend on the real-time bid. They are retained in the reward so that the reported profit represents the full economic outcome of delivery. Remark on the day-ahead and real-time coupling. In practice, the two markets operate at different temporal scales: the day-ahead market is cleared on day D−1D-1, while the real-time market is cleared at each interval of delivery day D. The commitment ui,h⁡(τ)dau^da_i,h(τ), binding schedule qi,h⁡(τ)daq^da_i,h(τ), and day-ahead price LMPn,h⁡(τ)daLMP^da_n,h(τ) are fixed parameters of the real-time clearing, while the real-time market prices deviations arising from uncertainty not captured by the day-ahead forecast. A.2.3 Partially Observable Stochastic Game The real-time balancing market can be also formulated as a general-sum partially observable stochastic game (POSG), rt=⟨ℐ,,ii∈ℐ,ii∈ℐ,P,rii∈ℐ,γ⟩,G^rt= \ I,\,S,\O_i\_i ,\A_i\_i ,\,P,\,\r_i\_i ,\,γ\ , (19) where ℐI is the set of strategic generators. One episodic market step τ corresponds to one 30-minute real-time interval Δrt ^rt. For a 24-hour delivery day, the real-time market therefore contains Trt=48T^rt=48 sequential clearing steps. State. The state at interval τ is sτ=(xτsys,xτda,xi,τi∈ℐ)∈,s_τ= (\,x^sys_τ,\,x^da_τ,\,\x_i,τ\_i \, ) , (20) where xτsysx^sys_τ contains the realized system conditions used for real-time clearing, i.e., the net bus demand d~n,τ d_n,τ. The day-ahead component is xτda=ui,h⁡(τ)da,qi,h⁡(τ)da,LMPn,h⁡(τ)dai,n,x^da_τ= \\,u^da_i,h(τ),\,q^da_i,h(τ),\,LMP^da_n,h(τ)\, \_i,n, (21) which contains the fixed commitment ui,h⁡(τ)dau^da_i,h(τ), binding generation schedules qi,h⁡(τ)daq^da_i,h(τ), and nodal prices LMPn,h⁡(τ)daLMP^da_n,h(τ) from the corresponding day-ahead period. The operating state of generator i is xi,τ=(pi,τ−1,vi,τda,wi,τda),x_i,τ= (\,p_i,τ-1,\,v^da_i,τ,\,w^da_i,τ\, ), (22) where pi,τ−1p_i,τ-1 is the realized dispatch from the previous real-time interval τ−1τ-1, and vi,τdav^da_i,τ and wi,τdaw^da_i,τ are the start-up and shut-down indicators implied by the mapped day-ahead commitment, which are used in the ramping constraint (16h) to account for output changes associated with start-up and shut-down transitions. Network parameters, generator operating limits, and cost parameters are fixed environment parameters and are omitted from the dynamic state. Observation. Each generator i observes its own operating information together with public market information. Its observation at real-time interval τ is oi,τ=(pi,τ−1,ui,h⁡(τ)da,qi,h⁡(τ)da,LMPbus⁡(i),h⁡(τ)da,xτpub)=Ωi(sτ)∈i,o_i,τ= (\,p_i,τ-1,\,u^da_i,h(τ),\,q^da_i,h(τ),\,LMP^da_bus(i),h(τ),\,x^pub_τ\, )= _i(s_τ) _i, (23) where pi,τ−1p_i,τ-1 is the generator’s realized dispatch in the previous real-time interval τ−1τ-1, ui,h⁡(τ)dau^da_i,h(τ) and qi,h⁡(τ)daq^da_i,h(τ) are its day-ahead commitment and schedule, and LMPbus⁡(i),h⁡(τ)daLMP^da_bus(i),h(τ) is the corresponding day-ahead LMP. The public real-time market information is denoted by xτpubx^pub_τ, which is the realized system conditions of net bus demand xτsys=d~n,τx^sys_τ= d_n,τ. A generator does not observe the private parameters or contemporaneous bids of other generators. The market is therefore partially observable from the perspective of each participant. Action. At each real-time interval, generator i submits a stepwise supply bid i,τ π_i,τ, obtained from a single markup αi,τ _i,τ on its truthful offer πi,10≤⋯≤πi,K0π^0_i,1≤·s≤π^0_i,K, in which each segment is priced at the unit’s true marginal cost MCiMC_i, ai,τ=αi,τ∈i=[1,α¯],πi,k,τ=αi,τπi,k0,k=1,…,K,a_i,τ= _i,τ _i=[1, α], _i,k,τ= _i,τ\,π^0_i,k, k=1,…,K, (24) where α¯ α is the largest admissible markup, subject to the admissible bid-price bounds and the monotonicity condition πi,1,τ≤πi,2,τ≤⋯≤πi,K,τ. _i,1,τ≤ _i,2,τ≤·s≤ _i,K,τ. (25) The segment quantities Gi,kG_i,k are fixed, so the strategic action determines the bid prices associated with these quantities. The joint action is aτ=(ai,τ)i∈ℐa_τ=(a_i,τ)_i . Market clearing and reward. Given the current state and joint bids, the market-clearing mechanism zτ=rt(sτ,aτ)z_τ=C^rt(s_τ,a_τ) (26) solves optimization (16) and returns the realized dispatch pi,τp_i,τ, accepted bid quantities gi,k,τg_i,k,τ, and real-time LMPs LMPn,τrtLMP^rt_n,τ. Each generator receives its realized profit as the stage reward, ri,τ r_i,τ =profiti,τ=revenuei,τda+revenuei,τrt−costi,τ =profit_i,τ=revenue^da_i,τ+revenue^rt_i,τ-cost_i,τ =ΔrtLMPbus⁡(i),h⁡(τ)daqi,h⁡(τ)da+ΔrtLMPbus⁡(i),τrt(pi,τ−qi,h⁡(τ)da) = ^rt\,LMP^da_bus(i),h(τ)\,q^da_i,h(τ)+ ^rt\,LMP^rt_bus(i),τ (p_i,τ-q^da_i,h(τ) ) −(ΔrtTCi(pi,τ)+ΔrtNLiui,h⁡(τ)da+Sivi,τda). - (\, ^rt\,TC_i(p_i,τ)+ ^rt\,NL_iu^da_i,h(τ)+S_iv^da_i,τ\, ). (27) where profiti,τprofit_i,τ is defined by the two-settlement rule in equation (18). The reward therefore accounts for the fixed day-ahead position, the real-time settlement of deviations, and the realized generation cost. State transition. The clearing result determines the operating state entering the next real-time interval. In particular, the realized dispatch pi,τp_i,τ becomes the previous-period dispatch in interval τ+1τ+1, so that xi,τ+1=(pi,τ,vi,τ+1da,wi,τ+1da).x_i,τ+1= (p_i,τ,v^da_i,τ+1,w^da_i,τ+1 ). (28) Thus, the dispatch determined at interval τ enters the ramping constraint of interval τ+1τ+1. The corresponding day-ahead commitment, schedule, and price are updated deterministically according to h⁡(τ+1)h(τ+1), while realized net demand evolves according to the exogenous data process. The overall state transition is therefore sτ+1∼P(⋅∣sτ,aτ).s_τ+1 P (· s_τ,a_τ ). (29) Agent objective. Each generator uses a policy σi(ai,τ∣oi,τ) _i(a_i,τ o_i,τ) to select its real-time bid. Over a horizon of H real-time intervals, generator i seeks to maximize its expected discounted profit, Ji(σi,σ−i)=⁡[∑τ=1Hγτ−1ri,τ],J_i( _i, _-i)=E [ _τ=1^Hγ^τ-1r_i,τ ], (30) where σ−i _-i denotes the joint policies of the other generators. Because both dispatch and real-time prices are determined jointly by all submitted bids, the return of each generator depends on both its own bidding policy and those of the other participants. Agents therefore learn their real-time bidding policies through repeated market interactions without observing competitors’ private information or policies. A.3 M3: Ancillary Services Market The ancillary services market procures operating reserve, where generation capacity is held available to respond to system contingencies. Unlike the energy market, it primarily trades availability: a provider is paid for maintaining reserve capacity that can be activated when needed but may never be called. In some market designs, operating reserve is co-optimized with real-time energy, so energy dispatch and reserve procurement are determined jointly for the same resources subject to shared operating constraints. This section presents the ancillary services market clearing model in subsection A.3.1, the pricing and settlement rules in subsection A.3.2, and the corresponding POSG in subsection A.3.3. A.3.1 Market model The market contains Nprod=2N^prod=2 reserve products distinguished by their response times: a fast product with θ1=10mins _1=10~mins and a slow product with θ2=30mins _2=30~mins. Before bids are submitted, the system operator publishes the reserve requirement for each product, which can be modeled as a fraction of forecast system demand, dj,τres=βj∑ndn,τforecast,d^res_j,τ= _j\, _nd^forecast_n,τ, (31) where dn,τforecastd^forecast_n,τ is the published demand forecast at bus n and βj _j is the requirement fraction for reserve product j. For each real-time interval, generator i submits the energy offer described in section A.2.1, together with a reserve offer price πi,j,τres≥0π^res_i,j,τ≥ 0 for each product j, expressed in $/MW-h. The energy and reserve markets are cleared jointly as min⁡∑i,kg,r,s,sres⁡πi,k,τgi,k,τ+∑i,jπi,j,τresri,j,τ+VOLL∑nsn,τ+VOLR∑jsj,τres, _\g,\,r,\,s,\,s^res\\; _i,k _i,k,τg_i,k,τ+ _i,jπ^res_i,j,τr_i,j,τ+VOLL _ns_n,τ+VOLR _js^res_j,τ, (32a) subject to pi,τ=p¯iui,h⁡(τ)da+∑kgi,k,τ, p_i,τ= p_i\,u^da_i,h(τ)+ _kg_i,k,τ, (32b) 0≤gi,k,τ≤Gi,kui,h⁡(τ)da, 0≤ g_i,k,τ≤ G_i,k\,u^da_i,h(τ), (32c) Pn,τ=∑i:bus⁡(i)=npi,τ+sn,τ−d~n,τ, P_n,τ= _i:bus(i)=np_i,τ+s_n,τ- d_n,τ, (32d) ∑nPn,τ=0(:λτ), _nP_n,τ=0 (: _τ), (32e) −F≤PTDF⋅Pτ≤F(:μτ+,μτ−), -F · P_τ≤ F (: _τ^+,\, _τ^-), (32f) 0≤sn,τ≤max(d~n,τ,0)(:ρn,τ), 0≤ s_n,τ≤ ( d_n,τ,0) (: _n,τ), (32g) −RidnΔrt−p¯iwi,τda≤pi,τ−pi,τ−1≤RiupΔrt+p¯ivi,τda, -R_i^dn ^rt- p_iw^da_i,τ≤ p_i,τ-p_i,τ-1≤ R_i^up ^rt+ p_iv^da_i,τ, (32h) 0≤ri,j,τ≤Riupθj, 0≤ r_i,j,τ≤ R_i^up _j, (32i) pi,τ+∑jri,j,τ≤p¯iui,h⁡(τ)da(:νi,τcap), p_i,τ+ _jr_i,j,τ≤ p_i\,u^da_i,h(τ) (:ν^cap_i,τ), (32j) dj,τres−∑iri,j,τ−sj,τres≤0(:λj,τres), d^res_j,τ- _ir_i,j,τ-s^res_j,τ≤ 0 (:λ^res_j,τ), (32k) 0≤sj,τres≤dj,τres. 0≤ s^res_j,τ≤ d^res_j,τ. (32l) The objective in equation (32a) minimizes the joint as-bid cost of energy dispatch and reserve procurement for each real-time interval Δrt ^rt, together with penalties for unserved energy and unmet reserve requirements. The first term represents the accepted energy offers, and the second term represents payments associated with reserve capacity held available for contingency response. The penalties VOLLVOLL and VOLRVOLR are assigned to load shedding and reserve shortage, respectively. We set VOLR<VOLLVOLR<VOLL so that, when available capacity is insufficient to satisfy both requirements, the model permits a reserve shortage before shedding firm load. Constraints (32b)–(32h) describe the energy dispatch and follow the real-time balancing model in section A.2.1. Constraint (32b) defines the output of generator i as its committed minimum output plus the accepted quantities from its energy-offer segments. Constraint (32c) bounds each accepted segment by its available quantity. Constraints (32d) and (32e) define nodal injections and enforce system-wide supply–demand balance, respectively. Constraint (32f) keeps transmission flows within the line ratings, and (32g) allows emergency load shedding when available generation is insufficient. Constraint (32h) limits the change in generator output between consecutive real-time intervals. Constraints (32i)–(32l) introduce the reserve products. Constraint (32i) limits the reserve award ri,j,τr_i,j,τ to the amount of additional output that generator i can deliver within the required response time θj _j. Therefore, a product with a longer response time allows a generator to provide more reserve capacity for a given ramp rate. Constraint (32j) couples energy and reserve procurement. The scheduled energy output and the total reserve capacity awarded to a generator must jointly remain below its committed capacity. Consequently, capacity used to produce energy cannot simultaneously be sold as reserve, creating an opportunity-cost trade-off between the two products. The dual variable νi,τcapν^cap_i,τ represents the marginal value of additional available capacity of generator i. Finally, constraint (32k) requires the total procured capacity for reserve product j to satisfy the published requirement dj,τresd^res_j,τ. The slack variable sj,τress^res_j,τ represents any unmet reserve requirement and is bounded by constraint (32l). The corresponding dual variable λj,τresλ^res_j,τ gives the marginal clearing price of reserve product j. A.3.2 Pricing and settlement Energy prices follow the same LMP formulation as in equation (17), evaluated using the dual variables of the joint energy–reserve clearing problem (32). The reserve price of product j is determined by the dual variable λj,τresλ^res_j,τ associated with the reserve requirement constraint (32k). It represents the marginal cost of increasing the system-wide requirement for reserve product j by one additional unit of capacity. A key feature of co-optimization is that the opportunity cost of reserving generation capacity is determined within the clearing problem. When the response-time constraint is non-binding, the marginal cost of providing reserve consists of two components: the submitted reserve offer and the opportunity cost of using generation capacity for reserve rather than energy. The latter is represented by the shadow price νi,τcapν^cap_i,τ of the shared capacity constraint. Hence, for a marginal reserve provider, λj,τres=πi,j,τres+νi,τcap.λ^res_j,τ=π^res_i,j,τ+ν^cap_i,τ. (33) If the same generator is also marginal in the energy market and no additional operating constraint is binding, the capacity shadow value corresponds to the energy margin, νi,τcap=LMPbus⁡(i),τrt−πi,k,τ,ν^cap_i,τ=LMP^rt_bus(i),τ- _i,k,τ, (34) and therefore λj,τres=πi,j,τres+(LMPbus⁡(i),τrt−πi,k,τ).λ^res_j,τ=π^res_i,j,τ+ (LMP^rt_bus(i),τ- _i,k,τ ). (35) Thus, the reserve price accounts not only for the submitted reserve offer but also for the value of energy production that the marginal provider gives up when capacity is allocated to reserve. Energy is settled according to the two-settlement rule in equation (18). Reserve capacity is settled separately at the clearing price of each product. For generator i, revenuei,τres=Δrt∑jλj,τresri,j,τ,revenue^res_i,τ= ^rt _jλ^res_j,τ\ r_i,j,τ, (36) and its total profit for the interval is profiti,τ=revenuei,τda+revenuei,τrt+revenuei,τres−costi,τ.profit_i,τ=revenue^da_i,τ+revenue^rt_i,τ+revenue^res_i,τ-cost_i,τ. (37) where revenuei,τda ^da_i,τ =ΔrtLMPbus⁡(i),h⁡(τ)daqi,h⁡(τ)da, = ^rt\,LMP^da_bus(i),h(τ)\,q^da_i,h(τ), (38) revenuei,τrt ^rt_i,τ =ΔrtLMPbus⁡(i),τrt(pi,τ−qi,h⁡(τ)da), = ^rt\,LMP^rt_bus(i),τ (p_i,τ-q^da_i,h(τ) ), (39) costi,τ _i,τ =ΔrtTCi(pi,τ)+ΔrtNLiui,h⁡(τ)da+Sivi,τda. = ^rt\,TC_i(p_i,τ)+ ^rt\,NL_iu^da_i,h(τ)+S_iv^da_i,τ. (40) Reserve availability itself does not consume fuel, so no additional generation cost is assigned to a reserve award unless the reserve is activated. Every cleared MW of the same reserve product is paid the same product price, so a provider whose reserve offer is below the clearing price earns an inframarginal margin. Remark on energy–reserve co-optimization. Constraint (32j) couples energy dispatch and reserve procurement because both compete for the same generation capacity. Increasing the reserve award can reduce the capacity available for energy production, while increasing energy output reduces the headroom available for reserve. The joint clearing therefore allows an agent’s energy and reserve offers to interact through this shared capacity constraint. The ancillary services market is implemented as a separate benchmark task that extends the real-time balancing market with reserve products. The day-ahead commitment, schedule, and prices enter as fixed inputs, while real-time energy and reserve are cleared jointly. Setting dj,τres=0d^res_j,τ=0 for all reserve products removes the reserve layer and recovers the real-time balancing model of section A.2. A.3.3 Partially Observable Stochastic Game The ancillary services market is formulated as a general-sum POSG, as=⟨ℐ,,ii∈ℐ,ii∈ℐ,P,rii∈ℐ,γ⟩,G^as= \ I,\,S,\,\O_i\_i ,\A_i\_i ,\,P,\,\r_i\_i ,\,γ\ , (41) where ℐI denotes the set of strategic generators. One episodic market step corresponds to one 30-minute real-time interval τ, during which energy and reserve are cleared jointly. State. The state at interval τ is sτ=(xτsys,xτda,xτres,xi,τi∈ℐ)∈,s_τ= (\,x^sys_τ,\,x^da_τ,\,x^res_τ,\,\x_i,τ\_i \, ) , (42) where xτsysx^sys_τ contains the realized system conditions of demand d~n,τ d_n,τ used in the joint energy–reserve clearing. The day-ahead component is xτda=ui,h⁡(τ)da,qi,h⁡(τ)da,LMPn,h⁡(τ)dai,n,x^da_τ= \\,u^da_i,h(τ),\,q^da_i,h(τ),\,LMP^da_n,h(τ)\, \_i,n, (43) which contains the fixed commitment, binding generation schedules, and nodal prices from the corresponding day-ahead period. The reserve component is xτres=(d1,τres,…,dNprod,τres),x^res_τ= (\,d^res_1,τ,…,d^res_N^prod,τ\, ), (44) where dj,τresd^res_j,τ is the system-wide requirement for reserve product j published before bidding. The operating state of generator i is xi,τ=(pi,τ−1,vi,τda,wi,τda),x_i,τ= (\,p_i,τ-1,\,v^da_i,τ,\,w^da_i,τ\, ), (45) where pi,τ−1p_i,τ-1 is the realized dispatch from the previous real-time interval, and vi,τdav^da_i,τ and wi,τdaw^da_i,τ are the start-up and shut-down indicators implied by the mapped day-ahead commitment. Network parameters, reserve-product response times, generator operating limits, and cost parameters are fixed environment parameters and are omitted from the dynamic state. Observation. Each generator observes its own operating information together with the public energy and reserve market information available before bidding. Its observation at interval τ is oi,τ=(pi,τ−1,ui,h⁡(τ)da,qi,h⁡(τ)da,LMPbus⁡(i),h⁡(τ)da,xτsys,xτres)=Ωi(sτ)∈i.o_i,τ= (\,p_i,τ-1,\,u^da_i,h(τ),\,q^da_i,h(τ),\,LMP^da_bus(i),h(τ),\,x^sys_τ,\,x^res_τ\, )= _i(s_τ) _i. (46) Thus, the generator observes the realized net-demand conditions and the published reserve requirements in addition to its own day-ahead position and previous dispatch. It does not observe competitors’ private cost information or their contemporaneous energy and reserve offers. The market is therefore partially observable from the perspective of each participant. Action. At each interval, generator i jointly submits an energy offer and one reserve offer price for each reserve product. Its action is ai,τ=(i,τen,i,τres)∈i,a_i,τ= (\, π^en_i,τ,\, π^res_i,τ ) _i, (47) where i,τen=(πi,1,τ,…,πi,K,τ),πi,k,τ=αi,τπi,k0, π^en_i,τ= ( _i,1,τ,…, _i,K,τ ), _i,k,τ= _i,τ\,π^0_i,k, (48) is the stepwise energy-offer price vector, obtained from a single markup αi,τ∈[1,α¯] _i,τ∈[1, α] on the truthful offer πi,10≤⋯≤πi,K0π^0_i,1≤·s≤π^0_i,K as in the real-time balancing market, and i,τres=(πi,1,τres,…,πi,Nprod,τres) π^res_i,τ= (π^res_i,1,τ,…,π^res_i,N^prod,τ ) (49) contains the reserve offer prices. The energy offer satisfies πi,1,τ≤πi,2,τ≤⋯≤πi,K,τ, _i,1,τ≤ _i,2,τ≤·s≤ _i,K,τ, (50) while 0≤πi,j,τres≤VOLR0≤π^res_i,j,τ for each reserve product j. The energy segment quantities Gi,kG_i,k are fixed, and no reserve quantity is submitted by the agent; the reserve awards are determined by the co-optimization. The joint action is aτ=(ai,τ)i∈ℐa_τ=(a_i,τ)_i . Market clearing and reward. Given the current state and joint energy–reserve offers, the market-clearing mechanism zτ=as(sτ,aτ)z_τ=C^as(s_τ,a_τ) (51) solves the co-optimization problem (32). The clearing outcome includes the energy dispatch pi,τp_i,τ, reserve awards ri,j,τr_i,j,τ, real-time LMPs LMPn,τrtLMP^rt_n,τ, and reserve prices λj,τresλ^res_j,τ. Each generator receives its total realized profit as the stage reward, ri,τ r_i,τ =profiti,τ=revenuei,τda+revenuei,τrt+revenuei,τres−costi,τ =profit_i,τ=revenue^da_i,τ+revenue^rt_i,τ+revenue^res_i,τ-cost_i,τ =ΔrtLMPbus⁡(i),h⁡(τ)daqi,h⁡(τ)da+ΔrtLMPbus⁡(i),τrt(pi,τ−qi,h⁡(τ)da) = ^rt\,LMP^da_bus(i),h(τ)\,q^da_i,h(τ)+ ^rt\,LMP^rt_bus(i),τ (p_i,τ-q^da_i,h(τ) ) +Δrt∑jλj,τresri,j,τ−(ΔrtTCi(pi,τ)+ΔrtNLiui,h⁡(τ)da+Sivi,τda). + ^rt _jλ^res_j,τ\ r_i,j,τ- (\, ^rt\,TC_i(p_i,τ)+ ^rt\,NL_iu^da_i,h(τ)+S_iv^da_i,τ\, ). (52) as defined in section A.3.2. The reward therefore captures both energy-market profit and reserve-capacity revenue. State transition. The clearing outcome determines the operating state entering the next real-time interval. In particular, xi,τ+1=(pi,τ,vi,τ+1da,wi,τ+1da),x_i,τ+1= (\,p_i,τ,v^da_i,τ+1,w^da_i,τ+1\, ), (53) so the realized dispatch pi,τp_i,τ enters the ramping constraint of the next interval. The corresponding day-ahead commitment, schedule, and price are updated deterministically according to h⁡(τ+1)h(τ+1), while the realized net demand and reserve requirements evolve according to their exogenous data processes. The overall transition is therefore sτ+1∼P(⋅∣sτ,aτ).s_τ+1 P\! (· s_τ,a_τ ). (54) Agent objective. Each generator uses a policy σi(ai,τ∣oi,τ) _i(a_i,τ o_i,τ) to jointly determine its energy and reserve offers. Over a horizon of H real-time intervals, generator i maximizes its expected discounted profit, Ji(σi,σ−i)=⁡[∑τ=1Hγτ−1ri,τ],J_i( _i, _-i)=E [ _τ=1^Hγ^τ-1r_i,τ ], (55) where σ−i _-i denotes the joint policies of the other generators. Because energy dispatch, reserve awards, and both energy and reserve prices are jointly determined by all submitted offers, the return of each generator depends on both its own bidding policy and those of other participants. Agents therefore learn how to allocate their available capacity between energy and reserve through repeated market interactions without observing competitors’ private information or policies. A.4 M4: Peer-to-Peer Local Energy Market Rooftop photovoltaic generation and behind-the-meter storage have enabled many consumers to become prosumers that can both consume and supply electricity. When surplus energy is exported directly to the grid, it is typically settled at an export price such as feed-in-tariff, which is below the retail tariff paid for grid consumption. A peer-to-peer (P2P) local energy market allows neighboring participants to trade their surplus and deficit locally at prices between these two outside options, creating the possibility of mutual benefits for buyers and sellers. Unlike the day-ahead and real-time markets in sections A.1 and A.2, which operate at the transmission level, the P2P market considered here operates at the end-use level and facilitates local energy exchange among prosumers. The P2P market uses 15-minute trading intervals. Before each auction, the net energy position of every participant is determined from its local generation, consumption, and storage operation. Hence, a participant knows the quantity it has available to buy or sell before submitting its bid, and its strategic decisions are its battery power and the submitted price. A local market operator collects the bids and offers, clears the auction, and settles the resulting transactions. The market does not issue physical dispatch instructions: the participants’ net quantities are fixed before clearing, so the auction determines which quantities are matched and at what price rather than how the underlying devices are operated. In our model, the P2P local energy market is implemented as a periodic double auction among a fixed population of NagentN^agent prosumers, each equipped with rooftop photovoltaic generation, an inelastic local load, and a battery. In each trading interval, a participant appears on one side of the market according to its net energy position: a surplus is offered for sale, whereas a deficit is submitted as demand. The local market operator collects one price–quantity pair from each participant and constructs aggregate supply and demand curves. The market clears at their intersection, with all matched buyers and sellers settling at a common market clearing price. Any unmatched deficit is supplied by the upstream grid at the retail tariff, while any unmatched surplus is exported to the grid at the export price. This section presents the market clearing model in subsection A.4.1, the pricing and settlement rules in subsection A.4.2, and the corresponding POSG in subsection A.4.3. A.4.1 Market model Let τ index the 15-minute P2P trading intervals, with Δp2p=0.25h ^p2p=0.25~h. Before each auction, the local generation, consumption, and battery operation of participant i determine its net power position, pi,τnet=−pi,τpv+pi,τch+di,τ−pi,τdis,p^net_i,τ=-p^pv_i,τ+p^ch_i,τ+d_i,τ-p^dis_i,τ, (56a) and the corresponding quantities available for P2P trading are qi,τsell=Δp2pmax⁡(−pi,τnet,0),qi,τbuy=Δp2pmax⁡(pi,τnet,0).q^sell_i,τ= ^p2p \! (-p^net_i,τ,0 ), q^buy_i,τ= ^p2p \! (p^net_i,τ,0 ). (56b) Here, pi,τpvp^pv_i,τ and di,τd_i,τ denote photovoltaic generation and local demand, respectively. pi,τdisp^dis_i,τ and pi,τchp^ch_i,τ denote battery discharging and charging power, respectively. Only one of qi,τsellq^sell_i,τ and qi,τbuyq^buy_i,τ can be positive. A participant with surplus energy is therefore a seller, while one with an energy deficit is a buyer. The battery is the only local device that carries a physical state between trading intervals. Its state of charge evolves according to soci,τ+1=soci,τ+Δp2pEi(ηichpi,τch−pi,τdisηidis),soc_i,τ+1=soc_i,τ+ ^p2pE_i ( _i^chp^ch_i,τ- p^dis_i,τ _i^dis ), (57) where EiE_i is the battery energy capacity, and ηich _i^ch and ηidis _i^dis are the charging and discharging loss efficiencies. The feasible charging and discharging powers are bounded by 0≤pi,τdis≤min⁡(p¯ibat,(soci,τ−soc¯i)EiηidisΔp2p),0≤ p^dis_i,τ≤ \! ( p_i^bat, (soc_i,τ- soc_i )E_i _i^dis ^p2p ), (58) and 0≤pi,τch≤min⁡(p¯ibat,(soc¯i−soci,τ)EiηichΔp2p),0≤ p^ch_i,τ≤ \! ( p_i^bat, ( soc_i-soc_i,τ )E_i _i^ch ^p2p ), (59) where p¯ibat p_i^bat is the battery power rating and [soc¯i,soc¯i][ soc_i, soc_i] is the admissible state-of-charge range. Charging and discharging are mutually exclusive within each trading interval. Battery operation is chosen by the participant before the P2P auction. Hence, pi,τnetp^net_i,τ and the corresponding trading quantity are fixed when the participant submits its bid, and within the auction only the submitted price is a strategic bidding decision. The seller and buyer sets in interval τ are τ=i:qi,τsell>0,ℬτ=i:qi,τbuy>0.S_τ= \i:q^sell_i,τ>0 \, _τ= \i:q^buy_i,τ>0 \. (60) Each participant submits one price πi,τ _i,τ: an ask if i∈τi _τ and a bid if i∈ℬτi _τ. Submitted prices are restricted by the outside options provided by the upstream grid, πexp≤πi,τ≤πret,π^exp≤ _i,τ≤π^ret, (61) where πretπ^ret is the retail tariff for grid imports and πexpπ^exp is the export price for grid injections, with πexp<πretπ^exp<π^ret. The submitted quantity is fixed by the participant’s net position, so πi,τ _i,τ is the only strategic variable in the auction. The local market operator clears the market through a periodic double auction. Seller asks are sorted in ascending order of price, while buyer bids are sorted in descending order of price, with ties broken by participant index. Let στs(j)σ^s_τ(j) denote the participant occupying position j in the sorted seller order and στb(j)σ^b_τ(j) the participant occupying position j in the sorted buyer order. The associated cumulative quantities are Qj,τsell=∑ℓ=1jqστs(ℓ),τsell,Qj,τbuy=∑ℓ=1jqστb(ℓ),τbuy.Q^sell_j,τ= _ =1^jq^sell_σ^s_τ( ),τ, Q^buy_j,τ= _ =1^jq^buy_σ^b_τ( ),τ. (62) The total quantities available on the two sides are Qτsell=∑i∈τqi,τsell,Qτbuy=∑i∈ℬτqi,τbuy.Q^sell_τ= _i _τq^sell_i,τ, Q^buy_τ= _i _τq^buy_i,τ. (63) The sorted submissions define the aggregate supply curve Sτ(x)S_τ(x) and aggregate demand curve Dτ(x)D_τ(x). The supply curve returns the ask price associated with the x-th unit of cumulative supply, while the demand curve returns the bid price associated with the x-th unit of cumulative demand. Both are step functions whose breakpoints are given by the cumulative quantities in equation (62). Let τX_τ denote the union of the supply and demand breakpoints, together with zero. The cleared P2P trading volume is Qτ⋆=maxx∈τ:x≤min(Qτsell,Qτbuy),Dτ(x)≥Sτ(x).Q _τ= \x _τ:x≤ \! (Q^sell_τ,Q^buy_τ ),\;D_τ(x)≥ S_τ(x) \. (64) Thus, trading continues while the marginal buyer’s bid is no lower than the marginal seller’s ask. If no bid meets an ask, Qτ⋆=0Q _τ=0. Individual awards follow the sorted orders. For the seller occupying position j, qστs(j),τsell⋆=min[max(Qτ⋆−Qj−1,τsell,0),qστs(j),τsell],q^sell _σ^s_τ(j),τ= [ \! (Q _τ-Q^sell_j-1,τ,0 ),q^sell_σ^s_τ(j),τ ], (65) with Q0,τsell=0Q^sell_0,τ=0. Similarly, the award to the buyer occupying position j is qστb(j),τbuy⋆=min[max(Qτ⋆−Qj−1,τbuy,0),qστb(j),τbuy],q^buy _σ^b_τ(j),τ= [ \! (Q _τ-Q^buy_j-1,τ,0 ),q^buy_σ^b_τ(j),τ ], (66) with Q0,τbuy=0Q^buy_0,τ=0. Equations (65) and (66) fill the most competitive submissions first: lower-priced sellers and higher-priced buyers receive priority. Since Qτ⋆Q _τ coincides with a cumulative-quantity breakpoint on at least one side of the market, all accepted participants on that side are either fully cleared or rejected. On the opposite side, only the marginal participant may be partially cleared. A.4.2 Pricing and settlement Let Sτ⋆S _τ and Dτ⋆D _τ denote the marginal accepted ask and bid, respectively, and let Sτ+S^+_τ and Dτ+D^+_τ denote the first rejected ask and bid beyond the cleared volume Qτ⋆Q _τ. A uniform clearing price must be high enough for all accepted sellers and low enough for all accepted buyers, while remaining unattractive to rejected submissions. This defines the feasible price interval λ¯τ=max⁡(Sτ⋆,Dτ+),λ¯τ=min⁡(Dτ⋆,Sτ+),λτloc=12(λ¯τ+λ¯τ). λ_τ= \! (S _τ,D^+_τ ), λ_τ= \! (D _τ,S^+_τ ), λ^loc_τ= 12 ( λ_τ+ λ_τ ). (67) The local market therefore publishes the midpoint of the admissible price interval. When an adjacent accepted or rejected submission does not exist because one side of the market is exhausted, the export price πexpπ^exp and retail tariff πretπ^ret provide the corresponding lower and upper price bounds. Consequently, πexp≤λτloc≤πret.π^exp≤λ^loc_τ≤π^ret. (68) Thus, a matched seller receives no less than the grid export price and a matched buyer pays no more than the retail tariff. If a marginal participant is only partially cleared, the admissible interval collapses to that participant’s submitted price. P2P awards are financially binding, while any unmatched quantity is settled with the upstream grid. Define the residual export and import quantities as qexi,τ=qselli,τ−qsell⋆i,τ,qimi,τ=qbuyi,τ−qbuy⋆i,τ.q^ex_i,τ=q^sell_i,τ-q^sell _i,τ, q^im_i,τ=q^buy_i,τ-q^buy _i,τ. (69) The settlement of participant i is then revenuei,τ _i,τ =λlocτqsell⋆i,τ+πexpqexi,τ, =λ^loc_τq^sell _i,τ+π^expq^ex_i,τ, (70a) costi,τ _i,τ =λτlocqi,τbuy⋆+πretqi,τim+cidegΔp2p(pi,τch+pi,τdis), =λ^loc_τq^buy _i,τ+π^retq^im_i,τ+c^deg_i ^p2p (p^ch_i,τ+p^dis_i,τ ), (70b) profiti,τ _i,τ =revenuei,τ−costi,τ. =revenue_i,τ-cost_i,τ. (70c) Here, cidegc^deg_i is the battery degradation cost in EUR/MWh of throughput. Photovoltaic generation is assumed to have zero marginal cost, while local demand is inelastic. A participant with a net energy deficit may therefore have negative profit, corresponding to a net electricity bill; maximizing profit in this case is equivalent to minimizing that bill. Because all matched buyers and sellers settle at the same local price, P2P payments are budget balanced within the local market, excluding transactions with the upstream grid and any network charges not modeled here. Remark on the P2P local market. The P2P market provides an intermediate trading opportunity between the two grid settlement prices. Sellers can receive more than the export price, while buyers can pay less than the retail tariff when mutually acceptable bids are matched. The market can also increase the share of locally produced renewable energy consumed within the local network. It therefore complements, rather than replaces, upstream electricity markets and grid transactions. A.4.3 Partially Observable Stochastic Game The P2P local energy market is formulated as a general-sum POSG, p2p=⟨ℐ,,ii∈ℐ,ii∈ℐ,P,rii∈ℐ,γ⟩,G^p2p= \ I,\,S,\O_i\_i ,\A_i\_i ,\,P,\,\r_i\_i ,\,γ\ , (71) where ℐI is the fixed set of NagentN^agent prosumers. One game step corresponds to one 15-minute P2P trading interval τ. State. The state at interval τ is sτ=(xi,τi∈ℐ,πexp,πret)∈,s_τ= (\,\x_i,τ\_i ,π^exp,π^ret\, ) , (72) where the local physical state of prosumer i is xi,τ=(soci,τ,pi,τpv,di,τ,pi,τch,pi,τdis).x_i,τ= (\,soc_i,τ,p^pv_i,τ,d_i,τ,p^ch_i,τ,p^dis_i,τ\, ). (73) These variables determine the participant’s net power position pi,τnetp^net_i,τ and therefore its available selling or buying quantity through equation 56b. The export price πexpπ^exp and retail tariff πretπ^ret define the outside settlement options with the upstream grid. Observation. Each participant observes its own physical state and net trading position, together with the public grid settlement prices. Its observation is oi,τ=(soci,τ,pi,τnet,qi,τsell,qi,τbuy,πexp,πret)=Ωi(sτ)∈i.o_i,τ= (soc_i,τ,p^net_i,τ,q^sell_i,τ,q^buy_i,τ,π^exp,π^ret )= _i(s_τ) _i. (74) Thus, each participant knows whether it is a seller or buyer and the quantity it has available to trade before submitting its price. It does not observe the contemporaneous bids, asks, or private local states of other participants. The market is therefore partially observable from the perspective of each individual prosumer. Action. The trading quantity is fixed by the battery power before market clearing, so the strategic action of participant i is its battery power together with its submitted price, ai,τ=(pi,τbat,πi,τ),πi,τ∈[πexp,πret],a_i,τ= (p^bat_i,τ, _i,τ ), _i,τ∈ [π^exp,π^ret ], (75) where pi,τbat=pi,τdis−pi,τchp^bat_i,τ=p^dis_i,τ-p^ch_i,τ is the signed battery power within the feasible range (58)–(59). If qi,τsell>0q^sell_i,τ>0, the submitted price is an ask; if qi,τbuy>0q^buy_i,τ>0, it is a bid. A participant with zero net position does not actively participate in the auction. The joint action is aτ=(ai,τ)i∈ℐa_τ=(a_i,τ)_i . Market clearing and reward. Given the participants’ net positions and joint submitted prices, the local market-clearing mechanism zτ=p2p(sτ,aτ)z_τ=C^p2p(s_τ,a_τ) (76) sorts the asks and bids and applies the double-auction clearing rules in Section A.4.1. The clearing outcome contains the total traded volume Qτ⋆Q _τ, the uniform local clearing price λτlocλ^loc_τ, and the individual awards qsell⋆i,τq^sell _i,τ and qbuy⋆i,τq^buy _i,τ. Any unmatched quantity is settled with the upstream grid. The stage reward of participant i is its realized profit, ri,τ r_i,τ =profiti,τ=revenuei,τ−costi,τ =profit_i,τ=revenue_i,τ-cost_i,τ =(λτlocqi,τsell⋆+πexpqi,τex)−(λτlocqi,τbuy⋆+πretqi,τim+cidegΔp2p(pi,τch+pi,τdis)) = (λ^loc_τq^sell _i,τ+π^expq^ex_i,τ )- (λ^loc_τq^buy _i,τ+π^retq^im_i,τ+c^deg_i ^p2p(p^ch_i,τ+p^dis_i,τ) ) (77) where profiti,τprofit_i,τ is defined by equation 70. For a seller, the reward reflects revenue from P2P sales and any residual export to the grid. For a buyer, the reward reflects the cost of P2P purchases and any residual import from the grid, together with battery degradation cost. A buyer may therefore receive a negative reward corresponding to its electricity bill, so maximizing reward is equivalent to minimizing that bill. State transition. The battery state of charge provides the physical coupling between consecutive trading intervals. Following equation 57, soci,τ+1=soci,τ+Δp2pEi(ηichpi,τch−pi,τdisηidis).soc_i,τ+1=soc_i,τ+ ^p2pE_i ( _i^chp^ch_i,τ- p^dis_i,τ _i^dis ). (78) Photovoltaic generation and local demand evolve according to their exogenous data processes, while battery operation for the next period is determined before the next P2P auction. The next net position and trading quantity are then obtained from equation 56b. The overall state transition is sτ+1∼P(⋅∣sτ,aτ).s_τ+1 P\! (· s_τ,a_τ ). (79) Under the market design considered here, the submitted P2P price affects market matching and settlement but does not directly alter the participant’s physical net position or battery dynamics. Agent objective. Each participant uses a policy σi(ai,τ∣oi,τ) _i(a_i,τ o_i,τ) to determine its submitted price. Over a horizon of H trading intervals, participant i seeks to maximize its expected discounted profit, Ji(σi,σ−i)=⁡[∑τ=1Hγτ−1ri,τ],J_i( _i, _-i)=E [ _τ=1^Hγ^τ-1r_i,τ ], (80) where σ−i _-i denotes the joint policies of the other participants. Because the cleared volume, matching outcome, and local clearing price depend on all submitted bids and asks, the return of each participant depends on both its own pricing policy and those of the other prosumers. Participants therefore learn bidding strategies through repeated local-market interactions without observing competitors’ contemporaneous submissions or private states. A.5 M5: Local Flexibility Market Growing electrification and distributed generation can place distribution networks under increasing operational stress, leading to feeder congestion and voltage violations. Reinforcing distribution infrastructure can be costly and slow, particularly when the underlying constraint occurs only during a limited number of periods. A local flexibility market provides an alternative means of managing these constraints by allowing the distribution system operator (DSO) to procure temporary changes in power injection or consumption from resources already connected to the affected network. The market operates at the distribution level and clears once per delivery period. It addresses local network constraints that are not represented in the transmission-level wholesale markets, while the upstream energy price faced by each aggregator is treated as exogenous. The DSO is the sole buyer, making the market a one-sided procurement auction. It specifies the locations and periods where flexibility is required, and accepts offers that restore a feasible distribution-network operating point at minimum procurement cost. The sellers are aggregators, with one aggregator at each bus, each representing a portfolio of distributed energy resources. Two features distinguish this market from the preceding settings. First, the traded product is a deviation from a predefined baseline rather than energy itself. Second, settlement follows a pay-as-bid rule: each accepted aggregator is paid its submitted offer price for the flexibility it provides, rather than a common market-clearing price. The settlement therefore does not rely on dual-based marginal prices. This section presents the market-clearing model in subsection A.5.1, the pricing and settlement rule in subsection A.5.2, and the corresponding POSG in subsection A.5.3. A.5.1 Market model We consider a radial distribution network with NbusN^bus buses supplied by a single substation, indexed by 00, whose voltage is fixed at v0=1v_0=1 p.u. Let τ index the local flexibility trading intervals, with duration Δflex=1hour ^flex=1~hour. Before the flexibility market clears, each aggregator has a baseline operating position determined by local photovoltaic generation, demand, and planned battery charging. With no flexibility activated, the baseline active-power injection at bus n is Pn,τbase,inj=∑i:bus⁡(i)=n(pi,τpv−di,τ−pi,τplan)−dn,τbg,P^base,inj_n,τ= _i:bus(i)=n (p^pv_i,τ-d_i,τ-p^plan_i,τ )-d^bg_n,τ, (81) where pi,τpvp^pv_i,τ is the photovoltaic output, di,τd_i,τ is the local demand of aggregator i, pi,τplanp^plan_i,τ is its planned battery charging power, and dn,τbgd^bg_n,τ denotes background demand not controlled by any aggregator. Under the radial-network approximation, the corresponding baseline line flows and squared voltages are obtained from Pbasel,τ=−∑m∈⁡(l)Pbase,injm,τ,P^base_l,τ=- _m (l)P^base,inj_m,τ, (82) and (vn,τbase)2=v02+2∑m(Rn,mPm,τbase,inj+Xn,mQm,τbase,inj), (v^base_n,τ )^2=v_0^2+2 _m (R_n,mP^base,inj_m,τ+X_n,mQ^base,inj_m,τ ), (83) where ⁡(l)D(l) denotes the set of buses downstream of line l. The quantities Rn,mR_n,m and Xn,mX_n,m are the sums of resistance and reactance over the network path shared by buses n and m, respectively. Reactive injections are determined from the active loads using a fixed power-factor assumption. Before offers are submitted, the DSO publishes indicators of the local flexibility requirement. For an undervoltage at bus n, the indicator is reqn,τv=max⁡[(v¯n+γv)2−(vn,τbase)2,0]2Rn,n,req^v_n,τ= \! [ ( v_n+γ^v )^2- (v^base_n,τ )^2,0 ]2R_n,n, (84) while the requirement associated with forward-flow congestion on line l is reql,τth=max⁡[Pl,τbase−(1−γth)P¯l,0].req^th_l,τ= \! [P^base_l,τ- (1-γ^th ) P_l,0 ]. (85) These quantities are published as market information for the aggregators; the market clearing itself enforces the full network constraints below. The traded product is upward flexibility, defined as an increase in net active power injection relative to the baseline. Aggregator i submits one price–quantity offer (πi,τ,q¯i,τ)( _i,τ, q_i,τ), where πi,τ _i,τ is its offer price and q¯i,τ q_i,τ is the maximum flexibility quantity offered. Any quantity qi,τ∈[0,q¯i,τ]q_i,τ∈[0, q_i,τ] may be accepted by the DSO. Unlike the stepwise offers used in the wholesale markets, each aggregator submits a single price–quantity pair in each flexibility trading interval. Upward flexibility can be delivered by reducing planned battery charging or, once planned charging has been fully removed, by discharging the battery. The offered quantity must therefore satisfy 0≤q¯i,τ≤qi,τphys,0≤ q_i,τ≤ q^phys_i,τ, (86) where qi,τphys=pi,τplan+min⁡(p¯idis,ηidisEi(soci,τ−soc¯i)Δflex).q^phys_i,τ=p^plan_i,τ+ \! ( p^dis_i,\, η^dis_iE_i (soc_i,τ- soc_i ) ^flex ). (87) Here, p¯idis p^dis_i is the battery discharge-power limit, EiE_i is its energy capacity, ηidisη^dis_i is its discharge efficiency, and soc¯i soc_i is the minimum admissible state of charge. Because qi,τphysq^phys_i,τ depends on the current state of charge, the amount of flexibility available from an aggregator varies across trading intervals. Given the submitted offers, the DSO selects flexibility awards by solving minq,s,Pinj,P,v⁡Δflex[∑iπi,τqi,τ+VOLL∑nsn,τ], _\q,s,P^inj,P,v\\; ^flex [ _i _i,τq_i,τ+VOLL _ns_n,τ ], (88a) subject to Pn,τinj=∑i:bus⁡(i)=n(pi,τpv−di,τ+qi,τ−pi,τplan)−dn,τbg+sn,τ, P^inj_n,τ= _i:bus(i)=n (p^pv_i,τ-d_i,τ+q_i,τ-p^plan_i,τ )-d^bg_n,τ+s_n,τ, (88b) Pl,τ=−∑m∈⁡(l)Pinjm,τ, P_l,τ=- _m (l)P^inj_m,τ, (88c) vn,τ2=v02+2∑m(Rn,mPm,τinj+Xn,mQm,τinj), v_n,τ^2=v_0^2+2 _m (R_n,mP^inj_m,τ+X_n,mQ^inj_m,τ ), (88d) (v¯n+γv)2≤vn,τ2≤(v¯n−γv)2,n≠0, ( v_n+γ^v )^2≤ v_n,τ^2≤ ( v_n-γ^v )^2, n≠ 0, (88e) −(1−γth)P¯l≤Pl,τ≤(1−γth)P¯l, - (1-γ^th ) P_l≤ P_l,τ≤ (1-γ^th ) P_l, (88f) 0≤qi,τ≤q¯i,τ, 0≤ q_i,τ≤ q_i,τ, (88g) 0≤sn,τ≤dbgn,τ+∑i:bus⁡(i)=ndi,τ. 0≤ s_n,τ≤ d^bg_n,τ+ _i:bus(i)=nd_i,τ. (88h) The objective (88a) minimizes the total procurement cost of local flexibility in each trading interval. The first term represents the pay-as-bid cost of accepted flexibility offers, while the second penalizes emergency load curtailment. The value of lost load VOLLVOLL is set above the admissible flexibility offer prices so that curtailment is used only when the available flexibility cannot restore a feasible network operating point. Constraint (88b) defines the active-power injection at each distribution bus. Without flexibility, the baseline injection consists of photovoltaic generation minus local demand and planned battery charging. An accepted flexibility quantity qi,τq_i,τ increases the bus injection relative to this baseline, either by reducing charging or by discharging the battery. Background demand dn,τbgd^bg_n,τ represents demand not controlled by the aggregators, while sn,τs_n,τ denotes emergency load curtailment. Constraint (88c) computes the active-power flow on each line as the total downstream withdrawal under the radial-network structure. Here, ⁡(l)D(l) denotes the set of buses downstream of line l. Constraint (88d) uses a linearized radial distribution power-flow model. The quantities Rn,mR_n,m and Xn,mX_n,m represent the resistance and reactance shared by the paths from the substation to buses n and m, respectively. The reactive injections Qm,τinjQ^inj_m,τ are determined from the fixed power-factor assumptions of the underlying loads and are not strategic decision variables. Constraints (88e) and (88f) enforce the distribution-network operating limits. The parameters γvγ^v and γthγ^th introduce safety margins by tightening the admissible voltage range and thermal line ratings used during clearing. Constraint (88g) allows the DSO to accept any quantity between zero and the quantity offered by aggregator i. The submitted quantity q¯i,τ q_i,τ must itself satisfy the physical deliverability condition (87). Consequently, the clearing may partially accept an offer when the full quantity is not required to restore network feasibility. Finally, constraint (88h) bounds emergency curtailment by the total active demand at each bus. Curtailment acts as a feasibility backstop when the submitted flexibility offers are insufficient to satisfy the tightened network constraints. Since the network equations (88b)–(88d) are affine in the clearing variables under the linearized radial power-flow model, the resulting flexibility-market clearing problem is a linear program. A.5.2 Payment and settlement Settlement follows a pay-as-bid rule. Each accepted aggregator is paid its submitted offer price for the quantity cleared by the DSO, revenuei,τ=Δflexπi,τqi,τ,profiti,τ=revenuei,τ−costi,τ,revenue_i,τ= ^flex _i,τq_i,τ, _i,τ=revenue_i,τ-cost_i,τ, (89) where costi,τcost_i,τ is the true delivery cost defined in Section A.5.3. Because settlement is pay-as-bid, two aggregators providing flexibility at the same bus and in the same interval may receive different payments if they submit different offer prices. Although the market imposes no uniform clearing price, the curtailment backstop creates an implicit upper value for flexibility. Emergency curtailment can relieve the same network constraints at a marginal cost of VOLLVOLL, so a flexibility offer is selected only when its network benefit justifies its submitted cost relative to this alternative. This value is inherently locational because the effect of an injection depends on where it enters the network. For a binding forward-flow constraint, an additional MW of injection downstream of the constrained line reduces the line flow by one MW under equation 88c, whereas an injection outside that downstream region does not relieve that constraint. For a binding voltage constraint, the value of flexibility depends on the voltage sensitivity of the participant’s bus, so injections at electrically more effective locations can support higher accepted offer prices than injections with weaker voltage impact. The flexibility product is defined as a deviation from the pre-market baseline. For aggregator i, this baseline is determined from its photovoltaic generation, local demand, and planned battery charging, pi,τbase=pi,τpv−di,τ−pi,τplan.p^base_i,τ=p^pv_i,τ-d_i,τ-p^plan_i,τ. (90) An accepted flexibility award qi,τq_i,τ therefore requires the aggregator to increase its net injection by exactly that amount relative to the baseline. In the benchmark, delivery is assumed to equal the cleared award, so no payment is made for flexibility that is not delivered. A.5.3 Partially Observable Stochastic Game The local flexibility market is formulated as a general-sum partially observable stochastic game (POSG), flex=⟨ℐ,,ii∈ℐ,ii∈ℐ,P,rii∈ℐ,γ⟩,G^flex= ,S,\O_i\_i ,\A_i\_i ,P,\r_i\_i ,γ , (91) where ℐI denotes the set of strategic aggregators. One game step corresponds to one local flexibility trading interval τ. State. The state at interval τ is sτ=(xτsys,xi,τi∈ℐ)∈,s_τ= (x^sys_τ,\x_i,τ\_i ) , (92) where xτsysx^sys_τ describes the distribution-network operating condition before flexibility procurement. It contains the baseline bus injections, line flows, voltages, and the flexibility-need indicators published by the DSO, xτsys=(τbase,inj,τbase,τbase,τv,τth).x^sys_τ= (P^base,inj_τ,P^base_τ,v^base_τ,req^v_τ,req^th_τ ). (93) The local physical state of aggregator i is xi,τ=(soci,τ,pi,τpv,di,τ,pi,τplan),x_i,τ= (soc_i,τ,p^pv_i,τ,d_i,τ,p^plan_i,τ ), (94) which, together with the planned charging in its action, determines its baseline injection and the maximum physically deliverable flexibility qi,τphysq^phys_i,τ through equation 87. Network parameters, battery ratings, and other fixed technical parameters are omitted from the dynamic state. Observation. Each aggregator observes its own operating state together with the public network information released by the DSO. Its observation is oi,τ=(soci,τ,pi,τpv,di,τ,τv,τth)=Ωi(sτ)∈i.o_i,τ= (soc_i,τ,p^pv_i,τ,d_i,τ,req^v_τ,req^th_τ )= _i(s_τ) _i. (95) Thus, each aggregator knows its own available flexibility and the locations and magnitudes of the network needs published before bidding. It does not observe the private operating states, delivery costs, or contemporaneous offers of other aggregators. The market is therefore partially observable from the perspective of each participant. Action. At each trading interval, aggregator i submits one price–quantity offer together with its planned battery charging, ai,τ=(πi,τ,q¯i,τ,pi,τplan)∈i,a_i,τ= ( _i,τ, q_i,τ,p^plan_i,τ ) _i, (96) subject to crep≤πi,τ≤π¯,0≤q¯i,τ≤qi,τphys,0≤pi,τplan≤p¯i,τch.c^rep≤ _i,τ≤ π, 0≤ q_i,τ≤ q^phys_i,τ, 0≤ p^plan_i,τ≤ p^ch_i,τ. (97) The offer price πi,τ _i,τ determines the payment requested per unit of flexibility, q¯i,τ q_i,τ specifies the maximum quantity the aggregator is willing to provide, and pi,τplanp^plan_i,τ enters the baseline injection (81). Here, crepc^rep is the replacement cost of one delivered MWh, i.e., the energy price and the degradation cost corrected for the round-trip efficiency, π¯<VOLL π<VOLL is the market price cap, and p¯i,τch p^ch_i,τ is the charging power the battery can absorb in the interval given its state of charge. The DSO may accept any quantity qi,τ∈[0,q¯i,τ]q_i,τ∈[0, q_i,τ]. The joint action is aτ=(ai,τ)i∈ℐa_τ=(a_i,τ)_i . Market clearing and reward. Given the current state and joint offers, the DSO applies the network-constrained clearing mechanism zτ=flex(sτ,aτ),z_τ=C^flex(s_τ,a_τ), (98) which solves equation 88. The clearing outcome contains the accepted flexibility quantities qi,τq_i,τ, resulting bus injections, line flows and voltages, together with any emergency load curtailment. Under the pay-as-bid settlement rule, the revenue of aggregator i is revenuei,τ=Δflexπi,τqi,τ.revenue_i,τ= ^flex _i,τq_i,τ. (99) Its stage reward is the realized profit, ri,τ=profiti,τ=revenuei,τ−Ciflex(qi,τ,xi,τ),r_i,τ=profit_i,τ=revenue_i,τ-C^flex_i (q_i,τ;x_i,τ ), (100) where Ciflex=Δflex[λτpi,τch+cicyc(pi,τch+pi,τdis)]C^flex_i= ^flex [ _τ\,p^ch_i,τ+c^cyc_i (p^ch_i,τ+p^dis_i,τ ) ] is the true cost of the interval, the energy bought for charging at the exogenous price λτ _τ plus the degradation cost cicycc^cyc_i per MWh of battery throughput, with pchp^ch and pdisp^dis given by the state transition below. This cost depends on the aggregator’s physical operating state and the quantity actually delivered rather than on its submitted offer price. State transition. An accepted flexibility award changes the battery operating trajectory and therefore affects the flexibility available in subsequent intervals. Let qi,τred=min⁡(qi,τ,pi,τplan)q^red_i,τ= \! (q_i,τ,p^plan_i,τ ) (101) denote the part of the flexibility award delivered by reducing planned charging, and let qi,τdis=max⁡(qi,τ−pi,τplan,0)q^dis_i,τ= \! (q_i,τ-p^plan_i,τ,0 ) (102) denote the remaining part delivered through battery discharge. The realized battery charging and discharging powers are therefore pi,τch=pi,τplan−qi,τred,pi,τdis=qi,τdis.p^ch_i,τ=p^plan_i,τ-q^red_i,τ, p^dis_i,τ=q^dis_i,τ. (103) The state of charge evolves according to soci,τ+1=soci,τ+ΔflexEi(ηichpi,τch−pi,τdisηidis).soc_i,τ+1=soc_i,τ+ ^flexE_i ( _i^chp^ch_i,τ- p^dis_i,τ _i^dis ). (104) Hence, providing flexibility in the current interval can reduce the battery energy available for future flexibility provision. Photovoltaic generation, local demand, planned battery operation, and the resulting network operating conditions evolve according to their exogenous data processes. The overall state transition is sτ+1∼P(⋅∣sτ,aτ).s_τ+1 P\! (· s_τ,a_τ ). (105) Agent objective. Each aggregator uses a policy σi(ai,τ∣oi,τ) _i(a_i,τ o_i,τ) to determine its flexibility offer. Over a horizon of H trading intervals, aggregator i seeks to maximize its expected discounted profit, Ji(σi,σ−i)=⁡[∑τ=1Hγτ−1ri,τ],J_i( _i, _-i)=E [ _τ=1^Hγ^τ-1r_i,τ ], (106) where σ−i _-i denotes the joint policies of the other aggregators. Because the DSO jointly selects offers according to their submitted prices, available quantities, locations, and effects on network constraints, the return of each aggregator depends on both its own bidding policy and those of the other participants. Aggregators therefore learn price–quantity offering strategies through repeated market interactions without observing competitors’ private states, costs, or contemporaneous offers. Appendix B Computation Performance This section compares the throughput of the same environment code under three layouts: all on the GPU; environment on the host CPU, policy on the GPU; all on the host CPU. With 64 parallel environments, the throughput of all on the GPU is 2.3 to 22 times that of all on the host CPU and 2.3 to 33 times that of environment on the host CPU, policy on the GPU, in every one of the five markets, and it does not change with the number of host threads allotted. Figure 7 compares the three layouts in the market with the most participants, the M4 peer-to-peer local energy market with 1 200 households; Figure 8 covers all five markets, and Figure 9 open-source tools. All experiments run under Python 3.11.15 with JAX and jaxlib 0.10.2, CUDA support added as the same-version plugin rather than by changing either, and double precision and highest matmul precision enabled, on a single shared workstation with three NVIDIA RTX 4500 Ada (24 GB) GPUs and an AMD Ryzen Threadripper PRO 7985WX host (64 cores, 128 threads); each measurement point is run three times and compilation time is excluded. Throughput is the number of environment steps advanced per second, summed over the parallel environments, where one step covers the market clearing and one forward pass of the policy, without gradient updates. B.1 Three layouts on the peer-to-peer market In the peer-to-peer local energy market with 64 parallel environments, all on the GPU advances 27 929 environment steps per second, 22 times the throughput of all on the host CPU and 33 times that of environment on the host CPU, policy on the GPU (Figure 7a). Raising the number of parallel environments from 64 to 1 024 leaves the throughput unchanged, and so does the number of host threads allotted (Figure 7b). Figure 7: Throughput of the peer-to-peer local energy market environment (1 200 households, 96 periods per episode) under three layouts: (a) against the number of parallel environments; (b) against the number of host CPU threads allotted, with 64 parallel environments. Each point is the mean of three runs with a bootstrap 95% confidence interval; each dotted line joins two points and the number beside it is their throughput ratio. B.2 Across the five markets All five markets show the speed and throughput advantage of PowerMarketJax: with 64 parallel environments, all on the GPU gives 2.3 to 22 times the throughput of all on the host CPU and 2.3 to 33 times that of environment on the host CPU, policy on the GPU (Figure 8a); all on the GPU, raising the number of parallel environments from 1 to 128 multiplies the throughput by 2.9 to 12 (Figure 8b). Figure 8: Throughput of the five market environments: (a) the three layouts with 64 parallel environments, the number above each host bar being the ratio of all on the GPU to that layout; (b) all on the GPU with 1 and with 128 parallel environments, the number above each pair being the ratio of the two bars. Each bar is the mean of three runs. B.3 Against open-source tools The clearing problem of the M2 real-time balancing market was also implemented in PyPSA (Brown et al., 2018) and in cvxpy (Diamond and Boyd, 2016; Agrawal et al., 2018), each in its default usage (default formulation and solver), stepping one environment at a time. All on the GPU with 64 parallel environments gives 71 times the throughput of PyPSA and 8.8 times that of cvxpy (Figure 9). Training PPO on the real-time balancing market for the same number of environment steps, PowerMarketJax all on the GPU is 3.8 times faster than Stable-Baselines3 (Raffin et al., 2021) with its default settings (wall-clock time, including gradient updates). Figure 9: Throughput of the real-time balancing market environment in three implementations: PyPSA 1.3.0 and cvxpy 1.9.3 in their default usage, one environment at a time, and PowerMarketJax all on the GPU with 1 and with 64 parallel environments. Each bar is the mean of three runs; the label above each grey bar is how many times slower it is than PowerMarketJax with 64 parallel environments. B.4 Implementation validation We validate the implementation in three ways: market clearing and settlement against a NumPy reference implementation; the dual prices of the JAX linear-program solver against HiGHS (Huangfu and Hall, 2018), an independent solver; and the three-stage clearing of the M1 day-ahead market against the exact solution of the same commitment problem, computed by HiGHS with the commitment decisions kept binary. Against a NumPy reference implementation. Each market has a separate NumPy reference implementation that rewrites the same clearing and settlement rules with explicit matrices and loops. We run both on every period of the evaluation days of each market under truthful offers and compare each quantity (Table 1). In M3, load shedding and reserve shortfall are compared as totals per period, because the optimum does not fix how they are split across buses and reserve products. Table 1: Largest difference between the JAX implementation and the NumPy reference. The relative difference divides the absolute difference by the largest absolute reference value. Market Quantity Data Absolute difference Relative difference M1 LMP 36 days × 24 h, case29gb 2.2×10−82.2× 10^-8 $/MWh 1.4×10−101.4× 10^-10 Award 1.3×10−111.3× 10^-11 MW 1.7×10−151.7× 10^-15 Objective value 9.3×10−109.3× 10^-10 $ 4.9×10−164.9× 10^-16 Settlement (profit) 2.8×10−92.8× 10^-9 $ 6.3×10−166.3× 10^-16 M2 LMP 36 days × 48 half-hours, case29gb 9.1×10−89.1× 10^-8 $/MWh 9.1×10−129.1× 10^-12 Award 1.6×10−111.6× 10^-11 MW 2.1×10−152.1× 10^-15 Settlement (profit) 1.5×10−101.5× 10^-10 $ 1.7×10−171.7× 10^-17 M3 LMP 36 days × 48 half-hours, case29gb 7.3×10−67.3× 10^-6 $/MWh 7.3×10−107.3× 10^-10 Reserve price 1.2×10−41.2× 10^-4 $/MWh 4.7×10−74.7× 10^-7 Award 5.1×10−95.1× 10^-9 MW 6.5×10−136.5× 10^-13 Load shed, total 1.4×10−111.4× 10^-11 MW 7.7×10−157.7× 10^-15 Reserve shortfall, total 9.3×10−129.3× 10^-12 MW 4.0×10−154.0× 10^-15 Objective value 3.0×10−43.0× 10^-4 $ 3.0×10−113.0× 10^-11 Settlement (profit) 4.7×10−24.7× 10^-2 $ 5.5×10−95.5× 10^-9 M4 Clearing price 64 episodes × 96 quarter-hours, 1 200 households 6.1×10−66.1× 10^-6 EUR/MWh 1.8×10−81.8× 10^-8 Traded volume 1.4×10−81.4× 10^-8 MWh 1.0×10−71.0× 10^-7 Award 1.9×10−81.9× 10^-8 MWh 3.9×10−63.9× 10^-6 Settlement (profit) 1.8×10−61.8× 10^-6 EUR 1.0×10−61.0× 10^-6 M5 Award 36 days × 24 h, feeder 459_0 3.4×10−93.4× 10^-9 MW 1.3×10−81.3× 10^-8 Load shed 1.5×10−91.5× 10^-9 MW 5.0×10−75.0× 10^-7 Objective value 5.2×10−75.2× 10^-7 CHF 1.1×10−91.1× 10^-9 Settlement (profit) 5.2×10−75.2× 10^-7 CHF 1.5×10−81.5× 10^-8 Dual prices against HiGHS. The prices of M1, M2 and M3 are duals of a linear program. On every period of the same evaluation days, we solve the same linear program with HiGHS and compare its prices with those of the JAX solver (Table 2). M4 and M5 set prices without a linear program. Table 2: Largest difference between the JAX solver and HiGHS. Market Test system Price Periods Price difference ($/MWh) Relative objective difference M1 case29gb LMP 864 2.6×10−82.6× 10^-8 3.1×10−163.1× 10^-16 M1 case73rts LMP 864 4.1×10−84.1× 10^-8 6.2×10−86.2× 10^-8 M1 case813nem LMP 864 3.5×10−73.5× 10^-7 1.2×10−71.2× 10^-7 M2 case29gb LMP 1 728 1.1×10−71.1× 10^-7 1.8×10−121.8× 10^-12 M3 case29gb LMP 1 728 7.3×10−67.3× 10^-6 3.1×10−113.1× 10^-11 M3 case29gb Reserve price 1 728 2.1×10−72.1× 10^-7 3.1×10−113.1× 10^-11 Gap to the exact optimum. The M1 day-ahead clearing has binary commitment variables, and no mixed-integer solver runs inside the compiled graph, so the market is cleared by a three-stage procedure: a linear program with the on/off decisions relaxed to values between 0 and 1, rounding of the relaxed commitment, and a re-solve of the economic dispatch with the commitment fixed. On the 36 evaluation days of each of the three M1 test systems, under truthful offers, we solve the same unit commitment problem exactly and offline with HiGHS (commitment variables 0 or 1) and compare its production cost with that of the three-stage procedure. Over the 36 evaluation days, the median excess production cost of the three-stage procedure over the exact optimum is 2.04% on case29gb, 0 on case73rts and 1.10% on case813nem, and the maximum is 3.78%, 1.79% and 1.95% (Figure 10). On 23 of the 36 evaluation days of case73rts, the three-stage commitment is the same as the exact optimum and the gap is 0. Figure 10: Production cost of the three-stage clearing above the exact optimum on the evaluation days of the three M1 test systems, under truthful offers. Each point is one evaluation day, in date order; the dashed line is the median. Appendix C Quickstart and API This section is a walk-through of how to install PowerMarketJax, run a minimal rollout, read the interface contract that the five markets share, construct each of them, find where an experimental configuration is written down, and reproduce a result from the command line. Listing C.2 is a complete runnable program, Listing C.3 a rollout function that takes an environment built by any of the five constructors, Listing C.5 is abridged from a released source file, and the bash blocks in Sections C.1 and C.6 cover installation and experiment script usage. The market models themselves, including the observation, action, clearing and settlement of each market, are in Appendix A; this section covers only the software interface to them. C.1 Installation PowerMarketJax requires Python 3.10–3.12 and has no build step. The base install ships a CPU-only JAX; CUDA 12 support is opt-in via the cuda extra, which adds the CUDA plugin alongside the already resolved jaxlib at the same version and therefore does not move the JAX version the suite is verified on. GPU support follows JAX’s own platform support: the CUDA build is available on Linux only, so on Windows the package runs on the CPU natively and on the GPU through WSL2. All reported results and test runs were obtained on Linux. The learning stack of rlax and distrax used by the P2P trading experiment script lives behind rl, and the figure scripts behind figs. Nothing under powermarketjax/ imports matplotlib, rlax or distrax, so the five markets and both learners are available on a dev install alone. Listing C.1: PowerMarketJax Installation commands ⬇ git clone https://github.com/powermarketjax/PowerMarketJax conda create -y -n powermarketjax python=3.11 && conda activate powermarketjax pip install -e ".[dev]" # CPU JAX, the five markets, both learners pip install -e ".[dev,cuda,rl,figs]" # Linux + CUDA 12, plus scripts and figures python -c "import powermarketjax, jax; print(powermarketjax.__version__, jax.__version__, jax.devices())" C.2 Minimal rollout Listing C.2 opens the day-ahead wholesale market on the 29-bus GB case, resets it, and runs a three-step rollout under jax.lax.scan with every agent bidding at twice its cost. One step is one market day, so the rollout clears three consecutive days of a multi-period security-constrained unit commitment; every transition, the clearing included, stays inside jit. Listing C.2: Minimal rollout on case29gb (day-ahead wholesale) ⬇ import jax, jax.numpy as jnp jax.config.update("jax_enable_x64", True) # M1/M2/M3/M5 refuse to build without it from powermarketjax.case import load_case from powermarketjax.envs.day_ahead import make_env, load_commitment, load_gb_demand from powermarketjax.utils.jax_utils import scan_rollout env, spec = make_env(load_case("29gb"), load_commitment(n_periods=24), load_gb_demand(), kind="markup", markup_max=2.0, cap_scale=0.60, ramp_scale=1.00) params = env.make_params(episode_len=3) obs, state = env.reset(jax.random.PRNGKey(0), params) actions = jnp.full((3, spec["n_agents"]), 2.0) # (n_steps, n_agents) final_state, obs_traj, reward_traj, cost_traj, done_traj, info_traj = jax.jit(lambda k, s: scan_rollout(env, k, s, params, actions) )(jax.random.PRNGKey(1), state) At this configuration the market has 66 agents, one per thermal unit: obs_traj is (3, 66, 112), reward_traj is (3, 66) and cost_traj is (3, 66, 2), and info_traj["converged"] reports per step whether the clearing met its tolerance. scan_rollout drives step_auto_reset, so episodes restart at their boundary without a Python conditional and the trajectory has a fixed length. The scenario parameters have no defaults. cap_scale and ramp_scale in Listing C.2 are required: omitting either raises ValueError. The two scale the line ratings and the ramp limits the case ships with, so cap_scale 1.00 means every line keeps the capacity written in the case file and in that network no line ever reaches its limit. With no line at its limit there is no congestion, and with no congestion every bus clears at the same price. Listing C.3 is the same contract on any of the five markets, vectorised over a batch of environments. unpack_env accepts whichever shape a market’s constructor returns (Section C.4) and yields the same four objects, so the rollout below is market-agnostic. Listing C.3: One rollout function for five markets: vmap over environments inside scan over time ⬇ import math, jax, jax.numpy as jnp, numpy as np from functools import partial from powermarketjax.learning import unpack_env from powermarketjax.resources.battery import make_battery_bundle from powermarketjax.envs.p2p import (make_p2p_env, make_p2p_params, load_fluvius_households) jax.config.update("jax_enable_x64", True) # M1/M2/M3/M5 refuse to build without it # ‘built‘ is whatever a market’s constructor returned: the (env, spec) pair for M1/M2, or the (reset, step, step_auto_reset, spec) tuple for M3-M5. Here it is M4, the P2P trading market, on 12 metered houses. N, DT, ETA = 12, 0.25, math.sqrt(0.85) series = load_fluvius_households(n_households=N) battery = make_battery_bundle(n_devices=N, dt_hours=DT, capacity_mwh=[0.01] * N, power_mw=[0.005] * N, eta_charge=ETA, eta_discharge=ETA, soc_min=0.15, soc_max=1.0, initial_soc=0.5, cycle_cost_per_mwh=0.0) params = make_p2p_params(p_pv=series.injection, load=series.offtake, battery=battery, kappa=np.full(N, 13.88, np.float32), learner_mask=np.ones(N, bool), episode_len=96) built = make_p2p_env(N, 73.0, 333.4, DT) # n_agents, export/retail, dt reset, step, step_auto_reset, spec = unpack_env(built) # unpack_env takes either shape and gives the four names below. def rollout(key, params, policy, n_envs, n_steps): k_reset, k_roll = jax.random.split(key) obs, state = jax.vmap(reset, in_axes=(0, None))( jax.random.split(k_reset, n_envs), params) def one_step(carry, k): obs, state = carry action = policy(obs) # (n_envs, *action_shape) obs, state, reward, costs, done, info = jax.vmap( step_auto_reset, in_axes=(0, 0, 0, None))( jax.random.split(k, n_envs), state, action, params) return (obs, state), (reward, costs, done) return jax.lax.scan(one_step, (obs, state), jax.random.split(k_roll, n_steps)) # The action shape comes from the market, not from the learner. zero = lambda obs: jnp.zeros((obs.shape[0], *spec["action_shape"]), jnp.float32) (obs, state), (reward, costs, done) = jax.jit( partial(rollout, policy=zero, n_envs=8, n_steps=6) )(jax.random.PRNGKey(0), params) The returned reward has shape (n_steps, n_envs, N) and costs has shape (n_steps, n_envs, N, C): time, parallel environments and market participants are three separate axes, and the second and third are the two the accelerator exploits. params is closed over as a broadcast argument, so a sweep over anything carried there is a loop over calls and not a recompilation. C.3 The environment interface Signatures. Every market exposes the same three pure functions: Listing C.4: Market environment signatures ⬇ obs, state = reset(key, params) obs, state, reward, costs, done, info = step(key, state, action, params) obs, state, reward, costs, done, info = step_auto_reset(key, state, action, params) With N agents and C constraint channels, obs is (N,d)(N,d), reward is (N,)(N,), costs is (N,C)(N,C) and done is a scalar boolean. The agent index is one array axis: observations, actions, rewards and costs are aligned on it, and N is fixed at compile time. Reward and constraint costs. The reward of an agent is the profit it earns in that step, settled on the quantities and prices produced by the market’s own clearing (Appendix A). Violations of physical or market constraints, such as shed load or an unmet reserve requirement, are returned separately in costs, one column per name in spec["cost_names"], and are never subtracted from the reward. A user decides how to handle them, for example by adding a penalty to the reward or by using a constrained learning method. Episode boundary. spec["termination"] tells a learner what the end of an episode means. In four markets it is "truncation": the episode is cut off in time, the market would continue, and the value of the last state should be bootstrapped. In the P2P market it is "terminal": the last reward already pays each household for the energy left in its battery, so bootstrapping would count that energy twice. C.4 Constructing the five markets Each market has its own constructor, since the scenarios differ: the wholesale markets take a network case, a precomputed day-ahead commitment and schedule, and a demand series; the ancillary market adds the reserve products; the P2P trading market takes a population size and the export and retail tariffs; the local flexibility market takes a feeder and the placement of the aggregators. The ancillary, P2P and local flexibility constructors return (reset, step, step_auto_reset, spec), and the day-ahead and real-time constructors return (env, spec); powermarketjax.learning.unpack_env converts either to the first form, as in Listing C.3. Anything that changes the size of the clearing problem, such as the network or the placement of agents, is fixed when the environment is built, and changing it requires recompiling; anything that varies within a market, such as the demand series or the episode length, is passed in params on every call. C.5 Scenario and learner configuration There is no configuration file format: an experimental configuration is Python. The learner configuration is tools/benchmark/hyperparams.py, abridged in Listing C.5. The scenario parameters are module-level constants of each market’s experiment script, each also exposed as a command-line flag. Listing C.5: The shared learner configuration ⬇ N_ENVS, MINIBATCHES = 64, 32 HORIZON = "01": 4, "02": 48, "03": 48 # steps per update, per market for _m, _h in HORIZON.items(): # the batch must divide into minibatches assert (N_ENVS * _h) % MINIBATCHES == 0 SHARED = IPPOConfig( n_envs=N_ENVS, horizon=HORIZON["02"], # ours, per market via shared_for() epochs=10, # update_epochs minibatches=MINIBATCHES,# num_minibatches lr=3e-4, # learning_rate (no annealing here) clip_eps=0.2, # clip_coef gamma=0.99, # gamma gae_lambda=0.95, # gae_lambda vf_coef=0.5, # vf_coef ent_coef=0.0, # ent_coef max_grad_norm=0.5, # max_grad_norm hidden=(64, 64), # Agent: two tanh layers of 64 weight_decay=0.0, # zero reproduces optax.adam exactly init_scale=1.0, ) def shared_for(market): return dataclasses.replace(SHARED, horizon=HORIZON[market]) C.6 Command-line usage Each market has one experiment script for its learning runs. The non-learning baselines are run by a separate script for the day-ahead, real-time and ancillary markets (tools/benchmark/run_eval_0*.py), by the learning script itself for the P2P trading market (--arm truthful) and for the local flexibility market, where they run alongside the learners. The learning script takes the learner (--algo ippo|sac), parameter sharing or not (--per-agent-params for one network per agent), the seed, the iteration count, the scenario parameters and an output directory. It trains on the training days only and then evaluates the mean action of the policy on the held-out days, so the reported number excludes exploration noise. Listing C.6: Command line for RL training ⬇ # One-year inputs for M1-M3 (2023-07-10 to 2024-07-08), built once. python tools/commitment/precommit.py --mode relax --start-day 5 --days 365 --cap-scale 0.6 --ramp-scale 1.0 --out commit_y1.npz python tools/commitment/da_position.py --case 29gb --fixture commit_y1.npz --out position_y1.npz # M1 day-ahead: parameter sharing, then one network per agent. python tools/benchmark/run_rl_01.py --fixture commit_y1.npz --algo ippo --seed 0 --iterations 400 --out-dir runs/m1-ippo-ps python tools/benchmark/run_rl_01.py --fixture commit_y1.npz --algo ippo --seed 0 --iterations 400 --per-agent-params --out-dir runs/m1-ippo-nops # M2 real-time and M3 ancillary settle against the day-ahead position. python tools/benchmark/run_rl_02.py --position position_y1.npz --seed 0 --out-dir runs/m2-ippo-ps python tools/benchmark/run_rl_03.py --position position_y1.npz --seed 0 --out-dir runs/m3-ippo-ps # M4 P2P, 1200 households. python tools/benchmark/run_rl_04.py --agents 1200 --seed 0 --out-dir runs/m4-ippo-ps # M5 local flexibility. PYTHONPATH=tools/flex_experiment python -m concentration_baseline --algo ippo --seeds 3 --out runs/m5-ippo-ps # Non-learning baselines for M1 on the held-out days. python tools/benchmark/run_eval_01.py --fixture commit_y1.npz --days eval --out-dir results/m1-baselines C.7 Cases and data powermarketjax.case.list_cases() returns the 18 built-in network cases, 10 transmission and 8 distribution, sorted by bus count and with aliases removed; load_case accepts a canonical identifier or an alias. The demand, generation, price and smart-meter series are carried in the repository as parquet files, so every listing above runs on a fresh checkout with no separate download step. C.8 Clearing and learning in a single computation graph A training iteration has two parts, a rollout and an update. In the rollout, N environments advance together for T steps. In each step the participants submit offers, the market clears, the awards are settled, and the next observation is constructed; the policy maps the observation to the action at every step, and the observation, action, reward and constraint cost of one step form one experience. The update takes gradient steps on the N×TN× T experiences of the rollout, split by epoch and minibatch. The whole iteration, including the clearing problem of every step, is compiled as one function on the GPU: the host makes one call and receives the training metrics, and every clearing, every policy forward pass and every gradient step in between runs on the GPU in sequence. Figure 11 shows the data flow of one iteration, and Listing C.7 writes the same iteration as loops, with the JAX primitive that implements each loop in the comment above it. The values of N and T for each market are in Table 4, the epochs and minibatches in Table 3, and n, the number of agents, in Appendix A. The loop over the N environments is vmap(step). The function step advances one environment by one step and is written for one environment only; vmap returns the function that advances N environments at once, whose input and output gain one axis of length N. The clearing, settlement and observation of the N environments are one batched computation over N sets of data, not N calls one after another. The forward pass of the policy goes through vmap in the same way, so the actions are produced inside the same function and passed directly to step, and the observations never reach the host. Figure 11: One training iteration of IPPO, compiled as a single function. Rollout: N environments advance in parallel for T steps; in each step the n agents submit offers, the market clears, the awards are settled and the next observation is constructed, and the policy maps the observation to the next action inside the same function. Update: the policy is updated on the N×TN× T experiences, with a second scan over epochs and minibatches (Table 3). The values of N and T for each market are in Table 4. Listing C.7: One training iteration as loops, with the primitive that implements each loop ⬇ # iteration = jit(iteration): the whole body below is one compiled # function on the GPU, called once per training iteration iteration(params, states, observations): # rollout # scan(one_step, carry, length=T): carry = (states, observations); # returns the final carry and the T records stacked for t = 1 .. T: # policy_N = vmap(policy), step_N = vmap(step): # each takes N inputs and returns N outputs at once for each of the N environments: for each of the n agents: action <- policy(observation) # step(state, actions), one environment offers <- submit(actions) awards, prices <- clear(offers) reward, cost <- settle(awards, prices) observation <- observe(the state after clearing) if the episode ended: reset this environment in place # one record per step, holding the N environments record (observations, actions, rewards, costs) # N x T experiences, each with n agents experiences <- the T records # update # scan(gradient_step, ...): the passes over the experiences, # by epoch and minibatch for each pass over the experiences: params <- optimiser step on the gradient of the loss return params, states, observations, training metrics The loop over the T steps is lax.scan(one_step, carry, length=T). It takes the loop body one_step, the initial carry, which is the N states and the N observations, and the number of steps T; runs the body T times, feeding the carry returned by one run into the next; stacks the record of each run into an array of length T; and returns the final carry together with this array. The horizon T is fixed, and an environment whose episode ends before step T is reset in place inside the body, so the input and output of the body have the same shape at every step, which is the condition under which lax.scan applies. The gradient steps of the update are a second lax.scan. None of these loops runs on the host, and the host does not see the boundary between two consecutive steps. The outer function is jit(iteration): the rollout followed by the update, compiled for the GPU at the first call and called once per iteration. The host passes in the parameters, the environment states and the observations, and receives the updated parameters, states, observations and training metrics; no intermediate result returns to the host, and the only loop left on the host is the loop over iterations. The iteration of SAC has the same form: the rollout is the same, and the update first places the experiences of the rollout into a replay buffer of fixed size, which also lives inside this function and is carried from one iteration to the next, then samples from it for the gradient steps, which are again a lax.scan. Appendix D Training Setup and Hyperparameters The results of Section 5 and Appendix E share four learners, one set of hyperparameters and one evaluation protocol; where a market departs from them, its subsection of Appendix E gives the difference. Appendix C.5 shows where these values are set in the code. D.1 Four learners Each market is run with four learners: IPPO-PS, IPPO-NoPS, SAC-PS and SAC-NoPS. All four are independent learners: every agent sees only its own observation and optimises only its own profit, with no centralised value network. PS and NoPS stand for parameter sharing and no parameter sharing. Under PS all agents share one set of networks and pool their samples for training; under NoPS each agent has its own set and trains only on its own samples. For IPPO the set is one actor and one value network that share two tanh layers, and the log standard deviation of the actor does not depend on the observation; for SAC it is one stochastic actor and a pair of Q-networks with their target networks, with an automatically tuned temperature. D.2 Hyperparameters The hyperparameters build on the defaults of the continuous-action PPO and SAC implementations in CleanRL; the five markets use one set, listed in Table 3, without tuning for each market. The profit of one step in M1 is of the order of 10510^5 to 10610^6 dollars and the Q regression of SAC is sensitive to this magnitude, so SAC divides the reward by a constant before the Q regression: the standard deviation of the rewards under truthful offers, computed once before training. SAC runs the gradient steps of an iteration as one block after the rollout. In all four learners each component of the observation is normalised with constants fixed before training. Table 3: Hyperparameters of the two IPPO and the two SAC learners, shared by the five markets. Hyperparameter IPPO SAC Optimiser Adam Adam Learning rate 3×10−43× 10^-4, constant actor 3×10−43× 10^-4, Q-networks and temperature 10−310^-3 Discount factor γ 0.99 0.99 Hidden layers two layers of 64 units, tanh two layers of 256 units, ReLU GAE parameter λ 0.95 — Clipping range 0.2 — Update epochs per iteration 10 — Minibatches per epoch 32 — Value loss coefficient 0.5 — Entropy coefficient 0 — Maximum gradient norm 0.5 — Initialisation scale of the actor output layer 1.0 — Replay buffer — 32 768 environment steps Batch size — 256 environment steps, each carrying the transitions of all agents Gradient steps per environment step — 1 Random-action steps before training — 0 Actor update interval — every gradient step Target smoothing coefficient τ — 0.005 Temperature — initial value 0.2, tuned automatically Target entropy — minus the action dimension Range of the log standard deviation — [−5,2][-5,2] The two SAC learners of M4 and the two IPPO learners of M5 differ from Table 3 in the settings given in Sections E.4.2 and E.5.2. D.3 Training scale Table 4 gives the training scale of each market. Table 4: Training scale. Market Seeds per learner Iterations Parallel environments and steps per iteration Environment steps per iteration M1 day-ahead wholesale 3 400 64 environments, 4 steps each 256 M2 real-time balancing 3 200 64 environments, 48 steps each 3 072 M3 ancillary services 3 200 64 environments, 48 steps each 3 072 M4 peer-to-peer local energy 3 400 64 environments, 96 steps each 6 144 M5 local flexibility 3 to 10 early stoppinga, at most 300 64 environments, 24 steps each 1 536 a M5 evaluates a checkpoint on the validation days every 10 iterations and stops early once the mean validation return of the last 5 checkpoints exceeds that of the 5 checkpoints before them by 0 to 5%. D.4 Evaluation protocol M1 to M4 evaluate the policy at the last iteration. In M1 to M3, evaluation takes the mean action of the policy and runs day by day on the 36 evaluation days of the one-year window of case29gb, 2023-07-10 to 2024-07-08 (the dates of the other two test systems are in Table 5). In M4, evaluation also takes the mean action and runs on 1 024 held-out episodes (Table 7). In M5, the checkpoint with the highest validation return is evaluated on the 36 test days with sampled actions (Section E.5.2). The training return is the return under sampled actions and includes exploration noise. The untrained network is the network of the same parameter layout (PS or NoPS) at iteration 0, evaluated in the same way. Truthful offers are the reference policy in which every agent offers at its true cost: in M1 to M3 every unit takes the markup αi=1 _i=1, with a reserve offer of 0 in M3; in M4 the batteries are kept still, sellers ask the export price and buyers bid the retail tariff; and in M5 every aggregator offers its full deliverable quantity at the replacement cost and plans no charging. Appendix E Additional Market Results E.1 M1: Day-ahead Wholesale Market This section first presents detailed data, training curves and evaluation results of the three test systems, and then elaborates the finding in Section 5.1 with the supply curves, the offer distribution and the joint sampling probability on the British test system. E.1.1 Market setting Each unit i offers a price–quantity curve with a single segment (K=1K=1 in Section A.1.1). Its offer price is αimi _im_i, where mim_i is its segment cost (the mean of its true marginal cost over its output range) and αi∈[1,α¯] _i∈[1, α] is its markup. At αi=1 _i=1 the offer is truthful, and α¯>1 α>1 is the markup cap. A markup above one represents economic withholding: the unit offers its capacity at a price above its cost. We consider three test systems. As shown in Table 5, each test system uses one year of data, of which 36 days are used for evaluation and the rest for training. On each test system, three selected days are taken from the 36 evaluation days: the days at the 10th, 50th and 90th percentiles of total daily demand, referred to below as the low-load day, the mid-load day and the high-load day. Table 5: Data of the three test systems. Test system Grid Demand data Time span Training / evaluation days Markup cap α¯ α case29gb A reduced 29-bus model of the GB transmission networka (network diagramb), 66 units GB transmission system demand (NESO)c, day-ahead forecast from Elexond 2023-07-10 to 2024-07-08 329 / 36 2 case73rts The RTS-GMLC test systeme, 73 units Load, wind, solar and hydro of RTS-GMLCf 2020-01-01 to 2020-12-31 330 / 36 1.4 case813nem An open grid model of the Australian National Electricity Marketg, 151 units Operational demand and day-ahead forecast of the four mainland regions from the Australian Energy Market Operator (AEMO)h 2025-02-01 to 2026-01-31 329 / 36 2 a https://webhomes.maths.ed.ac.uk/OptEnergy/NetworkData/reducedGB/; b https://webhomes.maths.ed.ac.uk/OptEnergy/NetworkData/reducedGB/GBreducednetwork.pdf; c https://w.neso.energy/data-portal/historic-demand-data; d https://bmrs.elexon.co.uk/; e https://github.com/GridMod/RTS-GMLC; f https://github.com/GridMod/RTS-GMLC/tree/master/RTS_Data/timeseries_data_files; g https://github.com/akxen/egrimod-nem-dataset; h https://visualisations.aemo.com.au/aemo/nemweb/index.html. In the British test system case29gb, generation lies mostly in the north and load mostly in the south, and power flows from north to south: the north has 63% of the total capacity but only 43% of the load. The three example days are 2024-06-23, 2024-03-16 and 2024-01-23. Under truthful offers, 2, 3 and 5 lines are congested on these days, all on central and southern corridors (Figure 12). Of the 66 units, 14 are nuclear, 23 coal and 29 gas. In order of segment cost, all 14 nuclear units are among the 16 cheapest units, and their total capacity is only 4.359 GW. These 16 units have a total capacity of 19.2 GW, while hourly system demand over one year ranges from 17.2 to 47.4 GW and is above 19.2 GW in almost every hour. The nuclear units therefore sit near the left of the supply curve with demand almost always to their right, which means a unilateral markup by one nuclear unit does not move the price, and only a joint markup raises it. Figure 12: The test system case29gb. Left: the 66 units at their buses, coloured by fuel and sized by maximum output; corridor width scales with capacity. Middle: annual mean bus load and the corridors congested under truthful offers on the three example days. Right: the supply curve ordered by segment cost, against the range of hourly system demand over one year. Generation sits mostly in the north and load mostly in the south, so the congested corridors are central and southern. All 14 nuclear units, 4 359 MW in total, are among the 16 cheapest units, which sum to 19.2 GW; hourly demand ranges from 17.2 to 47.4 GW and exceeds 19.2 GW in almost every hour. The nuclear units therefore sit near the left of the supply curve with demand almost always to their right: a unilateral markup does not move the price, while a joint markup lifts the whole curve and the price with it. In the RTS-GMLC test system, case73rts, wind, solar and hydro (referred to below as renewables) do not submit offers, and the demand cleared by the market is the net demand (total load minus renewable output). The three example days are 2020-02-24, 2020-03-24 and 2020-07-16. Under truthful offers, 0, 2 and 7 lines are congested on these days (Figure 13). Figure 13: The test system case73rts, drawn as Figure 12: the three-area RTS-GMLC system with 73 units. Left: units by fuel and maximum output. Middle: annual mean bus load and the lines congested under truthful offers on the three example days. Right: the supply curve ordered by segment cost against the range of hourly net demand over one year. Gas units make up most of the supply curve; no line is congested on the low-load day and seven are on the high-load day. Regarding the Australian test system, case813nem, the three example days are 2025-09-20, 2025-12-10 and 2025-06-24. Under truthful offers, 3, 4 and 4 lines are congested on these days (Figure 14). Figure 14: The test system case813nem, drawn as Figure 12: an open grid model of the mainland Australian National Electricity Market with 151 units. Only 7 of its 1 278 lines carry a published rating. Left: units by fuel and maximum output. Middle: annual mean bus load and the lines congested under truthful offers on the three example days. Right: the supply curve ordered by segment cost against the range of hourly demand over one year. Hydro and then coal fill the left of the supply curve; three lines are congested on all three days and a fourth on the mid- and high-load days. On all three test systems the market clears the demand that actually occurs on the day, while the units observe the day-ahead forecast before they submit offers (Figure 15). On case73rts, net demand has a floor of 2 500 MW, and it sits at that floor in more than half of the hours of the year. On case813nem, demand has a floor of 11 500 MW, reached in only a few hours. Figure 15: Demand on the three test systems, and the renewable output netted out on case73rts. Left: daily means over the one-year window, with the 36 evaluation days marked. Right: hourly values in the week around the mid-load example day. The thick black line is the demand the market clears; the dashed line is the day-ahead forecast the agents observe. On case73rts the market clears net demand, total load minus the renewable output shown, raised to a floor of 2 500 MW; it sits at that floor in more than half of the hours of the year. Demand on case813nem has a floor of 11 500 MW, reached in a few hours. E.1.2 Agent training The training setup of the four learners is that of Appendix D. On the British test system, the training return of IPPO-NoPS, SAC-PS and SAC-NoPS ends close to its highest level (top row of Figure 16). IPPO-PS stays below the other three and is the only one whose return rises and then falls; the markup it samples also rises to about 1.7 and then returns to about 1.5 (Figure 3a). On the other two test systems only IPPO-PS and IPPO-NoPS are trained (bottom row of Figure 16). The training return of both learners falls from its starting level. The fall comes from competition among the units: at the start of training every unit offers near the middle of its markup range, which amounts to a joint markup; during training the units that set the price learn to offer closer to cost, because a unit that stays high alone loses its output to its rivals, so the price falls while production cost does not, and profit falls with it. Figure 16: Training return on the three test systems. Top: the four learners on case29gb. Bottom: IPPO-PS and IPPO-NoPS on case73rts and case813nem. Each line is the mean of three seeds, each seed first smoothed by an 11-iteration centred moving mean, and the shaded band is the bootstrap 95% confidence interval. Return is the mean daily profit per generator over all sampled environment steps of an iteration. On case29gb IPPO-PS is the only learner that rises and then falls; on the other two systems the return of both learners falls from its starting level. E.1.3 Discussion of results On each test system, the policies learned by IPPO-PS and IPPO-NoPS are compared with truthful offers on the 36 evaluation days, each learner taken as the mean of three seeds. As shown in Figure 17, across all three test systems, the mean LMP under the learned policies (load-weighted over all periods and buses of the day) is higher than under truthful offers on every one of the 36 evaluation days (production cost barely changes) and the profit of all units rises accordingly. Averaged over the 36 days, the mean LMP under the policies learned by IPPO-PS and IPPO-NoPS is 1.50 and 1.56 times, respectively, its value under truthful offers on case29gb, 1.10 and 1.07 times on case73rts, and 1.23 and 1.29 times on case813nem. For both learners, the total production cost over the 36 days differs from that under truthful offers by less than 1%. The increase in mean LMP is smallest on case73rts, where the mean markups of IPPO-PS and IPPO-NoPS are only 1.19 and 1.21, respectively. Figure 17: Daily system quantities on the 36 evaluation days of each test system, truthful offers (black) against the learned offers of IPPO-PS (orange) and IPPO-NoPS (blue); each learned line is the mean of three seeds, with a bootstrap 95% confidence interval. Rows: daily demand, the load-weighted mean LMP, production cost, and the total profit of all units. E.1.4 Details of supply curves At the peak demand periods of the British test system, the committed units are sorted by offer into a stepped supply curve as shown in Figure 18. Under both truthful and learned offers, the steps of the 14 nuclear units lie far to the left of the demand line. The policy learned by IPPO-PS raises the whole supply curve while the demand line stays in place, and the price rises with it. Figure 18: Supply curves at the hour of highest demand on each example day of case29gb, the evaluation days at the 10th, 50th and 90th percentiles of daily demand (2024-06-23, 2024-03-16 and 2024-01-23): committed units sorted by offer under truthful (grey) and learned (orange) offers, with the steps of the 14 nuclear units drawn thicker and the demand in that hour as a dashed line; the shaded bands are the range of LMP over the 29 buses under each set of offers. The learned offers are the mean over three IPPO-PS seeds at the end of training, with a bootstrap 95% confidence interval for each step; the learned LMP band is the range over buses of the seed-mean LMP. The nuclear steps lie far to the left of demand under both sets of offers, and the learned offers raise the whole curve. E.1.5 Markups of different learners For each of the four learners the mean markup over the 66 units is near 1.5, but the spread across units differs. IPPO-PS gives almost the same markup to every unit; IPPO-NoPS has the widest spread, with some units near truthful offers and some near the cap; the two SAC learners fall between the two, as shown in Figure 19. For each algorithm, NoPS has a wider spread than PS. All four learners give the 14 nuclear units a mean markup below their own mean over all 66 units, and IPPO-NoPS gives these units markups close to truthful offers. Figure 19: Markup of each unit under the four learners on case29gb; rows are learners, each the mean over three seeds, columns the 66 units ordered by segment cost. Colour is the markup of that unit, averaged over the 36 evaluation days and the three seeds, on the full action range from 1.00 (truthful) to 2.00 (cap). Black triangles above the top row mark the 14 nuclear units. E.2 M2: Real-Time Balancing Market In this section, we discuss the result of the four learners in the RTM balancing market. E.2.1 Market setting The action of unit i is a markup αi∈[1,α¯] _i∈[1, α] on its generation cost (i.e., true marginal cost over its output range); αi=1 _i=1 is a truthful offer, and an offer is submitted once per half-hour period. The test systems of this market are case29gb and case73rts (detailed data in Table 5), with the markup cap α¯ α set to 2. The grids and units are shown in Figures 12 and 13. Most output is scheduled day ahead, so real-time offers price only small deviations: on the 36 evaluation days, the median absolute half-hourly demand deviation is 214 MW, less than 1% of mean demand (Figure 20). This limits the gain from raising an offer alone (Section E.2.4). Figure 20: Demand on case29gb. Left: the week around the mid-load day (2024-03-16): the half-hourly demand the real-time market clears (teal), the hourly day-ahead schedule summed over buses, which equals the hourly mean of that demand (purple steps), and the day-ahead forecast the units observe (dashed yellow). Right: the deviation of the demand in each half-hour from the mean of its hour, over all 365 days (grey) and the 36 evaluation days (teal). On the evaluation days the absolute deviation has a median of 214 MW, a 90th percentile of 638 MW and a maximum of 1 478 MW, against a mean demand of 27 730 MW. The day-ahead schedule follows the half-hourly demand to within a few hundred MW. E.2.2 Agent training The training return is the mean daily profit per unit under sampled actions, which is the objective of the learner itself. Under all four learners the mean markup of the 30 committed units starts between 1.48 and 1.61; at the end of training it is 1.04 under IPPO-PS, and 1.24, 1.40 and 1.39 under IPPO-NoPS, SAC-PS and SAC-NoPS; the markup along training is shown in Figure 4b. On case29gb the training return of IPPO-PS falls throughout and ends lowest of the four learners; IPPO-NoPS and SAC-PS also fall, by less; SAC-NoPS rises first and then stays level (top row of Figure 21). On case73rts IPPO-PS and IPPO-NoPS are trained, and the training return of both rises slowly (bottom row of Figure 21). Figure 21: Training return on two test systems in the real-time market. Top: the four learners on case29gb. Bottom: IPPO-PS and IPPO-NoPS on case73rts. Each line is the mean over seeds, smoothed by an 11-iteration moving mean, and the shaded band is the bootstrap 95% confidence interval of the mean. Return is the profit of a unit per day, averaged over units and over the environment steps of an iteration. On case29gb the return of IPPO-PS falls the most, and SAC-NoPS is the only learner whose return rises. On case73rts the return of both learners rises slowly. E.2.3 Discussion of results On case29gb, IPPO-PS returns to truthful offers, while the other three learners end with higher total profit and lower system cost, as shown in Figure 22. This position is not learned. The untrained networks already give the units different markups, and training only moves these markups towards truthful offers. The reason is that an independent learner sees only how its own return responds to its own action. A unit that raises its offer alone only loses output, as shown in section E.2.4. If its output depends on its offer, its gradient points towards truthful offers; if not, its gradient is zero. Without parameter sharing, each unit follows its own gradient: the first kind returns to truthful offers and the second stays where it started. With sharing, the updates from the first kind move all units, so every markup falls together. There are 36 units that are never committed, which give a clean test. Their offers never enter the clearing, so their own return carries no signal. Without information sharing, training leaves their markups unchanged (IPPO-NoPS from 1.48 to 1.49, SAC-NoPS from 1.50 to 1.51). With sharing, they fall with the committed units (IPPO-PS from 1.52 to 1.03, SAC-PS from 1.57 to 1.47). SAC-PS falls much less, because under SAC the committed units themselves stay far from truthful offers (Section 5.2). These results show an information limit in electricity markets: a unit learns a bidding strategy only when its offer changes its own outcome. Otherwise, its learned offer comes from the initialization (without parameter sharing) or from the other units (with it), and should not be read as a strategy. Figure 22: Learned offers against truthful offers on case29gb over the 36 evaluation days. Left: change in total profit of all 66 units compared to truthful offers (in million dollars per day); right: change in system cost compared to truthful offers (in percent). Each circle indicates one learner after training, placed at the mean markup of the 30 committed units. The bars are the 95% confidence interval of the mean over seeds; the dotted line is truthful offers. IPPO-PS returns to truthful offers; the other three learners end with higher total profit and lower system cost. E.2.4 Unilateral and joint markups We extend the discussion in Appendix E.2.3 in this section. A unit gains little by raising its offer alone. We let each of the 30 committed units raise its offer alone, at 21 markup levels from 1.00 to 2.00, while the other 65 units offer truthfully as shown in the left subfigure of Figure 23. For 19 units, the most profitable level is the truthful offer, and no unit gains more than $0.23 million per day. At a markup of 2.00, 27 of the 30 units lose profit, by up to $8.0 million per day. The unit that raises its offer produces less, and the other units take over both its output and its lost profit (Figure 23, right, slope −0.97-0.97). A unilateral markup therefore moves profit between units but leaves the total almost unchanged. Raising all offers together helps only if the markups differ across units. With the same markup for all 66 units, the order of the offers and the dispatch schedules do not change as shown in Figure 24. The reason is the two-settlement rule equation 18. Since the day-ahead market clears the actual hourly demand, output below the schedule in one half-hour is roughly offset by output above it in the other. A higher real-time price therefore moves money between units without raising the total. Different markups, in contrast, change the dispatch, raise total profit and lower system cost, as under IPPO-NoPS, SAC-PS and SAC-NoPS (as shown in Figure 22). This joint gain appears only when units offer differently, and no single unit sees it in its own profit. Figure 23: Each of the 30 committed units raises its offer alone while the others offer truthfully. Left: own profit change against markup (one line per unit, median in black). Right: own profit change against the profit change of the other units (with a least-squares fit, near-zero points are omitted). Means over the 36 evaluation days, in million dollars per day. Figure 24: All 66 units at the same markup, from 1.00 to 2.00. Left: change in total profit (million dollars per day). Right: change in system cost (%). Both are relative to truthful offers (dotted line), averaged over the 36 evaluation days. A common markup leaves both nearly unchanged. E.3 M3: Ancillary Services Market This section first gives the data, training curves and the discussion about results of the ancillary services market. E.3.1 Market setting We consider the setting in which M2 and M3 are jointly cleared. Therefore, the action of each unit i has two parts: an energy offer with markup αi∈[1,α¯] _i∈[1, α], and one reserve offer πi,j,tresπ^res_i,j,t per period t on each of the two reserve products j=1,2j=1,2, in £/MWh. The two products require a response within 10 and 30 minutes respectively. The reserve a unit is awarded on a product, called its reserve held, is at most the output it can ramp up within that time. The reserve requirement of each product is 5% of the demand forecast of the period. Unmet reserve requirement is priced at VOLRVOLR (as shown in Table 6), and the reserve price λj,tresλ^res_j,t does not exceed it. This market uses the same three test systems as the day-ahead market: case29gb, case73rts and case813nem. Table 5 describes their data, and Table 6 gives the markup cap and the value of lost reserve for each system. Figures 12 to 14 show the grids and units, and Figure 15 shows the demand. Table 6: Markup cap and value of lost reserve of the ancillary services market on the three test systems. Test system Markup cap α¯ α Value of lost reserve VOLRVOLR (£/MWh) case29gb 2 250 case73rts 2 136 case813nem 2 147 E.3.2 Agent training The four learners are trained as described in Appendix D, and Figure 25 shows their training return. On the case29gb test system, IPPO-PS is the only learner whose return falls, from −106-106 k £per generator per day over the first 5 iterations to −126-126 over the last 20. The other three rise slightly and stay close to where they started. In test systems case73rts and case813nem, the return of both IPPO learners rises. Figure 25: Training return on the three test systems. Top: the four learners on case29gb. Bottom: IPPO-PS and IPPO-NoPS on case73rts and case813nem. Return is the mean daily profit per generator in each iteration. Lines are the mean of three seeds, each smoothed over 11 iterations, and bands are bootstrap 95% confidence intervals. On case29gb, only the return of IPPO-PS falls, and on the other two systems, both learners rise. E.3.3 Discussion of results Training does not make the learners more profitable in this market. The untrained networks, with random reserve offers, already earn £2 million to £4.5 million per day more than truthful offers as shown in Figure 26. IPPO-PS gives almost all of this back: after training, it earns only £0.56 million per day more than truthful offers, £3.95 million less than its untrained network. The other three learners stay near their starting point, at £2.71 million to £3.69 million per day above truthful offers. No learner receives more gains compared to its own untrained network. A much larger gain is available, but no learner finds it. If all units raise their reserve offers together to £150/MWh, total profit rises by £9.05 million per day as shown in the subfigure c of Figure 4. Only £0.51 million to £1.63 million per day of the learners’ profit comes from reserve payments, and the rest comes from energy. Production cost stays within 0.6% of its truthful level as shown in the subfigure b of Figure 26. The other two test systems show the same results, but the differences are small: all changes in profit are within a few tens of thousands of dollars per day as shown in subfigures c and d of Figure 26. After training, IPPO-PS again earns the same as or less than before training, while IPPO-NoPS earns slightly more. Figure 26: Evaluation on the 36 evaluation days. (a) case29gb: change in total profit of all 66 units relative to truthful offers (million dollars per day), split into reserve payments and energy. (b) case29gb: change in production cost. Filled markers are trained policies; hollow markers are untrained networks. (c) case73rts, relative to the untrained network (truthful offers not evaluated), and (d) case813nem, relative to truthful offers: change in total profit against change in production cost for IPPO-PS (orange) and IPPO-NoPS (blue). The black square marks the reference. Markers are seed means with bootstrap 95% confidence intervals. On case29gb, all learners earn more than truthful offers at nearly unchanged cost, but only IPPO-PS earns less than its untrained network; on the other two systems, the differences are small. E.3.4 Reserve offer analysis In ancillary service markets, a unit gains almost nothing by raising its reserve offer alone, but all units gain a lot by raising it together as shown in Figure 27. When one of the 29 committed units raises its reserve offer from 0 to £150/MWh while the others offer truthfully, its own profit usually does not change. Across the 29 units, the median change is zero, the largest gain is £0.08 million per day, and the largest loss is £0.32 million per day. When all 66 units raise their reserve offers together, total profit rises linearly with the offer, by £9.05 million per day at £150/MWh, while production cost changes by less than 0.001%. Figure 27: Raising the reserve offer on case29gb from 0 to £150/MWh on both energy and reserve products. Teal: own profit change of each of the 29 committed units when it raises its offer alone, with the others truthful. Black: the median. Orange: total profit change of all 66 units when all raise their offers together. Values in million dollars per day. Raising alone gains almost nothing, while raising together gains £9.05 million per day at £150/MWh. E.4 M4: Peer-to-Peer Local Energy Market Section 5.3 finds that an arbitrage that pays for a single household (charging at noon and discharging in the evening) cancels itself when many households adopt it, and that whether a learner finds it depends on the parameter layout. This section first describes the community and its metering data (Section E.4.1), then gives the training curves (Section E.4.2) and the evaluation results on held-out episodes (Section E.4.3); the last section covers how the gain from arbitrage changes with the number of households arbitraging (Section E.4.4). E.4.1 Market setting The test system at community level has the 1 200 households with PV in the Fluvius 2024 metering data, as shown in Figure 28. Each household is considered as one agent. Each period is a quarter-hour, and an episode is the 96 periods of one day. The smart meters record only the injection and the offtake of each household, and the market takes their difference as the net position of each household. The export price is πexp=73.0π^exp=73.0 EUR/MWh and the retail tariff is πret=333.4π^ret=333.4 EUR/MWh. Each household has one battery with a capacity of 0.011 MWh, a round-trip efficiency of 0.85 and a battery degradation cost of 13.88 EUR per MWh of throughput; its state of charge (as a fraction of capacity) stays between 0.15 and 1.0 and starts at 0.5. In every period, each household chooses a battery power (charge or discharge) and a price between the two grid prices. A household with a positive net position sells, and the price is its ask; one with a negative net position buys, and the price is its bid. As shown in Figure 29, at noon most households inject more than they take off and are sellers: around 15:00 about 80% households are sellers, and at night injection is close to zero. Figure 28: The P2P market of the test community. Each of the 1 200 households has PV and a battery. In every quarter-hour, sellers ask and buyers bid in a community double auction, which sets the clearing price λtlocλ^loc_t between the export price πexpπ^exp and the retail tariff πretπ^ret; trades in the community are settled at λtlocλ^loc_t. Energy the auction leaves unmatched is settled with the grid. Table 7: Data of the test community. Data source Households Period covered Training and held-out Licence Quarter-hour injection and offtake from Fluvius smart metersa 1 200, all with PV 2024-04-01 to 2024-10-26, 209 days 3 consecutive days held out of every 15; 14 702 training and 2 702 held-out episode starts; evaluation on 1 024 held-out episodes, of which the figures below use 64 Fluvius open data licence a opendata.fluvius.be/explore/dataset/1_50-verbruiksprofielen-dm-elek-kwartierwaarden-voor-een-volledig-jaar/. Figure 29: The test community of 1 200 households: community injection and offtake (left) and the share of households injecting more than they take off (right), by hour of day over the 64 held-out episodes. E.4.2 Agent training The training setup of the four learners is that of Appendix D, and the two SAC learners differ from Table 3 only in using two hidden layers of 64 units, a replay buffer of 8 192 environment steps, a batch size of 16 environment steps and one gradient step every 4 environment steps. The training return of both IPPO learners rises fastest in the first 50 iterations, and both SAC learners level off within the first few dozen iterations. After that the return changes little, as shown in Figure 30: from iterations 100–149 to iterations 351–400, the mean return of each IPPO learner rises by only 0.0009 to 0.0027 EUR per household per quarter-hour. Figure 30: Training return of the four learners up to iteration 400; the policy at iteration 400 is the one evaluated throughout this section. Shown is the mean over three seeds of the 11-iteration moving mean of the return on 64 sampled training episodes, with the shading the bootstrap 95% confidence interval. Both IPPO learners rise fastest in the first 50 iterations and both SAC learners reach their level within a few dozen; after iteration 50 all four change little. E.4.3 Discussion of results We evaluate the results under truthful offers and under the considered learners. Truthful offers mean that households with batteries stay idle, and sellers ask the export price and buyers bid the retail tariff. As shown in Figure 31, IPPO-NoPS is the most consistent learner: all three of its seeds lower the clearing price and raise the profit per household on every one of the 64 episodes. IPPO-PS and SAC-NoPS are less consistent, with all three seeds earning more than under truthful offers on 54 and 57 episodes, respectively. Averaged over the 64 episodes, the profit per household is −1.201-1.201 EUR per episode under truthful offers and −0.769-0.769 to −0.685-0.685 EUR over the three seeds of IPPO-PS, the highest of the four learners. The three seeds of SAC-PS disagree on the clearing price: the mean clearing price of two of them is 270.3 and 272.9, against 238.1 EUR/MWh under truthful offers. Figure 31: Results on the 64 held-out episodes ordered by start time. Top: mean clearing price per episode. Bottom: profit per household per episode. Black is truthful offers; each learner is the mean of three seeds with a bootstrap 95% confidence interval. IPPO-NoPS lowers the price and raises the profit on every episode, while IPPO-PS and SAC-NoPS gain on 54 and 57 episodes. The seeds of SAC-PS disagree, and two of them raise the price. Figure 32 shows the actions each learner learns. The state of charge in each period of the day (the last 12 hours of each episode only: in every episode the learned policies first sell the energy stored at the initial state of charge of 0.5) splits the different learners into two behaviours according to whether they share parameters: IPPO-NoPS and SAC-NoPS charge in the hours of PV surplus and discharge in the evening, IPPO-NoPS by a large amount and SAC-NoPS by a small one; IPPO-PS and SAC-PS stay near the lowest state of charge. The learners also differ in their bidding prices. Under both IPPO learners, most sellers ask a price close to the export price. Buyers lean towards the retail tariff under IPPO-PS, but bid at all levels under IPPO-NoPS. Under SAC-NoPS, most asks and bids are close to one of the two grid prices, and few lie in between. The three seeds of SAC-PS learn very different prices, so SAC-PS has the widest confidence intervals in the figure. Figure 32: Learned actions of the four learners on the 64 held-out episodes (mean of three seeds, bootstrap 95% confidence interval). Top: state of charge averaged over households by hour of day (last 12 hours of each episode only). The band marks the PV surplus hours, those in which the community injects more than it draws on most days. Middle and bottom: asks and bids, scaled from the export price (0) to the retail tariff (1). E.4.4 Arbitrage gain and participation This section asks how the gain from arbitrage changes with the number of households arbitraging at the same time. The fixed schedule of Section 5.3 charges at 2.62 kW from 11:00 to 15:00 and discharges at 2.62 kW from 18:00 to 22:00, with all households offering truthfully; it is replayed on the 64 held-out episodes with k households on the schedule and the rest idle (Figure 33). The gain is the change in profit per household per episode (EUR) relative to all batteries idle. Arbitrage pays only when few households take part, as shown in the left subfigure in Figure 33. A single arbitraging household gains 1.129 EUR per episode, but the gain falls as more households join, turns negative at 210 to 240 households, and reaches −1.248-1.248 EUR when all 1 200 take part. The households that keep their batteries idle benefit instead: with 600 households arbitraging, each idle household gains 0.573 EUR. More specifically, arbitrage closes the price spread it relies on as shown in the right subfigure in Figure 33: charging at noon raises the noon price, and discharging in the evening lowers the evening price. Once about 240 households take part, the remaining spread no longer covers the round-trip losses and the degradation cost. This saturation comes from the market, not from the learner. For a single agent (household) the price is exogenous, but as more agents follow the same strategy, their actions change the price, which becomes endogenous. A profit measured for one agent, or against historical prices, therefore overstates what the strategy earns when many adopt it. Figure 33: On the 64 held-out episodes, k households follow a fixed arbitrage schedule (charge 11:00–15:00, discharge 18:00–22:00) and the rest keep their batteries idle. Left: gain per household, for those arbitraging and those idle, relative to all batteries idle. Right: mean clearing price in the charging and discharging windows. The horizontal axis of both panels is k. Arbitrage pays while few households take part and turns into a loss at 210 to 240 households, before the two window prices cross. E.5 M5: Local Flexibility Market In this appendix section, we provide detailed market simulation settings and the discussion of the performance of the four learners. E.5.1 Market setting The test system is the Swiss distribution feeder 459_0, as shown in Figure 34. The system has 129 buses and 128 lines, and we scale its load by 1.50. We consider an episode as one day, and the market is cleared hour by hour. Each aggregator operates one battery, and aggregator i submits three quantity bids at period t, including a price πi,t _i,t, an offered quantity, and planned charging. The distribution system operator (the operator) clears the market by minimising the cost of satisfying demand, as detailed in Appendix A.5. Each winning aggregator is paid its own offer. The price floor is set to the replacement cost crep=149.88c^rep=149.88 CHF/MWh. Figure 34: Feeder 459_0 in Switzerland. Bottom right: its location in the Lake Geneva region. Left: the feeder, with a small red box marking the congested line. Top right: an aerial image of the congested line. Lines behind the congested line are blue, and only batteries on these lines can relieve it. The system requires that the bus voltage stays above 0.9 per unit and that the flow of every line keeps within its maximum value. The operator buys flexibility to keep voltage within its range and the congested line within its rating. The requirement is how much the flow on the line would exceed 98% of its rating if the operator bought nothing. This flow includes the load and the charging the aggregators plan, so the aggregators can raise the requirement themselves as detailed in Section E.5.3. In the Swiss test system, voltage stays within its limits all year, so the line is the only reason to buy. The flexibility market trades much less than the P2P energy market, because it buys only what is needed to keep the network within its limits. With all batteries idle, it buys on only 9 of the 36 test days, and never more than 0.43 MW, as shown in the left subfigure of Figure 35. Only batteries downstream of the line can relieve it: 12 of 24 in the 2040 placement and 16 of 34 in the 2050 placement. The 12 batteries alone can supply 2.51 MW, almost six times over the largest requirement as shown in the right subfigure of Figure 35. Supply therefore far exceeds demand. Figure 35: Demand and supply of flexibility. Left: the requirement in each hour of the 36 test days, with all batteries idle. Right: the power of each of the 24 batteries in the 2040 placement, with the 12 downstream of the congested line first. Table 8: Overview of Swiss test system data. Data Source Licence Training days Validation days Test days Swiss distribution feeder 459_0 SwissDNa, Zapparoli et al. (2025)b C BY 4.0 293 days 36 days (the 5th, 15th and 25th of each month) 36 days (the 1st, 11th and 21st of each month) Energy price (category C2, 2026) ElComc opendata.swiss Open use — — — a https://doi.org/10.5281/zenodo.15056134; b https://doi.org/10.1038/s41597-025-05830-y; c https://energy.ld.admin.ch/elcom/electricityprice. E.5.2 Agent training The training setup is that of Appendix D, and IPPO differs from Table 3 only in the following: each iteration takes 4 full-batch gradient steps, the learning rate of 3×10−43× 10^-4 decays linearly to zero over 300 iterations, the log standard deviation of exploration is annealed linearly from −0.5-0.5 to −3.0-3.0 over the same span, and the initialisation scale of the actor output layer is 0.01 rather than 1.0. Training uses the training days of Table 8; the stopping rule and the checkpoint evaluated on the test days are given in Table 4. On the test days the selected checkpoint acts with sampled actions, IPPO at a log standard deviation of −3.0-3.0, the end point of its annealing, and SAC at the log standard deviation of its own actor. Figure 36: Training return of the four learners against training iteration; lines are the geometric mean over the seeds of each configuration and shaded bands the bootstrap 95% confidence interval, each seed held at its last value after it stops. All four learners reach a plateau within about 100 iterations. E.5.3 Analysis of the procurement requirement Under truthful offers, the operator buys flexibility in about 5% of periods: 43 of 864 hours across the 36 test days in the 2040 placement, as shown in Figure 35. All four learners increase this share. More than 80% of daily procurement lies above what truthful offers buy, and aggregator returns track procurement as shown in Figure 37. The excess comes from planned charging, which raises baseline flow on the congested line above 98% of its rating and leads the operator to buy flexibility from batteries behind it. Cleared prices remain near the floor: on the test days, all IPPO seeds and 75% of SAC seeds clear within 1% of it. In the two 3.9 MW configurations, replacing truthful-offer aggregators with SAC-PS learners raises the share of periods with flexibility procurement from 5.0% to 18.7% with 24 aggregators, and from 4.9% to 15.4% with 34 aggregators (Figure 6(c)). With the monitor on, all six paired runs in the 34-aggregator configuration show daily procurement falling from about 3.2 to 0.15 MWh and daily return per learner falling from about 118 to 1 CHF. Figure 37: Source of the flexibility requirement over the 36 test days of the four configurations. Bars show truthful offers, the untrained shared and per-agent networks pooled in one bar, and the four learners at validation-selected checkpoints (seed mean and bootstrap 95% confidence interval). Two SAC-PS return bars exceed the axis and are cut at its top, with their means shown above. Rows show the share of periods with procurement; daily procurement split into what truthful offers also buy and the excess; and daily return per aggregator. Learners raise procurement above the truthful-offer level, and their returns rise with it. E.5.4 Analysis of a unilateral markup In the 24-aggregator, 3.9 MW configuration, aggregator 7’s return from a unilateral offer rises with the offer up to the highest offer swept, 9 900 CHF/MWh, as shown in Figure 6(b). With all others at the floor, raising its offer to 150.25 CHF/MWh (0.25% above the floor) lets the cheaper aggregators behind the congested line clear first. On the example day, aggregator 7 is awarded only from 13:00 to 18:00, when their offers cannot meet the requirement as shown in the right subfigure of Figure 38. Its award over the 36 test days falls from 8.86 to 2.53 MWh and remains at 2.53 MWh even at 9 900 CHF/MWh. Its return therefore drops sharply (just above the floor), then rises almost linearly with its offer. For the curves in Figure 6(b), return exceeds its floor value again at about 481 and 489 CHF/MWh in the two 3.9 MW configurations, but only at about 4 590 CHF/MWh in the 34-aggregator, 6.3 MW configuration. Near the floor, the bidding offer becomes nearly flat: π=crep[1+softplus⁡(απ)]π=c^rep[1+softplus(α^π)] gives dπ/dαπ=crepsigmoid(απ)dπ/dα^π=c^repsigmoid(α^π), which approaches zero as απα^π decreases as shown in the left subfigure of Figure 38. This weakens the return gradient reaching the price action and helps explain why learners stay near the floor. Figure 38: Unilateral offer increase. Left: offer above the floor against price action (log scale). Near the floor, the gap shrinks by about a factor of e for each unit decrease in the action. Cleared prices for all IPPO seeds are within 1% of the floor. Right: aggregator 7’s hourly award on the example day at offers of the floor, 150.25 and 9 900 CHF/MWh, with all other aggregators at the floor.

Instant research discovery