Paper deep dive
Multi-agent Reinforcement Learning for Low-Carbon P2P Energy Trading among Self-Interested Microgrids
Junhao Ren, Honglin Gao, Lan Zhao, Qiyu Kang, Gaoxi Xiao, Yajuan Sun
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 94%
Last extracted: 4/14/2026, 1:37:42 AM
Summary
The paper proposes a multi-agent reinforcement learning (MARL) framework, specifically using a Multi-Agent Proximal Policy Optimization (MAPPO) algorithm with LSTM networks, to optimize peer-to-peer (P2P) electricity trading among self-interested microgrids. The approach addresses uncertainties in renewable generation and demand by enabling microgrids to learn adaptive bidding strategies for price, quantity, and energy storage control, ultimately improving renewable utilization and economic welfare while reducing carbon emissions.
Entities (5)
Relation Signals (3)
Microgrid → participatesin → P2P Electricity Market
confidence 95% · microgrids participating in peer-to-peer (P2P) electricity trading
MRDAC → clears → P2P Electricity Market
confidence 94% · the multi-round double auction clearing (MRDAC) mechanism... is employed to efficiently clear the bids
MAPPO → optimizes → Microgrid
confidence 92% · MMAPPO algorithm is presented to address P2P trading problem among NN microgrids
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Uncertainties in renewable generation and demand dynamics challenge day-ahead scheduling. To enhance renewable penetration and maintain intra-day balance, we develop a multi-agent reinforcement learning framework for self-interested microgrids participating in peer-to-peer (P2P) electricity trading. Each microgrid independently bids both price and quantity while optimizing its own profit via storage arbitrage under time-varying main-grid prices. A market-clearing mechanism coordinating trades and promoting incentive compatibility is proposed. Simulation results show that the learned bidding policy improves renewable utilization and reduces reliance on high-carbon electricity, while increasing community-level economic welfare, delivering a win-win situation in emission reduction and local prosperity.
Tags
Links
- Source: https://arxiv.org/abs/2604.08973v1
- Canonical: https://arxiv.org/abs/2604.08973v1
Trouble viewing inline? Open PDF directly →
Full Text
34,251 characters extracted from source content.
Expand or collapse full text
Multi-agent Reinforcement Learning for Low-Carbon P2P Energy Trading among Self-Interested Microgrids †thanks: This work is partially supported by the A*Star SIMTech research fund. †thanks: Corresponding Author: Gaoxi Xiao. 1st Junhao Ren 2nd Honglin Gao 3rd Lan Zhao 4th Qiyu Kang 5th Gaoxi Xiao 6th Yajuan Sun Abstract Uncertainties in renewable generation and demand dynamics challenge day-ahead scheduling. To enhance renewable penetration and maintain intra-day balance, we develop a multi-agent reinforcement learning framework for self-interested microgrids participating in peer-to-peer (P2P) electricity trading. Each microgrid independently bids both price and quantity while optimizing its own profit via storage arbitrage under time-varying main-grid prices. A market-clearing mechanism coordinates trades and promotes incentive compatibility. Simulations show that the learned bidding policy improves renewable utilization and reduces reliance on high-carbon electricity, while increasing community-level economic welfare, delivering a win–win in emission reduction and local prosperity. I Introduction Carbon emissions, the primary driver of the greenhouse effect, have severely constrained global sustainable development. To address this challenge, the Paris Agreement [4] set the goal of achieving net-zero emissions by 2050. Among various mitigation strategies, integrating renewable energy into conventional power networks has become a key approach. Renewable sources are expected to account for 25%25\%–41%41\% of global electricity generation by 2040 [2]. In practice, most microgrids procure the majority of their electricity through the day-ahead (DA) market based on forecasts of demand and renewable generation. However, due to the inherent uncertainty of renewable resources and load dynamics, significant deviations between day-ahead schedules and real-time realizations are inevitable though equiped with energy storage system can relieve these fluctuations [5]. As a result, microgrids frequently experience unexpected surpluses or deficits during operation, which cannot be efficiently resolved by the day-ahead market alone. To address this issue, intra-day peer-to-peer (P2P) electricity trading has been widely advocated as a flexible mechanism to rebalance local supply and demand among neighboring microgrids [9]. Nevertheless, designing effective P2P trading mechanisms remains challenging due to the decentralized market structure and inherently complex, stochastic, and highly nonconvex characteristics. Under such conditions, microgrids are required to make sequential and adaptive bidding and storage control decisions based on limited local information and repeated market interactions. This naturally leads to a decentralized sequential decision-making problem, for which multi-agent reinforcement learning (MARL) provides a principled and scalable solution framework. MARL enables microgrids to learn adaptive decision policies directly from repeated market interactions, without relying on accurate system models or complete information, thereby endowing them with the ability to respond autonomously to evolving market conditions and endogenous strategic dynamics [10]. This capability is particularly essential for realizing self-organized, low-carbon, and intelligent P2P energy trading systems. In this paper, we propose a MARL-based double auction bidding strategy for microgrids to maximize their own interests in a P2P electricity market. With the help of an effective market clearing mechanism, the learned bidding strategy can protect individual interests while also reducing overall carbon emissions in the community and ensure social welfare. Our contributions are concluded as follows. • We formulate the multi-microgrid P2P market as a decentralized partially observable Markov decision process (DEC-POMDP) and solve it under a centralized training with decentralized execution (CTDE) paradigm to mitigate day-ahead dispatch errors. After P2P clearing, any remaining surplus can be actively fed into the main grid at the feed-in tariff (FIT) without requiring storage to be full. • We design a multi-agent proximal policy optimization (MAPPO) framework that incorporates a Long Short-Term Memory (LSTM) network to extract multi-scale temporal features, thereby improving bidding decisions and storage management. In addition, effective market-clearing mechanisms are integrated to ensure system-level efficiency while preserving individual economic incentives. Unlike prior studies, each microgrid in the proposed framework simultaneously decides price, quantity, and the storage parameter to maximize its own profit in this electricity market. The remainder of the paper is organized as follows. Section I reviews existing methods and technologies employed in the P2P market of multi-microgrid systems. Section I presents the overall system formulation, including the structure of the multi-microgrid systems and the electricity market framework. Section IV describes the proposed MARL-based model for the P2P market. Section V provides simulation results and discussions. Section VI concludes the paper and outlines potential directions for future research. I Related Works Existing studies on trading problems in multi-microgrid systems mainly rely on two methodological paradigms: model-based optimization and model-free optimization (specifically, MARL). Model-based optimization refers to frameworks in which the objective functions, constraints, and system dynamics are explicitly known or can be analytically modeled, allowing optimal decisions to be directly derived from the model. In [6], a two-stage stochastic game model was proposed to determine the optimal energy trading strategy under supply and demand uncertainties. A distributed algorithm was further developed to obtain the stochastic Nash equilibrium using the sample average approximation technique. A stochastic cartel game was introduced in [12] to address cooperative bidding strategies among multiple microgrids in P2P energy transactions considering renewable generation and energy storage. A collaborative optimization problem for capacity planning of distributed generation units and P2P trading was investigated in [13], where a risk-averse stochastic programming framework and Nash bargaining approach were adopted to mitigate renewable energy uncertainties and ensure fairness in trading. In addition, a low-carbon P2P trading model based on a master–slave nested mixed game was proposed in [7], which explicitly incorporated carbon emission flows as constraints in the game formulation. Model-free optimization, on the other hand, enables agents to learn their individual trading strategies and energy schedules from historical data to maximize operational profitability without requiring explicit knowledge of system dynamics. In [10], a double-sided auction-based P2P electricity market was formulated as a MARL problem, and the DA-MADDPG algorithm was proposed to maximize the profits of prosumers in a dynamic electricity market. A two-level MARL framework was developed in [8] to design dynamic pricing policies that facilitate efficient on-site energy trading while supporting decarbonization and grid security objectives. In [14], a multi-stage dispatching method incorporating day-ahead and intra-day scheduling was proposed to address uncertainties in electricity–hydrogen integrated energy systems. Moreover, [1] introduced a multi-microgrid framework combining carbon capture and utilization technologies with P2P energy trading, trained using the MAPPO algorithm to reduce both cost and carbon emissions. A recurrent neural network (RNN)-based MAPPO algorithm was further presented in [17] to learn one-to-one clearing policies in the coupled energy and carbon trading markets. I Problem Statement In this paper, we investigate a network of interconnected microgrids, as illustrated in Fig. 1. The network consists of a main grid and N distributed microgrids, each representing different types of regions—such as residential, industrial, and commercial areas. The main grid comprises heterogeneous generation resources, and electricity procurement across different market stages is considered for assessing both community welfare (economic payoff) and societal welfare (carbon emissions). The heterogeneity of these resources implies that the main grid can supply electricity with varying carbon emission intensities, which will be detailed in Section I-B. Each microgrid is equipped with electrical loads, photovoltaic (PV) generation, and electricity storage (ES), which provide operational flexibility and resilience to emergency. For tractability, this work focuses on economic performance and carbon accounting, while the validation of physical power-flow constraints (e.g., line flows and AC power-flow equations) is omitted. Figure 1: The two-stage electricity market framework. I-A Microgrid System For the microgrid i, let Li,tL_i,t denote its electricity demand and Gi,tG_i,t denote its photovoltaic renewable electricity output at hour t. In operational process, microgrids prioritize using ES to balance their own deficits and surplus after P2P trading. This means ES acts as a power supply to discharge electricity if generation and day-ahead purchases are insufficient, otherwise it is the load to charge the oversupply. Therefore, the state of charge (SoC) of ES for each microgrid i at hour t is given by: Ei,t+1=Ei,t+βchrTi,tESΔtEi,if Ti,tES≥0,Ei,t+Ti,tESΔtβdisEi,if Ti,tES<0,E_i,t+1= casesE_i,t+ _chrT^ES_i,t tE^i,&if T^ES_i,t≥ 0,\\ E_i,t+ T^ES_i,t t _disE^i,&if T^ES_i,t<0, cases (1) where Ti,tEST^ES_i,t and EiE^i denote the electricity power and current capacity of ES for microgrid i at hour t, respectively. The initial capacity of microgrid i is denoted by E0iE^i_0. βchr _chr and βdis _dis refer to the charge and discharge efficiency of ES for microgrid i. The electricity power Ti,tEST^ES_i,t and SoC Ei,t+1E_i,t+1 should satisfy the following inequalities: T¯iES T^ES_i ≤Ti,tES≤T¯i,tES ≤ T^ES_i,t≤ T^ES_i,t (2) Ei,min E_i, ≤Ei,t≤αe,iEi,max ≤ E_i,t≤ _e,iE_i, where T¯iES T^ES_i and T¯i,tES T^ES_i,t denote the hourly minimum and maximum electricity power of ES for microgrid i. Ei,minE_i, , Ei,maxE_i, and αe _e are the minimum, maximum storage level, and control parameters of ES, respectively. Without loss of generality, we allows the storage device to simultaneously absorb and supply power within the same trading interval, as commonly observed in practical multi-port or bidirectional converter-based energy storage systems. I-B Electricity Market Framework In this work, we consider a two-stage electricity market for microgrid participants, consisting of a day-ahead (DA) stage and an intra-day stage (see Fig. 1). The day-ahead stage is used to provide the majority of the electricity of microgrids based on forecasts of demand and renewable generation, while the intra-day P2P market is to rebalance surpluses and deficits due to uncertainties of forecasts. I-B1 DA market mechanism The day-ahead market is widely recognized as an efficient clearing mechanism for electricity and other commodities. In this stage, the main grid announces three distinct price signals to microgrids: the day-ahead price pdaGp^G_da, the emergency price peGp^G_e, and the FIT pfGp^G_f. Since emergency electricity is typically generated by fuel-powered or diesel units with high carbon emissions, and excessive feed-in electricity from distributed sources may threaten grid stability, it is reasonable to assume that pfG≤pdaG≤peG,p^G_f≤ p^G_da≤ p^G_e, (3) which penalizes both emergency electricity usage and excessive feed-in to the main grid. Without loss of generality, it is assumed that pfGp^G_f is time-invariant while peGp^G_e is time-variant. By differentiating these prices, the main grid can ensure reliable baseline generation while simultaneously encouraging the integration of renewable resources. Each microgrid i then forecasts its hourly demand and renewable generation for the following day. However, due to the limited capacity of local renewable units and inevitable forecast errors, some microgrids may still experience electricity deficits that must be compensated by purchasing electricity from the main grid in the day-ahead market. The day-ahead procurement policy of microgrid i at hour t is modeled as qi,tda=max0,αi(L¯i,t−G¯i,t),q^da_i,t= \0, _i( L_i,t- G_i,t)\, (4) where L¯i,t≥0 L_i,t≥ 0 and G¯i,t≥0 G_i,t≥ 0 denote the forecast load and local renewable generation of microgrid i at hour t∈[0,24)t∈[0,24). Positive values of qi,tdaq^da_i,t indicate purchases from the main grid; sales to the main grid are excluded from the day-ahead stage and are addressed after the intra-day P2P market. αi _i denotes the procurement policy factor of microgrid i in the day-ahead stage. In this paper, the day-ahead procurement policy factor αi _i of the microgrid i remains fixed, and therefore we focus on using the intra-day P2P market to balance residual surpluses and deficits to improve the’ economic outcomes of the participants. I-C P2P market mechanism Due to uncertainties on forecasts of generation and loads, a double auction clearing mechanism is thereby needed to promote internal balance among these microgrids and the integration of renewable energy. In this stage, microgrids should make the decision to buy/sell how much electricity and at what price given the observed information. At each time step in the P2P market, three sequential events occur for each microgrid i: (1) submission of a quotation (pi,qi)(p_i,q_i), (2) matching of quotations through the P2P market mechanism, (3) electricity exchange and payment settlement. In the quotation (pi,qi)(p_i,q_i), pfG≤pi≤peGp^G_f≤ p_i≤ p^G_e represents the unit price at which the microgrid intends to buy/sell electricity, and 0≤qi≤q¯i,t0≤ q_i≤ q_i,t denotes the amount of electricity it wishes to trade while q¯i q_i represents the current maximum demand/supply electricity of microgrid i. This enables microgrids to offer more diverse quotations in transactions, thereby maximizing their own benefits. This bidding strategy allows microgrids to submit flexible and diverse offers in the market, thereby optimizing their own operational benefits and reflecting self-interested trading behavior. In the matching stage, the multi-round double auction clearing (MRDAC) mechanism proposed in [3] is employed to efficiently clear the bids. In the MRDAC mechanism, the auctioneer first collects all quotations, where nbn_b quotations are submitted by buyers and nsn_s quotations are submitted by sellers. The quotations from sellers are then arranged in ascending order of their offered prices, while those from buyers are arranged in descending order of their bid prices. For instance, suppose the auctioneer receives quotations from three sellers: (6,2)(6,2), (1,3)(1,3), and (4,2)(4,2), where the first element denotes the unit price and the second the quantity. The matching priority for these seller quotations is given by (1,3)>(4,2)>(6,2)(1,3)>(4,2)>(6,2), meaning that sellers offering lower prices are prioritized. A similar rule applies inversely to the buyer side. After ranking the quotations from buyers and sellers, the auctioneer matches the highest-priority seller i with the highest-priority buyer j. The transaction is then settled at the average price pi,j=pi+pj2=pj,i,p_i,j= p_i+p_j2=p_j,i, (5) and the traded quantity is determined by qi,j=min(qi,qj)=qj,i.q_i,j= (q_i,q_j)=q_j,i. (6) If a seller only sells part of its quoted quantity, the remaining quota is moved to the end of the sellers’ list to await subsequent matching rounds, while the next seller in priority begins trading with the next buyer. The same procedure applies symmetrically to buyers who do not fulfill their desired purchase quantities. During the matching process, a transaction between seller i and buyer j can be settled only if pi≤pjp_i≤ p_j. Otherwise, one of the two participants is randomly moved to the end of its respective list to await the next matching round. In addition, a seller (or buyer) is removed from the waiting list once its quoted quantity has been fully allocated, or when its unit price becomes higher (or lower) than all remaining buyers (or sellers). The matching process terminates when there are no remaining quotations in the waiting list for either buyers or sellers, or for both. At the end of the matching process, all microgrids with electricity deficits must purchase emergency electricity from the main grid at the price peGp^G_e, while those with electricity surpluses sell their excess energy to the main grid at the price pfGp^G_f. Overall, the MRDAC mechanism with an average-price settlement effectively enhances the economic incentives for small-scale microgrids to participate in the P2P market, as it is generally more cost-efficient than trading directly with the main grid after the P2P stage. IV MARL-based Bidding Strategy in Intra-day P2P Market In this section, the intra-day P2P trading problem is formulated as a DEC-POMDP, which is typically solved using MARL algorithms. In this work, we adopt a CTDE paradigm and propose a MAPPO framework integrated with a LSTM architecture (MMAPPO), which enables each microgrid to learn an autonomous bidding policy from repeated market interactions. IV-A Model of bidding strategy for microgrid i In the intra-day P2P market, each microgrid i aims to maximize its own economic benefit by optimizing its bidding strategy subject to operational and market constraints. The bidding strategy model for microgrid i can be formulated as follows: maxpi,t,qi,t,αe,i,t∑t=1T _p_i,t,\,q_i,t,\, _e,i,t _t=1^T (Pi,tG+Pi,tp2p) (P^G_i,t+P^p2p_i,t ) (7) s.t.Li,t+Ti,tES+qi,tfit+qi,ts= s.t. L_i,t+T^ES_i,t+q^fit_i,t+q^s_i,t= Gi,t+qi,tda+qi,tb+qi,te, G_i,t+q^da_i,t+q^b_i,t+q^e_i,t, (8) pfG≤pi,t p^G_f≤ p_i,t ≤peG, ≤ p^G_e, (9) −q¯i,t,s≤qi,t - q_i,t,s≤ q_i,t ≤q¯i,t,b, ≤ q_i,t,b, (10) 0≤αe,i,t 0≤ _e,i,t ≤1, ≤ 1, (11) Eq.(2). Eq.~ eq: ESconstraints. Here, Pi,tGP^G_i,t denotes the operational profit of microgrid i related to the main grid, and Pi,tp2pP^p2p_i,t represents the profit obtained from P2P trading. The variables qi,tfitq^fit_i,t, qi,teq^e_i,t, qi,tsq^s_i,t, and qi,tbq^b_i,t correspond to the quantities of feed-in electricity to the main grid, emergency electricity purchased from the main grid, electricity sold, and electricity bought in the P2P market, respectively. The parameters q¯i,t,s q_i,t,s and q¯i,t,b q_i,t,b denote the maximum supply and demand capacities of microgrid i when acting as a seller or buyer, respectively. In addition, for each microgrid i, Pi,tGP^G_i,t only considers the profit obtained from feed-in and emergency transactions, since the expected day-ahead expense remains constant when αi _i is fixed. Therefore, Pi,tG=pfGqi,tfit−peGqi,te.P^G_i,t=p^G_fq^fit_i,t-p^G_eq^e_i,t. (12) Similarly, the profit of microgrid i from P2P trading is given by Pi,tp2p=∑j∈nbpi,j,tqi,j,t−∑j∈nspi,j,tqi,j,t,P^p2p_i,t= _j∈ n_bp_i,j,tq_i,j,t- _j∈ n_sp_i,j,tq_i,j,t, (13) where nbn_b and nsn_s denote the sets of matched buyers and sellers, respectively, and nb+ns≤Nn_b+n_s≤ N. IV-B Model of DEC-POMDP The P2P trading problem can be formulated as a DEC-POMDP. A DEC-POMDP is defined by the tuple (,,i,i,ℛi,,γ), (N,S,A^i,O^i,R^i,P,γ ), where N, S, and P denote the set of agents, the global state space of the environment, and the state transition function, respectively. The components iA^i, i:×→[0,1]O^i:S×A×S→[0,1], and ℛi:×→ℝR^i:S×A represent the action space, the observation space, and the reward function for agent i∈i , respectively. The discount factor γ∈(0,1)γ∈(0,1) accounts for the agent’s preference for immediate versus future rewards. IV-B1 Agents Each agent i corresponds to a distinct microgrid in the system, where all agents collectively participate in the P2P trading process. IV-B2 Observation At time step t, the partial observation available to agent i is defined as ti=mt,Ei,t,hi,t,t,O^i_t=\m_t,E_i,t,h_i,t,t\, (14) where mtm_t represents the embedded market information reflecting the overall supply–demand condition in the P2P market; Ei,tE_i,t denotes the current energy storage (ES) capacity of microgrid i; and hi,t=qi,zda,L¯i,z,G¯i,z,pe,zGz=t−δ1t+δ2h_i,t=\q^da_i,z, L_i,z, G_i,z,p^G_e,z\_z=t- _1^t+ _2 is the noisy temporal observation embedding that includes the day-ahead procurement quantity, forecasted load, forecasted renewable generation, and the emergency electricity price within the time interval (t−δ1,t+δ2)(t- _1,t+ _2). The temporal window allows each agent to incorporate both historical and predictive information into its local decision-making process. IV-B3 State The global state of the environment at time t is denoted by t=s1,t,s2,t,…,sN,tS_t=\s_1,t,s_2,t,…,s_N,t\, which consists of all possible environment states determined by the agents’ actions and rewards. IV-B4 Action At each time step t, agent i selects an action ti=pi,t,qi,t,αe,i,t,A^i_t=\p_i,t,q_i,t, _e,i,t\, where pi,tp_i,t and qi,tq_i,t represent the bidding price and quantity in the P2P market, respectively, and αe,i,t _e,i,t denotes current ES control parameter of microgrid i. The joint action of all agents is t=t1,…,tNA_t=\A^1_t,…,A^N_t\. IV-B5 Reward After the environment transitions from tS_t to t+1S_t+1 due to the joint action tA_t, agent i receives an immediate reward ℛtiR^i_t defined as ℛti=Pi,tG+Pi,tp2p.R^i_t=P^G_i,t+P^p2p_i,t. (15) The proposed design seeks to maximize operational profitability through the integration of renewable energy sources while simultaneously reducing reliance on electricity resources characterized by high carbon emissions. IV-C MMAPPO-driven intra-day P2P market In this work, the MMAPPO algorithm is presented to address P2P trading problem among N microgrids. The MMAPPO framework consists of N decentralized policy networks, denoted by πθii=1N\ _ _i\_i=1^N, and a centralized critic network VϕV_φ, which are used to approximate the policy and value functions of the microgrids, respectively. To enhance temporal representation, an LSTM module is incorporated for sequential observation embedding, while a sinusoidal periodic encoder is employed to capture daily cyclical patterns. The overall training process of MMAPPO is summarized in Algorithm 1. For further architectural and implementation details of the policy and critic networks, readers are referred to [15]. Algorithm 1 MMAPPO for P2P Microgrids 1:Initialize simulation parameters. 2:Initialize policy networks πθii=1N\ _ _i\_i=1^N, critic network VϕV_φ, buffers ii=1N\D_i\_i=1^N. 3:for episode=1episode=1 to NepN_ep do 4: Reset the environment and Clear all buffers. 5: for t=1t=1 to T do 6: Actions sampling ti∼πθi(⋅|ti)i=1N\A^i_t _ _i(·|O^i_t)\_i=1^N. 7: Environment steps →t+1 _t+1, ℛtii=1N\R^i_t\_i=1^N. 8: Store oti,ati,ℛti,ot+1i,st,st+1→i\o^i_t,a^i_t,R^i_t,o^i_t+1,s_t,s_t+1\ _i. 9: end for 10: for epoch=1epoch=1 to K do 11: for each mini-batch ℬB from buffers do 12: Normalized advantages Atii=1N←Vϕ(st)\A^i_t\_i=1^N← V_φ(s_t). 13: Update each θi _i via per-agent policy loss. 14: Update shared ϕφ via critic loss. 15: end for 16: end for 17:end for V Numerical Simulation V-A Environment Setup We consider an electricity market comprising four microgrids interconnected through a shared main grid. The parameters of these four microgrids are listed in Table I. In the simulation, it is assumed that there is no energy loss during charging and discharging processes, i.e., βdis=βchr=1 _dis= _chr=1. The parameters of the observable time window for each microgrid are set as δ1=1 _1=1 and δ2=6 _2=6. Furthermore, the daily renewable generation and electricity load profiles of the four microgrids are sampled from the normalized data of four Australian residential households reported in [11]. The feed-in tariff (FIT) price of the main grid is fixed at 2$/kWh2~ /kWh, while the emergency price varies dynamically over time within the range of [15,35]$/kWh[15,35]~ /kWh. TABLE I: Simulation Parameters for the 4 Microgrids Parameter Grid 1 Grid 2 Grid 3 Grid 4 LimaxL_i^max (kWh) 25 6 40 5 GimaxG_i^max (kWh) 5 7 10 15 EmaxE_max (kWh) 8 15 15 30 T¯iES T^ES_i (kW) 4 5 8 10 −T¯iES- T^ES_i (kW) 4 5 8 10 E0E_0 (kWh) 0 2 0 20 V-B Baselines We compare the proposed framework against three existing multi-agent reinforcement learning (MARL) algorithms: (1) MIPPO — an improved independent PPO algorithm with an LSTM-based architecture; (2) MAPPO-one — a MAPPO variant that employs an LSTM encoder but shares a single global critic across all agents; and (3) MAPPO-s — a simplified MAPPO-one configuration with a smaller critic network. In addition, we evaluate the performance of different market-clearing mechanisms (Greedy and VDA [16]) against the existing MRDAC mechanism to assess the advantages of the proposed framework. Figure 2: Training curves of different MARL algorithm ++ MRDA. V-C Results & Discussions As shown in the training results in Fig. 2, the proposed MMAPPO algorithm achieves the highest total reward (approximately −15-15 on average) and demonstrates the fastest convergence within about 1000 episodes, outperforming MAPPO-s and MIPPO by roughly 20$20 per hour in the steady stage. In contrast, MIPPO quickly improves at the early stage but soon becomes trapped in a local optimum, while MAPPO-one converges more slowly, requiring around 2000 episodes. The performance of MAPPO-s declines significantly due to the limited capacity of its critic network. TABLE I: Daily Performance Comparison of MMAPPO under Different Market Mechanisms Metric MRDA VDA Greedy Total Profit ($) −123.81-123.81 −185.53-185.53 −221.02-221.02 Emergency Purchase (kWh) 30.3330.33 34.0034.00 34.3734.37 Total FIT (kWh) 18.9518.95 32.2432.24 43.7443.74 P2P Trade Volume (kWh) 10.5810.58 9.969.96 9.559.55 Average SoC (kWh) 16.5616.56 16.6916.69 16.6016.60 For performance comparison of 44 microgrids under different market-clearing mechanisms, MMAPPO was trained under each mechanism for approximately 50005000 episodes and evaluated over 960960 testing days. The average results are summarized in Table I. The table shows that the MRDAC mechanism under MMAPPO achieves the best economic and operational performance among the three mechanisms. Specifically, MRDAC yields the highest total profit (improving by 50%50\% over VDA and 78%78\% over Greedy), and the lowest emergency purchase (reducing by 12%12\% and 13%13\% compared with VDA and Greedy, respectively). Moreover, the MRDAC mechanism significantly increases the trading volume in the P2P market. These results demonstrate that MRDA, when integrated with MMAPPO, effectively enhances renewable energy utilization and maintains strong individual trading incentives. VI Conclusion In this paper, we proposed a double-auction-based P2P trading framework for multi-microgrid systems using the MMAPPO algorithm. The trading problem under the MRDAC mechanism was formulated as a DEC-POMDP, enabling each microgrid to make autonomous and coordinated decisions under uncertainty. By integrating MAPPO with an LSTM architecture, the framework achieved efficient and reliable decision-making through multi-scale feature extraction. Simulation results verified that the proposed MARL-based mechanism enhances the adaptability and intelligence of distributed energy management. Future work will extend this study to optimize day-ahead procurement policy and address uncertainty caused by renewable generation faults. References [1] M. Chen, Z. Shen, L. Wang, and G. Zhang (2024) Combined carbon capture and utilization with peer-to-peer energy trading for multimicrogrids using multiagent proximal policy optimization. IEEE Trans. Control Netw. Syst. 11 (4), p. 2173–2186. External Links: ISSN 2325-5870, Document Cited by: §I. [2] X. Deng and T. Lv (2020) Power system planning with increasing variable renewable energy: A review of optimization models. J. Cleaner Prod. 246, p. 118962. External Links: ISSN 09596526, Document Cited by: §I. [3] H. Haggi and W. Sun (2021) Multi-Round Double Auction-Enabled Peer-to-Peer Energy Exchange in Active Distribution Networks. IEEE Trans. Smart Grid 12 (5), p. 4403–4414. External Links: ISSN 1949-3053, 1949-3061, Document Cited by: §I-C. [4] C. A. Horowitz (2016) Paris agreement. Int. Leg. Mater. 55 (4), p. 740–755. Cited by: §I. [5] S. Kwon, L. Ntaimo, and N. Gautam (2017) Optimal Day-Ahead Power Procurement With Renewable Energy and Demand Response. IEEE Trans. Power Syst. 32 (5), p. 3924–3933. External Links: ISSN 0885-8950, 1558-0679, Document Cited by: §I. [6] C. Li, Y. Xu, X. Yu, C. Ryan, and T. Huang (2017) Risk-averse energy trading in multienergy microgrids: A two-stage stochastic game approach. IEEE Trans. Ind. Informat. 13 (5), p. 2620–2630. External Links: ISSN 1941-0050, Document Cited by: §I. [7] Z. Liang and L. Mu (2024) Multi-agent low-carbon optimal dispatch of regional integrated energy system based on mixed game theory. Energy 295, p. 130953. External Links: ISSN 03605442, Document Cited by: §I. [8] R. May and P. Huang (2023) A multi-agent reinforcement learning approach for investigating and optimising peer-to-peer prosumer energy markets. Appl. Energy 334, p. 120705. External Links: ISSN 03062619, Document Cited by: §I. [9] T. Morstyn, N. Farrell, S. J. Darby, and M. D. McCulloch (2018) Using peer-to-peer energy-trading platforms to incentivize prosumers to form federated power plants. Nat Energy 3 (2), p. 94–101. External Links: ISSN 2058-7546, Document Cited by: §I. [10] D. Qiu, J. Wang, J. Wang, and G. Strbac (2021) Multi-agent reinforcement learning for automated peer-to-peer energy trading in double-side auction market. In Proc. 30th Int. Joint Conf. Artif. Intell., p. 2913–2920. External Links: Document, ISBN 978-0-9992411-9-6 Cited by: §I, §I. [11] E. L. Ratnam, S. R. Weller, C. M. Kellett, and A. T. Murray (2017-09) Residential load and rooftop PV generation: an Australian distribution network dataset. Int. J. Sustain. Energy 36 (8), p. 787–806. External Links: ISSN 1478-6451, Document Cited by: §V-A. [12] L. Wang, Y. Zhang, W. Song, and Q. Li (2022) Stochastic cooperative bidding strategy for multiple microgrids with peer-to-peer energy trading. IEEE Trans. Ind. Informat. 18 (3), p. 1447–1457. External Links: ISSN 1941-0050, Document Cited by: §I. [13] Z. Wang, H. Hou, B. Zhao, L. Zhang, Y. Shi, and C. Xie (2024) Risk-averse stochastic capacity planning and P2P trading collaborative optimization for multi-energy microgrids considering carbon emission limitations: An asymmetric nash bargaining approach. Appl. Energy 357, p. 122505. External Links: ISSN 03062619, Document Cited by: §I. [14] Z. Yang, Z. Ren, H. Li, Z. Sun, J. Feng, and W. Xia (2024-10) A multi-stage stochastic dispatching method for electricity-hydrogen integrated energy systems driven by model and data. Appl. Energy 371, p. 123668. External Links: ISSN 03062619, Document Cited by: §I. [15] C. Yu, A. Velu, E. Vinitsky, J. Gao, Y. Wang, A. Bayen, and Y. WU (2022) The surprising effectiveness of PPO in cooperative multi-agent games. In Adv. Neural Inf. Process. Syst., Vol. 35, p. 24611–24624. Cited by: §IV-C. [16] Z. Zhao, C. Feng, and A. L. Liu (2023) Comparisons of auction designs through multiagent learning in peer-to-peer energy trading. IEEE Trans. Smart Grid 14 (1), p. 593–605. External Links: ISSN 1949-3061, Document Cited by: §V-B. [17] Y. Zhou, Z. Ma, T. Wang, J. Zhang, X. Shi, and S. Zou (2024) Joint energy and carbon trading for multi-microgrid system based on multi-agent deep reinforcement learning. IEEE Trans. Power Syst. 39 (6), p. 7376–7388. External Links: ISSN 1558-0679, Document Cited by: §I.