Paper deep dive

Deep Reinforcement Learning for Optimizing Energy Consumption in Smart Grid Systems

Abeer Alsheikhi, Amirfarhad Farhadi, Azadeh Zamanifar

Year: 2026Venue: arXiv preprintArea: cs.LGType: PreprintEmbeddings: 79

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 90%

Last extracted: 7/20/2026, 10:49:57 PM

Summary

This study addresses the sample inefficiency and computational cost of using Reinforcement Learning (RL) for Optimal Power Flow (OPF) in smart grids by introducing Physics-Informed Neural Networks (PINNs) as surrogate models. The authors demonstrate that PINNs, which incorporate physical laws like KKT conditions, outperform data-driven surrogates (e.g., XGBoost, Decision Trees) by enabling strong RL policy convergence without direct interaction with the true simulator. The approach accelerates RL training by 50% while maintaining performance comparable to the original complex simulator.

Entities (9)

Smart Grid · domain · 99%Physics-informed neural networks · method · 98%Reinforcement Learning · method · 95%Optimal Power Flow · problem · 92%Surrogate Model · concept · 90%Karush-Kuhn-Tucker conditions · constraint · 88%PPO · algorithm · 87%Gym-ANN · tool · 85%XGBoost · algorithm · 82%

Relation Signals (6)

Physics-informed neural networks → usedfor → Surrogate Model

confidence 95% · In this study, these challenges are addressed through the use of Physics-Informed Neural Networks (PINNs), which can replace conventional and costly smart grid simulators.

Reinforcement Learning → appliedto → Optimal Power Flow

confidence 93% · Although Reinforcement Learning (RL) has been proposed for solving Optimal Power Flow (OPF) problems...

Physics-informed neural networks → accelerates → Reinforcement Learning

confidence 92% · The results demonstrate that using PINN surrogates can accelerate training by 50% compared to RL training without a surrogate.

Physics-informed neural networks → outperforms → Data-Driven Surrogates

confidence 90% · The PINN-based surrogate was compared to other benchmark Data- Driven surrogates... it was found that the PINN surrogate is the only approach considered in this context that can obtain a strong RL policy even without having samples from the true simulator.

Karush-Kuhn-Tucker conditions → integratedinto → Physics-informed neural networks

confidence 88% · Specifically, the Karush-Kuhn-Tucker (KKT) conditions are integrated into the surrogate formulation to ensure the feasibility of generator and energy storage operations.

Smart Grid → contains → Optimal Power Flow

confidence 85% · The energy management problem in the context of smart grids is inherently complex... solving Optimal Power Flow (OPF) problems

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:The energy management problem in the context of smart grids is inherently complex due to the interdependencies among diverse system components. Although Reinforcement Learning (RL) has been proposed for solving Optimal Power Flow (OPF) problems, the requirement for iterative interaction with an environment often necessitates computationally expensive simulators, leading to significant sample inefficiency. In this study, these challenges are addressed through the use of Physics-Informed Neural Networks (PINNs), which can replace conventional and costly smart grid simulators. The RL policy learning process is enhanced so that convergence can be achieved in a fraction of the time required by the original environment. The PINN-based surrogate is compared with other benchmark data-driven surrogate models. By incorporating knowledge of the underlying physical laws, the results show that the PINN surrogate is the only approach considered in this context that can obtain a strong RL policy even without access to samples from the true simulator. The results demonstrate that using PINN surrogates can accelerate training by 50% compared to RL training without a surrogate. This approach enables the rapid generation of performance scores similar to those produced by the original simulator.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

77,402 characters extracted from source content.

Expand or collapse full text

Deep Reinforcement Learning for Optimizing Energy Consumption in Smart Grid Systems Abeer Alsheikhi 1 ,Amirfarhad Farhadi 2 , Azadeh Zamanifar 3 1,2 School of Computer Engineering, Iran university of Science and Technology, Tehran, Iran 3 Department of Computer Engineering, SR.C, Islamic Azad university, Tehran, Iran Graphical Abstract Highlight ▪ Novel PINN method for Smart Grid surrogates, addressing RL sample efficiency. ▪ PINN surrogates outperform data-driven models in RL training for Smart Grid management. PINN achieves 50 % faster RL policy training with comparable performance. ▪ Efficient RL training without costly simulator samples via physics-based surrogates. Abstract The energy management in the context of smart grids is a kind of real-world problem complex by nature and due to the interrelatedness among diverse elements that are part of these scenarios. Although Reinforcement Learning (RL) has been proposed for solving Optimal Power Flow (OPF) problems, the requirement of iterative interaction with an environment can force it to employ computationally complex simulators, generating significant sample inefficiency. In this study, these obstacles are overcome by the use of Physics-Informed Neural Networks (PINNs), which can replace conventional and expensive smart grid simulators. The RL policy learning is fine-tuned so that the convergent results be reached during a fraction of the time taken by the original environment. The PINN-based surrogate was compared to other benchmark Data- Driven surrogates. With the knowledge of the underlying physical laws, it was found that the PINN surrogate is the only approach considered in this context that can obtain a strong RL policy even without having samples from the true simulator. The results show that using PINN surrogates can accelerate training by 50% as compared to RL training without surrogate. This method allows for fast generation of performance scores similar to the original simulator output. Keywords: Smart grids control; Reinforcement learning; Physics-informed neural networks; Active network management; Optimal power flow; Surrogate models; Renewable energy 1.Introduction The development of "smart grids" is now driving the modernization of our electricity networks. These cutting-edge systems are a direct reaction to the worldwide push to reduce pollution, more efficiently utilize energy and maintain consistent electricity supply through load adjustment. The European Union is leading this transition and has targeted that at least 43% of its energy should be generated from renewables by 2030 [1]. They are already doing so to some extent: green energy reached 32.1% in 2021 [2]. Renewable energy sources like wind and sun may not always be generating power when we need them, so today’s society also requires a grid that can effectively prepare for and handle that variation. Such networks must be able to employ energy storage and demand side management tools in a controllable and intuitive manner, while remaining affordable and reliable for the general public [3]. This evolution has already gained significant traction in the EU with more than 13 countries rolling out smart meters to over 81% of homes. This transformation enables local energy communities, and flexible power markets [4], to further develop. And the digital revolution, which has already given us smart meters, enables more sophisticated methods for this “demand response.” Outside of energy systems, we have seen GenerativeAI used to automate administrative and workflow activities that are time-consuming in other high-stakes domains, thereby lightening the burden of operations while raising fairness, safety, and trust issues—suggesting a need for prudence when deploying AI in real-world decision support systems [54]. They do so by load-balance grid, decreasing the electricity usage at peak hours, and moving them to off-peak hours [5]. Yet, synthesizing digital models for these networks is a major challenge as we are now integrating more and more decentralized generation and storage, which has its counterpart at the Big Data level. To determine how best to operate such modern grids, scientists are turning to simulators on which they can try out data-driven Machine Learning (ML) algorithms. Classical mathematical equations have been the standard, but augmenting them with ML models like decision trees or neural networks has emerged as a powerful tool for generalizing across such diverse phenomena from individual lithium-ion batteries [7,8] to entire national power systems [9,10]. Reinforcement Learning (RL) has the potential for success in this domain as it provides a “trial- and-error” reward mechanism to identify best possible techniques for managing energy storage and power distribution [11,12]). Artificial Neural Networks as Other Predictive Modelling Tool When combined with RL, ANNs or other predictive modelling tools allow researchers to smartly design policies for consumer demand management [13,14]. The challenge is that RL requires constant engagement in the system to learn from it. If we attempted to teach such AI agents on an actual, operational electricity network, they could make a mistake that causes the grid to black out or equipment damage. This is why safe simulation environments are critically important for the training. More generally recent work on Generative AI has highlighted that while such models can be very flexible the deployment of them at scale in safety-critical applications must have frameworks to govern privacy, security and ethical risks [55]. In our work we developed a realistic grid model using Gym-ANN. The benefit of this allows for the incorporation of real-world physical constraints, and various types of physical hardware such as local generators and storage devices enabling training of RL policies that are fully deployable into practice [15]. The disadvantage is that the simulations are very low as one deals with complex patterns and high- resolution grids [16]. To circumvent this bottleneck, we suggest a novel methodology: Physics- Informed Neural Network (PINN). This serves as a high speed ‘’stand-in’ for the original simulator, yielding identical results within as small fraction of the time. With this alternative model, we are able to train a RL policy without the actual grid environment. By this strategy, the policy can achieve the same near-optimal performance level as the original system but at much less time. Our findings stress the importance of taking into account the true physical laws of a system when constructing a model. We demonstrated it by comparing our approach with other modern “data-only” models: our method works unlike theirs thanks to its underlying physics and not just raw data.” We show that by employing these physics-based substitutes (or PINNs), it is possible to achieve a 50% reduction in the training duration as compared with directly training within the real simulation. This means we can get the results of equal quality in half the time. This advance is one of the most significant steps toward smart grids of the future, because it will allow us to much more easily control these very high-voltage systems and be able to integrate new technologies like solar energy in with the older infrastructure that we’ve always used. To tackle the computational challenges and sample inefficiency associated with reinforcement learning-based optimal power flow solutions in smart grids, this study introduces a physics- informed surrogate modeling framework. A surrogate model based on Physics-Informed Neural Networks (PINNs) is developed to replicate smart grid dynamics while explicitly incorporating physical constraints derived from power system laws and the operational limits of devices. Specifically, the Karush-Kuhn-Tucker (KKT) conditions are integrated into the surrogate formulation to ensure the feasibility of generator and energy storage operations. This approach allows reinforcement learning policies to be trained without direct interaction with the original simulator, significantly reducing computational costs. A systematic comparison with state-of-the- art data-driven surrogate models shows that physics-informed surrogates offer enhanced robustness and generalization, particularly in regions of the state space that are either poorly sampled or completely unseen. Quantitative results indicate that the proposed method achieves comparable control performance while cutting reinforcement learning training time by approximately 50%, underscoring its potential for scalable and reliable smart grid control. 2. Background 2.1. Smart grid: Optimal power flow and control systems Conventional attempts for solving OPF problems adopted numerically based approaches, most were based on the use of Newton-Raphson method and some other mathematical programs such as recursive-based programming, Lagrange multipliers and linear approach [18–20]. Traditional dispatch models concerning power systems grounded on OPF techniques seek the optimal operation of generation resources to supply grid demand in a cost-efficient way, whilst meeting system stability and some relevant technical limitations related to its elements [17]. The transition of the grid paradigm, with increasing penetration of renewable generation and distributed generation (DG) systems, and also inclusion of energy storage systems (ESSs), in addition to demand response requirements smart grid dispatch are also faced as new challenges for tackling efficiently OPF problems. Such complications result from complex constraints and dependencies on stochastic variables such as weather or state of charge of batteries [20,21]. Other utilities have simplified the OPF problem, from an AC perspective, to speed execution time by going DC in an attempt to simplify computation. However, these simplifications caused sub-optimality in many cases and led to safety issues as well as non-realistic controllers [17]. On the other hand, data-centric machine learning methods provide a promising potential for computational burden reduction and system stability improvement. This requires a transition from online optimization methods to offline training with large amounts of historical or simulated data [17]. 2.2. Surrogate models With the increasing volume of information, efficient data reduction, such as summarization methods, has even been investigated to countermeasure computational and informational overloading, affirming the general motivation for compact representations and approximations in modeling complex systems [57]. Operating a modern smart grid is an extraordinary technical challenge, in part because the actual physical system is too complex to simulate with detailed models that can only be solved on supercomputing clusters. One way around this problem is with “surrogate models.” These are simple proxies for the actual system, but they return results far faster by making estimates about how the grid will operate [22]. For those trying to solve an optimal power flow (OPF), these models are orders of magnitude faster than old-school math formulas. These can more generally be grouped into two categories: Analytical (ASMs) and Learned (LSMs). Parallel concepts in the use of surrogate-assisted optimization are seen in other complex operational settings, such as supply chain management, for which adaptive neuro-fuzzy inference systems (ANFIS) and blockchain-based solutions have been investigated to enhance robustness and transparency within uncertain and nonlinear constraints [56]. For instance, some studies on the analysis and computation used the Kriging method to address the randomness of wind farms [23]. Others use it as a way to improve the "trial-and-error" math tools, such as increasing the speed and accuracy of Genetic Algorithms [24]. On the “Learned” side, there’s an even more unconventional approach. Others studied the possibility of getting rid from all data and use machine learning techniques in order to predict voltage levels in large networks consisting of more than 8,500 nodes [25]. These intelligent models can work twice as quickly as conventional methods such as Monte Carlo simulations. Of course, there is a catch. To achieve that speed, you sometimes have to trade little accuracy. It also requires an enormous amount of time and effort to “train” these models in the first place, especially if you want them to obey rigid physical constraints. But the trade-off is mostly worth it, most experts believe. When the model is set up, it responds to nearly all requests at near- instantaneous speed, allowing for constant re-evaluation of the grid state. 2.3. Physics-informed machine learning Data-driven surrogates, such as decision trees, random forests, XGBoost, and deep neural networks, may perform well within the training data distribution but struggle with out-of- distribution scenarios or underrepresented regions. PINNs offer an innovative approach that addresses many limitations of conventional surrogates [26]. Unlike traditional data-driven models, PINNs integrate physical equations, such as ordinary differential equations or partial differential equations, directly into the training loss function to ensure that the model’s output remains consistent with fundamental physical laws, leading to improved robustness in poorly sampled regions of the state space compared to purely data-driven models. This makes them particularly suitable for modeling systems with high complexity and variability, such as energy networks with dynamic load profiles and renewable energy generation. For example, PINNs have been used to model lithium batteries by including the governing PDEs of electrochemical processes of Li-ion batteries [27], or mathematical models for Li-ion batteries such as the Single Particle Model [28], or the pseudo-2D model [29]. PINNs were also used to model parts of the behavior of hydrogen electrifiers, such as the temperature fluctuation of the membranes [30], or the voltage losses during the electrification [31]. Regarding smart grids, in [32,33], PINNs were developed, including active and reactive power balance equations, to solve AC-OPF problems, and in all study cases shown, PINNs were more accurate than other conventional Artificial Neural Network (ANN) schemes. The advantages brought by PINNs can also be applied to solutions beyond classification or forecasting, such as Reinforcement Learning, which is a strategy widely used in energy systems due to its reward- focused approach, as explained in the introduction. Moreover, model-free RL tends to struggle with training efficiency, often requiring excessive computing time to achieve a satisfactory reward value, and it also tends to violate physical constraints [34]. As such, coupling PINNs with RL algorithms can increase training performance and has drawn attention in multiple complex environments. An example of which includes a physics-informed Deep Dyna Q RL for the control of building heating systems, in which PINNs were used for the planning phase of Dyna Q, which objective function combined a regression loss and a physics-based loss with the RC heating model [35]. Focusing more on the field of energy, PINNs were also used with PPO-based RL to extract parameters of PV, wind, and energy storage systems connected to a local grid, so that they could be used to accurately simulate and forecast the behavior of such grid and use the parameters to control and preserve the stability of the grid with RL [36]. 2.4. Reinforcement learning solution of the optimal power flow problem Reinforcement Learning (RL) is the methodology adopted in this study. In this configuration, an AI “agent” is trained on how to operate a smart grid through repeated interactions with it. The agent gets feedback — something like a grade report — on how well its actions did, and it wants to maximize this feedback over time. There are many researchers advocating RL use in power flow problems [37,38], however this is burdened with a fundamental challenge: complexity. Smart grids have too many moving parts, so the agent must train in a vast number of different situations. This results in a “curse of dimensionality”, because it requires too many data to fill up each bucket so that the training process can become slow or not work at all [39]. It is hard to find a sufficiently realistic simulation for such training. A very known framework (like) is Grid2Op for the most detailed models of transmission system including power lines and generators constraints [40]. For instance, some researches which employ an RL technique named “Soft Actor-Critic”(SAC) in Grid2Op to explore the optimal approach of power dispatch aims to reduce costs and make full use of renewable energy, are adopted [41]. Another, network-distribution-centric tool is Gym-ANN [15]. Similar resilience issues also arise in distributed cyber-physical networks, such as wireless sensor and actor networks, where connectivity restoration under node failures is concerned with a fast, distributed recovery mechanism that also becomes relevant to reliability considerations for power system operations over a smart grid [58]. This model encompasses numerous smart grid components, such as flexible batteries and distributed energy generation. It gives scientists a way to design specific goals — such as maintaining an embedded battery’s “State of Charge” (SoC) — that also respect the physical laws of the grid. Unfortunately, attempting to bring slow physics simulations into a RL framework involving large (MB-scale) and frequent data reads here is also prohibitively inefficient. To ameliorate this problem, some have proposed the use of Machine Learning “prostitutes” (proxies) as a substitute to the true simulation [42–43]. Our work shows that even if these surrogates aren’t perfect replicas, provided they adhere to some basic physical laws, they can still be highly effective. We think the correct answer is a hybrid that combines data-driven learning with basic physical understanding. That is the inspiration for this paper. Using Physics-Informed Neural Networks (PINNs) trained with real physical laws, we demonstrate that it is possible to create RL systems able to learn signifi- cantly faster and in a more reliable manner than ever before. Our study builds upon the work of Cestero et al. [44]. Though those authors were able to construct well-trained surrogate models, they reported a significant shortcoming: their models didn’t actually “understand” the physics of the system. Since those models considered only raw data, they were not suitable to train an AI for controlling a real-world system. We address this problem by creating a sur- rogate for the Gym-ANN environment similar to the one we developed herein, and compare its output to that of theirs in order to establish how much our new methodology is able to contribute. 3.Methods In this paper we propose to solve the OPF problem by using RL to learn a working policy which can operate in a smart grid good enough on order is avoid penalties and reduce the energy loss of the grid. For that purpose we also use a smart grid simulation environment as our RL environment: the ANM6-Easy [15]. While the rendering is true to life in terms of grid bouncing, expensive iterates of mathematics are required to solve the equations that this environment needs. This renders the environment infeasible for larger topologies, so that scalability of this (RL based) solution to solve the OPF problem of a smart grid is limited. To address this problem, we propose to employ surrogates of the environment. However, as discussed by Cestero et al. [44], however using data-driven techniques to construct surrogates, even though are robust in representing the state transitions, they present good metrics R², sometimes is not enough to accurately represent the physics of state transitions. To this end, we suggest to leverage PINNs as an emulator and compare it with other data-driven approaches. The architecture of our approach is depicted in Fig. 1. We consider two deferent ways of training the surrogate from its original environment: data-driven and PINN-based. For the PINN method, we take the physical laws from the original environments to derive them, and train a surrogate environment with no use of the original environment as detailed in [45]. On the other hand we employ some of the latest methods for constructing data-driven models (XGBoost, Decision Trees, etc.). We note that all the data-driven algorithms learn a training data set based on the original environment. By the methods of Cestero et al. [44], we have two types of datasets: a generative dataset, where we procure transitions from the environment not interacting among each other by computing the intermediate actions of a random sample across the state space of the environment; and an agent-based dataset, in which data was gathered using a random agent mimicking realistic trajectories based on interpolated action sequences (performed with higher-level actions). Then we train all data-driven models using these two datasets individually. We then train an RL policy over the surrogate environments on each of these trained models via PPO [46]. At training we evaluate the policy after every update using one episode of the original environment with the in-training policy. In the end, we save the score of the episode, and use it to analyze the policy's evolution learning in each surrogate method. All equations presented in this paper are based on the formulation in [REF] 3.1. Reinforcement learning training regimes for surrogate models In this work, we use a surrogate model as the environment of the RL algorithms. An algorithmic template was introduced to bridge the connection among the predictive model and RL logic. This framework was implemented in compliance with the interface of the gym [47] and it enabled us to interact with the PPO RL training algorithm [46] from Stable-baselines3 library [48], by fixing all default hyperparameter values. Figure 1 The two different ways to create the surrogate environment – PINN method and data-driven approach – are shown here. The PINN training flow is shown with bold dashed lines and the data-driven process are presented with thin dashed lines. This surrogate environment is then used to train the RL policy. So that the progress measurement would not drift, policy is validated from time to time in the original environment with each update. The activities in the software framework are performed as follows: • Initial state selection: The initial state of the agent is uniformly sampled from the original environment s space. Whenever the environment receives a reset signal, the initial state is sampled uniformly over all possible states. • State transition computations: Transitions between states are executed by the surrogate model. The internal representation is then processed with the current state and agent's action to form a single input vector. This data is then used by the surrogate model to predict how the next state and a reward will change. • Terminal states: The end of an environment episode is denoted by a “done” flag. The binary classifier based on gradient boosting is used, as the surrogate model cannot discern between solvable and unsolvable physical equations [49]. We use this classifier to recognize the terminal state with 99% accuracy. 3.2. Parallelization of the surrogate environment We leverage the inherent parallelism in ANN based surrogates that derives from linear algebra. This is very different from the initial setting where one use sequential, iterative algorithms [25], such as Newton-Raphson and which prohibits again parallel work. With the surrogate being inherently parallel, the training of RL policies learns much faster as environment samples can be collected orders of magnitude faster. Although (as in this study) some frameworks do not inherently facilitate parallelization, one way to overcome this is to use an ANN-based surrogate. A custom wrapper is developed to interface with the proposed framework and parallelized. This configuration brings in two specific structural parameters: n_envs: This quantifies the number of parallel environments the RL algorithm operates over. Larger number of environments may lead to faster policy convergence, but at the cost of computation. In the Stable-baselines3 implementation of PPO, samples are collected in a so-called rollout_buffer. The size of this buffer is calculated as: 푟표푙표푢푡 푏푢푓푒푟 푠푖푧푒 = 푛 푒푛푣푠 · 푏푢푓푒푟 푠푖푧푒 ..............1 With the inclusion of more environments, this overall buffer size will get larger and at some point potentially slow down training due to the sheer size of your dataset. buffer_size: When experimenting, we saw that how often you update your policy is more critical to learning than the total size of seen data. If the policy is updated often (with a smaller buffer_size) huge gains can be seen in performance early on. This parameter determines how many steps to forward before doing an update. So, after every parallel environment completes the steps set by buffer_size, The rollout buffer is rows full and we are ready to update the policy. This is done by vectorizing state transitions. This enables the surrogate model to process transitions with all parallel environments at once using the following function: 푓: ( 푆 푡 ,퐴 푡 ) → ( 푆 푡+1 ,푟 푡 ) .....................2 Into this equation, 퐴 푡 , and 푟 푡 are the (selected from all n_envs ways) matrices and vectors of collected data. 4. Application on smart grids The ANM6 was utilized in this investigation. This structure resembles an integrated smart grid with three separate buses. One bus serves a residential load with photovoltaic (PV) generation, another supports an industrial load paired with a wind generator, and the third connects an electric vehicle (EV) charging station to an energy storage system (ESS). All components are powered by a slack generator, which acts as the interface with the wider transmission network. The key objective in this setting is that of finding a policy to minimize the total energy wastage. This must imply a reduction in partial line losses and partial transformer loss, avoiding energy wasting, due to network overload and also to reduce curtailment of RES. These goals are achieved by controlling the ESS power flow and curtailing PV and wind generation, as necessary. The excess or deficit energy demand is met by the slack generator. A strong constraint is represented by the lines feeding industrial load and EV charging park, as these lines have lower available power than the possible peak demand of both loads. This reflects a typical recent issue in distribution networks, where the increasing penetration of electric vehicles often leads to expensive investments required for infrastructure upgradings to accommodate higher peak demands [50]. Yet it is possible to avoid these costs by actively controlling the local renewable sources and storage systems, leading to better generation costs and network efficiency. 4.1. ANM6-easy environment The tis thesis develops Reinforcement Learning environments to optimize the distribution grid control, employed using the Gym-ANM simulation environment. In this context, the use case ANM6-Easy is intended to model a variety of management tasks on a small distribution network [15]. Figure 2 Layout of the ANM6-Easy environment. The illustrated power system consists of a slack generator and two renewable generation units: wind farm and photovoltaic plant. The grid also contains three different passive loads including industrial, residential and EV charging station along with an individual energy storage unit. Tab.2ANM6-Easy model is composed of seven devices linked across six buses to take into account the distribution network. The devices in D are grouped into two meta-device groups: three generators (one is a slack generator) referred to as 퐷 퐺 ,, and three loads, which are referred to as 퐷 퐿 ,. Further, there is a distributed energy source (ES) acting as an energy storing in a network. The slack generator is connected to its own bus (index 1) in order to balance power flows and to keep a voltage reference of 1푝.푢 0 All parts such as devices, buses and branches are characterized with physical properties/constraints [15]. Going to focus this study, for the sake of simplicity, an environment which uses deterministic. The load demand and the peak generation capacity is both represented by two deterministic, 24 hours’ time series that carry on from one day to the other. The existence of such known patterns is postulated throughout the chapter—a property termed the "future-awareness hypotheses." The simulation time is discrete and expressed as Δ푡=15min. This future-awareness assumption enables to eliminate complexities in prediction errors. This enables a quantitative assessment of the PINN surrogate ability to fit well physical grid dynamics. Therefore, any policy performance or training speed gains can be justified in value of the quality of the surrogate model used, which we believe is the main merit of this paper. Figure 3 Daily Load Demand Profiles Figure 4 Daily Potential Generation Profiles This arrangement also creates a best-case performance; it describes the best one can do with perfect forecasts. Such an assumption is of course a must for the present study, but the framework allows for modularity so that implementation in practice can be done. In a real-world scenario, the given time series would be replaced with data from a forecasting module and we would train the RL agent in an end-to-end PINN surrogate using forecast, uncertain values. Production environment, the deterministic time series data would be replaced by the output of a dedicated forecasting module. The RL agent would then be trained within the PINN surrogate environment, but fed these uncertain, predicted values. The state of the system can be represented for each step of time by the following equation. 퐬 푡 =[푃 푑푡,푡 푑∈풟 ,푄 푑푡,푡 푑∈풟 ,SoC 푡 , 푃 푔,푡 (max ) 푔∈풟 퐺 −푔 slack , , aux 푡 ].......3 Where: • 푃 푑,푡 and 푄 푑,푡 : The active and reactive power responsive at time t for all device • SoC 푡 : Battery state of the charge. 푃 g,푡 (max) : Maximum capacity produced by the generator g at time t (푔∈풟 퐺 −푔 slack ) • Aux: extra time component to preserve the Markov property of this process. It is used as an index, ranging from 0 to 95(24, h/Δ푡), and facilitates obtaining the demand and the generation capacity for the next interval from pre-defined time series. • Options for setting control variables at different network elements are captured by the action vector 퐚 푡 at time step t. The particular form of 퐚 푡 is set as: 퐚 푡 =[푎 푃 8,푡 푔∈풟 퐺 −푔 slack , , 푎 푄 푔,푡 푔∈풟 퐺 −푔 slack , ,푎 푃 DES,푡 ,푎 푄 DES,푡 ]........4 where: • 푎 푃, and 푎 푄 g, , represent the active and reactive power setpoints, respectively, 퐷 퐺 : Group of generators and 푔 slack : Slack generator • 푎 푃 DES,푡 and 푎 푄 DES,푡 denote the active and reactive power modifications for the Distributed Energy Systems (DES), which can together inject and remove power to equilibrium the system 4.1.1. Transition dynamics Given the known future demand 푃 푑,푡+1 푑∈퐷 퐿 , and power factor pf of each load, reactive power of loads 푄 푑,푡+1 푑∈퐷 퐿 is calculated as: 푄 푑,푡+1 =푃 푑,푡+1 tan ( arccos ( 푝푓 )) ∀푑∈풟 퐿 .. ............5 For generators, constraints define feasible active and reactive power ranges, ensuring each device operates safely. The setpoints for active 푎 푃 g,푡 and reactive 푎 푄 g,푡 power is mapped to the feasible set ℛ g,푡, a convex polytope defined by generators physical parameters, network conditions, and external variables: ℛ 푔,푡. = (푃,푄)∈ℝ 2 ∣푃 푔. ≤푃≤푃 푔,푡. (max) , 푄 푔 ≤푄≤푄 ̅ 푔 , 푄≤휏 푔 (1) 푃+휌 푔 (1) , 푄≥휏 푔 (2) 푃+휌 푔 (2) , ...............6 The generator's reactive power injection flexibility constraints (line constraints) applied on a linear in system using known constants휏 푔 (1) ,휌 푔 (1) ,휏 푔 (2) , and 휌 푔 (2) , for instance when the generator is close to the maximum of its active Power Capability Curve are presented. This mapping is posed as a convex optimization problem. The most important criterion for this operation is to find the feasible point in ℛ 푔,푡 that distance from chosen setpoints (푎 푃 g,푖 ,푎 푄 g,푖 ) is minimized. (푃 g,푡+1 ,푄 g,푡+1 )=argmin (푃.푄)∈퐷 g,푡 ⊆ℝ 2 ‖ ( 푎 푃 g,푡 ,푎 푄 g,푡 )−.(푃.,푄)‖.. ......................7 Similar approach is used for the DES unit plus supplementary limitations based on the SoC are introduced. The feasible operational set ℛ DES ,푡 is given by: ℛ DES,푡 = (푃,푄)∈ℝ 2 ∣푃 푑 ≤푃≤푃 푑 (max) , 푄 푑 ≤푄≤푄 ̅ 푑 , 푄≤휏 푑 (1) 푃+휌 푑 (1) , 푄≥휏 푑 (2) 푃+휌 푑 (2) 푄≤휏 푑 (3) 푃+휌 푑 (3) , 푄≥휏 푑 (4) 푃+휌 푑 (4) , 푃≥ 1 휂Δ푡 (SoC 푡−1 −SoC) 푃≤ 휂 Δ푡 (SoC 푡−1 −SoC) , ...................8 In this expression, 휂 serves for both charge and discharge efficiency. Additionally, the power constraints are expressed as: (푃 DES,푡+1 ,푄 DES,푡+1 )=argmin (푃,푄)∈풟 DES,푡+1 ⊆ℝ 2 ‖(푎 푃 DES,푡 ,푎 푄 DES,푡 )−(푃,푄)‖..........9 The state of charge (SoC) for each DES unit is then updated as follows, based on the following equation: SoC 푡+1 = SoC 푡 −휂Δ푡푃 DES,푡+1 if 푃 DES,푡+1 ≤0 SoC 푡 − Δ푡 휂 푃 DES,푡+1 otherwise. ......10 After that, the active and reactive power values of each unit (excluding the slack generator) are found and added up for each bus to be the net injections at that bus which are denoted as 푃 푖,푡+1 (bus ) 푖=2 6 and 푄 푖,푡+1 (bus ) 푖=2 6 . . And the resulting equations for the power flow problem can be solved using these injections along with the slack bus voltage definition and branch admittance matrix 푌 푖푘 =퐺 푖푘 +푗퐵 푖푘 , as previously mentioned. After calculating the active and reactive power of each element (the slack generator is excluded), it is used to group and sum the numbers according to own bus where we obtain the net injections 푃 푖,푡+1 (bus) . And 푄 푖,푡+1. (bus) .By applying these injections with the known slack bus voltage and the branch admittance matrix 푌 푖푘 =퐺 푖푘 +푗퐵 푖푘 ,, the power flow equations yield: 푃 푖,푡+1 (bus) =∑ 6 푘=1 | 푉 푖,푡+1 || 푉 푘,푡+1 |( 퐺 푖푘, cos 휃 푖푘,푡+1. +퐵 푖푘 sin 휃 푖푘,푡+1 ) ........11 푄 푖,푡+1. (bus) =∑ 6 푘=1 | 푉 푖,푡+1 || 푉 푘,푡+1 |( 퐺 푖푘. sin 휃 푖푘,푡+1. +퐵 푖푘 cos 휃 푖푘,푡+1 ) .......12 where 푃 푖,푡+1 ( bus ) and 푄 푖,푡+1 ( bus ) denote the active and reactive power injections at bus 푖at time step 푡+1; ∣푉 푖,푡+1 ∣and ∣푉 푘,푡+1 ∣are the voltage magnitudes at buses 푖and 푘, respectively; 휃 푖푘,푡+1 =휃 푖,푡+1 − 휃 푘,푡+1 is the voltage angle difference between buses 푖and 푘; 퐺 푖푘 and 퐵 푖푘 are the conductance and susceptance entries of the bus admittance matrix 푌 푖푘 ; and the summation is carried out over all buses 푘=1,...,6. From this, the complex voltages 푉 푖,푡+1 = | 푉 푖,푡+1 | ∠휃 푖,퐼+1 and the active and reactive power injections of the slack generator are extracted. Then, the direction of branch currents 퐼 푖푗,푡+1 and their apparent power flow 푆 푖푗,푡+1,푙 are obtained from: ( 퐼 푖푗,푡+1 퐼 푗푖,푡+1 ) = ( 1 | 푡 푖푗 | 2 (푦 푖푗 +푦 푖푗 (푠ℎ) )− 1 | 푡 푖푗 ∗ | 2 푦 푖푗 − 1 | 푡 푖,푗 | 2 푦 푖푗, (푦 푖푗 +푦 푖푗 (푠ℎ) ) ) ( 푉 푖,푡+1 푉 푗,푡+1 ) ........13 | 푆 푖푗,푡+1,푙 | = | 푉 푖,푡+1 퐼 푖푗,푡+1 ∗ | .....................................................14 | 푆 푗푖,푡+1 ∗ | = | 푉 푗,푡+1 퐼 푗푖,푡+1 ∗ | .......................................................15 4.1.2. Reward function The reward function proposed in the ANM6-Easy environment is formulated based on a multi-objective criterion, and mainly centering on the minimization of equipment energy losses as well as slightly minimizing renewable energy curtailments: 푟 푓 푖 =−(Δ퐸 푡:푡+1 +휆Φ ( s 푓+1 ) )................16 The total energy loss term is broken down into three separate sub-terms, corresponding to the main sources of inefficiency in this smart grid system architecture Losses on transmission: They are the energy that is wasted by leaks and resistance in substations and lines of transmission. DES operational losses This takes into account the net energy that reaches a certain part of the Distributed Energy Storage (DES) unit. It is an approximation to the losses caused by battery charging and discharging inefficiencies, see [15]. 2 휃 1푘. =휃 푖, −휃 푘,. ............17 Curtailments Renewable Energy: These losses occur when the entire power output from renewable sources cannot be accommodated in the grid. This is usually due to either lack of demand or line In mathematical terms: Δ퐸 푡:퐼+1. =Δ퐸 푡:푡+1. (1) +Δ퐸 푡:푡+1 (2) +Δ퐸 푡:푡+1 (3) .................18 Δ퐸 푡:푡+1. (1) =Δ푡∑ 푑∈풟. 푃 푑,푡+1 ....................................19 Δ퐸 푖:푡+1. (2) =−Δ푡푃 DES,푡+1 ......................................20 Δ퐸 푡:푡+1 (3) =Δ푡∑ 푔∈퐷 푔 −푔 (slack ) . (푃 푔,푡+1 (max) −푃 푔,퐼+1 )......21 The penalty function includes two particular grid related constraints; the rated power of lines and substations and the bus voltage limits. These restrictions are imposed to avoid the overheating of equipment and the voltage stability of devices connected. Φ ( 퐬 푖+1 ) =Δ푡(∑ 6 푖=1 (| 푉 푖,푡+1 | −푉 ̅ 푖, ) + + ( 푉 푖, − | 푉 푖,푡+1 |) + +∑ 푖,푗 (| 푆 푖푗,푡+1 | −푆 ̅ 푖푗 ) + + ( 푆 푖푗 − | 푆 푖푗,푡+1 |) + ) + ...................22 In our situation 휆=100 and, for numerical constancy, 푟 푖 is clipped to the range [−100,100]. 4.1.3. Terminal state It is considered as a terminal state if there is no physically realizable le solution to equations (9)-(10) under selected actions. This phenomenon corresponds to grid failure, which is usually induced by the violation of voltage constraints. This is called a sag which is one of the major power quality disturbances [51] with important negative effects, especially in industry. 4.2. Surrogate model The surrogate environment for ANM6-Easy is designed to emulate state evolution in a Markov Decision Process (MDP). It is trained to forecast the next system state and the reward of both states: ( 푠 푡+1 ,푟 푡 ) =Surrogate ( 푠 푡 ,푎 푡 ) ..............22 The main role of this surrogate model is to speed up calculation; the transition speeds of such a surrogate model can be orders of magnitude faster than those recorded by the original simulation. The architecture consists of three specialized neural network modules for the power transition dynamics of non-slack generators, battery systems and the global power balance, respectively. Although power systems are highly interconnected entities characterized by non- linear descriptions, this structure preserves physical interpretation as an inferential procedure that depends on the sequential order of data rather than isolated parallel updates. However, our architecture is explicitly designed to maintain this physical coupling through a sequentially dependent inference process. The models do not operate in parallel isolation during the prediction of a state transition. Instead, they form a computational chain that respects the cause-and-effect relationships of the grid physics (see reference [45] for more details on equations in section 4.2) A Model for State Transition The simulation of a single state transition from 푠 푡 to 푠 푡+1 is implemented in the following stages: 1. Asset Power Forecasting: Two KKT-Informed Neural Networks [45] are first used to estimate the active and reactive power production of the non- slack generators, and battery respectively. These predictions are premised on control actions 푎 푡 and state variables from 푠 푡 ,, like the State of Charge (SoC). These networks are trained with a loss, which is KKT-informed in order to guarantee a strict fulfillment of the local operating constraints. 2. Aggregation To achieve this, the forecast power injections of the devices as well as all individual nodes are aggregated for each bus in the network. These output bus powers, 푃 푡+1 ( 푖 ) 푖=2 6 and 푄 푡+1 ( 푖 ) 푖=2 6 , are the direct outputs of the final module. 3. Power Balance and Voltage Prediction: The aggregated injections are processed by the power balance network to predict (the system wise states, 1) bus voltage ∣푉 푖 ∣ , 휃 푖 푖=2 6 and 2) the power consumption at the slack generator. This block is learned to approximate the nonlinear power flow equations by reducing the residuals of grid’s physical laws. This cascade connection allows the outputs of the asset models to directly influence the inputs of system-wide power balance model causally. By adding coupled modules, the physical connection of the grid is explicitly imposed in order that the surrogate can adequately reproduce how the network reacts to certain power injections. 4.2.1. Generators state As explained in section 4.1.1, transformation ( 푎 푃 8, ,푎 푄 8, ), to (푃 g,푡+1 ,푄 g,푡+1 ) is performed by solving the convex optimization problem formulated by (5). The problem presents in a standard form : min ( 푃 g,푡+1 ,푄 g,푡+1 ) ‖(푎 푃 g,푡 , ,푎 푄 g,푡 )−(푃 g,푡+1 ,푄 g,푡+1 )‖ s.t. 퐺(푃 g,푡+1 ,푄 g,푡+1 )−ℎ≤0 ...............23 퐺=( −11100−휏 g (1) 휏 g (2) 000−1111 ) 푇 ......................24 ℎ=( 푃 g 푃 ̅ g 푃 g,푡 (max) 푄 g 푄 ̅ g 휌 g (1) −휌 g (2) ) 푇 ...............25 Since the formulation in (21) is a convex program with Slater's condition (푃 푔,푡+1 ,푄 푔,푡+1 ) ∗ holding for its constraints, an optimal solution exists if and only if there exist dual variables 휆 ∗ ∈ ℝ 7 such that the following Karush–Kuhn–Tucker (KKT) conditions hold. We use a KKT-Informed Neural Network to predict (푃 푔,푡+1 ,푄 푔,푡+1 ) ∗ 푔∈풟 훿 − ( 푔 glack ,4 from the given inputs. Contrary to conventional data-driven models, this approach casts the KKT- optimality conditions as part of the loss function itself. 푎 푃 푔,푡 ,푎 푄 푔,푡 ,푃 푔,푡 (max) 푔∈풟 퐺 − ( 푔 slack , . .....................26 4.2.2. Battery state Likewise, the battery system operational constraints as stated in (7) are rewritten into the standard optimization form as: min ( 푃 DES,푡+1, ,푄 DES,푡+1 ) ‖(푎 푃 DES,푡 ,푎 푄 DES,푡 )−(푃 DES,푡+1 ,푄 DES,푡+1 )‖ .................................................27 s.t. 퐺(푃 DES,푡+1 ,푄 DES,푡+1 )−ℎ≤0 퐺= ( −1100−휏 DES (1) 휏 DES (2) 휏 DES (3) 휏 DES (4) −1 00−1111110 0 ) 푇 ............................28 ℎ= ( 푃 DES, 푃 ̅ DES 푄 DES, 푄 ̅ DES 휌 DES (1) −휌 DES (2) −휌 DES (3) 휌 DES (4) 1 휂Δ푡 (SoC 푡 −SoC), 휂 Δ푡 (SoC 푡 −SoC) ) 푇 . .......29 So a KKT-Informed Neural Network is trained to predict the optimal active and reactive power (푃 DES,푡+1 ,푄 DES,푡+1 ) ∗ using [푎 푃 DES,푡 ,푎 푄 DES,푡 ,SOC 푡 ] as the input vector. By incorporating the optimality conditions of the battery's feasible region and its dynamic limits (e.g. SoC), such a network forces the predicted charge/discharge dynamics to be consistent with those imposed by hardware design. 4.2.3. Power balance After the active and reactive power of each device (except for the slack generator) are solved, the results are gathered at all their corresponding buses to get coarse-grained bus-wise quantities 푃 푖 (bus) 푄 푖 (bus) 푖=2,...,6 The power balance network takes these accumulated powers as input to forecast the bus voltages | 푉 푖 | ,휃 푖 푖=푧 − 6 . Since the slack bus voltage is maintained fixed ( 푉 1 =1 p.u. and 휃 1 =0 ∘ ), one can calculate the residuals of power flow equations (9-10) for all other buses for 푖=2,...,6. The network learns to minimize the difference between the predictions and ground truth by minimizing the following loss per sample: ℒ=ℒ 1 +ℒ 2 ....................................................................................30 ℒ 1 = 1 5 ∑ 6 푖=2 ( 푃 푖 (bus ) −∑ 6 푘=1 | 푉 푖 || 푉 푘 |( 퐺 푖푘 cos 휃 푖푘 +퐵 푖푘 sin 휃 푖푘 ) ) 2 .................31 ℒ 2 = 1 5 ∑ 6 푖=2 ( 푄 푖 (bus ) −∑ 6 푘=1 | 푉 푖 || 푉 푘 |( 퐺 푖푘 sin 휃 푖푘 +퐵 푖푘 cos 휃 푖푘 ) ) 2 −⋯...........32 After obtaining | 푉 푖 | ,휃 푖 푖=1 6 the active and reactive power injection of the slack generator are obtained as: 푃 1 (bus ) =∑ 6 푘=1 | 푉 1 || 푉 푘 | (퐺 1,푘 cos 휃 1푘. +퐵 1푘 sin 휃 1푘 ).....................33 푄 1 (bus ) =∑ 6 푘=1 | 푉 1 || 푉 푘 |( 퐺 1푘. sin 휃 1푘. +퐵 1푘 cos 휃 1푘 ) ......................34 4.2.4. Next state Assuming perfect foresight for the future of load and generation, we treat푃 푑,푡+1 푑∈퐷 퐿. and 푃 푔,퐼+1 (max ) 푔∈퐷 퐺 −푔 slax ) , as known. Now, equation (3) and (8), with the above networks created in sections 4.2.1 - 4.2.3, result in: aux 푡+1 = ( aux 푡 +1 ) mod96............................35 There is already enough information in all these defects, to build the next state 퐬 푡+1 . 4.2.5. Reward All information to compute (11)-(19) is available. 5. Results and discussion 5.1. Surrogate training conditions Each of the three constituent modules of the surrogate is defined as a multilayer perceptron with three hidden layers of 512 neurons each, a residual connection between the inputs and outputs of the latter, and a Leaky RELU (slope of 0.01) as the activation function. Each module is optimized with Adam W, with a learning rate of 1×10 −5 . An early stopping condition is imposed: training stops when no progress is observed in the last 5000 steps. At each training step, a batch of 64 samples is sampled from a Sobol sequence of dimensionality 21. All dimensions are scaled to provide inputs for each part of the surrogate, as outlined in Section 4.2. For more information on the scaling process, Our proposed PINN-based surrogate model has been analyzed against other methods from the state of the art, considering seven types of surrogates trained from two different kinds of datasets. The studied surrogates are: Deep Neural Networks (DNN), XGBoost (XGB), Decision Trees (DT), Random Forest (RF), Linear Regression (LR), and our proposed method, with PINNs. These models (except for the PINN) were trained using two types of datasets: Generative and Agent-based. Following the naming convention established in [44], a Generative dataset is constructed by sampling transitions across the state space using a general sampling method, in this case Sobol sampling [52]. In contrast, an Agent-based dataset is derived from the agent interaction of a specifically designed agent with the environment, represented here by a Random agent. Both datasets consist of 100,000 samples. The loss function used in the training process is the Mean Absolute Error (MAE) for each model. However, the coefficient of determination metric (R²) has been used to evaluate the performance of the trained models. The results reported in the following sections have been obtained by training the models on an Intel Core i5-9499F CPU. 5.2. Surrogate training results This section presents the results regarding the surrogates’ training metrics. Figure 4 shows trends of losses for the generators and the DES network, as defined in [45], and for the power balance solver, over the training. Additionally, the speed increase of the surrogate compared to the original environment is calculated by performing 1000 transitions with both. As shown in Fig. 5, there is a median boost of almost 10 times. Figure 7 shows the accuracy comparison between all the studied models in terms of R² and MAE. In line with the conclusions of Cestero et al. [44], these results generally show that agent- based sampling produces better performance than generative sampling in most cases. This is clearly seen in the MAE metric, which is plotted on a logarithmic scale, and the error of the models trained on the generative dataset is generally much higher than that of the other. XGB, followed closely by the PINN model, from these results, we can expect that XGB and PINN models perform best as a proper environment for RL training, while the models DNN and LR may have the least favorable performance. However, this is not what happens in practice, as further results reveal. Once the models operate directly within the environment, their performance shifts significantly. The overall accuracy of the models in terms of R² suggests that many models, in principle, should be able to represent the transitions of the environment accurately. To validate this, we run random episodes using two different agents: a pre-trained agent that is able to follow the optimal policy in the environment, and an expert agent, i.e., an agent that takes actions according to the environment dynamics. We then calculate the MAE of all the surrogates throughout the episodes of each agent. Figure 7 shows the averaged overall MAE of each surrogate model. The results are obtained by running all the surrogates in parallel with the real environment and following the real transitions from the environment, calculating transitions with the surrogates too. Then, the error is calculated as the MAE of the surrogate transitions compared to the real ones, and the MAE is averaged between several episodes. The error is presented in Fig. 8 as averaged MAE of surrogate transitions across the episodes, calculated as the MAE of the difference between surrogate predictions and real transitions. In addition, there is no accumulated error in this figure, since the surrogates do not simulate the episode independently; they are only used as a transition predictor. To calculate the transition 푠 푡+1 =푓(푠 푡 ,푎 푡 ), the surrogate functions 푓 surrogate are evaluated at each step, and the real state taken from the real environment is used for the next step. As per the figure, our method shows the least error, even though Fig. 5 shows that other models display more accurate overall results. These results highlight that PINNs present the most suitable method for modeling datasets with physical extrapolation capabilities, while the PINN model is reliable across the entire state space. Figure 5 Loss function components. The training metrics are such that: Generators Loss is a sum of KKT residuals associated with the generator-related KKT-N, DES Loss is similarly formed by summing up KKT residuals from battery-related KKT-N and Power Balance Loss is computed as the error in power balance equations (27) coupled with voltage related PINN. Figure 6 Analysis of Inference time (1,000 transition). This is a large difference in computational cost, since the PINN has a median inference time of 0.0021 s and the plain environment a median one of 0.0194 s. Figure 7 Model performance assessment evaluation of surrogate models with $R^2$ and MAE metrics. Table 1 Compares the accuracy of different surrogate models, 푅 2 is shown in linear scale and MAE on log scale(options include Genertive and Agent- based datasets). Although XGB models return the best performance in terms of accuracy which is accompanied by the smallest error—barely outperforming PINN—all except P NJ b1 and 8 are considered to be incapable for RL training. Figure 8 Distribution of MAE by training dataset. The errors of the various model are computed from training data (Agent-based or GAN) and averaged over a random episode, using both an Expert Agent (EA), as well as Random policy. In all these cases the value of the error is minimized by the model with PINNs. Observethat “Oth r” in dicates using no train d taset for the surrogate. Important conclusions about the performance of surrogates trained on Agent-based versus Generative data are provided in this paper. Nevertheless previous work in the group of Cestero et al. [44], the proposed method does not use a mixed strategy (random, expert, and search policies) to create the dataset but instead utilizes only a random policy. In contrast, the Generative dataset (generated using Sobol) provides consistent results for all surrogates and an indication of a more uniform coverage of the state space. In comparison, the Agent-based dataset exhibits large differences across explored regions when it is probed with expert or random policies. The convergence of the Mean Absolute Error (MAE) for the PINN surrogate in episodes where expert and random policies are used is shown in Fig. 8. While we play representative single episodes, the behavior was consistent across multiple runs. There is low error in the random episodes throughout; distinct features of daily periodicity and two-stage development (initial slope followed by stability) are present. These variations in the error profile are explained as periodic patterns in the grid. The small spikes in the MAE correspond to particular state variables which are under-represented the most often—corresponding to transient energy generation or consumption peaks. Inspection of a simulation rendering shows that this periodicity is determined by two main factors: • The battery configuration shows a mean charge that continues to grow linearly in time the long run while oscillating (in daily scale), it eventually saturates at steady state. • The cycle between the daily peaks of energy consumption, such as the evening peak for charging your car, versus a minimum in demand during the dead of night. Even with these occasional fluctuations, the PINN surrogate maintains a lower error rate than other algorithms. The discrepancy between the performance of the two agents is due to a difference in regions of state space visited; random agents remain near wellvisited initial states, while expert agents follow directed trajectories into states that are often underrepresented in the training data [44]. As a result, the accuracy of PINN is usually higher for states that are visited by a random agent rather than those traversed by an expert. Figure 9 Progression of the MAE for the PINN surrogate. Error evolution of the PINN agent is studied for a sample episode which takes place with both type of agents -one EA and one RA. It is noted that EA results exhibit a unique 24 daily periodicity in error and a linear increase of an average error followed by plateau. This tendency is mostly due to environmental environment, when stability will be achieved while load and battery will take the "best ones". On the contrary, the Random agent has a small overall error magnitude but with no clear periodicity. 5.3. Optimization of environment structural parameters The parallelizable nature of surrogate models is leveraged by predicting multiple transitions at once, enabling to grid multiple environment into the same operation. However, it is reported that the use of a large number of parallel environments can lead to training speed saturation and performance decay for PPO from Stable-baselines3. This impairment takes place in the network training phase, when we have the model with an increasing number of samples and thus it gets easily saturated and asymmetrically polarized. As a result, the algorithm’s rollout buffer size is said to be a key parameter affecting the training speed. It is worth noting that reducing the rollout buffer can make updates faster but also unstable if smaller than an episode duration. An investigation has been performed for various options of the numbers of these two variables to search out the optimal combination, and their illustration is provided in Fig. 9. A single episode in this environment has 3,000 steps, and smaller buffer sizes do not finish a full episode without updating. However, the data clearly indicates that a full episode is not needed for good policy learning; the best configuration of structural parameters turns out to be 100 parallel environments and a buffer size of 30. Early Termination: Similar early stopping techniques are also employed to terminate experiments at a performance plateau or after ax number of steps. A more in-depth analysis of these structural parameters includes computing their relationship to the mean reward per episode and total training time, as shown in Fig. 10. There is a strong negative correlation between the mean reward and the buffer size, which are the same effects observed in our early attempts on parallelization. Moreover, the mild impact of environment number on reward is less significant than that of buffer size. As for the buffers, we find that their total training time grows proportionally with both buffer size and number of environments, with them having roughly equal impact on the length of time for training. Figure 10 Results of the reward acquisition optimization for the structure parameter. The shown results are the mean reward of each parameter combination over the last ten evaluations. For the sake of consistency, some trials were stopped when they arrived at a maximum step counter or when convergence was obtained. Note that effective rollout buffer size will be the product of the number of envs and the hyperparam in this paper. Figure 11 Correlation analysis of structural parameters. The links among buffer size, n_envs, mean reward and duration of training are visualized. A negative correlation is detected among the reward, the buffer size and the number of environments, where impact of buffer size is greater. Furthermore, the two parameters present a similar positive relationship with total training time. With the previously determined optimal structural parameters, each of the investigated models was used as a surrogate habitat to train a PPO agent on an optimal policy. Fig. 11 shows the evolution of the training reward of multiple surrogates compared to a baseline (i.e., training a PPO agent on the original environment with no surrogate). For efficiency, an early stopping condition was added on all agents which stopped the experiment when rewards began to saturate (when the reward had not been improved upon for 20 episodes). The results show that our method with PINN-trained surrogate is much faster than the baseline with a comparable performance. It is worth pointing out that the PINN surrogate is found to be the only model (when trained with policy optimization) able to effectively generate a primitive policy which achieves a performance gain over random action. That this model can capture the background physics of the environment, and not just mimic what is observed, explains this success. In terms of operation efficiency, the smart grid controlled by the PINN agent is able to reduce and achieve much faster and reliable reductions in energy losses than all other models including the baseline. Examination of simulation renders shows that the agent correctly recognises and prevents the main sources for power loss. To be more precise, the battery is regularly used between 20% state of charge (SOC) and 80% SOC because these two boundary points are where you lose less energy and put less wear on your battery. Additionally, the agent shows good knowledge about physical limitations. For example, in the illustrated installation, power supply line to the bus from battery and EV chargers are rated for a capacity less than or equal to peak of EV demand. Even so, the agent is able to use the battery to fill in for power that the line, from a physical standpoint, cannot deliver. Yet another is the agent's extreme disinclination to renounce a considerable amount of renewable energy, even at the expense of diverting power from the slack generator and increasing some losses. This may indicate that the agent has found a sweet spot between voltage penalties and power losses, showing that the reward function effectively values grid safety and equipment lifespan more than consumption of as much renewable energy as possible. Although Fig. 6 one might also argue that the XGB model would perform best because of its almost perfect $R^2$ value but as we see in contrast to RF, prediction errors increase during long- term simulations with XGB. This causes the RL agent to fail to learn a stable policy. On the contrary, for the PINN surrogates, this problem can be mitigated by learning state transitions deeply and enforcing physical laws strictly so that physical predictions can remain realistic even in such regions of the state space not being visited (see Fig. 7. On a practical note, the extrapolation ability of PINN surrogate enables the manipulation of demand and generation profiles while no retraining is necessary. In practice, consumption on distribution networks often changes due to new loads or tenants and generators can increase their rated generation. The ability to obtain reliable simulations under such new conditions without additional training time or computational burden is a clear advantage of the PINN approach. 5.4. Discussion The findings given in Section 5.3 show that the PINN-based surrogates perform for reinforcement learning policies similar to the training in the original environment, reducing computational time considerably. Instead of elaborating on the error dynamics seen on Expert Agent and Random Agent trajectories, the discussion will look to the potential benefits of these findings for surrogate-assisted reinforcement learning to be used in smart grid control. A crucial observation is that surrogate accuracy is evaluated through traditional regression metrics, like 푅 2 or absolute mean error computed on static data sets, is often not sufficient for successful policy learning. Several data-driven surrogates have good predictive strength as a function of working offline and still suffer from the problem of the fact that errors accumulate (accumulation) over long horizon paths in reinforcement learning. This behavior highlights a critical limitation of purely data-driven models: small local inaccuracies can compound over time when the model is repeatedly queried within a closed-loop control setting. By contrast, the PINN-based surrogate maintains physically consistent state transitions even when extrapolating beyond densely sampled regions of the state space. By explicitly embedding power flow equations and device-level operational constraints into the learning process, the surrogate keeps feasibility and stability properties which are important for reinforcement learning. This may explain why the PINN surrogate works well when the policy goes under scenarios that the training data does not have very much coverage of, while pure data surrogate departs from this paradigm only after similar conditions. The learned policies demonstrate behavior matching established engineers’ practices from a control perspective. Given the endurance of the battery within intermediate state-of-charge ranges and prioritizing grid safety above aggressive renewable utilization, it can say that the reward formulation along with the physics-informed surrogate provides the means for physically meaningful control strategies. Significantly, these behaviors arise without the explicit hard- coding, indicating that the framework is capable of learning from the system constraints and internalizing them. Another implication is adaptability. Since the PINN surrogate encodes governing physical relationships rather than historical paths, it does not need to be retrained in order to support changes of demand profiles or generation patterns, unlike its ancestors. This is not only useful for real-world distribution networks, where operating conditions change over time due to new load factors, generation assets or user behavior. In conclusion, in the present study, we assume a deterministic forecasting to separate the impact of the surrogate model, but our framework will apply to stochastic settings, automatically by introducing forecasting uncertainty in the state representation. These results imply that physics-guided alternative scenarios can provide a strong platform for scalable, data-efficient reinforcement learning in complex energy systems. Conclusion In this work, we develop the capability of Physics-Informed Neural Networks (PINNs) as surrogate models for effective energy management in Smart Grids based on Reinforcement Learning (RL). We developed a Smart Grid simulation using the Gym-ANM framework and trained an RL agent for grid adaptation. The proposed PINN-based surrogate method significantly outperforms traditional approaches to achieve improved computational efficiency and a 10× speed-up in inference time and 50% reduction in policy training time, preserving policy accuracy and reliability. Classical surrogate models did not converge, whereas the PINN surrogate provides stable, efficient learning to cope with the increasing complexity in modern Smart Grids, especially given the high penetration of renewable energy and dynamic consumption. These results exhibit obvious benefits for the optimization of Smart Grid in that their training and inference time is much shorter with very low error which reduces computational cost. The PINN surrogate accurately captures the dynamics of the grid while dynamically changing the demand and generation, thus making it safe and effective to operate at lower maintenance costs and improved system reliability. Hence, this study demonstrates the usefulness of PINN-based surrogate models for speeding up the development of efficient, cost-effective and sustainable management strategy of Smart Grid with real-time operation. These models can be applied to larger systems for the scalability of the approach (e.g., on demand response and energy pricing scenarios for more extensive assessment) to evaluate their performance in real-world uncertain conditions. Adding forecasting models and measuring policy generalization will be key steps toward field implementation. Appendix A. Ranges of inputs for surrogate training Refer to [15] for a description of the physical parameters that outline the subsequent ranges. 푎 푃 푔,푡 ∈ [ 푃 푔 ,푃 ̅ 푔 ] 64 ∀푔∈풟 푔 푎 푄 푔,푡 ∈ [ 푄 푔 ,푄 ̅ 푔 ] 64 ∀푔∈풟 푔 푎 푃 DES,푡 ∈ [ 푃 DES ,푃 ̅ DES ] 64 푎 푄 DES,푡 ∈ [ 푄 DES ,푄 ̅ DES ] 64 푃 푔,푡 (max ) ∈ [ 푃 푔 ,푃 ̅ 푔 ] 64 ∀푔∈풟 푔 푃 푖 (bus) ∈ [ 푃 푖 (bus) ,푃 ̅ 푖 (bus) ] 64 푖=1,...,6 푄 푖 (bus ) ∈ [ 푄 푖 (bus ) ,푄 ̅ 푖 (bus ) ] 64 푖=2,...,6 SoC ∈ [ SoC , SoC 64 푖=2,...,6 Reference [1] European Commission, The European Green Deal, COM(2019)640, Final. Communication from the Commission to the European Parliament, the European Council, the Council, the European Economic and Social Committee, and the Committee of the Regions, Brussels, 2019. [2] Hassan Q., Nassar A., Al-Jibouri A.K., Viktor P., Telba A.A., Awwad E.M., et al., Mapping Europe, renewable energy landscape: insights into solar, wind, hydro, and green hydrogen production, Technol. Soc., 2024;77:102535. https://doi.org/10.1016/j.techsoc.2024.102535 [3] Bari A., Jiang J., Saad W., Jaekel A., Challenges in the smart grid applications, IEEE Wirel. Netw., 2014;20(2):974682. https://doi.org/10.1155/2014/974682 [4] Rebenaque O., Schmitt C., Schumann K., Dronne T., Roques F., Success of local energy market implementation: a review of current projects, Utilities Policy, 2023;80:101491. https://doi.org/10.1016/j.jup.2023.101491 [5] Du P., Li N., Zhong H., Overview of demand response, Springer International Publishing, 2019. p. 1– 27. https://doi.org/10.1007/978-3-030-19769-8_1 [6] Zio E., Aven T., Uncertainties in smart grids behavior and modeling: what are the risks and vulnerabilities?, How to analyze them? Energy Policy, 2013;91:308–320. https://doi.org/10.1016/j.enpol.2016.01.032 [7] Yang P., Mao Q., Zhang J., Bao G., Cheng K.W., Lam K.H., Novel joint algorithm for state-of-charge estimation of rechargeable batteries based on the back-propagation neural network combining ultrasonics detection method, J. Energy Storage, 2024;59:113894. https://doi.org/10.1016/j.est.2023.113894 [8] Wang X., Hu S., Xu Z., Zhai D., State-of-health estimation for lithium-ion batteries using random forests and gated recurrent unit, J. Energy Storage, 2024;76:109796. https://doi.org/10.1016/j.est.2023.109796 [9] Rahman J., Feng C., Zhang H., Machine learning-aided security constrained optimal power flow, In: 2020 IEEE Power & Energy Society General Meeting (PESGM), IEEE, 2020. p. 1–5. https://doi.org/10.1109/PESGM41954.2020.9281941 [10] Rahman J., Feng C., Zhang J., Learning-augmented approach for AC optimal power flow, Int. J. Electr. Power Energy Syst., 2021;130:106908. https://doi.org/10.1016/j.ijepes.2021.106908 [11] Ali N., Wahid A., Shaw R., Mason K., Reinforcement learning approach to dairy farm battery management using Q-learning, J. Energy Storage, 2024;93:112031. https://doi.org/10.1016/j.est.2024.112031 [12] Mussi M., Pellegrino L., Pindzola J., Resilience to voltage drop in smart grids, J. Energy Storage, 2024;82:110577. https://doi.org/10.1016/j.est.2024.110577 [13] Giraldo-Pérez J.P., Mejía-Gutiérrez R., Aguilar J., A reinforcement learning-based energy optimization approach for household fridges, Sustain. Energy Grids Netw., 2023;36:101174. https://doi.org/10.1016/j.segan.2023.101174 [14] da Costa Paulo B., Agarwal S., Del Barrio L., Quartulli M., Camberlin H., Surrogate model of HVAC system for self-consumption maximisation, Energy Convers. Manag., 2023;291:119036. https://doi.org/10.1016/j.enconman.2023.119036 [15] Henry R., Ernst D., Gym-ANN: reinforcement learning environments for active network management tasks in electricity distribution systems, Energy, 2021;51:100092. https://doi.org/10.1016/j.energy.2021.100092 [16] Azim S.M., Feng Z., Yed D., Burt G., High-fidelity validation with smart grid modeling complexity considerations on emerging solutions, In: 2023 IEEE Int. Conf. Communications, Control, and Computing Technologies for Smart Grids (SmartGridComm), IEEE, 2023. p. 1–7. https://doi.org/10.1109/SmartGridComm57358.2023.10333914 [17] Khaloie H., Dohani M., Tohidi S.F., Valle E., Review of machine learning techniques for optimal power flow, Sustain. Renew. Energy, 2024. https://doi.org/10.1213/srmk4681955 [18] Sarsenov E., Di Silvestre M., Baldaneni R., Nguyen N., Guerrero J., Energy management of islanded microgrids using a simple distributed algorithm, Energies, 2018;11(3):649. https://doi.org/10.3390/en11030649 [19] Esfahani T., Soroush M., Power flow optimization for smart microgrids by sparse relaxation and linear networks, IEEE Trans. Smart Grid, 2013;4(2):751–762. https://doi.org/10.1109/TSG.2012.2222677 [20] Abdi H., Beigvand S., Saffar N., A review of optimal power flow studies applied to smart grids and microgrids, Renew. Sustain. Energy Rev., 2017;71:742–766. https://doi.org/10.1016/j.rser.2016.12.102 [21] Hasan F., Karam A., Mohammad A., A survey on applications of machine learning for optimal power flow, In: 2020 IEEE Texas Power and Energy Conference (TPEC), 2020. https://doi.org/10.1109/TPEC48276.2020.9042547 [22] Mohammadi S., Bui V.H., Su W., Wang J., Surrogate modeling for solving OPF: a review, Sustainability, 2024;16(22):9851. [23] Li L., Fan Y., Xu X., Qiu G., Probabilistic load flow calculation of power systems integrated with wind farm based on Kriging model, Energy, 2012;118(3):565–580. https://doi.org/10.1016/j.energy.2017.03.085 [24] Deng Z., Rotaru M.D., Sykulski J.K., Kriging assisted surrogate evolutionary computation to solve optimal power flow problems, IEEE Trans. Power Syst., 2020;35(1):372–381. https://doi.org/10.1109/TPWRS.2019.2936999 [25] Memon Z., Trinchero R., Manfredi P., Canavero S., Compressed machine learning models for accurate prediction of node voltage distribution networks, Energies, 2020;13(8):1815. [26] Raissi M., Perdikaris P., Karniadakis G.E., Physics-informed neural networks: a deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations, J. Comput. Phys., 2019;378:686–707. https://doi.org/10.1016/j.jcp.2018.10.045 [27] Wang Q., Zhang P., Qiu W., Feng L., Stable and accurate representation of species diffusion in multilayer composite electrodes using physics-informed neural networks, J. Energy Storage, 2024;78:110016. [28] Hassani M., Weddle P., King R.N., DesDoos A., Randall C.R., PINN surrogate of Li-ion battery models for parameter inference, Part I: implementation and multi-fidelity hierarchies, J. Energy Storage, 2024;98:113103. [29] Hassani M., Weddle P., King R.N., DesDoos A., Randall C.R., PINN surrogate of Li-ion battery models for parameter inference, Part I: regularization and application of the pseudo-2D model, J. Energy Storage, 2024;98:113104. [30] Zerrougui L., Zhan B., Xie S., Physics-informed neural networks for modeling temperature fluctuations in proton exchange membrane electrolyzers, Appl. Energy, 2020;278:115730. [31] Chen X., Rex A., Woelk J., Eckert G., Buechner A., Knowledge-integrated framework for modeling electrolyzer stacks, J. Energy Storage, 2024;37:123550. [32] Sun Y., Fan X., Huang Q., Li X., Huang R., Yin T., et al., Local feature sufficiency exploration for predicting security-constrained generation dispatch in multi-area power systems, In: 2018 IEEE Int. Conf. on Machine Learning and Applications (ICMLA), IEEE, 2018. p. 1283–1288. https://doi.org/10.1109/ICMLA.2018.00208 [33] Nellikath R., Chatzivasilis S., Physics-informed neural networks for AC optimal power flow, Electr. Power Syst. Res., 2022;212:108412. https://doi.org/10.1016/j.epsr.2022.108412 [34] Xia Q., Wang Q., Zou Y., Chi Y., Yan Z., Meng Q., et al., Physical model-assisted deep reinforcement learning for energy management optimization of industrial electric-hydrogen coupling system with hybrid energy storage, J. Energy Storage, 2024;100:113477. [35] Hafeez M., Kazmi H., Deconinck G., Energy & buildings dual-PINN: physics-informed deep dyna-Q reinforcement learning control of building heating systems, Energy Build., 2024;324:113489. https://doi.org/10.1016/j.enbuild.2024.113489 [36] Feng R., Wajid K., Hafeez M., Wang J., Subhani S., Uniform physics-informed neural network framework for microgrid applications, IEEE Access, 2025;13:31471–31485. https://doi.org/10.1109/ACCESS.2025.31471 [37] Wu T., Scaglione A., Arnold D., Constrained reinforcement learning for stochastic dynamic optimal power flow control, In: 2023 IEEE Power & Energy Society General Meeting (PESGM), IEEE, 2023. p. 1–5. https://doi.org/10.1109/PESGM52003.2023.10253087 [38] Han X., Mu C., Yan J., Niu Z., An autonomous control technology based on deep reinforcement learning for optimal active power dispatch, Int. J. Electr. Power Energy Syst., 2023;145:108686. [39] Barto A.G., Mahadevan S., Recent advances in hierarchical reinforcement learning, Discrete Event Dyn. Syst., 2003;13:41–77. [40] Donnot B., Grondin-Pasquier L., A testbed platform to model sequential decision-making in power systems, 2022, https://github.com/rte-france/Grid2Op [41] Han X., Mu C., Yan J., An autonomous control technology based on deep reinforcement learning for optimal active power dispatch, Int. J. Electr. Power Energy Syst., 2023;145:108686. [42] Pinto G., DeConinck G., Data-driven district energy management with surrogate models and deep reinforcement learning, Appl. Energy, 2021;304:117642. [43] Kaewboonchan N., Srithapon C., Chattapun K., Electric distribution network with multi-microgrids management using surrogate-assisted deep reinforcement learning optimization, IEEE Access, 2022;10:130739. [44] Cestero J., Quartulli M., Rizzoni M., Building surrogate models using trajectories of agents trained by reinforcement learning, In: Int. Conf. on Artificial Neural Networks, Springer, 2024. p. 340–355. [45] Delle Femmine C., Frumento A., Hervé M., et al., KKT-informed neural networks, arXiv:2409.09087 [Preprint], 2024. [46] Schulman J., Wolski F., Dhariwal P., Radford A., Klimov O., Proximal policy optimization algorithms, arXiv:1707.06347 [Preprint], 2017. [47] Brockman G., Cheung V., Pettersson L., et al., OpenAI Gym, arXiv:1606.01540 [Preprint], 2016. [48] Raffin A., Hill A., Gleave A., Kanervisto A., Ernestus M., Dormann J., Stable-Baselines3: reliable reinforcement learning implementations, JMLR, 2021;22(268):1–8. [49] Chen T., Guestrin C., XGBoost: a scalable tree boosting system, In: Proc. of the 22nd ACM SIGKDD Int. Conf. on Knowledge Discovery and Data Mining (KDD), ACM, 2016. p. 785–794. [50] Hotz E., Baroni A., Burmester D., Alemany I., Approaches for managing energy consumption, generation, storage, and grid-independent micro-grids optimized by meta-heuristic optimization algorithms, Energy Convers. Manag., 2019;200:112105. [51] Almeida A. de, Almeida A. de, Power quality problems and new solutions, RE&PQJ, 2023;11. https://doi.org/10.24084/repqj01.00 [52] Burhenne S., Jacob D., Henze G.P., Sampling-based on Sobol’ sequences for Monte Carlo techniques applied to building simulations, Int. Build. Simulation, 2011;4(2):206–213. [53] Snoek J., Larochelle H., Adams R.P., Practical Bayesian optimization of machine learning algorithms, Adv. Neural Inf. Process. Syst., 2012;25. [54] Taheri, Alireza, Amirfarhad Farhadi, and Azadeh Zamanifar. "Application of GenAI in Clinical Administration Support." Application of Generative AI in Healthcare Systems. Cham: Springer Nature Switzerland, 2025. 91-117. [55] Rashidieranjbar, Fatemeh, Amirfarhad Farhadi, and Azadeh Zamanifar. "Revolutionizing healthcare with generative artificial intelligence technologies." Generative Artificial Intelligence (AI) Approaches for Industrial Applications. Cham: Springer Nature Switzerland, 2025. 189-221. [56] Farhadi, Amirfarhad, et al. "Leveraging blockchain and ANFIS for optimal supply chain management." arXiv preprint arXiv:2408.17161 (2024). [57] ofighy, M., Kashefi, O., Zamanifar, A. and Haj Seyyed Javadi, H., 2011, November. Persian text summarization using fractal theory. In International conference on informatics engineering and information science (p. 651-662). Berlin, Heidelberg: Springer Berlin Heidelberg. [58] Zamanifar, A., Sharifi, M. and Kashefi, O., 2009, April. Self actor-actor connectivity restoration in wireless sensor and actor networks. In 2009 First Asian Conference on Intelligent Information and Database Systems (p. 442-447). IEEE

Instant research discovery