Paper deep dive
Fully Decentralized and Safety-Aware Multi-Agent Reinforcement Learning for Control on Networks
Theodore Rogalski, Shirantha Welikala
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 95%
Last extracted: 9/30/2026, 6:24:50 AM
Summary
This paper introduces a fully decentralized multi-agent reinforcement learning (MARL) algorithm for discrete-time control on networks, specifically applied to the persistent monitoring problem. The method integrates deep reinforcement learning with safety constraints using a Discrete-Time Control Barrier Function (DCBF). The architecture features a graph encoder for structural correlation, a state estimator for uncertainty prediction, and an actor-critic network. The algorithm achieves 26.3% lower average uncertainty than centralized policies and matches the performance of more complex attention-based algorithms.
Entities (10)
Relation Signals (7)
Shirantha Welikala → affiliatedwith → Stevens Institute of Technology
confidence 99% · Shirantha Welikala Affiliation: Electrical & Computer Engineering Department Stevens Institute of Technology
Theodore Rogalski → affiliatedwith → Stevens Institute of Technology
confidence 99% · Theodore Rogalski Affiliation: Electrical & Computer Engineering Department Stevens Institute of Technology
Fully Decentralized MARL → solves → Persistent Monitoring
confidence 95% · This paper develops a safe and fully decentralized multi-agent reinforcement learning (MARL) algorithm to solve a class of discrete-time control problems on networks, including the persistent monitoring problem.
Fully Decentralized MARL → outperforms → Centralized Control Policy
confidence 94% · achieves 26.3 percent lower average uncertainty than a centralized control policy
Discrete-Time Control Barrier Function → reduces → likelihood of neglect
confidence 93% · passed through a discrete-time control barrier heuristic to reduce the likelihood that any node will be neglected.
State Estimator → predicts → Uncertainty
confidence 92% · a state estimator, which predicts the uncertainty at each node in the graph.
Graph Encoder → provides → structural information
confidence 90% · the graph encoder, which adds structural information and correlations among nodes
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:This paper develops a safe and fully decentralized multi-agent reinforcement learning (MARL) algorithm to solve a class of discrete-time control problems on networks, including the persistent monitoring problem. Fully decentralized control of agents, while offering numerous benefits, faces issues such as exponentially increasing sample complexity, lack of global information about the system, and challenges in coordinating between agents. To address these issues, this paper introduces a fully decentralized multi-agent reinforcement learning algorithm that integrates deep reinforcement learning with safety considerations. This method feeds a history of local observations of the network's state into two parallel neural-network branches: the graph encoder, which adds structural information and correlations among nodes, and a state estimator, which predicts the uncertainty at each node in the graph. Additionally, the result of feeding that input into an actor-critic network is passed through a discrete-time control barrier heuristic to reduce the likelihood that any node will be neglected. This approach enables teams of fully decentralized agents to solve challenging problems by increasing system awareness and incorporating built-in safety measures to prevent the adoption of potentially harmful control policies. Numerical results from a custom simulation environment demonstrate that the proposed algorithm achieves 26.3 percent lower average uncertainty than a centralized control policy and is within 1 percent of the uncertainty performance of a more computationally complex algorithm with added attention layers.
Tags
Links
- Source: https://arxiv.org/abs/2609.36292v1
- Canonical: https://arxiv.org/abs/2609.36292v1
Trouble viewing inline? Open PDF directly →
Full Text
38,975 characters extracted from source content.
Expand or collapse full text
Fully Decentralized and Safety-Aware Multi-Agent Reinforcement Learning for Control on Networks Theodore Rogalski Affiliation: Electrical & Computer Engineering Department Stevens Institute of Technology Hoboken, USA trogalsk@stevens.edu Shirantha Welikala Affiliation: Electrical & Computer Engineering Department Stevens Institute of Technology Hoboken, USA swelikal@stevens.edu Abstract This paper develops a safe and fully decentralized multi-agent reinforcement learning (MARL) algorithm to solve a class of discrete-time control problems on networks, including the persistent monitoring problem. Fully decentralized control of agents, while offering numerous benefits, faces issues such as exponentially increasing sample complexity, lack of global information about the system, and challenges in coordinating between agents. To address these issues, this paper introduces a fully decentralized multi-agent reinforcement learning algorithm that integrates deep reinforcement learning with safety considerations. This method feeds a history of local observations of the network’s state into two parallel neural-network branches: the graph encoder, which adds structural information and correlations among nodes, and a state estimator, which predicts the uncertainty at each node in the graph. Additionally, the result of feeding that input into an actor-critic network is passed through a discrete-time control barrier heuristic to reduce the likelihood that any node will be neglected. This approach enables teams of fully decentralized agents to solve challenging problems by increasing system awareness and incorporating built-in safety measures to prevent the adoption of potentially harmful control policies. Numerical results from a custom simulation environment demonstrate that the proposed algorithm achieves 26.3% lower average uncertainty than a centralized control policy and is within 1% of the uncertainty performance of a more computationally complex algorithm with added attention layers. I Introduction Multi-agent reinforcement learning (MARL) [1] is a powerful method for solving discrete-time control problems autonomously, but current approaches rely on unrealistic assumptions such as perfect state information and various forms of communication among agents. Those centralized MARL approaches contrast with fully decentralized MARL, an approach that replaces a single MARL controller that has access to full-state information and action selections for each agent with independent controllers with no information sharing during training or evaluation, including each agent’s local representation of the state, network parameters, and all other pieces of data. Fully decentralized approaches remove the overhead of communicating and sharing parameters with other agents but make coordination more difficult and lack convergence guarantees. Replacing the centralized controller with independent controllers for each agent presents three main issues: the lack of global state renders traditional methods for ensuring convergence of the algorithm to an optimal policy moot, the fact that each controller is independent removes inherent coordination from the architecture, and the curse of multi-agency leads to exponential sample complexity with respect to the number of agents for fully decentralized algorithms [2]. To address these challenges, decentralized algorithms may utilize inter-agent communication [3, 4, 5, 6], but its use incurs computational and implementation costs. Deep reinforcement learning (DRL) [7], an approach that incorporates deep learning components, such as neural networks, into reinforcement learning algorithms, enables strong generalization without requiring comprehensive system information [8]. Another decentralized method for addressing those challenges is safe reinforcement learning, referred to as ”safety” in this work. Complementary to DRL, safety methods [9, 10] enable deterministic behavior design and agent coordination in control architectures, even without the ability to communicate or share information. The stochastic nature of DRL is augmented by the predictable nature of safety methods, which we propose may increase the reliability of fully decentralized control policies. However, even with the added reasoning ability of DRL and safety integration, agents optimize for local sub-goals rather than the global picture, necessitating the selection of a problem whose solution fits that paradigm. The persistent monitoring problem is a problem formulation in which a team of agents must be coordinated to minimize a global measure of node uncertainty by monitoring the state of certain valuable points of interest [11]. Persistent monitoring is uniquely suited to a solution that optimizes local sub-goals, as it covers a large area that agents can sample and monitor efficiently. However, a realistic persistent monitoring scenario involves a large number of system variables that are abstracted away by the model, such as electronic noise and physical constraints. This motivates representing the environment as a network of nodes and edges, with points of interest mapped to nodes and important spatial information mapped to edges between nodes. Each agent is allowed to move along these edges to these nodes, preserving the important connections between points in space while reducing unnecessary computation. This paper is organized as follows. Section I discusses the particulars of controlling a decentralized team of agents and the selected persistent monitoring problem as the specific control problem to address. The proposed algorithm architecture is discussed in Section I, with supporting experiments and results detailed in Section IV. I Problem Formulation I-A Control of Agents on Networks We consider a dynamic environment with decision-making agents ≜i:i∈ℕNA \A_i:i _N\ (where ℕN≜1,2,…,NN_N \1,2,...,N\) that can be modeled as a network ≜(,ℰ)G (V,E). The nodes ≜m:m∈ℕMV \V_m:m _M\ and the edges ℰ⊂×E ×V represent entities of interest and their interconnections within this environment. In addition, each node mV_m may have features ℱm∈ℝnF_m ^n, which can respond to agent behavior around the node. At the kkth time-step k∈ℕKk _K, agent iA_i makes an action selection ai,k∈Ai,ka_i,k∈ A_i,k where Ai,kA_i,k is the local action space which is dependent on its local observations Ωi,k _i,k and iA_i’s observation at k of its one-step neighborhood i,kN_i,k alongside its current position ϕi,k∈ _i,k , including edges and nodes Ωi,k≜i,k,ϕi,k _i,k \N_i,k, _i,k\. At each time-step k, an agent iA_i may choose to move to a node directly connected to its current node ϕi,k _i,k or remain at ϕi,k _i,k. Each node mV_m is associated with a feature vector ℱm,kF_m,k. These features may be used to model arbitrarily complex systems with arbitrarily complex dynamics; for example, a team of autonomous robots operating within a rugged three-dimensional space may be modeled as a network whose nodes and edges represent important spatial information. Fitting engineering problems to this formulation reduces irrelevant complexities; in this case, the persistent monitoring problem is abstracted as a representative example of complex problems. I-B Persistent Monitoring The persistent monitoring problem, wherein a team of one or more agents is controlled to minimize uncertainty about an environment (mission space), has many characteristics that naturally allow it to be modeled using the above formulation, particularly because the node-edge connection structure lends itself to representing points of interest. The persistent monitoring problem serves as a strong representative for complex problems, meaning that a strong solution to it suggests applicability to other,t similar problems. Applications of persistent monitoring include the monitoring of a physical area with ground-based robots [12] as well as the autonomous coordination of surveillance drones [13], a burgeoning field with various applications including in agricultural, military, and urban surveillance [14]. In order to solve the persistent monitoring problem using simulation, an accurate and abstract formulation must first be established: in the network G, certain nodes T (targets) on the network ≜(,ℰ)G (V,E) have uncertainty values (e.g., umu_m for a target node mV_m) that increase when they are not being monitored and decrease when they are monitored (i.e., visited ) such that the uncertainty um,k+1u_m,k+1 at a time-step k+1k+1 is given by um,k+1=0,tm=0,um,k+(1−2pm,k),tm=1,um,k>0,um,k+(1−pm,k),tm=1,um,k=0,u_m,k+1= cases0,&t_m=0,\\ u_m,k+(1-2p_m,k),&t_m=1,u_m,k>0,\\ u_m,k+(1-p_m,k),&t_m=1,u_m,k=0,\\ cases (1) where pm,k∈0,1p_m,k∈\0,1\ denotes the presence of an agent at node mV_m at time-step k and m∈0,1t_m∈\0,1\ denotes whether or not each node is a target. 1 shows an example of a persistent monitoring system with six nodes and two agents transitioning from time-step k to k+1k+1. Fig. 1: A persistent monitoring system with six nodes and two agents at discrete time steps k and k+1k+1. I Methodology I-A Algorithm Architecture Fig. 2: At time-step k, the agent iA_i receives ℳi,kM_i,k, which is processed in the following manner: concatenation with ℒiL_i, transformation by Graph Attention (GAT), concatenation with ℒi(ℳi,k)L_i(M_i,k) and then transformation by the Actor network. The result of the processed observation is passed through a safety layer and then sent to iA_i as a probability mass function over possible actions. To address the persistent monitoring problem using decentralized agents, this project developed an algorithm that incorporates the following deep learning components: a lightweight mental map architecture and a discrete control barrier function (DCBF) heuristic, combined with existing deep reinforcement learning (DRL) algorithms. Each agent constructs an input to its deep network by concatenating the observations it has received, which is then augmented with the graph Laplacian. In parallel, its observation is sent to a neural network state estimator, which predicts the amount of uncertainty at each node at each time step. Then, the output from a Graph Attention network, concatenated with the state estimator result, is fed to an actor-critic network. The actor’s output (the policy π(a|s)π(a|s)) is then fed into the DCBF heuristic, which removes the probabilities of agents taking actions that move to lower-uncertainty nodes. I-B Mental Map Firstly, each agent iA_i is given a mental map ℳi,kM_i,k of the network to allow for reasoning over time. If each iA_i has a potentially limited ability to observe the network G at a time step k (i.e., Ωi,k⊂ _i,k ), then its state signal will be an incomplete representation of the network. To address incomplete system information, at each time step the mental map is updated with the latest local observation: newly observed features replace their stored values, while unobserved features retain their previous values. The mental map is initialized with empty nodes (edges are added during training/test time) to enable statically sized neural networks, saving computational bandwidth. I-C Graph Laplacian Then, the agent’s transformed state signal is augmented with the graph Laplacian of Mi,k,Li,k=Di,k−Ai,kM_i,k,L_i,k=D_i,k-A_i,k, where D is the degree matrix and A is the adjacency matrix of the mental map at k. The concatenation of L to M allows the agent to have built-in structural information about the graph, enabling it to move between nodes without having full information. I-D Graph Attention After receiving the enhanced state signal ℳi,kM_i,k concatenated with L, iA_i reasons over it using the graph attention mechanism to learn correlations among neighboring nodes. Graph attention [15] is defined as the following f→m′=∥h=1Hσ(∑j∈mαmjhhf→j), f _m= _h=1^Hσ ( _j _m _mj^h W^h f_j ), (2) where fm′f _m are the transformed features of mV_m ℱmF_m, σ is a non-linear function (in this case, LeakyReLU), mN_m is each node connected to mV_m via an edge, αm,j _m,j is a softmax over the result of an attention mechanism between neighboring node vjv_j and the node umu_m, WkW^k and h∈Hh∈ H are attention heads that learn different associations between the features of mV_m and that of surrounding nodes. I-E State Estimator At any given moment, the total ground-truth information an agent has about the system is limited to a composition of local observations of nodes and edges Ωi _i. Given that a subset of the network may not be visible to each agent due to that limited capacity to observe the environment, the estimation of such invisible node features ℱm,∀m∈−Ωi,kF_m, _m - _i,k is an important way for the agent to reason over the graph. At each time step, iA_i uses its own Multilayer Perceptron (MLP) state estimator ℒiL_i for this estimation task. Each state estimator is updated according to the mean-squared prediction loss: loss(ℒi)=1||(ℒi(ℳi,k−1)−ℳi,k)2loss(L_i)= 1|V|(L_i(M_i,k-1)-M_i,k)^2 (3) where s and s′s are the current and next states, respectively. ℒL allows the agent to learn a mapping between the dynamics of any given state to the next state over time. Because ℒL is a deep neural network, similar states may be mapped to similar outputs, allowing for a degree of generalization [4], [16]. In previous works, communication assumptions eliminated the property of full decentralization from these algorithms, but the ability to guide training remains very useful in a fully decentralized setting. After the graph encoder and state estimator representations are concatenated, the data is fed into a linear policy head to learn a complex environment. However, the fitness of π is not available to the actor, so a learned value function vπ(ℳ)v_π(M), implemented as a multi-layer perceptron, is used to evaluate the current policy and improve it during training. Feeding the concatenation of the graph encoder and graph Laplacian to the actor-critic layer allows the agent to learn control policies faster, reducing the training time required to achieve good results and addressing the curse of multi-agency. It is implemented using two linear models: the policy head has input dimensions of 5*number of nodes, and the critic is a linear model with 3*number of nodes and 1 output layer for probabilities. The use of linear layers keeps computational cost low while allowing for learnable weights. I-F Discrete-Time Control Barrier Function The use of a Discrete-Time Control Barrier Function (DCBF) heuristic to ensure agents select actions that preserve safety, in this case, selecting nodes with the highest uncertainty, allows for greater control over the learned policy. Safety is defined as ensuring that no target exceeds a certain uncertainty threshold (set here to k), with the aim of reducing the number of unattended nodes. The safe set of actions i,kS_i,k represents all actions for agent i at time step k that do not violate the principle of reducing uncertainty to a preset level γ∈ℝ≥0γ _≥ 0. The prediction of all uncertainties u^k u_k at time step k for the mental map ℳM is used as the estimate of the state. The safety value hi,k=γk−max(u^k)h_i,k= _k-max( u_k), where γk _k represents a threshold after which each node must be attended to by an agent, given that the agent can estimate the uncertainty to be at that level; this threshold is equal to the number of time-steps during training and testing. Given these variables, the following process is taken to transform the probability distribution outputted by π(a|s)π(a|s) to the safe distribution: I-F1 Masking of Nodes The selection of an action by an agent is limited to the set of all nodes on or one step away from its current node; the action at k is ak∈i,k,ϕi,ka_k∈\N_i,k, _i,k\. Therefore, all nodes not in this set are unreachable, and the probability of moving to them is set to 0. I-F2 Lower Bound Computation First, the predicted uncertainty ℒ(m)L(V_m) of each node is calculated. Then, the maximum of these expected values is multiplied by 1−η1-η, η∈[0,1]η∈[0,1] (η is chosen to be 0.1) to ensure that any nodes whose uncertainty is not at least within 0.9 of the max uncertainty are moved to. ht=max(ℒ(m),m∈)(1−η)h_t=max(L(V_m),V_m∈V)(1-η) (4) I-F3 Unsafe Node Determination After hth_t is determined, each prediction u^i,m,k u_i,m,k in the graph is looped through, checking if its uncertainty prediction is greater than or equal to hth_t; if it is, its probability of being selected as an action is multiplied by one; otherwise, it is multiplied by zero. If no feasible action survives the filter, the agent selects a feasible destination node with the highest estimated uncertainty, where ties are broken randomly. Additionally, the model freezes at the current node for a number of time steps (selected to be 10 in the implementation) to ensure that uncertainty is reduced by at least a minimum amount at each node. This model architecture, portrayed in Figure 2, means that the algorithm is designed to overcome major obstacles facing decentralized control algorithms, including the exponential sample complexity with the increase in the number of agents [2], non-stationarity from each agent’s perspective [17], and coordination among decentralized agents [18]. I-G Reinforcement Learning Design I-G1 Reward In Reinforcement Learning, the reward function ri,kr_i,k represents the quantity that each agent seeks to maximize over time. Each agent is given its own separate reward function, which its controller uses to modify its behavior. The reward function is the mechanism by which the environment provides behavioral feedback to the agent; it is the means by which decentralized agents can be compelled to achieve a global goal through coordination. In this paper, the reward function for each agent is given by ri,k≜Wcolcoli+Wchangechangei+Wmommomi,tr_i,k W_ col col_i+W_ change change_i+W_ mom mom_i,t (5) where A,B,C∈ℝA,B,C . The terms coli (collision), changei change_i, and momi,t mom_i,t (momentum) are defined as coli=−3,pi∈P−i,0,otherwise, col_i= cases-3,&p_i∈ P_-i,\\ 0,&otherwise,\\ cases (6) where P is the set of all current nodes each agent is positioned at and pip_i is the position of iA_i, changei=U¯−Ui, change_i= U-U_i, (7) where Ui=∑k=1K∑m=1Mum,k(e)U_i= _k=1^K _m=1^Mu^(e)_m,k, as in UiU_i equals the sum of uncertainty over all time-steps and nodes of ℳM_i for the current episode, and U¯ U is the average of UiU_i over the past W∈ℤW episodes with W=10W=10, and momi,t+1=momi,t+1,ui>0,momi,t−1,otherwise, mom_i,t+1= cases mom_i,t+1,&u_i>0,\\ mom_i,t-1,&otherwise,\\ cases (8) where uiu_i represents the uncertainty at the node the agent remains; momentum resets at the start of each episode. Additionally, the terms A,B,CA,B,C are each set to 0.050.05 to avoid excessively large reward signals, as higher values caused unstable training, based on inspection of the shape of the reward curves during training. I-H Modeling with Networks The applicability of this algorithm is not limited to scenarios that are immediately conducive to the usage of networks (e.g., power grids). This is because any n-dimensional rectangular environment can be divided into discrete components, whose spatial relationships can be represented as edges and whose space may be modeled as a graph with nodes V representing parts of the space. Given n dimensions of length dnd_n: d1,d2,…,dn,d∈ℝd_1,d_2,...,d_n,d and a desired number of chunks taken from dnd_n, mn,mn∈ℝ,dn%mn=0m_n,m_n ,d_n\%m_n=0, Algorithm 1 is able to decompose that space into a graph that represents all of the space as nodes and contact between parts of the space as edges. Algorithm 1 Partitioning an n-rectangular region into graph nodes and edges 1: Input: n-dimensional rectangular space with dimension lengths d1:nd_1:n 2: ⊳ A hyperrectangle is given to the algorithm, where it is partitioned into a list of nodes representing equally spaced portions of space in the input region. 3: listOfNodes←[]listOfNodes←[\,] ⊳ A list containing each node is initialized 4: for M1 in range(1,d1m1)M_1 in range(1, d_1m_1) do 5: for M2 in range(1,d2m2)M_2 in range(1, d_2m_2) do 6: ⋱ 7: for Mn in range(1,dnmn)M_n in range(1, d_nm_n) do 8: OPENCk≜((Mk−1)mk,Mkmk)),∀k∈ℕn C_k ((M_k-1)m_k,M_km_k)),∀ k _n 9: region ←(C1,C2,…,Cn)← ( C_1, C_2,..., C_n ) 10: ⊳ Coordinates at a given portion of a region are assigned to a tuple. 11: nodeM1,M2,…,Mn←node_M_1,M_2,…,M_n← region 12: ⊳ Each node is iteratively assigned coordinates from the lengths, ensuring the entire region is recorded as a node. 13: listOfNodes.append(node)listOfNodes.append(node) 14: end for 15: end for 16: end for 17: edges←∅edges← 18: edge ← True 19: for node in listOfNodes do 20: for i1∈−1,0,1i_1∈\-1,0,1\ do 21: ⋱ 22: for in∈−1,0,1i_n∈\-1,0,1\ do 23: for coordinate in node.coordinates do 24: if index(coordinate) + i1:n≥−1i_1:n≥-1 then 25: edge←Trueedge 26: continue 27: else 28: edge←Falseedge 29: break 30: end if 31: end for 32: if edge=Trueedge=True then 33: edges←edges+(node,node(i1:n))edges +(node,node(i_1:n)) 34: end if 35: end for 36: end for 37: end for 38: Output: listOfNodes, edges 39: ⊳ The algorithm outputs a list of nodes indexed by space, as well as a list of edges connecting the nodes, given by their indices. IV Numerical Results To test the proposed algorithm, we modeled the problem in a Python simulation and collected data on various performance metrics, including average uncertainty within the system over time and the maximum number of time steps between node visits, to assess its effectiveness. To create an environment that accurately reflected the persistent monitoring problem in networks, we used various software packages, including NetworkX [19], PyTorch [20], and PettingZoo [21]. The following experiments were run using the custom simulation software11 1 Available publicly at https://github.com/theohrogalski/dpmrl, with the compute comprising an AMD Ryzen 9 CPU and NVIDIA GeForce RTX 4070 Laptop GPU. Training and testing took approximately 24 and 12 hours, respectively. This paper includes in Table I and Table I the results of two experiments to assess the proposed algorithm: the first, when compared to other versions of the same algorithm, and the second, when compared to variations of itself, suggesting optimality among various possible designs. IV-A Metrics Two metrics were collected on a collection of randomly seeded solution architectures: • Average uncertainty over time: The sum of each node’s uncertainty, averaged over time using the arithmetic mean, to provide a clearer statistical picture of uncertainty reduction. • Max. time between any target visit: The maximum number of time steps between an agent visiting a node, capturing the tendency of a policy to neglect nodes by keeping agents in a small area. These metrics capture key aspects of the persistent monitoring problem, providing insight into the relative performance of a given control policy. IV-B Experimental Setup A number of system parameters, including the number of targets and nodes, are varied to assess the algorithm’s effectiveness across different scenarios with four agents. Each network is constructed according to a random process in which the probability of an edge between two nodes is 20%, and the probability that a node is a target is set to approximately 16%. Three representative situations are tested: the first with 50 nodes and 8 target nodes; the second with 100 nodes, 100 agents, and 17 target nodes; and the third with 200 nodes, 200 agents, and 32 target nodes, each with 4 agents. The uncertainty at a given node is capped at 100 to reflect the minimal difference in information loss from neglecting a node for 100 or more time steps in a real-world scenario. Each agent is trained for 100 episodes, with 500 moves per episode, a number of steps that reflects a finite-time monitoring scenario such as information gathering in a limited time span over a certain area. Fig. 3: Network with 50 nodes, eight targets (highlighted in peach) and connections between 20% of nodes. The number of agents is held constant to reduce computational complexity and assess a realistically sized team’s ability to tackle increasingly large-scale problems. Figure 3 illustrates the environment in which each agent reasons. Each graph keeps the same structure during training and testing. Fig. 4: Various reward (above) and reward (below) curves for different parts of the training process, with curves (a) and (e) representing episode one, curves (b) and (d) representing episode seven, and curves (c) and (f) representing episode 15, demonstrating the process of moving away from a random control policy to a more effective one over time Fig. 5: An exponentially increasing pattern for 3A_3 for one episode (a), a pattern from 3A_3 demonstrating a low-loss pattern for one episode (b) & state estimator loss at one early episode with the proposed model architecture on a graph with 100 nodes for four agents (c) Hyperparameters for training and testing include a learning rate of 11000 11000 for the optimizer (Adam). Uncertainty curves for one episode alongside the corresponding reward for 3A_3, areshown in Figure 4. The loss of the state estimator for four agents over a single episode is shown in Figure 5. The data in the following tables were collected as follows: ten runs of 500 steps of the model in the environment were collected for three seeds and three graphs, giving 90 data points for each model (excepting the centralized model, which was run for one seed due to computational constraints leading to around a fivefold increase in training time, possibly due to the relative sparsity of the decentralized mental maps). IV-C Experiment One: Performance Benchmarking We compare against baseline algorithms with various capabilities, including an omniscient centralized algorithm (grazing), a random action-selection policy (random policy), and a simple policy of selecting a target to stay at until the run completes (sit-on-nodes policy). A centralized algorithm with a shared mental map ℳM, shared state estimator ℒL, and shared observation processing network similar to [8] is also tested, allowing for a strong comparison between the effect of full decentralization on the performance of a given algorithm. The implementation in this paper differs from [8] in that it does not incorporate state history, spatio-temporal attention, or other components that rely on the environment presented in that work. Each algorithm represents a different approach to persistent monitoring, as they are either fully centralized/omniscient algorithms or simple policies. TABLE I: Simulation Performance Comparison Algorithm Avg. Uncertainty Std. Dev. (Uncertainty) Avg. Neglect Time Std. Dev. (Neglect) Proposed Algorithm 1736.3 895 1176.6 561.6 Grazing 1128.1 1078.4 276.7 195.2 Random 3006 1147.9 5001* 0 Sit on Nodes 3006 1147.9 5001* 0 Centralized Algorithm 2356.7 1120 322 213 * Value reached saturation From Table I, the average uncertainty of the grazing algorithm is the best (lowest), while the proposed and centralized algorithms have average uncertainties 53.9% and 108.9% higher than Grazing, respectively. However, because the grazing algorithm assumes perfect knowledge of the state, it is impractical for real-world applications, implying that the proposed algorithm may perform better in practice. The random and sit-on-nodes algorithms fail to perform, resulting in the worst possible value (saturation) occurring in both cases, due to the random algorithm’s tendency to move to a connected node rather than stay at the current node. The centralized and Grazing algorithms reduce the mean maximum inter-visit time by 72.6% and 76.5% respectively relative to the proposed algorithm, suggesting that the decentralized algorithm focuses on optimizing a small local environment rather than exploring the global environment. The less-complex sit-on-nodes and random algorithms both performed extremely poorly on this task, achieving the same maximum score. This implies that they were not able to reduce the uncertainty of many, if any, nodes. IV-D Experiment Two: Variational and Ablative Analysis Though the algorithm performs within the same region as algorithms that contain unrealistic assumptions or exhibit minimal performance (grazing and random), it may be that certain components are unnecessary, or that a slight modification to the algorithm may improve performance. Given that, a variational and ablative study was conducted to assess unnecessary components and potential improvements to the algorithm. These removals and additions include the removals of the DCBF heuristic and the state estimator, a modified DCBF-inspired heuristic that prevents collisions between agents (occupation of the same node at the same time) and the addition of numerous multi- and single-headed attention layers to process the data. A single-factor ANOVA on the data yielded a p-value of 0.012 and an F statistic exceeding the critical threshold, indicating a statistically significant result. This suggests that the data each model produces come from substantively different distributions. TABLE I: Ablation and Variant Analysis Algorithm Avg. Uncertainty Std. Dev. (Uncertainty) Avg. Neglect Time Std. Dev. (Neglect) Proposed Algorithm 1736.3 895 1176.6 561.6 No DCBF 3006.0 994 10001 * 0 No State Est. 1770.0 831 1241.3 667.4 No Collisions 1750.4 847 1139.3 564.3 Extra Attention 1734.6 830 1103.1 439.5 * Value reached saturation Table I demonstrates that while the uncertainty performance for the algorithm with extra attention layers is high, it is less than 1% better performing than the proposed algorithm. However, the extra attention algorithm requires an additional single-head attention and graph transform layer (5 layers and 5×5, respectively, with 1 head each) to function. For both metrics collected, the No Collisions algorithm performed better and had a similar or lower standard deviation, suggesting that further exploration of it is justified. The No Collisions DCBF and No State Estimator algorithms both performed about 1% less effectively than the proposed algorithm, suggesting potential use cases in situations where collisions are unacceptable or compute resources are limited. V Conclusion This paper presents a fully decentralized, safety-aware MARL algorithm for coordinating a team of non-communicating agents over a network. Incorporating deep reinforcement learning, control barrier functions, and full decentralization enabled strong performance in the persistent monitoring problem. Additionally, this paper elaborates on the importance of problem selection for key algorithms, drawing on prior research [22], and demonstrates the potential for fully decentralized solutions to address problems where local sub-objectives are key to global solutions. The performance of the proposed algorithm exceeds its centralized counterpart in terms of average uncertainty, indicating that its applicability to real-world drone- and robot-based persistent monitoring is plausible. Future directions for this research include identifying additional applications for fully decentralized algorithms, establishing stronger principles for the convergence of fully decentralized systems, and further investigating the impact of safety on control algorithms. References [1] C. Amato (2025) An initial introduction to cooperative multi-agent reinforcement learning. External Links: 2405.06161, Link Cited by: §I. [2] L. Shi, J. Gai, E. Mazumdar, Y. Chi, and A. Wierman (2025) Breaking the curse of multiagency in robust multi-agent reinforcement learning. External Links: 2409.20067, Link Cited by: §I, §I-F3. [3] S. Lu, K. Zhang, T. Chen, T. Başar, and L. Horesh (2021) Decentralized policy gradient descent ascent for safe multi-agent reinforcement learning. Proceedings of the AAAI Conference on Artificial Intelligence 35 (10), p. 8767–8775. External Links: Link, Document Cited by: §I. [4] M. S. Oh, Z. Zhang, F. Hairi, A. Velasquez, and J. Liu (2025) Consensus-based decentralized multi-agent reinforcement learning for random access network optimization. External Links: 2508.07001, Link Cited by: §I, §I-E. [5] Z. Zhang, M. S. Oh, F. Hairi, Z. Luo, A. Velasquez, and J. Liu (2025) Finite-time global optimality convergence in deep neural actor-critic methods for decentralized multi-agent reinforcement learning. In Forty-second International Conference on Machine Learning, External Links: Link Cited by: §I. [6] K. Yuan, Q. Ling, and W. Yin (2015) On the convergence of decentralized gradient descent. External Links: 1310.7063, Link Cited by: §I. [7] X. Wang, S. Wang, X. Liang, D. Zhao, J. Huang, X. Xu, B. Dai, and Q. Miao (2024) Deep reinforcement learning: a survey. IEEE Transactions on Neural Networks and Learning Systems 35 (4), p. 5064–5078. External Links: Document Cited by: §I. [8] X. Zhang, Y. Wang, and G. Sartoretti (2025) COMPASS: cooperative multi-agent persistent monitoring using spatio-temporal attention network. In 2025 IEEE International Symposium on Multi-Robot and Multi-Agent Systems (MRS), Vol. , p. 1–7. External Links: Document Cited by: §I, §IV-C. [9] F. P. Bejarano, L. Brunke, and A. P. Schoellig (2024) Safety filtering while training: improving the performance and sample efficiency of reinforcement learning agents. IEEE Robotics and Automation Letters 10 (1), p. 788–795. Cited by: §I. [10] S. Gu, L. Yang, Y. Du, G. Chen, F. Walter, J. Wang, and A. Knoll (2024) A review of safe reinforcement learning: methods, theory and applications. External Links: 2205.10330, Link Cited by: §I. [11] Q. Guo, W. Xu, J. Peng, H. Li, and Z. Xiang (2023) Persistent monitoring for points of interests with different priorities using multiple UAVs. In 2022 IEEE 28th International Conference on Parallel and Distributed Systems (ICPADS), Vol. , p. 427–434. External Links: Document Cited by: §I. [12] S. L. Smith, M. Schwager, and D. Rus (2012) Persistent robotic tasks: monitoring and sweeping in changing environments. IEEE Transactions on Robotics 28 (2), p. 410–426. External Links: ISSN 1941-0468, Link, Document Cited by: §I-B. [13] M. Boldrer, V. Kratky, and M. Saska (2026) Aerial robots persistent monitoring and target detection: deployment and assessment in the field. External Links: 2504.18832, Link Cited by: §I-B. [14] S. Kumar, A. Tiwari, Y. Ahirwar, G. Kumar, and M. Y. Arafat (2025) The rise of UAV-based smart surveillance: a systematic review of trends and technologies. IEEE Access 13 (), p. 181553–181575. External Links: Document Cited by: §I-B. [15] P. Veličković, G. Cucurull, A. Casanova, A. Romero, P. Liò, and Y. Bengio (2018) Graph attention networks. ICLR. External Links: 1710.10903, Link Cited by: §I-D. [16] W. Fang, D. Lyu, Y. Zhang, J. Zhang, Z. Wang, X. Zhang, and J. Wang (2025) Reinforcement learning-based model predictive control method for high-dynamic electromagnetic aircraft recovery platform. In 2025 6th International Conference on Electrical Technology and Automatic Control (ICETAC), Vol. , p. 488–492. External Links: Document Cited by: §I-E. [17] H. Nekoei, A. Badrinaaraayanan, A. Sinha, M. Amini, J. Rajendran, A. Mahajan, and S. Chandar (2023) Dealing with non-stationarity in decentralized cooperative multi-agent deep reinforcement learning via multi-timescale learning. External Links: 2302.02792, Link Cited by: §I-F3. [18] K. Zhang, Z. Yang, and T. Başar (2019) Decentralized multi-agent reinforcement learning with networked agents: recent advances. External Links: 1912.03821, Link Cited by: §I-F3. [19] A. Hagberg, P. Swart, and D. Chult (2008) Exploring network structure, dynamics, and function using NetworkX. p. . External Links: Document Cited by: §IV. [20] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Köpf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala (2019) PyTorchs: an imperative style, high-performance deep learning library. External Links: 1912.01703, Link Cited by: §IV. [21] J. K. Terry, B. Black, N. Grammel, M. Jayakumar, A. Hari, R. Sullivan, L. Santos, R. Perez, C. Horsch, C. Dieffendahl, N. L. Williams, Y. Lokesh, and P. Ravi (2021) PettingZoo: gym for multi-agent reinforcement learning. External Links: 2009.14471, Link Cited by: §IV. [22] A. Akella (2025) On the Fundamental Limitations of Decentralized Learnable Reward Shaping in Cooperative Multi-Agent Reinforcement Learning. External Links: 2511.00034, Link Cited by: §V.