Paper deep dive
Communication-Aware Multi-Agent Reinforcement Learning for Decentralized Cooperative UAV Deployment
Enguang Fan, Yifan Chen, Zihan Shan, Matthew Caesar, Jae Kim
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 94%
Last extracted: 3/22/2026, 5:02:11 AM
Summary
The paper introduces a graph-based multi-agent reinforcement learning (MARL) framework for decentralized cooperative UAV deployment. It utilizes centralized training with decentralized execution (CTDE) and a dual-attention architecture (agent-entity attention and neighbor self-attention) to handle partial observability and distance-limited communication. The method is evaluated on cooperative relay deployment (DroneConnect) and adversarial engagement (DroneCombat), demonstrating high coverage, scalability, and zero-shot generalization to unseen team sizes.
Entities (5)
Relation Signals (3)
CTDE → enables → Decentralized Execution
confidence 95% · a centralized critic and global state are available only during training, while each UAV executes a shared policy using local observations
DroneConnect → uses → CTDE
confidence 95% · We evaluate our approach primarily on a cooperative relay deployment task (DroneConnect) ... We train the swarm under CTDE.
Dual-Attention Graph Encoder → improves → DroneConnect
confidence 92% · the proposed method achieves high coverage under restricted communication and partial observation
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Autonomous Unmanned Aerial Vehicle (UAV) swarms are increasingly used as rapidly deployable aerial relays and sensing platforms, yet practical deployments must operate under partial observability and intermittent peer-to-peer links. We present a graph-based multi-agent reinforcement learning framework trained under centralized training with decentralized execution (CTDE): a centralized critic and global state are available only during training, while each UAV executes a shared policy using local observations and messages from nearby neighbors. Our architecture encodes local agent state and nearby entities with an agent-entity attention module, and aggregates inter-UAV messages with neighbor self-attention over a distance-limited communication graph. We evaluate primarily on a cooperative relay deployment task (DroneConnect) and secondarily on an adversarial engagement task (DroneCombat). In DroneConnect, the proposed method achieves high coverage under restricted communication and partial observation (e.g. 74% coverage with M = 5 UAVs and N = 10 nodes) while remaining competitive with a mixed-integer linear programming (MILP) optimization-based offline upper bound, and it generalizes to unseen team sizes without fine-tuning. In the adversarial setting, the same framework transfers without architectural changes and improves win rate over non-communicating baselines.
Tags
Links
- Source: https://arxiv.org/abs/2603.16141v1
- Canonical: https://arxiv.org/abs/2603.16141v1
Trouble viewing inline? Open PDF directly →
Full Text
31,881 characters extracted from source content.
Expand or collapse full text
t0.75in Communication-Aware Multi-Agent Reinforcement Learning for Decentralized Cooperative UAV Deployment Enguang Fan†*, Yifan Chen†*, Zihan Shan†, Matthew Caesar†, Jae Kim§ *These authors contributed equally to this work. Abstract Autonomous Unmanned Aerial Vehicle (UAV) swarms are increasingly used as rapidly deployable aerial relays and sensing platforms, yet practical deployments must operate under partial observability and intermittent peer-to-peer links. We present a graph-based multi-agent reinforcement learning framework trained under centralized training with decentralized execution (CTDE): a centralized critic and global state are available only during training, while each UAV executes a shared policy using local observations and messages from nearby neighbors. Our architecture encodes local agent state and nearby entities with an agent–entity attention module, and aggregates inter-UAV messages with neighbor self-attention over a distance-limited communication graph. We evaluate primarily on a cooperative relay deployment task (DroneConnect) and secondarily on an adversarial engagement task (DroneCombat). In DroneConnect, the proposed method achieves high coverage under restricted communication and partial observation (e.g. 74%74\% coverage with M=5M=5 UAVs and N=10N=10 nodes) while remaining competitive with a mixed-integer linear programming (MILP) optimization-based (offline) upper bound, and it generalizes to unseen team sizes without fine-tuning. In the adversarial setting, the same framework transfers without architectural changes and improves win-rate over non-communicating baselines. I Introduction Unmanned Aerial Vehicles (UAVs), commonly known as drones, are increasingly deployed as mobile sensing and communication platforms. A prominent application is to use autonomous UAVs as rapidly deployable aerial relays when terrestrial infrastructure is damaged by natural disasters or overloaded during crowded events [21, 5]. In scenarios such as wildfire monitoring and battlefield surveillance, UAVs may also operate beyond an operator’s control radius, requiring on-board autonomy and peer-to-peer coordination. In such scenarios, UAV teams must decide where to position themselves to maximize sensing or communication coverage over dynamic areas of interest. These coverage and placement decisions naturally give rise to optimization formulations. Many multi-UAV coverage and deployment tasks can be viewed through the lens of the Maximum Coverage Location Problem (MCLP), which is NP-hard and becomes computationally intractable in large or dynamic environments [16]. Moreover, real deployments are characterized by partial observability (each UAV can only sense nearby entities) and communication constraints (only nearby UAVs can exchange messages), making purely centralized controllers fragile and difficult to scale. To address these challenges, we develop a scalable multi-UAV deployment control system by integrating Multi-Agent Reinforcement Learning (MARL) with a graph-based environment representation. We adopt centralized training with decentralized execution (CTDE): during training, a centralized critic can access global information to stabilize learning, while during execution each UAV runs a shared policy using only local observations and peer-to-peer messages from communication neighbors. Concretely, we model the environment as an agent–entity graph and use attention-based embeddings to represent the state of each agent. We evaluate our approach primarily on a cooperative relay deployment task (DroneConnect) under full/partial observability and unrestricted/restricted communication, and we include a secondary mixed cooperative–competitive task (DroneCombat) to demonstrate applicability beyond fully cooperative settings. To summarize, our contributions are as follows. • We propose a multi-agent reinforcement learning (MARL) framework for multi-UAV deployment. The proposed framework is trained with centralized training decentralized execution (CTDE), under partial observability and distance-limited communication constraints. • We introduce a dual-attention graph encoder which encodes: (i) agent–entity attention for local environment embedding and (i) neighbor self-attention for inter-agent message sharing aggregation. • We demonstrate high-coverage decentralized relay deployment in a fully cooperative task (DroneConnect) and show the same framework transfers to an adversarial engagement task (DroneCombat) without architectural changes. The rest of this paper is structured as follows. Section I reviews prior research in learning and wireless communication for multi-UAV systems. Section I introduces our environment embedding, message sharing, and CTDE learning design. Section IV presents the simulation scenarios. We report evaluation results in Section V and conclude in Section VI. TABLE I: Key notation used throughout the paper. Symbol Description M / N number of UAVs / number of nodes (entities) i(t)p_i(t) / j(t)u_j(t) position of UAV i / node j at time t rsr_s / rcr_c / rcovr_cov sensing radius / communication radius / coverage radius s(i)N_s(i) entities sensed by UAV i (within rsr_s) c(i)N_c(i) UAV neighbors of i that can communicate (within rcr_c) hih_i / mim_i latent embedding of UAV i / aggregated message FO/PO full / partial observability UC/RC unrestricted / restricted communication I Related Work This section reviews existing research related to our work. We first summarize reinforcement-learning approaches for UAV control, then discuss communication mechanisms and graph-based representations for multi-agent coordination. Maximum Coverage Location Problem for multiple UAVs: Early research on UAV deployment has commonly framed the problem as a variant of the Maximum Coverage Location Problem (MCLP). When applied to UAV systems, MCLP captures the core challenge of selecting drone locations that maximize sensing or communication coverage under resource constraints. A representative formulation is the Maximum Coverage Facility Location Problem with Drones (MCFLPD) proposed by Chauhan et al.[3], which models UAV deployment as a static mixed‑integer program incorporating battery‑limited range, energy consumption, and facility capacities; despite its expressiveness, the MCFLPD quickly becomes computationally expensive and requires specialized heuristics for a tractable solution. Reinforcement Learning for Multi-UAV Systems: Researchers have demonstrated the utility of reinforcement learning algorithms in UAV-assisted wireless networks [9, 20]. Lee et al. [13] introduced the DroneDR framework for UAV deployment within a centralized architecture. Still, this approach poses a single point of failure and lacks practicality in satellite-denied environments. Kaviani et al. [12] proposed DeepCQ+, a deep reinforcement-learning-based routing protocol for highly dynamic mobile ad hoc networks. Our work addresses similar scenarios but adopts a distributed approach. UAVs are also utilized for wildfire monitoring in [11], where Julian et al. employed deep Q-learning for path planning. At the same time, the field of adversarial multi-UAV environments remains relatively unexplored. Multi-Agent Reinforcement Learning: MARL encompasses three interaction categories: fully cooperative, fully competitive, and mixed. Our objective is to develop algorithms suitable for all agent interactions. Moreover, MARL approaches can be categorized into centralized, decentralized, and hybrid methods. Centralized approaches use a single agent to formulate multi-agent systems, which is challenging to scale as the number of agents increases. Decentralized approaches, such as Tampuu et al.’s Q-learning [18], employ independent Q-value functions for each agent but struggle in non-stationary environments. Centralized learning with decentralized execution is another approach, represented by algorithms such as COMA [6], BiCNet [15], and MADDPG [14], in which a centralized critic is accessible only during training. However, such approaches often assume a fixed number of agents, limiting their applicability in dynamic environments with varying agent counts. Communication Mechanisms Between Agents: Many MARL approaches neglect explicit inter-agent communication. Differentiable communication protocols, as seen in CommNet [17] and VAIN [8], have improved communication using attention mechanisms. We apply scaled dot-product attention for inter-agent communication. Real-world scenarios often impose communication limits based on proximity, as seen in TarMAC [4]. DGN [10] allows agents to communicate with their nearest neighbors, aligning with practical drone swarm operations. We introduce distance-based communication restrictions, leading to multiple connected components in the agent graph and thereby increasing complexity and cooperation within the team. Graph Neural Networks: Graphs naturally model multi-agent systems, with nodes representing agents. GNNs, such as message-passing neural networks [7] and the Graph Attention Network (GAT) [19], employ trainable weights for feature propagation among nodes. Agarwal et al. [1] introduced entity graphs for environment integration, focusing on fully cooperative settings. OpenAI explored multi-agent reinforcement learning for the emergence of complex behavior [2]. In our work, we adopt an agent-entity graph to aggregate environment information across diverse settings, including cooperative, competitive, and mixed environments. I Environment Modeling To model large multi-agent environments efficiently, we represent the swarm and its surroundings as an agent–entity graph G=(V,E)G=(V,E), where vertices correspond to UAV agents and observable environment entities (e.g., ground nodes), and edges encode sensing and communication relationships. Each UAV i forms (i) a sensed-entity set s(i)N_s(i) from entities within its sensing radius and (i) a communication-neighbor set c(i)N_c(i) from UAVs within its communication radius. This separation allows us to model partial observability (via sN_s) and restricted communication (via cN_c) in a unified way. I-A Message Passing Over the Communication Graph We use message passing to aggregate information among UAVs over the communication graph. Let hi(k)h_i^(k) denote UAV i’s latent embedding after k rounds of communication message passing, with hi(0)h_i^(0) initialized from its local observation embedding (Section I-B). A generic message passing round can be written as mi(k) m^(k)_i =fagg(k)(hi(k−1),hj(k−1):j∈c(i)),1≤k≤K =f_agg^(k)\! (h^(k-1)_i,\;\h^(k-1)_j:j _c(i)\ ), 1≤ k≤ K (1) hi(k) h^(k)_i =fupd(hi(k−1),mi(k)), =f_upd\! (h^(k-1)_i,\;m^(k)_i ), (2) where mi(k)m^(k)_i is the aggregated message and K is the number of message passing rounds. Here, fagg(k)(⋅)f_agg^(k)(·) denotes a permutation-invariant aggregation operator over neighbor embeddings (instantiated as attention-weighted aggregation in Section I-C), and fupd(⋅)f_upd(·) is a learnable update function (e.g., an MLP or GRU) that fuses the previous embedding with the aggregated message. In unrestricted communication (UC), c(i)N_c(i) contains all other UAVs; in restricted communication (RC), c(i)=j≠i:‖i−j‖2≤rcN_c(i)=\j≠ i:\|p_i-p_j\|_2≤ r_c\. I-B Environment Embedding Each UAV i maintains a local state SiS_i (e.g., position and velocity) and observes a variable-size set of entities within its sensing range. We encode the UAV state and entity features via ia ^a_i =fa(Si), =f_a(S_i), (3) i,l _i,l =fe(i,l),l∈s(i), =f_e(x_i,l), l _s(i), (4) where i,lx_i,l denotes the feature vector of entity l as observed by UAV i. To obtain a fixed-size environment summary that is invariant to the number of sensed entities, we apply scaled dot-product attention with the UAV embedding as the query and entity embeddings as keys/values: i _i =qia, =W_qh^a_i, (5) i,l _i,l =ki,l,l∈s(i), =W_ke_i,l, l _s(i), (6) i,l _i,l =vi,l,l∈s(i), =W_ve_i,l, l _s(i), (7) αi,l _i,l =exp(i⊤i,l/dk)∑l′∈s(i)exp(i⊤i,l′/dk), = \! (q_i k_i,l/ d_k ) _l _s(i) \! (q_i k_i,l / d_k ), (8) iagg ^agg_i =∑l∈s(i)αi,li,l, = _l _s(i) _i,l\,v_i,l, (9) hi(0) h^(0)_i =[ia;iagg]. = [h^a_i;E^agg_i ]. (10) where dkd_k is the key dimension and [;][;] denotes concatenation. The resulting hi(0)h^(0)_i is used as the per-agent input to the policy and as the initialization for communication message passing. I-C Inter-Agent Message Sharing UAV i aggregates messages from its communication neighbors c(i)N_c(i) using self-attention. Let ~c(i)=c(i)∪i N_c(i)=N_c(i)∪\i\ denote the neighbor set with a self-loop. For round k, we compute ic ^c_i =qchi(k−1), =W^c_qh^(k-1)_i, (11) jc ^c_j =kchj(k−1),j∈~c(i), =W^c_kh^(k-1)_j, j∈ N_c(i), (12) jc ^c_j =vchj(k−1),j∈~c(i), =W^c_vh^(k-1)_j, j∈ N_c(i), (13) βi,j _i,j =exp((ic)⊤jc/dk)∑j′∈~c(i)exp((ic)⊤j′c/dk), = \! ((q^c_i) k^c_j/ d_k ) _j ∈ N_c(i) \! ((q^c_i) k^c_j / d_k ), (14) mi(k) m^(k)_i =∑j∈~c(i)βi,jjc, = _j∈ N_c(i) _i,j\,v^c_j, (15) hi(k) h^(k)_i =fupd(hi(k−1),mi(k)) =f_upd\! (h^(k-1)_i,\;m^(k)_i ) (16) During decentralized execution, each UAV performs this aggregation using only messages received from c(i)N_c(i), matching the RC setting. I-D Centralized Training with Decentralized Execution (CTDE) We train the swarm under CTDE. During execution, each UAV i samples actions from a decentralized actor ai(t)∼πθ(⋅∣oi(t),hi(K)(t)),a_i(t) _θ\! (· o_i(t),h^(K)_i(t) ), (17) where oi(t)o_i(t) is the local observation and hi(K)(t)h^(K)_i(t) is the final embedding after K communication rounds. During training, we additionally use a centralized critic that has access to global information, e.g., Vϕ((t)),(t)=Si(t)i=1M∪j(t)j=1N.V_φ\! (s(t) ), (t)=\S_i(t)\_i=1^M∪\u_j(t)\_j=1^N. (18) The critic is used only for learning; at test time, UAVs execute πθ _θ without access to (t)s(t) or any centralized coordinator. IV Scenarios and Tasks We study multi-UAV deployment under partial observability and distance-limited communication. Our primary focus is a cooperative relay deployment task (DroneConnect), and we include a secondary mixed cooperative–competitive task (DroneCombat) to demonstrate that the same CTDE graph-based framework applies beyond purely cooperative settings. We also compare against a static optimization-based formulation as a reference upper bound. IV-A Optimization-Based Static View A common abstraction of coverage and relay placement is to maximize the amount of demand covered within a service radius while penalizing relocation costs, where facilities represent drones and demand represents ground nodes. Let i0p_i^0 be the current position of UAV i, and let ip_i be its placement decision. A simplified maximum-coverage objective can be written as maxii=1M _\p_i\_i=1^M\;\; ∑j=1Nwjzj−α∑i=1M‖i−i0‖2 _j=1^Nw_j\,z_j\;-\;α _i=1^M \|p_i-p_i^0 \|_2 (19) s.t. zj=[mini∈1,…,M‖j−i‖2≤rcov],j=1,…,N z_j=I\! [ _i∈\1,…,M\ \|u_j-p_i \|_2≤ r_cov ],\;\;j=1,…,N where wjw_j is a node priority weight and rcovr_cov is the service/coverage radius. This problem is NP-hard; to provide an optimization-based reference, we discretize candidate UAV locations and solve a mixed-integer linear programming (MILP) formulation of (19) offline, which serves as an approximate upper bound on the attainable coverage for static snapshots. We highlight that MILP requires intensive computation and therefore not suitable for online control. IV-B DroneConnect Scenario DroneConnect models a team of UAV relays repositioning to provide coverage to mobile ground nodes (Fig. 1). IV-B1 Action Space For UAV i, the continuous action is a 2D force (or acceleration command) ai=(Fx,Fy)a_i=(F_x,F_y) that updates its velocity and position. IV-B2 Observability and Communication Settings We evaluate four settings that combine observation and communication constraints: • FO (full observability): each UAV observes all ground-node states (and UAV states). • PO (partial observability): each UAV observes only entities within sensing radius rsr_s (i.e., s(i)N_s(i)). • UC (unrestricted communication): all UAVs can exchange messages (complete communication graph). • RC (restricted communication): UAVs communicate only if within radius rcr_c (i.e., c(i)N_c(i)). IV-B3 Reward The DroneConnect task is fully cooperative: all UAVs share the same team reward during centralized training, and execute decentralized policies at test time. Let dj(t)=mini‖j(t)−i(t)‖2d_j(t)= _i\|u_j(t)-p_i(t)\|_2 be the distance from node j to its nearest UAV, and let cj(t)=[dj(t)≤rcov]c_j(t)=I[d_j(t)≤ r_cov] indicate whether node j is covered. We use the normalized reward r(t)=λcov⋅1N∑j=1Ncj(t)−λdist⋅1N∑j=1Ndj(t)rcov,r(t)= _cov· 1N _j=1^Nc_j(t)\;-\; _dist· 1N _j=1^N d_j(t)r_cov, (20) where λcov _cov and λdist _dist trade off coverage quantity and service quality. Figure 1: DroneConnect environment with 2 UAV relays and 4 mobile nodes. IV-C DroneCombat Scenario To demonstrate generality beyond fully cooperative settings, we also consider a lightweight adversarial engagement environment DroneCombat as a secondary task. Drones are split into two teams, and each team aims to eliminate all opponents by firing directional laser beams. We emphasize that DroneCombat is used here only as an auxiliary scenario; our primary evaluation is DroneConnect. IV-C1 Action Space Each drone uses a 3D continuous action ai=(Fx,Fy,Frot)a_i=(F_x,F_y,F_rot) that controls planar motion and rotation (firing direction). IV-C2 Reward We use a sparse-and-dense shaped reward (Table I) that encourages successful hits, discourages wasted firing, and rewards winning quickly. Event Reward Drone i emits a laser beam -0.1 Drone i emits a laser but misses any target -1 Drone i’s laser hits an opponent +3 Drone i gets hit by a laser beam -3 Each timestep −50/T-50/T All opponent drones are eliminated +20 TABLE I: Rewards for the secondary DroneCombat task (T is episode length). V Evaluation We evaluate the proposed CTDE graph-based MARL framework on aforementioned two scenarios in Section IV. Our primary focus is the cooperative DroneConnect task while DroneCombat is reported as a secondary demonstration of generality. V-A Experimental Setup We report representative settings here to support reproducibility. In DroneConnect, UAVs and nodes move in a bounded 2D area of size 100×100100× 100 with timestep Δt=0.1 t=0.1s and episode length T=200T=200 steps. Each UAV senses entities within radius rsr_s and communicates within radius rcr_c (RC); in UC we allow all-to-all messaging. We train using CTDE actor–critic (PPO-style) with a centralized critic and decentralized actors, for 2×1062× 10^6 environment steps and evaluate over 50 episodes. Unless otherwise stated, we average results over 3 random seeds. For communication, each UAV transmits a d=64d=64-dimensional message embedding to each neighbor per timestep; thus the per-step communication cost scales with the average degree of the RC graph. V-B DroneConnect Results We use M to represent the number of UAVs and N to represent the number of ground nodes to cover. The evaluation metric is the average coverage ratio over an episode, defined as the number of covered nodes divided by the total number of nodes. Figure 2: Average coverage per timestep as the number of UAVs (M) and nodes (N) vary (FO+UC setting). V-B1 Coverage Results and Ablations Table I summarizes coverage under different observability and communication constraints. Overall, our CTDE approach maintains strong coverage under partial observation and restricted communication, while remaining competitive with the static MILP upper bound. We also report two minimal ablations: disabling inter-UAV communication and replacing entity attention with mean pooling; both degrade performance in the challenging RC+PO (restricted communication and partial observability) setting. Method M N Comm Obs Coverage Ratio Ours (CTDE, dual-attention) 3 6 UC FO 0.76±0.010.76± 0.01 Ours (CTDE, dual-attention) 3 6 RC FO 0.74±0.020.74± 0.02 Ours (CTDE, dual-attention) 3 6 UC PO 0.72±0.020.72± 0.02 Ours (CTDE, dual-attention) 3 6 RC PO 0.71±0.020.71± 0.02 Ours (CTDE, dual-attention) 5 10 UC FO 0.79±0.020.79± 0.02 Ours (CTDE, dual-attention) 5 10 RC FO 0.77±0.020.77± 0.02 Ours (CTDE, dual-attention) 5 10 UC PO 0.76±0.020.76± 0.02 Ours (CTDE, dual-attention) 5 10 RC PO 0.74±0.020.74± 0.02 Ablation: no inter-communication 5 10 RC PO 0.65±0.030.65± 0.03 Ablation: no entity attention 5 10 RC PO 0.63±0.030.63± 0.03 Static MILP upper bound 3 6 – – 0.770.77 Static MILP upper bound 5 10 – – 0.800.80 Centralized single-agent RL 3 6 – FO 0.75±0.020.75± 0.02 Centralized single-agent RL 5 10 – FO 0.79±0.020.79± 0.02 TABLE I: DroneConnect coverage results (mean ± std over 3 seeds). UC/RC: unrestricted/restricted communication; FO/PO: full/partial observability. V-B2 Zero-Shot Policy Generalization We evaluate zero-shot generalization by applying a policy trained with M=5M=5 UAVs directly to scenarios with different team sizes (RC+PO), without fine-tuning. Table IV reports representative coverage ratios. Overall, performance remains stable across team sizes, suggesting that the graph-based representation helps the policy generalize across varying numbers of agents. TABLE IV: Coverage results with zero-shot transfer learning in DroneConnect. Num_drones Num_nodes Comm Obs Coverage Ratio M−2M-2 6 RC PO 0.70 M−1M-1 6 RC PO 0.72 M=5M=5 10 RC PO 0.74 M+1M+1 10 RC PO 0.80 M+2M+2 10 RC PO 0.82 V-B3 Qualitative Coordination and Overlap In addition to coverage ratio, we quantify coordination by measuring the coverage overlap rate, defined as the fraction of covered nodes that are simultaneously within rcovr_cov of more than one UAV. In the challenging RC+PO setting with M=5,N=10M=5,N=10, our learned policy achieves a low overlap rate 0.120.12 compared to the no-communication ablation 0.190.19, indicating better division of coverage responsibilities. Figure 3 illustrates representative trajectories and communication links under different team sizes. ((a)) M=3,N=6M=3,N=6 ((b)) M=3,N=10M=3,N=10 ((c)) M=4,N=8M=4,N=8 ((d)) M=5,N=10M=5,N=10 Figure 3: Representative DroneConnect snapshots showing division of coverage tasks. Red links indicate in-range (RC) inter-UAV communication. V-C DroneCombat Results We briefly report results on the DroneCombat task (Section IV-C) to demonstrate that the same CTDE graph-based architecture can be applied to a mixed cooperative–competitive environment and leave detailed study to future work. We evaluate a 5-vs-5 setting and compare against simple baselines. Method 5v5 win rate 5v5 avg. episode steps Ours (CTDE, dual-attention) 0.62±0.050.62± 0.05 140±18140± 18 No-communication ablation 0.49±0.040.49± 0.04 168±22168± 22 Independent agents (no CTDE) 0.42±0.030.42± 0.03 175±25175± 25 TABLE V: DroneCombat results (mean ± std over 3 seeds). Figure 4: Sample progression of a 5-vs-5 DroneCombat episode (attackers in pink, defenders in cyan). Eliminated drones are shown in darker colors. VI Conclusion We presented a centralized training with decentralized execution (CTDE) multi-agent reinforcement learning framework for cooperative UAV deployment under partial observability and communication constraints. Our method represents the environment as an agent–entity graph and uses dual attention: agent–entity attention for local environment embedding and neighbor self-attention for inter-UAV message aggregation. During execution, each UAV runs a decentralized policy using only local observations and peer-to-peer messages, without any centralized coordinator. In the cooperative DroneConnect task, our approach achieves high coverage under restricted communication and partial observability while remaining competitive with the static MILP upper bound. We also showed that the learned policy can generalize zero-shot to different team sizes in DroneConnect. Finally, we included a mixed cooperative–competitive DroneCombat scenario to illustrate that the same architecture can be applied beyond purely cooperative settings. We leave detailed wireless QoS models and explicit communication-cost constraints analysis for future work. References [1] A. Agarwal, S. Kumar, K. Sycara, and M. Lewis (2020) Learning transferable cooperative behavior in multi-agent teams. In Proceedings of the 19th International Conference on Autonomous Agents and MultiAgent Systems, AAMAS ’20, Richland, SC, p. 1741–1743. External Links: ISBN 9781450375184 Cited by: §I. [2] B. Baker, I. Kanitscheider, T. Markov, Y. Wu, G. Powell, B. McGrew, and I. Mordatch (2019) Emergent tool use from multi-agent autocurricula. In International conference on learning representations, Cited by: §I. [3] D. Chauhan, A. Unnikrishnan, and M. Figliozzi (2019) Maximum coverage capacitated facility location problem with range constrained drones. Transportation Research Part C: Emerging Technologies 99, p. 1–18. External Links: ISSN 0968-090X, Document, Link Cited by: §I. [4] A. Das, T. Gervet, J. Romoff, D. Batra, D. Parikh, M. Rabbat, and J. Pineau (2019) Tarmac: targeted multi-agent communication. In International Conference on machine learning, p. 1538–1546. Cited by: §I. [5] E. Fan, A. Peng, M. Caesar, J. Kim, J. Eckhardt, G. Kimberly, and D. Osipychev (2023) Towards effective swarm-based gps spoofing detection in disadvantaged platforms. In MILCOM 2023 - 2023 IEEE Military Communications Conference (MILCOM), Vol. , p. 722–728. External Links: Document Cited by: §I. [6] J. N. Foerster, G. Farquhar, T. Afouras, N. Nardelli, and S. Whiteson (2018) Counterfactual multi-agent policy gradients. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference and Eighth AAAI Symposium on Educational Advances in Artificial Intelligence, AAAI’18/IAAI’18/EAAI’18. External Links: ISBN 978-1-57735-800-8 Cited by: §I. [7] J. Gilmer, S. S. Schoenholz, P. F. Riley, O. Vinyals, and G. E. Dahl (2017) Neural message passing for quantum chemistry. In Proceedings of the 34th International Conference on Machine Learning - Volume 70, ICML’17, p. 1263–1272. Cited by: §I. [8] Y. Hoshen (2017) Vain: attentional multi-agent predictive modeling. Advances in neural information processing systems 30. Cited by: §I. [9] J. Hu, H. Zhang, and L. Song (2019) Reinforcement learning for decentralized trajectory design in cellular uav networks with sense-and-send protocol. IEEE Internet of Things Journal 6 (4), p. 6177–6189. External Links: Document Cited by: §I. [10] J. Jiang, C. Dun, T. Huang, and Z. Lu (2018) Graph convolutional reinforcement learning. arXiv preprint arXiv:1810.09202. Cited by: §I. [11] K. D. Julian and M. J. Kochenderfer (2019) Distributed wildfire surveillance with autonomous aircraft using deep reinforcement learning. Journal of Guidance, Control, and Dynamics 42 (8), p. 1768–1778. External Links: Document, Link, https://doi.org/10.2514/1.G004106 Cited by: §I. [12] S. Kaviani, B. Ryu, E. Ahmed, K. Larson, A. Le, A. Yahja, and J. H. Kim (2021) DeepCQ+: robust and scalable routing with multi-agent deep reinforcement learning for highly dynamic networks. In MILCOM 2021 - 2021 IEEE Military Communications Conference (MILCOM), Vol. , p. 31–36. External Links: Document Cited by: §I. [13] I. Lee, V. Babu, M. Caesar, and D. Nicol (2021) Deep reinforcement learning for uav-assisted emergency response. In MobiQuitous 2020 - 17th EAI International Conference on Mobile and Ubiquitous Systems: Computing, Networking and Services, MobiQuitous ’20, New York, NY, USA, p. 327–336. External Links: ISBN 9781450388405, Link, Document Cited by: §I. [14] R. Lowe, Y. I. Wu, A. Tamar, J. Harb, O. Pieter Abbeel, and I. Mordatch (2017) Multi-agent actor-critic for mixed cooperative-competitive environments. Advances in neural information processing systems 30. Cited by: §I. [15] P. Peng, Y. Wen, Y. Yang, Q. Yuan, Z. Tang, H. Long, and J. Wang (2017) Multiagent bidirectionally-coordinated nets: emergence of human-level coordination in learning to play starcraft combat games. arXiv preprint arXiv:1703.10069. Cited by: §I. [16] M. Sobouti, R. Mahapatra, and M. A. Rahman (2024) Utilizing uavs in wireless networks: advantages, challenges, objectives, and solution methods. Vehicles 6 (3), p. 764–789. External Links: Document Cited by: §I. [17] S. Sukhbaatar, R. Fergus, et al. (2016) Learning multiagent communication with backpropagation. Advances in neural information processing systems 29. Cited by: §I. [18] A. Tampuu, T. Matiisen, D. Kodelja, I. Kuzovkin, K. Korjus, J. Aru, J. Aru, and R. Vicente (2017-04) Multiagent cooperation and competition with deep reinforcement learning. PLOS ONE 12 (4), p. 1–15. External Links: Document, Link Cited by: §I. [19] P. Veličković, G. Cucurull, A. Casanova, A. Romero, P. Lio, and Y. Bengio (2017) Graph attention networks. arXiv preprint arXiv:1710.10903. Cited by: §I. [20] C. Wang, J. Wang, Y. Shen, and X. Zhang (2019) Autonomous navigation of uavs in large-scale complex environments: a deep reinforcement learning approach. IEEE Transactions on Vehicular Technology 68 (3), p. 2124–2136. External Links: Document Cited by: §I. [21] S. Yin, Z. Qu, and L. Li (2018) Uplink resource allocation in cellular networks with energy-constrained uav relay. In 2018 IEEE 87th Vehicular Technology Conference (VTC Spring), p. 1–5. Cited by: §I.