Paper deep dive
Threat-Aware Energy-Efficient Deployment for Dynamic UAV Networks: A Multi-Agent RL Approach
Faisal Al-Kamali, Hussein A. Ammar, Francois Chan, James H. Bayes, Yasser Gadallah, Mohamed H. Ahmed
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%
Last extracted: 9/28/2026, 3:22:02 AM
Summary
This paper proposes a threat-aware, energy-efficient deployment framework for multi-UAV networks acting as aerial base stations in hazardous environments. The framework utilizes a three-stage approach: Threat-Aware K-Means (TAKM) for safe initial clustering and UAV count determination, an optimal matching stage for energy-efficient assignment, and a Multi-Agent Twin Delayed Deep Deterministic Policy Gradient (MATD3) algorithm for dynamic trajectory, power, and user association optimization. The method ensures zero safety violations while achieving superior energy efficiency and faster convergence compared to heuristic baselines like GPSO and OPSO.
Entities (7)
Relation Signals (6)
Proposed Framework → ensures → Zero Safety Violations
confidence 95% · Simulation results show that the proposed framework achieves zero observed safety violations
TAKM → isvariantof → K-Means
confidence 95% · threat-aware K-means (TAKM) algorithm... replaces standard clustering
TAKM → usedfor → Safe Initial Placement
confidence 95% · TAKM algorithm determines the minimum required UAVs and computes safe initial placements.
MATD3 → usedfor → Dynamic Optimization
confidence 95% · MATD3 algorithm dynamically optimizes trajectories, power, and user associations.
Proposed Framework → hascomparableperformanceto → OPSO
confidence 90% · achieves performance comparable to that of the optimized PSO (OPSO)
Proposed Framework → outperforms → GPSO
confidence 90% · the proposed framework outperforms the greedy particle swarm optimization (GPSO)
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Ensuring operational safety in threat-prone environments remains a critical challenge for multi-UAV networks serving as aerial base stations. This paper proposes an efficient framework to maximize global energy efficiency (EE) while promoting safe operation through threat-aware clustering and reward-based safety enforcement. The proposed framework is executed in three steps. First, a threat-aware K-means (TAKM) algorithm determines the minimum required UAVs and computes safe initial placements. Second, an optimal matching stage assigns physical UAVs to these centroids to minimize energy expenditure. Third, a threat-aware multi-agent twin delayed deep deterministic policy gradient (MATD3) algorithm dynamically optimizes trajectories, power, and user associations. Simulation results show that the proposed framework achieves zero observed safety violations in the considered scenarios while achieving superior EE and faster convergence than other learning methods and non-clustering baselines. Compared to heuristic optimization, the proposed framework outperforms the greedy particle swarm optimization (GPSO) and achieves performance comparable to that of the optimized PSO (OPSO), while incurring significantly lower online deployment computational complexity. Furthermore, the proposed framework demonstrates effective generalization to unseen user distributions, large UAV fleets, and different threat geometries, while maintaining zero safety violations.
Tags
Links
- Source: https://arxiv.org/abs/2609.30690v1
- Canonical: https://arxiv.org/abs/2609.30690v1
Trouble viewing inline? Open PDF directly →
Full Text
104,517 characters extracted from source content.
Expand or collapse full text
Threat-Aware Energy-Efficient Deployment for Dynamic UAV Networks: A Multi-Agent RL Approach Faisal Al-Kamali Hussein A. Ammar Francois Chan James H. Bayes Yasser Gadallah Mohamed H. Ahmed †thanks: Manuscript received x, x.†thanks: Faisal Al-Kamali, Francois Chan, and Mohamed H. Ahmed are with the School of Electrical Engineering and Computer Science, University of Ottawa, Ottawa, ON K1N 6N5, Canada (e-mails: falkamal@uottawa.ca, chan-f@rmc.ca, and mahme3@uottawa.ca).†thanks: Faisal Al-Kamali is also with the Department of Electrical Engineering, Ibb University, Ibb, Yemen.†thanks: Francois˜Chan, Hussein˜A.˜Ammar, and James˜H.˜Bayes are with the Department of Electrical and Computer Engineering, Royal Military College, Kingston, ON, K7K 7B4, Canada (e-mails: chan-f@rmc.ca, hussein.ammar@rmc-cmr.ca, james.bayes@rmc-cmr.ca†thanks: Yasser Gadallah is with the Department of Electronics and Communications Engineering, The American University in Cairo, New Cairo 11835, Egypt (email: ygadallah@ieee.org). Abstract Ensuring operational safety in threat-prone environments remains a critical challenge for multi-UAV networks serving as aerial base stations. This paper proposes an efficient framework to maximize global energy efficiency (E) while promoting safe operation through threat-aware clustering and reward-based safety enforcement. The proposed framework is executed in three steps. First, a threat-aware K-means (TAKM) algorithm determines the minimum required UAVs and computes safe initial placements. Second, an optimal matching stage assigns physical UAVs to these centroids to minimize energy expenditure. Third, a threat-aware multi-agent twin delayed deep deterministic policy gradient (MATD3) algorithm dynamically optimizes trajectories, power, and user associations. Simulation results show that the proposed framework achieves zero observed safety violations in the considered scenarios while achieving superior E and faster convergence than other learning methods and non-clustering baselines. Compared to heuristic optimization, the proposed framework outperforms the greedy particle swarm optimization (GPSO) and achieves performance comparable to that of the optimized PSO (OPSO), while incurring significantly lower online deployment computational complexity. Furthermore, the proposed framework demonstrates effective generalization to unseen user distributions, large UAV fleets, and different threat geometries, while maintaining zero safety violations. Index Terms: Unmanned aerial vehicle (UAV), trajectory design, resource allocation, threat-aware clustering, multi-agent reinforcement learning. I Introduction I-A Background Unmanned aerial vehicles (UAVs), commonly known as drones, have emerged as an important technology for next-generation wireless communication networks. Owing to their mobility, flexible deployment, and relatively low operational cost, UAVs can provide rapid and adaptive connectivity in scenarios where terrestrial infrastructure is unavailable, damaged, or impractical [1, 2]. Their ability to establish favorable line-of-sight (LoS) links with ground users (GUs) reduces propagation loss and transmit power requirements, thereby enabling reliable and energy-efficient communication. These advantages make UAV-assisted systems a promising solution for diverse applications, including disaster recovery, emergency response, rural connectivity, and temporary hotspot provisioning [3]. UAVs can be used as temporary aerial base stations, such as in major events or emergencies, to support wireless communication, since they can be flexibly deployed and repositioned to provide on-demand connectivity for GUs. Consequently, many works have focused on single-UAV deployment and placement optimization in two- or three-dimensional space [4, 5]. These studies have shown clear advantages, such as improved coverage, reliable LoS links, and enhanced network performance. UAVs can also play a vital role in enhancing security within threat-prone environments. In [6], we present a novel UAV placement strategy that improves secure wireless communication by enforcing spatial threat exclusion. Nevertheless, single-UAV systems remain constrained by their limited communication, caching, and computational capabilities, which restrict their ability to support mission-critical services or serve a large number of GUs [7]. To overcome these limitations, the deployment of multi-UAV communication systems has been proposed [8, 9]. In such systems, multiple UAVs cooperatively serve GUs distributed over a large area, enabling simultaneous service. This cooperative approach not only increases system throughput but also reduces latency and improves scalability, thereby addressing challenges that single-UAV systems cannot overcome. However, the deployment of multi-UAV communication system entails several technical challenges, including optimal placement, trajectory design, and energy management [10, 11]. In particular, the limited on-board battery capacity significantly restricts their ability to provide sustained coverage. I-B Motivation and Novelty Ensuring the survivability and operational integrity of UAVs while maintaining reliable connectivity for GUs constitutes the primary motivation of this work. Specifically, we aim to deploy a multi-UAV network capable of providing reliable communication services in environments characterized by stringent safety constraints and the presence of threat zones. To address this challenge, the problem is formulated as a joint optimization of UAV trajectory design, transmit power allocation, and dynamic user association. Although this problem has attracted increasing research attention [12], [13, 14], most existing studies do not incorporate threat avoidance as a fundamental design constraint, thereby limiting their applicability in safety-critical scenarios. TABLE I: Comparison Between the Proposed Framework and Related Works Aspect [7] [15] [16] [17] [18] This Work User distribution Uniform Uniform Uniform Uniform Real-world Non-uniform Threat-zone awareness ✗ ✗ ✗ ✗ ✗ ✓ UAV count optimization ✗ ✗ ✗ ✗ ✗ ✓ User association strategy Dynamic Fixed Dynamic Fixed Fixed Dynamic Clustering scheme None MEM None Hotspot-based GAK-means TAKM Operational area 1000×10001000× 1000 m2 1000×10001000× 1000 m2 100×100100× 100 m2 500×500500× 500 m2 2000×20002000× 2000 m2 4000×40004000× 4000 m2 Conventional optimization and iterative methods [19], [20, 21] focus strictly on maximizing coverage and connectivity while ignoring the physical safety of the UAVs. Although, heuristic methods like particle swarm optimization (PSO) are more efficient for joint trajectory and power optimization, they are also unsuitable for real-time deployment. Their re-optimization-based nature requires iterative updates from scratch for every environmental change, incurring latency that exceeds the network’s coherence time in high-mobility scenarios. Reinforcement learning (RL) has recently emerged as an effective approach in multi-UAV networks, capable of learning and adapting from experience without requiring complete knowledge of the network environment [7], [15, 9]. Despite these advantages, existing RL approaches for UAV trajectory and resource optimization often neglect critical safety considerations, such as threat-zone restrictions, which are essential for practical deployment in risk-sensitive environments. As a result, these approaches are not suitable for scenarios that involve threat zones such as wildfire regions or hazardous areas. Moreover, conventional K-Means (CKM) prioritize geometric distance alone, which often leads to deploying UAVs inside threat zones. This creates an immediate risk of mission failure that current RL models cannot fix, as they typically lack a safety-aware initialization phase. To bridge this gap, we propose a three-stage framework that integrates threat avoidance into every phase. The main novelty lies in the threat-aware K-means (TAKM) algorithm, which replaces standard clustering with a projection-based approach that forces UAV centroids to remain within verified safe regions. Unlike previous methods, TAKM embeds threat avoidance directly into the optimization process. This safe deployment is then bridged via an optimal matching scheme to a multi-agent twin delayed deep deterministic policy gradient (MATD3) for dynamic control. The key novelty is the integration of threat-aware clustering, optimal deployment, and multi-agent RL (MARL) into a unified framework that enforces safety constraints throughout all phases of UAV network operation. I-C Related Works Recent research has extensively investigated UAV-assisted wireless communications with a focus on trajectory and power optimization. The study in [8] considers a multi-UAV wireless communication system and proposes an iterative algorithm that jointly optimizes user scheduling, UAV trajectory, and power control to maximize the minimum downlink throughput. Beyond throughput optimization, several studies have focused on secure UAV communications by co-designing trajectories and transmit power. In[12], a four-node system with a mobile UAV relay is investigated, employing an alternating optimization framework to maximize secrecy rate. Similarly, Zhang et al. [13] propose a joint trajectory and power control approach to improve secrecy performance, while the work in [19] considers UAV-ground communications in the presence of no-fly zones and develops a low-complexity algorithm to enhance both secrecy rate and computational efficiency. In addition to security-focused works, [14] investigates a UAV relay network with amplify-and-forward operation and proposes a low-complexity joint trajectory and power control design to minimize outage probability. Furthermore, [21] examines a UAV-assisted backscatter communication network, with simulations showing significant gains over other benchmark schemes. In [22], the authors jointly optimize UAV 3D trajectory, transmit power, and scheduling variables to improve system performance. However, conventional optimization-based methods have high computational complexity and require full knowledge of environmental models, which may not be available in practice. To overcome these limitations, RL-based approaches have been increasingly considered for scenarios with partial or no channel state information. RL can be applied in both single-agent [23] and multi-agent [15] settings, enabling UAVs to learn optimal policies for trajectory, power control, and resource allocation without requiring full knowledge of the environment. Also, tabular Q-learning methods have been applied for UAV placement and energy-efficient user association, as in [23], [24, 25]. However, the discrete state-action representation of these methods limits their applicability to continuous control problems [26], such as power allocation and trajectory design, and lead to high space/computational complexity [27]. Recently, the actor-critic framework [28] has been employed in UAV trajectory and resource optimization. In [29], deep deterministic policy gradient (DDPG) with deterministic policies provides a stable solution, and its multi-agent extension (MADDPG) [30] enables coordinated optimization in multi-UAV scenarios. MADDPG has been effectively utilized for joint UAV trajectory design, user fairness, load balancing, and energy efficiency [16], as well as for secure multi-UAV communications, including combined trajectory and transmit power optimization under minimum quality of service (QoS) requirements [31, 17]. In [15], MATD3 has been introduced, where a clustering-aided approach is developed to jointly optimize UAV trajectory, and transmit power, achieving faster convergence and improved performance. Recent advancements in DRL-aided UAV networks have explored various facets of system optimization. For instance, learning-based radio resource management was investigated in [32], while NOMA-assisted trajectory scheduling was explored in [33]. Meanwhile, path planning and trajectory control frameworks for MEC-integrated UAVs were proposed in [34]. More recently, the focus has shifted toward distributed many-agent perspectives to handle scalability and safety constraints in large-scale swarms [35]. Hybrid discrete-continuous MARL frameworks for UAV swarms were investigated in [36], where joint trajectory control, power allocation, and communication resource management were optimized using hybrid action representations. Unlike existing DRL- and MARL-based methods that assume threat-free settings, the key novelty of this work is its end-to-end energy-efficient approach for UAV network optimization in threat-prone environments. This approach enforces safety constraints throughout all phases of UAV network operation. Specifically, we integrate the TAKM algorithm, optimal matching of UAVs to safe centroids, and threat-aware MATD3 into a unified framework. As highlighted in Table I, which compares our framework with existing works [7, 15], [16, 17], [18], our proposed approach introduces several key innovations. While [15] presented a clustering-aided MARL method, it assumes uniformly distributed users and does not consider threat zones. In contrast, our framework is designed for non-uniform user distributions and is the only one that incorporates threat-zone awareness. Furthermore, our work addresses the optimization of the number of UAVs, a practical necessity absent in other referenced studies. Unlike the static user associations used in [15, 17], our method supports dynamic association updates at each time step based on system conditions. Finally, our work replaces conventional clustering schemes with the safety-critical TAKM algorithm, ensuring safety during both initialization and iterative updates. These enhancements, summarized in Table I, result in a practical and safety-aware UAV deployment framework suitable for real-world constrained environments. I-D Contribution The primary contribution of this paper is a risk-aware RL-based UAV placement framework that adapts to dynamic user association while maintaining low online complexity. The contributions of this paper are as follows: 1. We formulate an energy-efficiency maximization problem for UAV-assisted networks under explicit safety and QoS constraints, including minimum rate requirements, inter-UAV collision avoidance, and threat-zone avoidance. The safety and QoS requirements are formulated as hard constraints in the underlying optimization problem. However, MATD3 handles safety through a reward-based penalty rather than a hard action-space constraint. TAKM provides safe initialization and centroid updates, while the safety penalty discourages trajectory violations during learning. 2. We propose a TAKM clustering algorithm that embeds safety constraints directly into the clustering process, ensuring that all intermediate and final UAV placement points remain outside unsafe regions. TAKM is further integrated with a constraint-aware procedure to determine the minimum feasible number of UAVs required to satisfy coverage and QoS constraints. 3. We develop a hybrid deployment and control framework that combines TAKM-based safe pre-placement with a MATD3-based learning approach, where UAV trajectories and transmit powers are optimized through continuous-action learning and user associations are deterministically updated. I-E Organization and Notation The rest of the paper is organized as follows. Section I presents the system model. Section I formulates the problem. Section IV introduces the clustering and UAV matching through an Algorithm. Section V describes the proposed threat-aware MARL learning process and presents the computational complexity analysis. Section VI provides the simulation settings and performance evaluation. Finally, Section VII concludes the paper and outlines future research directions. The list of notations is illustrated in Table I. TABLE I: List of Notations Symbol Description Symbol Description Symbol Description a,ba,b Environment-dependent parameters am,k[t]a_m,k[t] User association indicator [t]A[t] User association matrix m[t]a_m[t] Action of UAV m ArA_r Rotor disc area B System bandwidth c1,c2c_1,c_2 PSO Acceleration coefficients mC_m TAKM cluster centroid m Cm[t]C_m[t] Total power consumption of UAV m dm,k[t]d_m,k[t] Distance between UAV m and user k dmind_ Minimum separation distance dsafed_safe Safety margin around threat zones dfd_f Fuselage drag ratio DnD_n Minimum distance to threat n ng_n Center of threat zone n gm,k[t]g_m,k[t] Channel power gain HmH_m Altitude of UAV m I Max iterations in TAKM K Number of GUs La,LcL_a,L_c Layers in Actor/Critic M Number of UAVs N Number of threat zones Na(i),Nc(j)N_a^(i),N_c^(j) Neuron counts in layers NmN_m Number of users served by UAV m m[t]o_m[t] Observation of UAV m pm[t]p_m[t] Transmit power of UAV m P(v[t])P(v[t]) UAV propulsion power P0P_0 Blade profile power PiP_i Induced power in hovering PmaxP_ Maximum UAV transmit power Psaf[t]P_saf[t] Safety violation indicator πm(⋅) _m(·) Policy of UAV m kq_k Horizontal position of user k Qm(⋅)Q_m(·) Action-value function r[t]r[t] Global reward rnr_n Radius of threat zone n Rk[t]R_k[t] Data rate of user k RminR_ Minimum required data rate [t]s[t] Global environment state srs_r Rotor solidity t Time-slot index tmaxt_ Total number of time slots m[t]u_m[t] Horizontal position of UAV m UtipU_tip Rotor tip speed v0v_0 Mean rotor induced velocity VmaxV_ Maximum UAV speed ω PSO inertia weight ξ1,ξ2 _1, _2 PSO random variables ζe,ζc,ζsaf _e, _c, _saf Reward weighting factors α Learning rate γ Discount factor γLoS _LoS Additional loss for LoS γNLoS _NLoS Additional loss for NLoS δt _t Duration of a time slot ηm,k[t] _m,k[t] SINR between UAV m and user k θmπ _m^π Policy network parameters ρ Air density σe2 _e^2 Exploration noise variance σn2 _n^2 Noise power τ Target network update factor I System Model Fig. 1: The proposed scenario. We consider a downlink multi-UAV-assisted wireless communication system consisting of M UAVs, represented by the set ℳ=1,2,…,MM=\1,2,…,M\, and K GUs, represented by the set =1,2,…,KK=\1,2,…,K\. The UAVs deliver mission-critical traffic in a disaster area, including voice calls and messaging, with a guaranteed minimum data rate per user. The GUs are non-uniformly distributed to represent a stabilized post-disaster snapshot. As illustrated in the 3D plot in Section VI, this model captures a scenario where a small number of users are trapped within hazardous zones while the majority have concentrated into emergency assembly points. Thus, the stationary assumption is reasonable for both the trapped users and the assembly points. The UAVs act as aerial base stations to provide wireless service to the GUs over a time horizon =1,2,…,tmaxT=\1,2,…,t_ \. The proposed scenario, illustrated in Fig. 1, includes a ground base station that establishes a backhaul connection to the UAVs. The horizontal coordinates of UAV m at time step t are represented by m[t]=(xm[t],ym[t])u_m[t]=(x_m[t],y_m[t]), with a fixed altitude HmH_m. The horizontal position of GU k is denoted by k=(xk,yk)q_k=(x_k,y_k). The GUs are assumed to be stationary, and each user is served by a single UAV using frequency-division multiple access (FDMA). In addition to the communication links, the environment contains N threat zones, represented by =1,2,…,NN=\1,2,…,N\, each characterized by its center coordinate n=(xn,yn)g_n=(x_n,y_n) and radius rnr_n. These zones may represent hazards such as fire areas, restricted or no-fly regions, and other dangerous zones, and UAVs must avoid entering them for safety reason. Although UAVs are strictly prohibited from entering threat zones, GUs may be located both inside and outside these areas. Consequently, UAV trajectories must be designed to ensure coverage for all users while keeping the UAVs outside the boundaries of the threat zones. The UAV-GUs association at time step t is represented by a binary matrix [t]=[am,k[t]]∈0,1M×KA[t]=[a_m,k[t]]∈\0,1\^M× K, where am,k[t]=1a_m,k[t]=1 if UAV m serves GU k at time t, and am,k[t]=0a_m,k[t]=0 otherwise. Each GU is served by exactly one UAV per time slot: ∑m=1Mam,k[t]=1,∀k∈,t∈. _m=1^Ma_m,k[t]=1, ∀ k ,\;t . (1) The distance between the UAV m and the GU k is given by: dm,k[t]=‖m[t]−k‖2+Hm2,d_m,k[t]= \|u_m[t]-q_k\|^2+H_m^2, (2) where ‖⋅‖ \|· \| denotes the Euclidean norm, and ‖m[t]−k‖\|u_m[t]-q_k\| represents the horizontal distance between the UAV m and GU k at time step t. The UAV trajectory must satisfy several safety constraints. First, the maximum speed constraint is imposed as ‖m[t+1]−m[t]‖≤Vmaxδt,∀m∈ℳ,t∈,\|u_m[t+1]-u_m[t]\|≤ V_ _t, ∀ m , t , (3) where VmaxV_ denotes the UAVs’ maximum allowable speed, and δt _t is the duration of the time slot. [t]u[t] and [t+1]u[t+1] are the UAV positions at time steps t and t+1t+1, respectively. To prevent collisions, the minimum separation between any two UAVs is enforced by ‖i[t]−j[t]‖≥dmin,∀i≠j,t∈,\|u_i[t]-u_j[t]\|≥ d_ , \ ∀ i≠ j, t , (4) with dmind_ is the minimum distance between the UAVs to avoid collision. In addition, the trajectories must avoid threat zones by maintaining a safe horizontal distance from each threat location, expressed as ‖m[t]−n‖≥Dn,∀m∈ℳ,n∈,t∈,\|u_m[t]-g_n\|≥ D_n, ∀ m ,\;n ,\;t , (5) where DnD_n represents the minimum allowable horizontal distance to threat zone n. The distance DnD_n is given by Dn=rn+dsafeD_n=r_n+d_safe, with dsafed_safe representing the safety margin. The UAV-to-ground channel is modeled using a probabilistic pathloss formulation that accounts for both LoS and non-LoS (NLoS) links, each occurring with different probabilities [37, 5]. The probability of a LoS connection between the UAV m and the GU k is given by [5]: Pm,kLoS[t]=11+aexp(−b(180π(θm,k[t])−a],P_m,k^LoS[t]= 11+a (-b ( 180π( _m,k[t])-a ] \, (6) where a and b are constants that depend on the environment. The elevation angle θm,k[t] _m,k[t] from GU k to UAV m is given by θm,k[t]=tan−1(Hm‖m[t]−k‖). _m,k[t]= ^-1 ( H_m\|u_m[t]-q_k\| ). (7) The probability of NLoS propagation is given by Pm,kNLoS[t]=1−Pm,kLoS[t]P_m,k^NLoS[t]=1-P_m,k^LoS[t]. The mean path loss using the LoS and NLoS between UAV m and GU k at time step t can be expressed as Lm,k[t]=Lm,kLoS[t]×Pm,kLoS[t]+Lm,kNLoS[t]×Pm,kNLoS[t],L_m,k[t]=L_m,k^LoS[t]× P_m,k^LoS[t]+L_m,k^NLoS[t]× P_m,k^NLoS[t], (8) where Lm,kLoSL_m,k^LoS and Lm,kNLoSL_m,k^NLoS denote the pathloss models, in dB, for the LoS and NLoS links, respectively, between UAV m and GU k, and are given by [37, 5]: Lm,kLoS[t]=20log(4πfcdm,k[t]c)+γLoS,L_m,k^LoS[t]=20 ( 4π f_cd_m,k[t]c )+ _LoS, (9) and Lm,kNLoS[t]=20log(4πfcdm,k[t]c)+γNLoS,L_m,k^NLoS[t]=20 ( 4π f_cd_m,k[t]c )+ _NLoS, (10) where fcf_c is the carrier frequency, c is the speed of light. γLoS _LoS and γNLoS _NLoS are the additional losses for LoS and NLoS, respectively. The channel power gain between UAV m and user k at time slot t can be modeled as [18] gm,k[t]=(c4πfcdm,k[t])2(Pm,kLoS[t]γLoS+Pm,kNLoS[t]γNLoS)−1,g_m,k[t]= ( c4π f_cd_m,k[t] )^2 (P^LoS_m,k[t]\, _LoS+P^NLoS_m,k[t]\, _NLoS )^-1, (11) where the first bracket denotes the free-space pathloss term, accounting for distance-dependent attenuation, and the second bracket represents the shadowing factor under LoS and NLoS conditions. As each UAV is allocated an orthogonal channel, co-channel interference can be considered negligible. Consequently, when GU k is served by UAV m at time step t, the corresponding signal-to-interference-plus-noise ratio (SINR) is expressed as: ηm,k[t]=gm,k[t]am,k[t]pm[t]σn2, _m,k[t]= g_m,k[t]a_m,k[t]p_m[t] _n^2, (12) where pm[t]p_m[t] is the transmit power of UAV m at time step t, and it satisfies 0≤∑k=1Kam,k[t]pm[t]≤Pmax,0≤ _k=1^Ka_m,k[t]\,p_m[t]≤ P_ , where PmaxP_ denotes the maximum allowable transmit power of a UAV. The term σn2 _n^2 denotes the noise power. In this paper, a single power level is assumed for each UAV m, which is equally shared among all its associated users. am,k[t]a_m,k[t] denotes the binary user association indicator. Since UAVs are allocated orthogonal channels, the inter-UAV co-channel interference can be neglected. Accordingly, the achievable data rate of GU k at time step t is given by Rk[t]=∑m=1M(B/Nm)log2(1+ηm,k[t]),R_k[t]= _m=1^M(B/N_m) _2 (1+ _m,k[t] ), (13) where B is the total bandwidth and NmN_m is the number of users served by UAV m. We consider a power consumption model that includes both the communication power for transmission and the propulsion power for UAV mobility, as in [15]. Accordingly, the total power consumption of UAV m at time step t is expressed as: Cm[t]=∑k=1Kam,k[t]pm[t]+P(v[t]),C_m[t]= _k=1^Ka_m,k[t]\,p_m[t]+P(v[t]), (14) where P(v[t])P(v[t]) denotes the flight power of the UAV as a function of its speed. The flight power consumption model P(v[t])P(v[t]) for rotary-wing UAV is given by [11]: P(v[t])=P0(1+3v2[t]Utip2)+Pi1+v4[t]4v04−v2[t]2v02+12dfρsrArv3[t], splitP(v[t])&=P_0 (1+ 3v^2[t]U_tip^2 )+P_i 1+ v^4[t]4v_0^4- v^2[t]2v_0^2\\ & + 12d_fρ s_rA_rv^3[t], split (15) where P0P_0 denotes the blade profile power, and PiP_i corresponds to the induced power during hovering. The tip speed of the rotor blade is represented by UtipU_tip, while v0v_0 indicates the mean velocity induced by the rotor. The parameters dfd_f, ρ, srs_r, and ArA_r denote the fuselage drag ratio, air density, rotor solidity, and rotor disc area, respectively. I Problem Formulation Our aim is to provide the GUs with high QoS while minimizing the energy consumption of the UAVs. To this end, we formulate a joint optimization problem that simultaneously considers the number of UAVs, user association, trajectory planning, and transmit power allocation. The primary objective is to maximize the global energy efficiency (E) of the system, defined as the sum rate of all GUs over the total energy consumed by all UAVs. The proposed optimization is subject to critical operational and safety constraints, including guaranteeing full coverage and a minimum QoS rate for all users, strictly avoiding UAV collisions, prohibiting entry into designated threat zones with a specified safety margin, and satisfying maximum UAV velocity and transmit power limits. The proposed problem can be formulated as: maxM,[],, _M,\,A[t],\,Q,\,P 1tmax∑t=1tmax[∑k=1KRk[t]∑m=1MCm[t]] 1t_ _t=1^t_ [ _k=1^KR_k[t] _m=1^MC_m[t] ] (16) s.t. am,k[t]∈0,1,∀m,k,t, a_m,k[t]∈\0,1\, \ ∀ m,k,t, (16a) ∑m=1Mam,k[t]=1,∀k,t, _m=1^Ma_m,k[t]=1, \ ∀ k,t, (16b) ‖m[t+1]−m[t]‖≤Vmaxδt,∀m,t, \|u_m[t+1]-u_m[t]\|≤ V_ _t,\ ∀ m,t, (16c) ‖i[t]−j[t]‖≥dmin,∀i,j,t,i≠j, \|u_i[t]-u_j[t]\|≥ d_ ,\ \ ∀ i,j,t,i≠ j,\; (16d) 0≤∑k=1Kam,k[t]pm[t]≤Pmax,∀m,t, 0≤ _k=1^Ka_m,k[t]p_m[t]≤ P_ , ∀ m,t, (16e) ‖m[t]−n‖≥Dn,∀m,n,t, \|u_m[t]-g_n\|≥ D_n, \ ∀ m,n,t, (16f) xmin≤xm[t]≤xmax,∀m,t, x_ ≤ x_m[t]≤ x_ , ∀ m,t, (16g) ymin≤ym[t]≤ymax,∀m,t, y_ ≤ y_m[t]≤ y_ , \ ∀ m,t, (16h) Rk[t]≥Rmin,∀k,t, R_k[t]≥ R_ , ∀ k,t, (16i) where the matrix Q, of size M×tmaxM× t_max, represents the trajectories of all M UAVs, with each element corresponding to the 2D horizontal coordinates of UAV m at time step t. The matrix P, of dimension M×tmaxM× t_max, denotes the transmit power allocation for all UAVs, with each entry pm[t]p_m[t] specifying the transmission power level of UAV m at time step t. It is important to note that the E metric considers only UAV power, since the grid-powered base station is not energy-limited like the UAVs. The objective function in (16) is subject to a set of operationally realistic constraints. User association is enforced through binary assignment variables (16a) and exclusive connectivity requirements (16b). UAV mobility is restricted by maximum velocity limits (16c) and collision-avoidance constraints (16d). Power allocation adheres to per-UAV transmit limits (16e). Operational safety is maintained by enforcing threat-zone avoidance (16f) and adherence to geographical boundaries (16g)–(16h). Finally, constraint (16i) ensures that all users receive a guaranteed minimum quality of service. Collectively, these constraints establish a formulation that integrates system performance with the practical requirements of real-world deployment. The optimization problem in (16) is a mixed-integer non-linear programming problem (MINLP), characterized by its combinatorial nature due to the binary user association variables in A (constraint (16a)) and the continuous variables in both the trajectory set Q and power allocation matrix P. The objective function, which represents the E of the system, is a fractional and non-convex function, further complicating the optimization process. Additionally, constraints such as the non-convex collision avoidance requirement in (16d), the threat-zone avoidance in (16f), and the minimum rate requirement in (16i) contribute to the overall non-convexity of the problem. These attributes collectively make (16) NP-hard and computationally challenging to solve using conventional optimization techniques. To address these challenges, the problem in (16) is decomposed into three subproblems. First, the required number of UAVs is determined and the proposed TAKM algorithm is applied to establish the initial user associations, and generate an initial safe UAV placement that satisfies geographical constraints. Second, UAVs are deployed to the identified initial clusters by minimizing the total flight distance and positioned at the cluster centroids. This pre-placement strategy narrows the search space and accelerates the convergence of the learning process. Finally, a threat-aware MARL scheme is proposed to optimize user associations, UAVs trajectories and power control. IV Clustering and UAV Matching IV-A Threat-Avoidance K-means Clustering Algorithm The conventional K-means clustering algorithm has been widely adopted for UAV placement due to its computational efficiency and effectiveness in minimizing the total within-cluster variance. The algorithm iteratively performs two main steps: (1) assignment of users to their nearest centroid (UAV), and (2) recalculation of centroids as the mean position of assigned users. This process converges to a locally optimal solution that minimizes the sum of squared Euclidean distances between users and their serving UAVs. However, CKM algorithm suffers from a significant drawback in threat-prone environments: the computed mean positions may fall within prohibited threat zones, violating safety constraints. The CKM algorithm lacks any mechanism to incorporate geographical constraints, making it unsuitable for real-world operations where airspace restrictions must be strictly observed. To address these limitations, we introduce a TAKM algorithm, which integrates threat avoidance mechanisms into the CKM algorithm through the following key enhancements: 1. Safe Initialization: The TAKM algorithm begins by defining threat buffers around each threat zone, extending the original radius by a safety margin dsafed_safe. A comprehensive grid scan S is performed across the operational area to precompute all safe positions that lie outside all threat buffers. Initial centroids are then randomly selected exclusively from this set of safe positions, ensuring the optimization process starts from feasible solutions. 2. Constrained Centroid Update: During each iteration, after computing the conventional mean position μm _m for each cluster, the algorithm performs a safety verification check: IsSafe(μm)=true,if ‖μm−m‖>rn+dsafe∀n=1,…,N,false,otherwise.IsSafe( _m)= casestrue,&if \| _m-g_m\|>r_n+d_safe\\[-2.0pt] & ∀ n=1,…,N,\\[6.0pt] false,&otherwise. cases (17) If the mean position is safe, it is accepted as the new centroid. If unsafe, the algorithm projects the mean to the nearest safe position: m(i+1)=argminp∈‖p−μm‖C_m^(i+1)= _p \|p- _m\| (18) This projection ensures feasibility while minimizing deviation from the optimal cluster mean. The proposed TAKM algorithm maintains the convergence properties of the CKM algorithm while ensuring all intermediate and final solutions satisfy the safety constraints. The termination condition considers both the stabilization of centroid positions and the maintenance of safety guarantees. The complete pseudocode of the proposed TAKM clustering is shown in Algorithm 1. Algorithm 1 The TAKM Clustering Algorithm 1: Input: M (number of clusters), K (number of GUs), GUs positions, threat zones (n,rn)n=1N\(g_n,r_n)\_n=1^N, safety margin dsafed_safe, area boundaries, ϵε 2: Step 1: Safe Initialization 3: Create a 2D grid of points over the area 4: for each threat n=1,…,Nn=1,…,N do 5: Mark grid points inside rn+dsafer_n+d_safe of threat n as unsafe 6: end for 7: Randomly select M safe positions as initial cluster centroids mm=1M\C_m\_m=1^M 8: Step 2: Constrained Clustering 9: for iteration i=1i=1 to ImaxI_max do 10: Assign each GU to the nearest cluster centroid 11: Store previous centroids for convergence check: mprev=mm=1M\C_m^prev=C_m\_m=1^M 12: for each cluster m=1,…,Mm=1,…,M do 13: Compute the mean of assigned GUs: μm _m 14: if μm _m is outside all threat zones + safety margin then 15: Update centroid m←μmC_m← _m 16: else 17: Project μm _m to nearest safe position from precomputed safe positions 18: end if 19: end for 20: Check convergence 21: for each cluster m=1,…,Mm=1,…,M do 22: Compute: Δm=‖m−mprev‖ _m=\|C_m-C_m^prev\| 23: end for 24: if Δmm=1M<ϵ\ _m\_m=1^M<ε then 25: break 26: end if 27: end for 28: Return: Centroid positions mm=1M\C_m\_m=1^M IV-B Joint Determination of the Minimum Number of UAVs and Initial User Association Determining the minimum number of UAVs is a critical step for cost-effective deployment, but existing approaches often assume a fixed number. To address this limitation, we introduce a systematic, constraint-aware procedure that guarantees feasibility while minimizing deployment costs. As outlined in Algorithm 2, our method iteratively applies the TAKM clustering algorithm (Algorithm 1), starting with M=1M=1 UAV and incrementing M until the resulting cluster configuration satisfies all coverage, minimum rate, and threat-avoidance constraints. This process simultaneously yields the minimum number of UAVs M, the initial safe UAV positions, and the corresponding initial user association matrix A. A key constraint ensures that each GU achieves at least the minimum data rate RminR_min. For each candidate clustering, the per‑user transmit power is set to pm=Pmax/Nmp_m=P_ /N_m. The achievable rate RkR_k is then computed using the SINR expression in (12) and the rate formula (13). A clustering is valid only if Rk≥RminR_k≥ R_min holds for every user k. Algorithm 2 repeats this verification while incrementing M until the rate requirement is met for all users while ensuring UAV safety. The output of this algorithm is the minimum number of UAVs M, the initial safe UAV positions mm=1M\C_m\_m=1^M (obtained via TAKM), and the corresponding user association matrix A. The complete procedure is detailed in Algorithm 2. IV-C UAV Matching Following the execution of Algorithm 2, which outputs the safe target cluster centroids, each physical UAV must be assigned to a specific location. Since the UAVs are homogeneous but may be initially dispersed, this task is treated as a minimum total distance assignment problem. The primary objective is to minimize the cumulative Euclidean distance from the UAVs’ initial launch positions to their designated centroids, thereby ensuring the most energy-efficient initial deployment possible. To achieve this, we adopt an optimal matching scheme from the literature based on the Hungarian algorithm (also known as the Kuhn-Munkres algorithm). This method solves the assignment task in polynomial time, guaranteeing a one-to-one mapping that minimizes the global movement cost, thereby making the system more energy-efficient during the deployment phase. V Threat-Aware MARL for user association, trajectory design and power control Once the minimum required number of UAVs (M) and their initial safe placements are determined in Section IV, the original optimization problem in (16) can be refined for a machine training/learning phase. In this stage, M is no longer a decision variable but a fixed parameter. The objective remains to maximize the system’s global E by optimizing the remaining variables: the user association matrix [t]A[t], the trajectory set Q, and the power allocation matrix P. The resulting problem is nonlinear and nonconvex, making traditional optimization methods intractable. Therefore, a MARL framework is proposed in this section to obtain a feasible solution. Moreover, the problem constitutes a hybrid optimization problem involving discrete user association variables [t]A[t] and continuous power and trajectory variables, [t]P[t] and [t]Q[t]. Due to this mixed structure, the problem cannot be treated purely as a continuous control problem or solely as a discrete assignment problem; instead, it requires a solution capable of handling both types of variables simultaneously. V-A Markov Decision Process Formulation The movement of UAVs directly impacts the data rate of GUs as well as the allocated transmission power. Hence, the actions chosen by UAVs affect both the immediate outcome and the future state of the system. This interaction can be effectively modeled through a partially observable Markov decision process (POMDP), providing a rigorous framework for sequential decision-making in environments characterized by uncertainty. The reinforcement learning parameter settings can be detailed as follows: Algorithm 2 Joint Determination of the Minimum Number of UAVs and Initial User Association 1: Input: K, GU positions, threat zones (n,rn)n=1N\(c_n,r_n)\_n=1^N, dsafed_safe, area boundaries, PmaxP_ , σn2 _n^2, B, RminR_ , channel parameters. 2: M←1M← 1 3: repeat 4: Apply TAKM (Algorithm 1) to obtain centroids m\C_m\ and A 5: all_covered ← true 6: for each user k=1,…,Kk=1,…,K do 7: Find the number of served users NmN_m for the UAV m serving user k 8: Compute the transmit power per user: pm=Pmax/Nmp_m=P_ /N_m 9: Compute channel gain gm,kg_m,k using (12) 10: Compute SINR ηm,k _m,k using (13) 11: Compute achievable rate RkR_k using (14) 12: if Rk<RminR_k<R_ then 13: all_covered ← false 14: break 15: end if 16: end for 17: if not all_covered then 18: M←M+1M← M+1 19: end if 20: until all_covered 21: Return: M, A, mm=1M\C_m\_m=1^M Fig. 2: Threat-aware MATD3 structure. • State: The state comprises the coordinates of all UAVs. It is a continuous variable, with bounds determined by the dimensions of the service area. • Observation: Each agent can only observe a subset of the state. For UAV m, only its own horizontal coordinate is observable, i.e., om=qmo_m=\q_m\. As a subset of the state, the observation is also a continuous and finite variable, with bounds defined by the coordinates of the service area. • Action: The action vector m[t]a_m[t] for UAV m at time step t is defined by its flight speed, flight angle, and transmit power, i.e., m[t]=[am(v)[t],am(ϕ)[t],am(p)[t]].a_m[t]= [a_m^(v)[t],a_m^(φ)[t],a_m^(p)[t] ]. (19) Given the flight speed vm[t]=am(v)[t]⋅Vmaxv_m[t]=a_m^(v)[t]· V_ and the flight angle ϕm[t]=am(ϕ)[t]⋅2π _m[t]=a_m^(φ)[t]· 2π, the UAV position is updated as xm[t+1] x_m[t+1] =xm[t]+vm[t]cos(ϕm[t])⋅δt, =x_m[t]+v_m[t] ( _m[t])· _t, (20) ym[t+1] y_m[t+1] =ym[t]+vm[t]sin(ϕm[t])⋅δt. =y_m[t]+v_m[t] ( _m[t])· _t. (21) • Reward: We propose the following reward function: r[t]= r[t]= ζeEmax⋅∑m=1M∑k=1Kam,k[t]Rk[t]∑m=1MCm[t] _eE_ · _m=1^M _k=1^Ka_m,k[t]\,R_k[t] _m=1^MC_m[t]~ (22) −ζc⋅[1−∑m=1M∑k=1Kam,k[t]K]−ζsaf⋅Psaf[t], - _c· [1- _m=1^M _k=1^Ka_m,k[t]K ]- _saf· P_saf[t], where EmaxE_ is a fixed normalization constant used to scale the E term in the reward. The variable Psaf(t)P_saf(t) is a binary safety flag that takes the value 1 when a collision occurs, a UAV enters a threat zone, or a UAV flies outside the defined operational boundaries at time step t, and 0 otherwise. The coefficients ζe _e, ζc _c, and ζsaf _saf are positive weighting parameters that balance E, coverage, and safety in the reward function. They were tuned empirically to ensure stable learning and constraint satisfaction, with ζsaf _saf set higher to strongly penalize safety violations. The POMDP is formulated to select a policy π(o1,…,oM)π(o_1,…,o_M) that maximizes the cumulative reward, expressed as G[t]=∑t′=t∞αt′−tr[t],G[t]= _t =t^∞α^\,t -tr[t], (23) where α∈[0,1[α∈[0,1[ represents the discount factor that balances immediate and future rewards. Given the current observation set o[t]=o1,…,oMo[t]=\o_1,…,o_M\ and the corresponding action a[t]=a1,…,aMa[t]=\a_1,…,a_M\, the expected cumulative return under the policy π(o1,…,oM)π(o_1,…,o_M) can be formulated as Q(o[t],a[t])=[G∣o[t],a[t]].Q(o[t],a[t])=E [G [t],a[t] ]. (24) The Bellman equation is then used to determine the optimal action-value function called the Q-value and defined as: Q∗(o[t],a[t]) Q^*(o[t],a[t]) =r+αmaxa[t+1]Q∗(o[t+1],a[t+1]), =r+α _a[t+1]Q^*(o[t+1],a[t+1]), (25) where o[t+1]=o1′,…,oM′o[t+1]=\o_1 ,…,o_M \ and a[t+1]=a1′,…,aM′a[t+1]=\a_1 ,…,a_M \ correspond to the subsequent observations and actions. An important point is that the maximization in (25) can introduce a bias, often resulting in an overestimated Q-value. V-B Multi-Agent RL Method Since the observations and actions in our UAV placement and resource allocation problem are continuous variables, conventional tabular methods (e.g., Q-learning) are not suitable for solving the optimization task. To address this, MARL algorithms such as MADDPG and MATD3 have been proposed for continuous state-action spaces. Among these, we employ MATD3 because it effectively mitigates the Q-value overestimation problem inherent in MADDPG through the use of double critics and delayed policy updates. To maintain stability and avoid the complexity of hybrid action spaces [36], we decompose the problem into a continuous control task and a deterministic assignment sub-problem. Because MATD3 handles only continuous actions, the agents do not explicitly optimize the association variables through direct action outputs. Instead, the associations are determined deterministically by the environment based on the UAV positions and transmit power levels. Since the global reward is computed after the association update, the agents indirectly learn deployment and power-control strategies that produce favorable user associations. Therefore, the framework learns the effects of the association mechanism implicitly through reward feedback, rather than explicitly learning the association matrix itself. The overall architecture of the proposed threat-aware MATD3 is illustrated in Fig. 2. Algorithm 3 The Proposed MATD3 with TAKM clustering and pre-placement 1: Input: Actor and critic network structures; UAV initial placement; exploration variance σe2 _e^2; decay factor δ; environment parameters; GUs positions; threat zones (n,rn)n=1N\(g_n,r_n)\_n=1^N; parameters in Table IV 2: Initialize actor network weights mU_m and critic network weights j,j∈1,2W_j,\;j∈\1,2\ 3: Initialize target networks: m′←mU _m _m, j′←jW _j _j 4: Initialize replay memory ←∅D← 5: for episode =1,2,…,L=1,2,…,L do 6: Initialize UAV positions and exploration noise 7: for t=1,2,…,tmaxt=1,2,…,t_ do 8: Each UAV agent observes its local state omo_m 9: Select exploratory action with noise: am[t]=clip(πm(om)+(0,σe2),0,1)a_m[t]=clip ( _U_m(o_m)+N(0, _e^2),0,1 ) 10: Execute joint action a[t]a[t], observe next state o′[t]o [t] 11: User Association: Update [t]A[t] based on current UAV positions and channel conditions 12: Check constraints (collision, threat-zone, operational boundaries, rate requirements) 13: Compute immediate global reward r using (22) Shared among all UAV agents 14: Store transition ([t],[t],r,[t+1])(o[t],a[t],r,o[t+1]) in replay memory D 15: if D has sufficient samples then 16: Sample mini-batch of experiences from D 17: Critic Update: Update 1,2W_1,W_2 using gradient descent on (29) 18: Actor Update (delayed): If policy update condition met, update mU_m using (30) 19: Target Update: Soft-update target networks m′U _m, 1′W _1, 2′W _2 using (32) 20: end if 21: end for 22: Anneal exploration variance: σe2←δσe2 _e^2←δ _e^2 23: end for 24: Return: m,1,2U_m,W_1,W_2 The learning process in MATD3 follows the centralized training and decentralized execution (CTDE) paradigm. Each UAV agent maintains its own deterministic policy πm(m) _U_m(o_m), parameterized by the actor weights mU_m, which maps its observation mo_m to a continuous action vector ma_m. During training, a centralized critic evaluates the joint state–action pair ([t],[t])(o[t],a[t]). The critic networks are parameterized by the weight vectors jW_j (j=1,2)(j=1,2), corresponding to the two critics in the twin delayed architecture. The vectors m′U _m and j′W _j (j=1,2)(j=1,2) denote the target actor and target critic weights, respectively. During experience collection, agents balance exploitation and exploration by following the current policy with some probability and adding noise to encourage exploration. This is given by am=clip(πm(om)+(0,σe2), 0, 1),a_m=clip ( _U_m(o_m)+N(0, _e^2),\;0,\;1 ), (26) where σe2 _e^2 is the variance of the exploration noise, πm _U_m is the policy, and the clip function bounds actions within [0,1][0,1]. While the observation space om[t]o_m[t] is restricted to local coordinates to ensure communication efficiency during execution, environmental awareness is achieved through a two-tiered safety mechanism. First, the TAKM algorithm provides a safety-aware initial deployment, ensuring agents begin in feasible regions. Second, the CTDE paradigm allows the centralized critics to utilize global state information, including threat boundaries and neighbor positions, to shape the reward signal. By incorporating a binary safety penalty PsafP_saf for constraint violations, the framework implicitly encodes the global environmental geometry into the decentralized policy weights. This allows each agent to navigate complex manifolds and avoid localized threats using only its own state, significantly reducing real-time coordination overhead. This observation design is consistent with [15] and is justified for the fixed scenario considered in this work, where users and threat zones are stationary and known. V-B1 Decentralized Execution In the execution step, each UAV operates independently using only its local observation. Given an observation mo_m, the actor network generates a continuous action vector ma_m, corresponding to the UAV’s velocity, flight angle, and transmit power. After executing the action, the UAV interacts with the environment, which produces a new observation and an immediate reward r. At every time step the environment deterministically recomputes the association matrix based on the updated UAV positions, transmit power levels, and the channel conditions. This ensures that each user is associated with the most appropriate UAV according to the current system state and guarantees consistency with the rate and coverage constraints. The interaction of all UAVs at time step t produces the joint experience tuple ([t],[t],r,[t+1],[t+1]),(o[t],\;a[t],\;r,\;o[t+1],\;a[t+1]), which is stored in a replay memory pool. This storage mechanism allows the learning process to reuse past transitions, break temporal correlations, and improve sample efficiency. V-B2 Centralized Training During training, a centralized critic is employed to evaluate the joint state-action pair (o[t],a[t])(o[t],a[t]), allowing the framework to capture inter-agent dependencies such as collision avoidance. To reduce overestimation bias, MATD3 introduces two critics and computes the target Q-value as Qt=r+αminQ1′(o[t+1],a[t+1]),Q2′(o[t+1],a[t+1]).Q_t=r+α \Q _1(o[t+1],a[t+1]),\;Q _2(o[t+1],a[t+1]) \. (27) Training of the critic network is carried out by reducing the squared difference between the QtQ_t and its evaluated, which is formulated using the mean squared error (MSE) as follows: L(Wj)=[(Qj(o[t],a[t])−Qt)2],j=1,2.L(W_j)=E [ (Q_j(o[t],a[t])-Q_t )^2 ],j=1,2. (28) Then, the critic network weights WjW_j are updated as j←j−η∇jL(j),j=1,2,W_j _j-η _W_jL(W_j), j=1,2, (29) where η is the learning rate. Then, the actor network weights UmU_m are updated as m←m+η∇mJ(m),m=1,…,M,U_m _m+η _U_mJ(U_m), m=1,...,M, (30) where ∇mJ(m) _U_mJ(U_m) is given by [15]: ∇mJ(m)≈[∇mQ1(o[t],a[t])]. _U_mJ(U_m) [ _U_mQ_1(o[t],a[t]) ]. (31) Finally, soft updates are applied to the target networks for both actors and critics [38]: m′ _m ←τm+(1−τ)′m,m=1,2,…,M, ← _m+(1-τ)U _m, m=1,2,...,M, (32) j′ _j ←τj+(1−τ)′j,j=1,2, ← _j+(1-τ)W _j, j=1,2, where τ≪1τ 1 is the soft update factor. The combination of delayed policy updates, double critics, and target policy smoothing stabilizes the training process, enabling UAV agents to learn cooperative strategies that maximize system performance while discouraging violations of the safety constraints through the reward-based penalty. In this paper, we employ uniform mini-batch sampling for the experience replay buffer, consistent with [15]. The TAKM-based initialization ensures the buffer is quickly populated with high-quality transitions, reducing the need for sophisticated prioritization. Nevertheless, advanced techniques such as prioritized experience replay (PER) [39] remain a promising direction for more complex environments. The presented MATD3 architecture builds upon the foundational MADDPG framework but introduces critical enhancements to improve learning stability. A key distinction lies in the treatment of the critic networks. In MADDPG, each agent is equipped with its own centralized critic, which is used to compute target Q-values. However, this single-critic design is susceptible to overestimation bias. In contrast, MATD3 employs two centralized critics shared among all agents, and the minimum of their outputs is used to form the target Q-value. This twin-critic mechanism effectively mitigates overestimation by taking their minimum, as defined in Eq. (27). The target Q-value in the MADDPG can be written as: Qt=r+αQ′([t+1],[t+1]).Q_t=r+α Q (o[t+1],a[t+1] ). (33) In addition, MATD3 algorithm employs delayed policy updates, where the actor network is updated less frequently than the critics. This mechanism stabilizes the learning process and ensures that target values remain reliable for multi-UAV optimization. The pseudocode of the proposed MATD3 with TAKM and pre-placement is shown in Algorithm 3. The RL agent is trained offline using simulated environments. During deployment, the learned policy is executed online with low computational complexity, as explained in the next section. TABLE I: Computational Complexity Comparison of Different Algorithms Algorithm Computational Complexity Approximate Operations MATD3 (MKI)+(M3)+(Ltmax(2M∑i=0LaNa(i)Na(i+1)+4∑j=0LcNc(j)Nc(j+1)))+(LtmaxMK)O(MKI)+O(M^3)+O\! (Lt_ \! (2MΣ _i=0^L_aN_a^(i)N_a^(i+1)+4Σ _j=0^L_cN_c^(j)N_c^(j+1) ) )+O(Lt_ MK) Offline Training: 6.4×10106.4× 10^10, Online Deployment: 4.7×1054.7× 10^5 MADDPG (MKI)+(M3)+(Ltmax(2M∑i=0LaNa(i)Na(i+1)+2∑j=0LcNc(j)Nc(j+1)))+(LtmaxMK)O(MKI)+O(M^3)+O\! (Lt_ \! (2MΣ _i=0^L_aN_a^(i)N_a^(i+1)+2Σ _j=0^L_cN_c^(j)N_c^(j+1) ) )+O(Lt_ MK) Offline Training: 5.8×10105.8× 10^10, Online Deployment: 4.7×1054.7× 10^5 PSO (MKI)+(M3)+(IPSONp(MK+M2+MN)tmax)O(MKI)+O(M^3)+O\! (I_PSON_p(MK+M^2+MN)t_ ) Online Deployment: 1.6×10101.6× 10^10 V-C Meta-heuristic Solution To establish a performance benchmark, we implement PSO, a population-based meta-heuristic that solves the joint trajectory and power allocation problem by simulating collective social behavior. It is used solely as a baseline for comparison and is not part of the proposed framework. Each particle j updates its velocity jv_j and state js_j based on its personal best, best,jp_ best,j, and the swarm’s global best, bestg_ best, using: j[k+1]=ωj[k]+c1ξ1(best,j−j[k])+c2ξ2(best−j[k]),v_j[k+1]= _j[k]+c_1 _1(p_ best,j-s_j[k])+c_2 _2(g_ best-s_j[k]), (34) j[k+1]=j[k]+j[k+1],s_j[k+1]=s_j[k]+v_j[k+1], (35) where j=[x1,y1,p1,…,xM,yM,pM]s_j=[x_1,y_1,p_1,…,x_M,y_M,p_M] is the candidate solution vector for all M UAVs. Here, ω is the inertia weight, c1c_1 and c2c_2 are acceleration coefficients, and ξ1,ξ2 _1, _2 are random values in [0,1][0,1]. For a fair comparison, the initial positions of the particles are derived from the TAKM algorithm, ensuring the baseline begins from the same threat-aware configuration as the proposed MARL scheme. At each time step, the PSO performs an iterative search to maximize the reward function subject to velocity and power constraints, providing a rigorous standard to evaluate the learning-based framework. Specifically, the proposed scheme is compared against two benchmarks: Greedy PSO (GPSO), which optimizes only the UAV trajectories and user associations while transmitting at fixed maximum power, and Optimized PSO (OPSO), which jointly optimizes transmit power in addition to trajectory planning and user association. V-D Computational Complexity and Scalability The computational complexity of our proposed framework is compared with the PSO as a baseline in Table I. The complexity of the proposed MATD3 with TAKM clustering and pre-placement is partitioned into two distinct phases: a one-time offline training phase and a real-time online execution phase. The offline training complexity is given by (MKI)O(MKI) for clustering, (M3)O(M^3) for UAV matching, (Ltmax(2M∑i=0LaNa(i)Na(i+1)+4∑j=0LcNc(j)Nc(j+1)))O (Lt_ (2M _i=0^L_aN_a^(i)N_a^(i+1)+4 _j=0^L_cN_c^(j)N_c^(j+1) ) ) [15] for reinforcement learning updates, and (LtmaxMK)O(Lt_ MK) for calculating user associations during training iterations. The term I is the number of iterations in the TAKM algorithm, LaL_a and LcL_c are the number of layers in the actor and critic networks, respectively, and Na(i)N_a^(i) and Nc(j)N_c^(j) represent the neuron counts in their respective layers. Crucially, this intensive computation is performed only once to establish the neural policy. Once the weights are learned and the model is deployed, the critic and target networks are discarded, and only the actor networks are utilized for online operations. Consequently, the online computational complexity scales linearly only with the network architecture, requiring a single forward pass of (∑Na(i)Na(i+1))O(Σ N_a^(i)N_a^(i+1)) operations. This makes the online execution independent of the total number of GUs K and eliminates the need for the iterative calculation of the user association matrix A during flight. In contrast, the PSO baseline functions as an iterative search-based optimizer. As shown in Table I, its complexity is (MKI)+(M3)+(IPSONp(MK+M2+MN)tmax)O(MKI)+O(M^3)+O\! (I_PSON_p(MK+M^2+MN)t_ ). Unlike the proposed learning-based schemes, the PSO must re-execute its entire iterative search, including the costly MKMK user-association evaluations for every particle in the swarm, at every time slot to adapt to the dynamic environment. As illustrated in Table I, the proposed MATD3 requires approximately 6.4×10106.4× 10^10 operations for one-time offline training but only ≈4.7×105≈ 4.7× 10^5 operations for total swarm online execution per time step. These values are obtained by evaluating the complexity expressions in Table I using the simulation parameters from Section VI (e.g., M=4M=4 UAVs, K=400K=400 users, tmax=100t_max=100 time steps, and the neural network architectures described in Table VII). Conversely, the PSO baseline requires approximately 1.6×10101.6× 10^10 operations to solve the same problem. Thus, while the proposed framework has a higher initial training cost, it provides a massive reduction in onboard computational burden, enabling real-time autonomous adaptation on resource-constrained UAV hardware. Notably, the offline complexity of the MADDPG is approximately 5.8×10105.8× 10^10 operations, which is slightly lower than the 6.4×10106.4× 10^10 required by MATD3 because MADDPG utilizes only a single critic per agent rather than a twin-critic architecture. However, during the online phase, both MADDPG and MATD3 exhibit identical computational complexity, as both rely on the forward pass of the same actor network architecture. Compared to an offline PSO design, the proposed offline MATD3 offers two main advantages. First, MATD3 learns a state-feedback policy that adapts its actions to the current environment. An open-loop PSO plan uses fixed precomputed trajectories and powers and cannot handle deviations from the offline design assumptions (e.g., wind conditions). Such deviations may drive UAVs into threat regions, because the PSO trajectory cannot be corrected once deployed, whereas the proposed MATD3 can adjust its actions online. Second, if fewer users require service at deployment than assumed during the design stage, MATD3 can adapt its actions to the actual active user set through its state-dependent policy. In contrast, an offline PSO plan cannot update its fixed trajectories and may waste energy flying toward inactive users. Although the proposed MATD3 framework scales efficiently during decentralized execution, the centralized critic employed under the CTDE paradigm introduces increased training complexity as the number of UAVs grows. Specifically, the critic input dimension scales linearly with the joint observation and action spaces of all UAVs, which may lead to slower convergence during training for very large-scale swarms. Nevertheless, the execution time of the decentralized actors remains independent of the total swarm size since each UAV only requires local observations. To address these centralized training bottlenecks, recent research has explored distributed many-agent perspectives that utilize localized coordination and networked Markov games to decouple learning complexity from the total agent count [35]. By focusing on local neighborhood interactions rather than a global state, such distributed architectures provide a scalable roadmap for massive-scale UAV-assisted IoT networks. VI Simulation Results In our simulation environment, we deploy M UAVs to provide communication services to K GUs distributed non-uniformly across a 4000×40004000× 4000 m2 operational area, which contains L=2L=2 distinct threat zones. Specifically, Threat 1 is centered at (550,1200)(550,1200) m with a radius of 500500 m, while Threat 2 is centered at (2250,2500)(2250,2500) m with a radius of 950950 m. The threat-zone locations were selected to emulate hazardous regions that partially overlap with high-density user areas in practical disaster-response scenarios. The adopted radii (500 m and 950 m) represent threat regions of different spatial scales within the considered operational area, enabling evaluation of the proposed framework under safety constraints. An urban environment is considered. All UAVs operate at a fixed altitude and strictly observe a safety margin around each threat boundary. Unless otherwise specified, the simulation parameters are listed in Table IV [23, 15]. TABLE IV: Simulation Parameters Parameter B K N f Value 10 MHz 400 2 1 GHz Parameter σn2 _n^2 PmaxP_ VmaxV_ δt _t Value -111 dBm 33 dBm 10 m/s 300 ms Parameter dsafed_safe ζsaf _saf dmind_ H Value 10 m 5 10 m 300 m Parameter a b γLoS _LoS γNLoS _NLoS Value 9.61 0.16 1 dB 20 dB Parameter P0P_0 PiP_i UtipU_tip v0v_0 Value 9.1827 W 11.5274 W 60 m/s 2.4868 m/s Parameter dfd_f ρ srs_r ArA_r Value 0.5017 1.205 kg/m3 0.0832 0.2827 m2 VI-A The Comparison of CKM and TAKM Clustering Algorithms Figs. 3 and 4 illustrate the results of applying the CKM and the TAKM algorithms for user clustering, with the distances from the centroids to each threat summarized in Table V. It is important to note that the number of UAVs, which is four, is obtained using Algorithm 2 and represents the minimum number of UAVs required to cover the considered area. The analysis assumes a minimum data rate of Rmin=0.128R_ =0.128 Mbps. Therefore, M=4M=4 will be used for the rest of the experiments. Fig. 3: Clustering using the CKM Algorithm. Fig. 4: Clustering using the proposed TAKM Algorithm. TABLE V: Distance Between CKM/TAKM Cluster Centroids and Threats Cluster CKM TAKM Threat 1 Threat 2 Threat 1 Threat 2 Cluster 1 3269.50 1134.33 3274.17 1138.50 Cluster 2 2754.85 2139.65 2766.54 2150.54 Cluster 3 151.31 (UNSAFE) 2285.86 510 1784.66 Cluster 4 2018.21 1668.77 2240.12 1785.94 The key advantage of the proposed TAKM method, as visually confirmed in Fig. 4, is its ability to maintain safety guarantees without significantly compromising clustering quality. While the CKM algorithm (Fig. 3) minimizes within-cluster variance without regard to constraints, leading to unsafe UAV placements, the proposed TAKM approach incorporates threat avoidance during centroid updates. This ensures compliance with safety requirements while still preserving the convergence properties and computational efficiency of CKM algorithm. Based on the distance analysis in Table V, the comparison between CKM and the proposed TAKM clustering highlights the critical advantage of incorporating threat awareness into the clustering process. Under CKM, Cluster 3 is positioned only 151.31 m from Threat 1, placing the centroid inside the 500 m threat radius and resulting in an unsafe configuration. This unsafe placement reflects a key limitation of CKM in threat-constrained environments, as it forms clusters solely based on geometric proximity. In contrast, the proposed TAKM algorithm successfully relocates all centroids, including Cluster 3, to safe positions, with the TAKM centroid of Cluster 3 lying 510 m from Threat 1, thereby completely outside the hazardous zone. As summarized in Table VI, TAKM achieves this safety-aware repositioning with only slight redistribution in GU assignments (e.g., Cluster 2 changes from 200 to 198 GUs, Cluster 3 from 52 to 55), while maintaining overall cluster balance. These results show that TAKM ensures safe, balanced, and effective UAV clustering while avoiding threat regions. The matching assignment of UAVs to TAKM centroids is assumed to be optimally solved using the Hungarian algorithm, a well-known method. Since this step incurs a fixed, one-time energy cost and does not affect the comparative evaluation of TAKM and MARL performance, it is not simulated and our experiments focus on the clustering and learning stages only. TABLE VI: Cluster Position and Number of GUs Cluster Cluster Centroid Number of GUs CKM TAKM CKM TAKM Cluster 1 [3224.35, 3080.82] [3224.35, 3088.92] 65 65 Cluster 2 [3239.34, 602.82] [3249.8, 596] 200 198 Cluster 3 [909.04, 1145] [1060, 1170] 52 55 Cluster 4 [739.47, 3209.3] [727.2, 3433.1] 83 82 TABLE VII: MARL Simulation Parameters Parameter Value Number of episodes (L) 400 Episode duration 30 s (100 time slots) Discount factor α 0.950.95 Soft updated factor τ 0.0050.005 Actor network 3 hidden layers, 256 neurons each Critic network 3 hidden layers, 256 neurons each Actor input UAV 2D coordinates Actor output Speed, Flight angle, transmit power Critic input 20 neurons for M=4, 25 neurons for M=5, and 45 neurons for M=9 Critic output Q-value of state–action pair Activation (hidden layers) ReLU Actor output activation Sigmoid Critic output activation Linear Optimizer Adam [40] Learning rate γ 0.00001 Replay memory size 20,000 EmaxE_max 106b/J10^6b/J VI-B Performance of the Proposed MATD3 with TAKM Clustering and Pre-placement Scheme In this section, we evaluate the performance of the proposed MATD3 with TAKM clustering and pre‑placement scheme and compare it with other RL and meta-heuristic benchmark methods. The UAVs are initialized at the centroids obtained from the TAKM algorithm (Table VI). Both the decentralized actor and centralized critic are implemented as fully connected neural networks with three hidden layers. Rectified linear unit (ReLU) activation is used in the hidden layers, with a sigmoid activation at the actor’s output and a linear activation at the critic’s output. Training employs the Adam optimizer [40] and an exploration factor initialized at 0.4, which is multiplied by 0.995 at the beginning of each episode. Detailed training parameters are summarized in Table VII. To verify the effectiveness of the reward function design, an ablation study was conducted. Under the baseline configuration (ζe=1,ζc=1,ζsaf=5 _e=1, _c=1, _saf=5), the system achieves an average transmit power of 7373 W while maintaining full coverage. Removing the energy efficiency component (ζe=0 _e=0) resulted in an increase in power consumption to 8080 W, representing a nearly 10% degradation in efficiency. Similarly, disabling the coverage term (ζc=0 _c=0) led to a reduction in the percentage of served users to 98%. These results confirm that each reward component is essential for guiding the agents toward an optimized state that balances connectivity and resource conservation. Furthermore, to verify the robustness of the safety weight selection, we performed a sensitivity sweep of ζsaf _saf over 3,5,10,15\3,5,10,15\. In all cases, the trained agents maintained the required safety distances and exhibited zero entries into prohibited threat regions throughout the testing phase, with negligible variation in E performance. This confirms that the chosen weighting ζsaf=5 _saf=5 is robust, while sufficiently large safety weights are essential to enforce collision avoidance and threat evasion. To demonstrate the unsuitability of CKM clustering in threat-constrained UAV deployment scenarios, Fig. 5 illustrates the safety violations versus training episodes for the MATD3 scheme initialized with CKM clustering and pre-placement. As shown, the safety violation rate remains persistently high, reaching about 70% even after 400 episodes. This occurs because CKM places UAVs inside threat zones, preventing the MARL agent from learning safe trajectories and resulting in repeated violations even with an advanced algorithm like MATD3. These results highlight a key limitation of conventional clustering in constrained environments: without built-in safety awareness, even sophisticated MARL methods cannot achieve safe UAV deployment. In contrast, the proposed TAKM algorithm explicitly enforces safety during initialization, allowing MARL to focus on optimizing performance rather than escaping unsafe regions. Fig. 6 compares the average reward performance of the proposed MATD3 with TAKM clustering and pre-placement against five benchmark schemes: MADDPG with TAKM clustering and pre-placement, MATD3 with only pre-placement, MADDPG with only pre-placement, MATD3 with random pre-placement, and MATD3 with CKM clustering and pre-placement. Each curve is obtained from one independent training run of 400 episodes, as in [15], and following Algorithm 3. The proposed method achieves the highest final E among the evaluated schemes and exhibits smooth convergence, reaching a final E of 4.48±0.25×1064.48± 0.25× 10^6 b/J, where the reported mean and standard deviation are calculated over the 100 time slots of the best episode. This represents a notable improvement over MATD3 with only pre-placement (4.25±0.26×1064.25± 0.26× 10^6 b/J) and MATD3 with random pre-placement (3.66×1063.66× 10^6 b/J). The proposed method also achieves a competitive E compared to MADDPG with TAKM clustering and pre-placement (4.41±0.28×1064.41± 0.28× 10^6 b/J), while offering faster convergence and more stable learning behavior. The reported standard deviations reflect temporal variations across the time slots of the selected episode, rather than variability across independent training runs. While MATD3 with CKM clustering and pre-placement achieves an E of 4.45×1064.45× 10^6 b/J, it leads to persistent safety violations, underscoring that threat-aware initialization is essential for real-world deployment where safety is encouraged via a heavily weighted penalty in the reward function. The results confirm that the TAKM-based initialization is essential for producing safe, threat-aware UAV placements, enabling the reinforcement learning framework to achieve high E without safety violations. Fig. 5: Safety violations versus Episodes for MATD3 with CKM clustering and Pre-placement. Fig. 6: The average reward comparison between the proposed scheme and the benchmark schemes. Fig. 7 presents a comprehensive comparison of safety-violation behavior across different schemes, highlighting the clear advantage of the proposed TAKM clustering. The results in Figs. 7(a) show that MATD3 with TAKM clustering and pre-placement achieves the fastest reduction in violations, approaching nearly zero after roughly 300 episodes. In comparison, MADDPG with TAKM clustering and pre-placement exhibits more safety violation as shown in Fig. 7(b), even after 300 episodes, reflecting its sensitivity to Q-value overestimation and gradient noise. Figs. 7(c) and 7(d) illustrate the safety-violation behavior of MATD3 and MADDPG, respectively, when only pre-placement is employed without TAKM clustering. In both cases, frequent safety violations are observed during the training episodes. Although the overall number of violations decreases as training progresses, neither algorithm succeeds in fully eliminating unsafe behavior, even at episode 400, indicating the absence of convergence to a strictly safe policy. MATD3 in Fig. 7(c) exhibits slightly improved stability compared to MADDPG in Fig. 7(d); however, this improvement is insufficient to guarantee safety without threat-aware clustering. These results clearly demonstrate that pre-placement alone is inadequate, and that TAKM clustering is essential to provide structured, threat-aware initial UAV positions that suppress unsafe exploration and enable near-zero safety violations. (a) (b) (c) (d) Fig. 7: Safety violations versus Episodes for the proposed scheme and the benchmark schemes. Fig. 8: Rate violations percentage vs. episodes for the proposed scheme and benchmark schemes. Fig. 8 shows the rate violation percentage versus training episodes for the proposed MATD3-based schemes and the MADDPG benchmark schemes, both with and without TAKM clustering and pre-placement. During the early stages of training, all schemes experience a high rate violation percentage due to random exploration and untrained policies. However, a rapid decline in violations is observed within the initial episodes. Notably, the MADDPG-based schemes exhibit a faster initial reduction in rate violations compared to the proposed MATD3 schemes. Nevertheless, the proposed MATD3 with TAKM clustering and pre-placement demonstrates stable convergence, similar to MADDPG with TAKM clustering and pre-placement, and achieves comparable violation levels in later episodes. After approximately 150–200 episodes, the rate violation percentages of all schemes converge toward zero, indicating that all users are successfully served. Fig. 9: The average reward performance with different altitudes for the proposed scheme. Fig. 10: Number of served users at different altitudes for the proposed scheme. Fig. 11: The average reward performance with different number of users for the proposed scheme. Figs. 9 and 10 collectively emphasize the critical role of UAV altitude in optimizing overall system performance. The considered UAV altitudes (100–300 m) represent practical low-to-moderate altitude operating ranges commonly adopted in urban UAV communication studies. These values enable evaluation of the tradeoff between blockage reduction and LoS improvement at higher altitudes versus the increased propagation distance and energy consumption associated with excessive elevation. In particular, the choice of the maximum altitude H=300H=300 m is grounded in the system configuration adopted in [15]. Fig. 9 shows that an altitude of 300 m delivers the highest average reward, substantially outperforming 250 m, 200 m and 100 m. This superior performance is primarily due to improved channel conditions at higher altitudes: as UAVs gain height, the probability of establishing LoS links with GUs increases according to the probabilistic channel model. Higher LoS probability reduces path loss, enhances signal quality, and allows for higher data rates with more efficient power consumption. Fig. 10 provides complementary insights by highlighting the effect of altitude on user coverage. At 300 m, the UAVs can consistently serve all 400 GUs, whereas the other altitudes cannot fully cover all users. This demonstrates that the superior reward at 300 m is not only a result of better channel conditions but also due to meeting the system’s primary requirement of universal service. Together, these figures illustrate the same altitude that ensures full coverage simultaneously maximizes reward, channel rate, and E. In threat-constrained environments with non-uniform user distribution, higher altitudes therefore offer the dual benefit of achieving complete service coverage and optimal communication performance, making them the most effective choice for UAV deployment. Given these results, an altitude of 300 m is selected for the proposed scheme in this paper. Fig. 11 illustrates the average reward obtained under different numbers of users and shows that the reward decreases as the number of users grows. This behavior is due to the system’s operational constraints: as the number of users increases, the UAV transmit power is allocated among more GUs. Given the fixed maximum transmit power per UAV, the power allocated to each user decreases. Moreover, each user is allocated less bandwidth, which reduces the average channel capacity and consequently lowers the reward. These results demonstrate that the proposed learning-based approach can adapt to scenarios with varying user densities while maintaining efficient operation. It is also noted that the coverage rate remains at 100%100\% for 200 and 400 users, but decreases to 95%95\% for 600 users with M=4M=4. However, when the number of UAVs is increased to M=5M=5 (determined via Algorithm 2 for K=600K=600), the coverage is restored to 100%100\% and the average reward increases compared to the case with M=4M=4. This improvement occurs because with more UAVs, each UAV serves fewer users, leading to higher transmit power per user, better channel conditions, and more efficient bandwidth allocation, thereby enhancing overall system E. To further evaluate scalability, we consider a larger scenario with K=1500K=1500 users and the result is shown in Fig. 11. Algorithm 2 first determines that M=9M=9 UAVs are required to satisfy the network demand, after which MATD3 optimizes their deployment and resource allocation. For the M=9M=9 scenario, the critic input dimension was scaled to 4545. This scaling is consistent with the centralized critic architecture described in Section V-B. Under this configuration, the proposed framework achieves 100%100\% user coverage with zero safety violations and zero rate violations. Despite the larger swarm size, the CTDE-based MATD3 maintains stable convergence, indicating that the centralized critic remains computationally manageable for the considered deployment scale. The scalability results in Fig. 11 demonstrate that the proposed framework scales effectively by first determining the required number of UAVs and their initial safe positions via Algorithm 2, and then optimizing their trajectories, transmit powers, and user associations with MATD3. Consequently, the framework preserves its safety, coverage, and QoS guarantees as the network size increases. TABLE VIII: Comparison of User Distribution. Method Cluster 1 Cluster 2 Cluster 3 Cluster 4 SDNU CKM 65 200 52 83 58.78 TAKM 65 198 55 82 57.40 MATD3 64 183 67 86 48.66 Fig. 12 illustrates the UAV trajectories generated by the proposed MATD3 scheme with TAKM clustering and pre-placement. The initial UAV positions correspond to the TAKM centroids in Table VI, where the four clusters contain 65, 198, 55, and 82 GUs, respectively. These values represent the initial user grouping using TAKM. However, as the UAVs follow the learned trajectories, they gradually reposition themselves toward more energy-efficient locations, which reshapes the effective coverage regions and results in new user grouping. This dynamic re-clustering is reflected in the final user distributions, where the UAVs ultimately serve 64, 183, 67, and 86 GUs, and their final horizontal positions are (3429.8, 3013.3) m, (3467.4, 514.6) m, (1251.3, 1082.9) m, and (945.2, 3360.8) m, respectively. The changes in user assignment demonstrate that the proposed method does not rely on fixed clustering; instead, the UAVs adapt their service regions to maximize E while satisfying LoS and safety constraints. UAV 2, initially responsible for 198 GUs, ends up serving 187 users by shifting slightly toward the densest region, whereas UAVs 1, 3, and 4 experience moderate adjustments as their trajectories refine the cluster shapes. Fig. 12: The trajectories of UAVs for the proposed scheme. Fig. 13: Distribution of user rates at final UAV positions. The standard deviation of the number of users (SDNU) across clusters is presented in Table VIII for CKM, TAKM, and MATD3. The results show that users are distributed more evenly by the MATD3 framework, with the SDNU decreasing from 58.78 (CKM) to 57.40 (TAKM) and finally to 48.66 (MATD3). This progressive improvement of 10.12% from CKM to MATD3 occurs because the MATD3 framework dynamically adjusts user associations during trajectory optimization. Compared to static clustering, MATD3 further considers channel conditions and UAV positioning, resulting in a more balanced workload distribution. Fig. 13 illustrates the distribution of user rates achieved at the final point of the UAV trajectories under the proposed MATD3 with TAKM clustering and pre-placement scheme. The red line in the figure indicates the minimum rate RminR_ . Fig. 13 shows that the distribution exhibits a pronounced right-skew, concentrated above RminR_ and extending toward higher rate values, indicating that most users enjoy robust channel conditions while a small subset benefits from particularly favorable links. A per-UAV breakdown further reveals: UAV 1 serves 64 users with an average rate of 1.31 Mbps, UAV 2 serves 183 users at average rate of 0.45 Mbps, UAV 3 serves 67 users at average rate of 1.06 Mbps, and UAV 4 serves 86 users at average rate of 0.96 Mbps, illustrating how the framework dynamically balances load while maintaining high per-user throughput. Fig. 14: Energy efficiency versus time steps for the final trajectories in the proposed and PSO schemes. Fig. 14 compares the E of the proposed MATD3 scheme with that of the GPSO and OPSO benchmarks. GPSO denotes a baseline in which each UAV transmits at its maximum power, whereas OPSO jointly optimizes the UAV trajectories and transmit powers at every time step. For the PSO benchmarks, the numbers of particles and iterations were selected before the final performance comparison through a predefined parameter search. Specifically, we evaluated Np∈100,150,250,300N_p∈\100,150,250,300\ and IPSO∈100,200,300,400,500I_PSO∈\100,200,300,400,500\, while fixing the inertia weight to ω=0.7ω=0.7 and the acceleration coefficients to c1=c2=1.5c_1=c_2=1.5. The configuration Np=250N_p=250 and IPSO=400I_PSO=400 was selected, as further increasing the search budget yielded negligible E improvement. These parameters were then fixed for all benchmark comparisons and were not adjusted based on MATD3 results. Under this setting, OPSO achieved an E of 5.30×1065.30× 10^6 b/J, whereas the GPSO baseline achieved 3.90×1063.90× 10^6 b/J. The obtained results from one training run indicate that MATD3 achieves a higher E than the GPSO baseline. This performance difference arises because MATD3 functions as an intelligent learner: its trained actor network predicts velocity and power settings based on a learned policy, whereas GPSO follows a rigid, precomputed path. Although OPSO attains a higher E, its computational demands make it impractical for onboard UAV implementation, as explained in Section V-D. In contrast, MATD3 combines low computational complexity, requiring only a single forward pass through the neural network during online execution, with competitive performance compared with traditional heuristic approaches. The validation results presented in Table IX demonstrate the robustness and reproducibility of the proposed MATD3 framework. In this evaluation setup, the random number generator (rng) seed determines the spatial distribution of GUs within the simulation area. To assess training stability, the proposed method is evaluated across multiple random seeds; the results show only minor performance variation, indicating that the learned policy consistently converges to near-optimal solutions and is not sensitive to stochastic initialization. Initially, the policy was evaluated on the same user distribution used during training (rng 1). In this case, the learned policy achieves 4.24×1064.24× 10^6 b/J, which corresponds to 94.6%94.6\% of the peak training performance (4.48×1064.48× 10^6 b/J). This minor gap is expected when removing exploration noise during the testing phase and confirms that the policy is stable rather than overfitted to exploratory behaviors. To further verify generalization, the framework was tested across diverse spatial realizations without retraining or fine-tuning. As shown in Table IX, while the peak training performance reached 4.48×1064.48× 10^6 b/J, the testing trials yielded a high average E of approximately 4.13×1064.13× 10^6 b/J. By retaining over 92%92\% of the training performance across all unseen distributions, these results demonstrate that the MATD3 framework, integrated with TAKM clustering, learns generalized decision-making strategies rather than merely memorizing specific user coordinates. The training peak of 4.48×1064.48× 10^6 b/J in Table IX represents the maximum E achieved over a single episode during training, whereas the testing values represent averages over multiple independent test runs with exploration noise removed. TABLE IX: Reproducibility and Robustness Evaluation of the MATD3 Framework Scenario User Distribution E (b/J) Training rng (1) 4.48×1064.48× 10^6 Testing (Same) rng (1) 4.24×1064.24× 10^6 Testing (Different) rng (4) 4.10×1064.10× 10^6 Testing (Different) rng (10) 4.06×1064.06× 10^6 Testing (Different) rng (11) 4.12×1064.12× 10^6 Testing (Different) rng (14) 4.22×1064.22× 10^6 Testing (Different) rng (17) 4.14×1064.14× 10^6 Testing (Different) rng (22) 4.14×1064.14× 10^6 Testing (Different) rng (27) 4.07×1064.07× 10^6 Testing (Different) rng (30) 4.14×1064.14× 10^6 To assess the safety guarantees of the proposed framework, we tested it under a new threat configuration with different placements and sizes. Specifically, the original two threat zones were replaced by new regions centered at (1000,2200)(1000,2200) m and (2800,1500)(2800,1500) m with radii of 800800 m and 600600 m, respectively, while all other simulation parameters remained unchanged. Under this new configuration, the proposed TAKM algorithm successfully places all UAVs in safe positions, as shown in Fig. 15. The results show that the proposed TAKM-MATD3 framework achieved an energy efficiency of 4.13×1064.13× 10^6 b/J, maintained 100%100\% user coverage, and exhibited zero safety violations, with safe trajectories. These results confirm that the proposed threat-aware framework generalizes well to different threat geometries while consistently satisfying the safety constraints. To study the robustness of the proposed restricted observation design, we conducted a nested ablation study in which the observation was progressively extended to include other UAVs’ positions, threat-zone locations, and user positions. As shown in Fig. 16, all four observation configurations converge to nearly identical average reward. During the earlier training episodes, configurations with additional observed information exhibit marginally faster and smoother convergence, while the restricted observation shows larger transient oscillations before stabilizing. However, since this difference is confined to the early training phase and vanishes at convergence, all four configurations maintain zero safety violations and 100%100\% user coverage throughout. Consequently, the restricted observation in the proposed design is sufficient for the fixed scenario considered, and is chosen to reduce communication overhead and execution complexity. Fig. 15: UAV trajectories under a new threat configuration with different zone centers and radii. Fig. 16: Average reward versus episode for the proposed scheme at different observation scenarios. VII Conclusion In this paper, we proposed a threat-aware, energy-efficient MARL framework for deploying multiple UAVs in constrained and hazardous environments. First, we introduced a safety-critical TAKM clustering algorithm (Algorithm 1), which embeds safety constraints directly into the clustering process by ensuring that all UAV placements remain outside threat zones with a prescribed safety margin. Second, we integrated TAKM with a systematic procedure (Algorithm 2) to determine the minimum number of UAVs and the initial safe UAV positions required to satisfy the design constraints such as rate, and safety. Third, building upon this safe and efficient initial configuration, we proposed a threat-aware MATD3 framework (Algorithm 3) to jointly optimize UAV trajectories, transmit power allocation, and user associations, learning cooperative policies that maximize long-term E while continuously respecting all operational constraints. Extensive simulation results demonstrated that the proposed MATD3 framework with TAKM-based clustering and pre-deployment initialization outperforms benchmark schemes by achieving higher E, faster convergence, and zero safety violations. The proposed approach surpassed GPSO and approached OPSO performance while requiring significantly lower online deployment computational complexity. The results further demonstrated robust generalization to unseen user distributions, retaining more than 92% of the training performance without retraining while maintaining zero safety violations under different threat configurations. These findings demonstrate that integrating TAKM with MATD3 provides a practical and scalable solution for safe and energy-efficient UAV deployment in threat-constrained environments. The results also highlight that learning-based algorithms without TAKM clustering are insufficient to guarantee safety, underscoring the critical role of TAKM in safety-aware UAV deployment. While the framework was validated in simulation using well-established system models, real flight effects such as wind, GPS drift, and latency were not considered. Consistent with existing research, hardware validation on Pixhawk-based multi-UAV platforms under real-world disturbances is therefore left for future work. Future work may also extend this framework to more complex scenarios such as dynamic hazards, dynamic users, while integrating PER to enhance learning efficiency. References [1] B. Li, Z. Fei, and Y. Zhang (2019) UAV communications for 5g and beyond: recent advances and future trends. IEEE Internet of Things Journal 6 (2), p. 2241–2263. External Links: Document Cited by: §I-A. [2] F. Al-Kamali, F. Chan, H. A. Ammar, J. H. Bayes, and C. D’Amours (2026) UAV-mounted aerial relays in military communications: A comprehensive survey. IEEE Open Journal of the Communications Society 7 (), p. 1096–1136. External Links: Document Cited by: §I-A. [3] Y. Zeng, R. Zhang, and T. J. Lim (2016) Wireless communications with unmanned aerial vehicles: opportunities and challenges. IEEE Communications Magazine 54 (5), p. 36–42. External Links: Document Cited by: §I-A. [4] M. Alzenad, A. El-Keyi, F. Lagum, and H. Yanikomeroglu (2017) 3-D placement of an unmanned aerial vehicle base station (UAV-BS) for energy-efficient maximal coverage. IEEE Wireless Communications Letters 6 (4), p. 434–437. External Links: Document Cited by: §I-A. [5] A. Al-Hourani, S. Kandeepan, and S. Lardner (2014) Optimal lap altitude for maximum coverage. IEEE Wireless Communications Letters 3 (6), p. 569–572. External Links: Document Cited by: §I-A, §I, §I. [6] F. Al-Kamali, F. Chan, J. H. Bayes, and C. D’Amours (2026) Optimizing uav placement with spatial threat exclusion for secure and energy-efficient wireless communications. IEEE Open Journal of the Communications Society 7 (), p. 539–558. External Links: Document Cited by: §I-A. [7] Z. Chang, H. Deng, L. You, G. Min, S. Garg, and G. Kaddoum (2023) Trajectory design and resource allocation for multi-UAV networks: deep reinforcement learning approaches. IEEE Transactions on Network Science and Engineering 10 (5), p. 2940–2951. External Links: Document Cited by: §I-A, §I-B, §I-C, TABLE I. [8] Q. Wu, Y. Zeng, and R. Zhang (2018) Joint trajectory and communication design for multi-UAV enabled wireless networks. IEEE Transactions on Wireless Communications 17 (3), p. 2109–2121. External Links: Document Cited by: §I-A, §I-C. [9] C. H. Liu, X. Ma, X. Gao, and J. Tang (2020) Distributed energy-efficient multi-UAV navigation for long-term communication coverage by deep reinforcement learning. IEEE Transactions on Mobile Computing 19 (6), p. 1274–1285. External Links: Document Cited by: §I-A, §I-B. [10] M. Mozaffari, W. Saad, M. Bennis, and M. Debbah (2017) Mobile unmanned aerial vehicles (UAVs) for energy-efficient internet of things communications. IEEE Transactions on Wireless Communications 16 (11), p. 7574–7589. External Links: Document Cited by: §I-A. [11] Y. Zeng, J. Xu, and R. Zhang (2019) Energy minimization for wireless communication with rotary-wing UAV. IEEE Transactions on Wireless Communications 18 (4), p. 2329–2345. External Links: Document Cited by: §I-A, §I. [12] Q. Wang, Z. Chen, H. Li, and S. Li (2018) Joint power and trajectory design for physical-layer secrecy in the UAV-aided mobile relaying system. IEEE Access 6 (), p. 62849–62855. External Links: Document Cited by: §I-B, §I-C. [13] G. Zhang, Q. Wu, M. Cui, and R. Zhang (2019) Securing UAV communications via joint trajectory and power control. IEEE Transactions on Wireless Communications 18 (2), p. 1376–1389. External Links: Document Cited by: §I-B, §I-C. [14] S. Zhang, H. Zhang, Q. He, K. Bian, and L. Song (2018) Joint trajectory and power optimization for UAV relay networks. IEEE Communications Letters 22 (1), p. 161–164. External Links: Document Cited by: §I-B, §I-C. [15] S. Zhou, Y. Cheng, X. Lei, Q. Peng, J. Wang, and S. Li (2022) Resource allocation in UAV-assisted networks: A clustering-aided reinforcement learning approach. IEEE Transactions on Vehicular Technology 71 (11), p. 12088–12103. External Links: Document Cited by: §I-B, §I-C, §I-C, §I-C, TABLE I, §I, §V-B2, §V-B2, §V-B, §V-D, §VI-B, §VI. [16] L. Wang, K. Wang, C. Pan, W. Xu, N. Aslam, and L. Hanzo (2021) Multi-agent deep reinforcement learning-based trajectory planning for multi-UAV assisted mobile edge computing. IEEE Transactions on Cognitive Communications and Networking 7 (1), p. 73–84. External Links: Document Cited by: §I-C, §I-C, TABLE I. [17] N. Zhao, Z. Liu, and Y. Cheng (2020) Multi-agent deep reinforcement learning for trajectory design and power allocation in multi-UAV networks. IEEE Access 8 (), p. 139670–139679. External Links: Document Cited by: §I-C, §I-C, TABLE I. [18] X. Liu, Y. Liu, Y. Chen, and L. Hanzo (2019) Trajectory design and power control for multi-UAV assisted wireless networks: A machine learning approach. IEEE Transactions on Vehicular Technology 68 (8), p. 7957–7969. External Links: Document Cited by: §I-C, TABLE I, §I. [19] Y. Gao, H. Tang, B. Li, and X. Yuan (2019) Joint trajectory and power design for UAV-enabled secure communications with no-fly zone constraints. IEEE Access 7 (), p. 44459–44470. External Links: Document Cited by: §I-B, §I-C. [20] Y. Wu, J. Xu, L. Qiu, and R. Zhang (2018) Capacity of UAV-enabled multicast channel: joint trajectory design and power allocation. In 2018 IEEE International Conference on Communications (ICC), Vol. , p. 1–7. External Links: Document Cited by: §I-B. [21] G. Yang, R. Dai, and Y. Liang (2021) Energy-efficient UAV backscatter communication with joint trajectory design and resource optimization. IEEE Transactions on Wireless Communications 20 (2), p. 926–941. External Links: Document Cited by: §I-B, §I-C. [22] Y. Sun, D. Xu, D. W. K. Ng, L. Dai, and R. Schober (2019) Optimal 3D-trajectory design and resource allocation for solar-powered UAV communication systems. IEEE Transactions on Communications 67 (6), p. 4281–4298. External Links: Document Cited by: §I-C. [23] Z. Kaleem, W. Khalid, A. Ahmad, H. Yu, A. M. Almasoud, and C. Yuen (2024) Reinforcement learning for energy-efficient user association in UAV-assisted cellular networks. IEEE Transactions on Aerospace and Electronic Systems 60 (2), p. 2474–2481. External Links: Document Cited by: §I-C, §VI. [24] Y. Cao, S. Lien, and Y. Liang (2021) Deep reinforcement learning for multi-user access control in non-terrestrial networks. IEEE Transactions on Communications 69 (3), p. 1605–1619. External Links: Document Cited by: §I-C. [25] W. Zhang, Q. Wang, X. Liu, Y. Liu, and Y. Chen (2021) Three-dimension trajectory design for multi-UAV wireless network with deep reinforcement learning. IEEE Transactions on Vehicular Technology 70 (1), p. 600–612. External Links: Document Cited by: §I-C. [26] H. A. Ammar, R. Adve, S. Shahbazpanahi, G. Boudreau, and I. Bahceci (2025) Handoff design in user-centric cell-free massive MIMO networks using DRL. IEEE Transactions on Communications 73 (11), p. 11368–11384. External Links: Document Cited by: §I-C. [27] H. A. Ammar, R. Adve, S. Shahbazpanahi, G. Boudreau, and K. V. Srinivas (2024) Handoffs in user-centric cell-free MIMO networks: A POMDP framework. IEEE Transactions on Wireless Communications 23 (8), p. 10319–10335. External Links: Document Cited by: §I-C. [28] N. Heess, D. Silver, and Y. W. The (2012) Actor-critic reinforcement learning with energy-based policies. In Proc. Eur. Workshop Reinforcement Learn., Edinburgh, Scotland, p. 1–11. Cited by: §I-C. [29] T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra (2015) Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971. External Links: Link Cited by: §I-C. [30] R. Lowe, Y. Wu, A. Tamar, J. Harb, P. Abbeel, and I. Mordatch (2017) Multi-agent actor-critic for mixed cooperative-competitive environments. In Proc. Advances in Neural Information Processing Systems (NeurIPS), p. 6382–6393. Cited by: §I-C. [31] Y. Zhang, Z. Mou, F. Gao, J. Jiang, R. Ding, and Z. Han (2020) UAV-enabled secure communications by multi-agent deep reinforcement learning. IEEE Transactions on Vehicular Technology 69 (10), p. 11599–11611. External Links: Document Cited by: §I-C. [32] S. Yin and F. R. Yu (2022) Resource allocation and trajectory design in UAV-aided cellular networks based on multiagent reinforcement learning. IEEE Internet of Things Journal 9 (4), p. 2933–2943. External Links: Document Cited by: §I-C. [33] X. Dai, Z. Lu, X. Chen, X. Xu, and F. Tang (2024) Multiagent RL-based joint trajectory scheduling and resource allocation in NOMA-assisted UAV swarm network. IEEE Internet of Things Journal 11 (8), p. 14153–14167. External Links: Document Cited by: §I-C. [34] Z. Gao, Q. Xie, L. Yang, Y. Dai, and W. Wang (2026) CPID-MAAC: RL for joint user association and trajectory control in UAV-assisted MEC systems. IEEE Transactions on Networking 34 (), p. 2316–2331. External Links: Document Cited by: §I-C. [35] A. Termehchi, A. Syed, W. S. Kennedy, and M. Erol-Kantarci (2024) Distributed safe multi-agent reinforcement learning: joint design of THz-enabled UAV trajectory and channel allocation. IEEE Transactions on Vehicular Technology 73 (10), p. 14172–14186. External Links: Document Cited by: §I-C, §V-D. [36] T. Sun, N. Guo, Y. Cui, and Z. Feng (2026) Multi-agent reinforcement learning based on hybrid action representation for uav swarms’ integrated communication and control. IEEE Wireless Communications Letters 15 (), p. 2080–2084. External Links: Document Cited by: §I-C, §V-B. [37] A. Al-Hourani, S. Kandeepan, and A. Jamalipour (2014) Modeling air-to-ground path loss for low altitude platforms in urban environments. In 2014 IEEE Global Communications Conference, Vol. , p. 2898–2904. External Links: Document Cited by: §I, §I. [38] S. Fujimoto, H. V. Hoof, and D. Meger (2018) Addressing function approximation error in actor-critic methods. Note: arXiv preprint arXiv:1802.09477 External Links: Link Cited by: §V-B2. [39] T. Schaul, J. Quan, I. Antonoglou, and D. Silver (2016) Prioritized experience replay. arXiv preprint arXiv:1511.05952. Cited by: §V-B2. [40] D. P. Kingma and J. Ba (2014) Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980. External Links: Link Cited by: §VI-B, TABLE VII.