Paper deep dive
Merging model-based control with multi-agent reinforcement learning for multi-agent cooperative teaming strategies
Christian Llanes, Spencer W. Jensen, Samuel Coogan
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 7/9/2026, 1:53:10 AM
Summary
The paper introduces MA-AC-MPC, a hybrid framework that integrates multi-agent reinforcement learning (MARL) with model-based control (MPC) to enable safe, dynamically feasible, and cooperative multi-agent policies. By extending actor-critic MPC to multi-agent settings using MAPPO for training and differentiable MPC solvers (leap-c/acados) for the actor layer, the method bridges the gap between long-horizon cooperative learning and short-horizon constraint satisfaction. Experiments in pursuit-evasion and drone-rover cooperative landing scenarios demonstrate that MA-AC-MPC significantly outperforms pure learning-based baselines (MA-AC-MLP), achieving a 100% hardware success rate compared to 60%, while maintaining robustness and satisfying physical constraints.
Entities (14)
Relation Signals (9)
Christian Llanes → affiliatedwith → Georgia Tech
confidence 95% · Christian Llanes, Georgia Tech and Sandia National Labs.
MA-AC-MPC → combines → MARL
confidence 95% · In this work, we propose a framework that combines multi-agent reinforcement learning (MARL) with model-based control to achieve safe, dynamically feasible actions in cooperative multi-agent tasks.
MA-AC-MPC → outperforms → MA-AC-MLP
confidence 95% · We demonstrate the robustness of the proposed MA-AC-MPC algorithm in hardware for both environments... 100% success rate in hardware for MA-AC-MPC compared to 60% for MA-AC-MLP.
MA-AC-MPC → appliedto → Cooperative Landing Scenario
confidence 90% · We also provide an example with a heterogeneous environment where a drone and omni-wheeled rover cooperate to achieve repeatable and successful landing
MA-AC-MPC → appliedto → Pursuit-Evasion Scenario
confidence 90% · We demonstrate the capabilities of this algorithm by applying it to a multi-agent pursuit-evasion scenario.
MA-AC-MPC → extends → AC-MPC
confidence 90% · We extend the AC-MPC algorithm to multi-agent reinforcement learning problems with cooperating agent teams working together to maximize their total rewards.
Pursuers → uses → Augmented Proportional Navigation
confidence 90% · The pursuer team uses augmented proportional navigation as it is accepted as an advanced adversarial control law.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:In this work, we propose a framework that combines multi-agent reinforcement learning (MARL) with model-based control to achieve safe, dynamically feasible actions in cooperative multi-agent tasks. Multi-agent reinforcement learning provides the advantage of learning cooperative policies for multi-agent teams from discrete non-differentiable rewards in a long planning horizon. Model-predictive control is robust and offers safe, dynamically feasible actions in a fast replanning framework for short horizons. We propose an algorithm that extends actor-critic model predictive control for MARL which we refer to as multi-agent actor-critic model predictive control (MA-AC-MPC). We demonstrate the capabilities of this algorithm by applying it to a multi-agent pursuit-evasion scenario. Specifically, we compare the evader team's strategy using the MA-AC-MPC model and a multi-layer perceptron model (MA-AC-MLP). The pursuer team uses augmented proportional navigation as it is accepted as an advanced adversarial control law. We also provide an example with a heterogeneous environment where a drone and omni-wheeled rover cooperate to achieve repeatable and successful landing with 100% success rate in hardware for MA-AC-MPC compared to 60% for MA-AC-MLP. We demonstrate the robustness of the proposed MA-AC-MPC algorithm in hardware for both environments.
Tags
Links
- Source: https://arxiv.org/abs/2606.06011v1
- Canonical: https://arxiv.org/abs/2606.06011v1
Trouble viewing inline? Open PDF directly →
Full Text
68,370 characters extracted from source content.
Expand or collapse full text
Christian Llanes, Georgia Tech and Sandia National Labs. Merging model-based control with multi-agent reinforcement learning for multi-agent cooperative teaming strategies Christian Llanes1,21,2affiliationmark: Spencer W. Jensen22affiliationmark: and Samuel Coogan11affiliationmark: 11affiliationmark: School of Electrical and Computer Engineering, Georgia Institute of Technology, Atlanta, GA 30332, USA. 22affiliationmark: Sandia National Laboratories, Albuquerque, NM 87123, USA. christian.llanes@gatech.edu, christianllanes96@gmail.com Abstract In this work, we propose a framework that combines multi-agent reinforcement learning (MARL) with model-based control to achieve safe, dynamically feasible actions in cooperative multi-agent tasks. Multi-agent reinforcement learning provides the advantage of learning cooperative policies for multi-agent teams from discrete non-differentiable rewards in a long planning horizon. Model-predictive control is robust and offers safe, dynamically feasible actions in a fast replanning framework for short horizons. We propose an algorithm that extends actor-critic model predictive control for MARL which we refer to as multi-agent actor-critic model predictive control (MA-AC-MPC). We demonstrate the capabilities of this algorithm by applying it to a multi-agent pursuit-evasion scenario. Specifically, we compare the evader team’s strategy using the MA-AC-MPC model and a multi-layer perceptron model (MA-AC-MLP). The pursuer team uses augmented proportional navigation as it is accepted as an advanced adversarial control law. We also provide an example with a heterogeneous environment where a drone and omni-wheeled rover cooperate to achieve repeatable and successful landing with 100%100\% success rate in hardware for MA-AC-MPC compared to 60%60\% for MA-AC-MLP. We demonstrate the robustness of the proposed MA-AC-MPC algorithm in hardware for both environments. keywords: Collaborative Aerial Systems, Reinforcement Learning, Machine Learning for Robot Control, Aerial Systems: Mechanics and Control 1 Introduction Figure 1: Screenshots of both a homogeneous and heterogeneous multi-agent environment where multi-agent actor-critic model predictive control (MA-AC-MPC) is implemented. The top image is a screenshot at the final step of a two pursuer versus two evader environment trajectory before the pursuers collide into each other. The bottom screenshot is the final step of a drone landing on a rover environment where both agents learn their own AC-MPC policy. Both screenshots are from MuJoCo for rendering. Multi-agent reinforcement learning (MARL) (Busoniu et al., 2008) has emerged as a tool for modeling and control of environments with multiple agents acting as decision makers. MARL has been driven by robotics applications where coordination among autonomous agents such as warehouse robots, vehicle fleets, and robotic swarms is essential. Its capabilities have also been demonstrated in developing policies for heterogeneous multi-agent video games with complex tasks and constraints (Samvelyan et al., 2019). However, despite their success, pure learning-based techniques suffer from well-known limitations in maintaining robustness in out-of-distribution scenarios, large sample complexity, and lack of physical real-world constraints. All of which deter the use of pure learning-based methods in safety-critical applications. Model-based control provides several key advantages by leveraging the known physics and constraints of the robotic application to compute a dynamically feasible and safe action. Model-predictive control (MPC) recomputes the optimization problem over a receding horizon while continuously updating the solution. This enables fast online replanning and enhances robustness. However, this requires that the objective function is differentiable and that the solution be computed quickly enough for the desired robotics application. This also means that the objective function must be manually crafted to fit the desired task which is not feasible for a large array of tasks especially in a multi-agent cooperation problem. To bridge the gap, we propose combining model-based control and multi-agent reinforcement learning for computing safe dynamically feasible actions that respect the dynamic constraints of each agent, maintain robustness, and are capable of maximizing complex, non-differentiable, and cooperative team rewards. To accomplish this, we extend Actor-Critic Model Predictive Control (AC-MPC) (Romero et al., 2025) for multi-agent reinforcement learning which we refer to as multi-agent actor-critic model predictive control (MA-AC-MPC). To test the capabilities of the MA-AC-MPC we apply it to a multi-agent pursuit-evasion (MAPE) scenario (Llanes et al., 2026) and compare it to the results from using a multi-agent actor-critic multilayer perceptron (MA-AC-MLP) model. We summarize our contributions as follows: 1. We extend the AC-MPC algorithm to multi-agent reinforcement learning problems with cooperating agent teams working together to maximize their total rewards. 2. We evaluate the MA-AC-MPC algorithm across two distinct benchmarks: (i) a 2 vs. 2 multi-agent pursuit-evasion scenario, where evaders use MA-AC-MPC to induce pursuer collisions before reaching a goal; and (i) a cooperative landing scenario where a drone and a rover must synchronize their maneuvers to achieve a successful landing on the moving rover platform. 3. We implement MA-AC-MPC using the open-source project, leap-c (Fichtner et al., 2025), which is faster than the previously used differentiable MPC module, mpc.pytorch, in AC-MPC (Romero et al., 2025) and works for general nonlinear optimal control problems with general nonlinear inequality constraints by using acados (Verschueren et al., 2021). 4. We compare our MA-AC-MPC approach with an MA-AC-MLP model to demonstrate improved robustness by incorporating MPC as an actor layer. This is done by comparing the success rate of tasks for different variations of independent agent masses between MA-AC-MPC and MA-AC-MLP. We additionally provide an example where MA-AC-MPC resulted in higher success rate in experiments compared to MA-AC-MLP. 2 Related Works The literature on MPC and RL can be generally categorized into three approaches (Reiter et al., 2025b): using MPC as an expert policy, using MPC in the critic, and using MPC directly in the policy. Methods that use MPC as an expert policy typically use some form of imitation learning (Hoffmann et al., 2024) to match the MPC policy exactly or use the possibly suboptimal MPC to guide the RL exploration (Schulz et al., 2024). Some methods use MPC as an expert critic (Ghezzi et al., 2023; Anand et al., 2023) which allows for defining the constraint satisfaction in the objective. Methods that use MPC directly in the policy either learn a parametrized dynamical model (Lambert et al., 2019), the stage cost to encode complex task-specific behavior (Song and Scaramuzza, 2022; Tao et al., 2024; Romero et al., 2025), or the terminal value function (Moreno-Mora et al., 2023; Reiter et al., 2025a) which relates more to stability guarantees. Some methods also use MPC directly as an optimization layer in the actor policy during RL training which results in a closed-loop or end-to-end learning approach (Romero et al., 2025). This approach can require computing the gradients of the MPC during back-propagation. Several differentiable MPC methods have been developed (Amos et al., 2018; Oshin et al., 2024) with a tradeoff between gradient accuracy and computation time. Literature that embeds RL with MPC is typically focused on single-agent RL problems while literature on the multi-agent RL with MPC problem is limited. Mallick et al. (2024) proposes a multi-agent RL algorithm that uses distributed MPC for each agent that contributes to the global value function in a centralized learning framework. However, learning is not done in an end-to-end framework which limits its capabilities for learning complex behaviors efficiently such as in cooperation. We contribute to this line of research by extending the single-agent AC-MPC to multi-agent environments. 3 Preliminaries 3.1 Decentralized Markov Decision Processes A multi-agent system defines a collection of agents interacting jointly within a shared environment, collaboratively or competitively, to meet some objectives. Often, this requires some form of communication and coordination between agents to complete a shared task. Communication can be in the form of a complete observation of the states of other agents and the global state of the environment or a partial observation with only public states of agents being shared while private states being hidden. For decentralized agents with partial observability the Markov decision process (MDP) is known as a decentralized partially observable Markov decision process (dec-POMDP) (Oliehoek and Amato, 2016) defined by the tuple M:=(,,,P,,O,R,γ)M:=(K,S,A,P,O,O,R,γ) (1) where =1,..,KK=\1,..,K\ is the set of K>1K>1 agents, S is the global state space of the environment, and =(i)i=1KA=\A^(i)\_i=1^K is the set of action spaces from which an agent can select an action a(i)∈(i)a^(i) ^(i). Given the global state st∈s_t and the set of agent actions =a(1),…,a(K)a=\a^(1),...,a^(K)\, then the probability of transitioning to the next state st+1s_t+1 is defined by the state transition probability function P(st+1∣st,)P(s_t+1 s_t,a). =(i)i=1KO=\O^(i)\_i=1^K is the set of individual agent observation spaces (i)O^(i). Each agent receives an observation ot(i)∈(i)o^(i)_t ^(i) from t=ot(1),…,ot(K)o_t=\o^(1)_t,...,o^(K)_t\ given the observation probability function O(t+1∣t,st+1)O(o_t+1 _t,s_t+1). The immediate reward function for each agent is defined by R(i)(st,t,st+1)R^(i)(s_t,a_t,s_t+1) with a discount factor of γ∈[0,1)γ∈[0,1). In a shared reward (Jiang et al., 2024) or fully cooperative setting (Gronauer and Diepold, 2022) the reward is equally shared between all agents R=R(i)=R(K)R=R^(i)=R^(K). The actions of the individual agent are sampled from a decentralized policy π(i)(a(i)∣o(i))π^(i)(a^(i) o^(i)). The objective is to find a joint policy Π=π(1),…,π(K) =\π^(1),...,π^(K)\ that maximizes the discounted expected cumulative reward maximize Πst+1∼P,t∼Π[∑t=0∞γt∑i=1KR(i)(st,t,st+1)]. maximize E_s_t+1 P,a_t [ _t=0^∞γ^t _i=1^KR^(i)(s_t,a_t,s_t+1) ]. (2) For a team of collaborating homogeneous agents, all agents are of the same type and can therefore use a shared policy network that is executed in a distributed manner with each agent acting on its own local observation. Parameter sharing is an effective way (Gupta et al., 2017) to speed up the learning process when agents have a common goal and need to learn similar behaviors. Synthesizing a policy to solve (2) can be a challenging task. The first reason is that if we consider agents to learn independently and observe other agents as part of the environment, then the environment becomes non-stationary in the perspective of each agent and will be harder to learn a policy that converges to an optimal solution. This is known as fully decentralized learning and execution. An approach to improve the learning of cooperative tasks between agents is incorporating centralized training (Gronauer and Diepold, 2022). This is typically done by including additional global state information in the learning phase. If the additional global state information is discarded and only local agent information is available at execution time, then this is known as centralized training with decentralized execution (CTDE) (Jiang et al., 2024). If a joint policy is available for all units collectively and they all fully share information with each other, then this is known as centralized training and execution (CTE) (Amato, 2024). However, inter-agent communication can be expensive, particularly when bandwidth or latency limitations are present. One way to address this is to learn a communication protocol between agents (Sukhbaatar et al., 2016; Foerster et al., 2016). Additional solutions (Santos et al., 2025) have been proposed for dropout prone communication networks where the communication level is not known at the execution time. For a team of collaborating homogeneous agents, all agents are of the same type and can therefore use a shared policy network that is executed in a distributed manner with each agent acting on its own local observation. Parameter sharing is an effective way (Gupta et al., 2017) to speed up the learning process when agents have a common goal and need to learn similar behaviors. 3.2 Nonlinear Model Predictive Control For a given discrete-time dynamical system xk+1=fk(xk,uk;θMPC)x_k+1=f_k(x_k,u_k; _MPC) with state xk∈x_k , control uk∈u_k , and parameters θMPC _MPC we define a nonlinear MPC formulation as an optimal control problem (OCP) πMPC(x;θMPC)=argminu∑k=0N−1Lk(xk,uk;θMPC)+M(xN;θMPC)s.t.x0=xxk+1=fk(xk,uk;θMPC),k=0,…,N−10≥hk(xk,uk;θMPC),k=0,…,N−10≥hN(xk;θMPC) split _MPC(x; _MPC)&= *arg\,min_u _k=0^N-1L_k(x_k,u_k; _MPC)+M(x_N; _MPC)\\ s.t. x_0&=x\\ x_k+1&=f_k(x_k,u_k; _MPC), k=0,…,N-1\\ 0&≥ h_k(x_k,u_k; _MPC), k=0,…,N-1\\ 0&≥ h_N(x_k; _MPC) split (3) where the state and control constraints are contained in the path constraint function hk(xk,uk;θMPC)h_k(x_k,u_k; _MPC) and the terminal constraint function hN(xk;θMPC)h_N(x_k; _MPC). The cost function for this optimal control problem consists of the stage cost Lk(xk,uk;θMPC)L_k(x_k,u_k; _MPC) and the terminal cost M(xN;θMPC)M(x_N; _MPC). In a trajectory tracking MPC formulation, the cost function may be written in a nonlinear least-squares form, such as Lk(xk,uk;θMPC)=12∥yk(xk,uk;θMPC)−yk,ref(θMPC)∥W2M(xN;θMPC)=12∥yN(xN;θMPC)−yN,ref(θMPC)∥WN2 splitL_k(x_k,u_k; _MPC)&= 12 y_k(x_k,u_k; _MPC)-y_k,ref( _MPC) ^2_W\\ M(x_N; _MPC)&= 12 y_N(x_N; _MPC)-y_N,ref( _MPC) ^2_W_N split (4) for stage reference yk,ref∈ℝnyy_k,ref ^n_y, terminal reference yN,ref∈ℝny,Ny_N,ref ^n_y,N, and weighting matrices W∈ℝny×nyW ^n_y× n_y, WN∈ℝny,N×ny,NW_N ^n_y,N× n_y,N from the weighted L2-norm, i.e. ∥x∥W2=xTWx x ^2_W=x^TWx. This formulation can also be written as linear least-squares through a linear transformation for yk(xk,uk;θMPC)y_k(x_k,u_k; _MPC) and yN(xN;θMPC)y_N(x_N; _MPC). The cost function may also be written as a quadratic cost function such as Lk(xk,uk;θMPC)=12[xkuk]⊤Qk(θMPC)[xkuk]+pk(θMPC)⊤[xkuk],L_k(x_k,u_k; _MPC)= 12 bmatrixx_k\\ u_k bmatrix Q_k( _MPC) bmatrixx_k\\ u_k bmatrix+p_k( _MPC) bmatrixx_k\\ u_k bmatrix, (5) where Qk(θMPC)Q_k( _MPC) and pk(θMPC)p_k( _MPC) denote the quadratic and linear cost coefficients at prediction stage k, respectively. The vector θMPC _MPC contains the learned parameters that determine these cost coefficients. As shown above, parameters may appear in the cost, dynamics, or constraints. There can also be global parameters such as mass, moment of inertia, spring constants, or fixed cost function coefficients. By defining both the system dynamics and the cost function as neural networks, we treat their underlying weights and biases as tuneable parameters. Adjusting the parameters may lead to improved real-world tracking performance through manual tuning or gradient descent techniques. To perform gradient descent, we require the computation of the sensitivity of the MPC optimal solution with respect to the parameter set. Consequently, this necessitates a differentiable MPC framework that allows gradients to propagate through the optimization solver. Additionally, the differentiable MPC framework should work with existing learning-based libraries such as PyTorch. Amos et al. (2018) proposed a differentiable MPC framework called mpc.pytorch that computes gradients of a loss function with respect to the MPC parameters by implicitly differentiating KKT conditions at a fixed point of a box-constrained iterative LQR solver. However, their implementation is restricted to quadratic cost functions such as in (5) and box constraints on the control input. Other differentiable optimal control frameworks have been proposed (Dinev et al., 2022; Adabag et al., 2025; Jin et al., 2021; Bounou et al., 2023), however, their solution sensitivities are often degraded or restricted to unconstrained optimal control problems. Additionally, the above approaches cannot handle general nonlinear parametric inequality constraints with states and controls. The open-source software framework acados (Verschueren et al., 2021) is designed for optimal control problems with nonlinear and parametric costs, dynamics, and inequality constraints with the speed required for real-time control applications. Sensitivity propagation has also been implemented in the acados framework (Frey et al., 2023, 2025) for computing derivatives of the optimal solution with respect to parameters and initial state. Recently, the open-source project leap-c has provided differentiable interfaces for the acados sensitivity implementation within machine learning frameworks such as PyTorch and JAX. (Fichtner et al., 2025). Figure 2: Architecture diagram for multi-agent actor-critic model predictive control (MA-AC-MPC). 4 Methodology In this section, we discuss our implementation of actor-critic model predictive control for multi-agent reinforcement learning problems. 4.1 Actor-Critic Multi-Agent Reinforcement Learning In order to find a joint policy Π=π(1),…,π(K) =\π^(1),...,π^(K)\ that maximizes the discounted expected cumulative returns from (2) we turn towards the actor-critic structure in reinforcement learning. To approximate the true value function, Vπ(s)=[∑t=0T−1γtR(st,t,st+1)|s0=s]st+1∼P,t∼Π splitV_π(s)= E [& _t=0^T-1γ^tR(s_t,a_t,s_t+1) |s_0=s ]\\ &s_t+1 P,a_t split (6) we use a feed-forward neural network that computes a centralized value function for the multi-agent scenario during training given the full state of the environment which may include information not provided at execution time to the policy. To train the actor-critic structure, we use multi-agent proximal policy optimization (MAPPO) (Yu et al., 2022). In MAPPO, the general actor loss for potentially heterogeneous agents is defined by the sum of the clipped surrogate objective and the entropy loss Lactor(i)(θ(i))=Lclip(i)(θ(i))+λentropyLentropy(i)(θ(i))L^(i)_actor(θ^(i))=L^(i)_clip(θ^(i))+ _entropyL^(i)_entropy(θ^(i)) where the hyperparameter λentropy _entropy is the coefficient of entropy loss. The clipped surrogate objective is defined as Lsurrogate(i)(θ(i))=rθ,b(i)A^b(i)Lclipped surrogate(i)(θ(i))=clip(rθ,b(i),1−ϵa,1+ϵa)A^b(i)Lclip(i)(θ(i))=−1B∑b=1B[min(Lsurrogate(i),Lclipped surrogate(i))] splitL^(i)_surrogate(θ^(i))&=r^(i)_θ,b A^(i)_b\\ L^(i)_clipped surrogate(θ^(i))&=clip(r^(i)_θ,b,1- _a,1+ _a) A^(i)_b\\ L^(i)_clip(θ^(i))=- 1B _b=1^B [& (L^(i)_surrogate,L^(i)_clipped surrogate ) ] split (7) for batch of size B, PPO clipping term ϵa _a, and rb(i)(θ(i))=πθ(i)(ab(i)|ob(i))πθold(i)(ab(i)|ob(i))r^(i)_b(θ^(i))= π^(i)_θ(a^(i)_b|o^(i)_b)π^(i)_ _old(a^(i)_b|o^(i)_b). The clipped surrogate objective updates the actor in a way that maximizes the probability that actions lead to better rewards while clipping the update to stabilize the learning process. The advantage estimate A^b(i) A^(i)_b is calculated using the generalized advantage estimate (GAE) (Schulman et al., 2018) where γGAE _GAE is the reward discount factor and λGAE _GAE is the smoothness factor. The second term in the actor loss function is the entropy loss defined as Lentropy(i)(θ(i))=−1B∑b=1Bπθentropy(i)(ob(i))L^(i)_entropy(θ^(i))=- 1B _b=1^Bπ^(i)_ _entropy(o^(i)_b) (8) which prevents getting stuck in a local optimum and reinforces exploration. The critic loss function is defined as Vϕ,clipped(i)(sb)=Vϕold(i)(sb)+clip(Vϕnew(i)(sb)−Vϕold(i)(sb),−ϵc,ϵc)Lcritic(i)(ϕ(i))=1B∑b=1B(Vϕ,clipped(i)−R^b(i))2 splitV^(i)_φ,clipped(s_b)&=V^(i)_ _old(s_b)+clip(V^(i)_ _new(s_b)-V^(i)_ _old(s_b),- _c, _c)\\ L^(i)_critic(φ^(i))&= 1B _b=1^B(V^(i)_φ,clipped- R^(i)_b)^2 split (9) for critic clipping term ϵc _c and discounted reward-to-go R^b(i)=A^b(i)+Vϕold(i)(sb) R^(i)_b= A^(i)_b+V^(i)_ _old(s_b). The total loss function is defined as L(Θ,Φ)=∑i=1KLactor(i)(θ(i))+λcriticLcritic(i)(ϕ(i))L( , )= _i=1^KL^(i)_actor(θ^(i))+ _criticL^(i)_critic(φ^(i)) for Θ=θ(1),…,θ(K) =\θ^(1),...,θ^(K)\ and Φ=ϕ(1),…,ϕ(K) =\φ^(1),...,φ^(K)\ where the hyperparameter λcritic _critic is the critic loss coefficient. For homogeneous agent environments with shared actor networks the loss function can be reduced to L(θ,ϕ)=1K∑i=1KLactor(i)(θ)+λcriticLcritic(i)(ϕ)L(θ,φ)= 1K _i=1^KL^(i)_actor(θ)+ _criticL^(i)_critic(φ). The parameters Θ and Φ are updated via gradient descent on the joint loss function L(Θ,Φ)L( , ). To ensure training stability, the gradient norm is clipped to a maximum threshold of λgrad _grad, so that the update follows g^=gif ‖g‖2≤λgradλgrad‖g‖2gif ‖g‖2>λgrad g= casesg&if \|g\|_2≤ _grad\\ _grad\|g\|_2g&if \|g\|_2> _grad cases (10) where g=∇Θ,ΦL(Θ,Φ)g= _ , L( , ). In certain multi-agent environments, agents may be terminated or deactivated mid-episode, often as a consequence of events such as collisions. It has been shown by Yu et al. (2022) that updating the actor and critic while an agent is inactive without any changes to the agent observation or global state results in poor training results. This leads us to the concept of death masking in which an inactive or dead agent has its observation or states masked by a zero vector. Yu et al. (2022) proposed an explanation in which a distribution shift occurs in the input of the value network when an agent becomes deactivated, but containing it to a single vector of zeros allows the value network to better reason about the future rewards of the environment with deactivated agents in the zero vector input. Additionally, they achieved better results by also including an agent identifier with death masking. Therefore, in our implementation we use GAE bootstrap cutoff for inactive agents, masking inactive agent states and observations, one-hot encoded agent ID, and a binary flag indicating whether an agent is active. 4.2 Augmenting MARL with MPC The MPC problem defined in (3) is defined as a single-agent MPC problem. Transforming this problem as a multi-agent MPC problem can we done in several ways. One method is to have a centralized MPC solver that controls agents externally. However, this approach scales poorly with the number of agents as the number of decision variables is increased and is not effective for autonomous environments where a centralized computer is required. Additionally, this approach requires reliable communications as control actions are published through the network. Even then, latency remains a problem, limiting the effective capabilities of each agent. A secondary approach is a distributed MPC formulation where a centralized problem can be distributed into several smaller subsystems (Negenborn et al., 2009) that communicate with each other. Agents that do not interact with each other are known as decentralized agents. The challenge in designing a distributed MPC framework is constructing a proper objective for each agent that formalizes the general cooperative task. Additionally, the objective and constraints must remain tractable enough to compute the optimal control in real-time. In the context of (3) the cooperative task must be encoded through the MPC parameters θMPC _MPC. More specifically, the stage and terminal cost are designed for some cooperative objective. In our approach, we use a linear least-squares formulation for the cost function where the tuneable parameters are the stage reference yk,refy_k,ref, the terminal reference yN,refy_N,ref, and the weighting matrices W, WNW_N. More formally, we use an AC-MPC structure with a hierarchical actor such that the first layer is a neural network that outputs the MPC parameters πθ(i)(yk,ref(i),yN,ref(i),W(i),WN(i)|o(i))π^(i)_θ(y^(i)_k,ref,y^(i)_N,ref,W^(i),W^(i)_N|o^(i)) which behaves as more of a planner and the second layer is the differentiable MPC module using leap-c to generate feasible control actions. The deployment is distributed, however, the neural network cost function takes observations that contain information about the agent’s own state and partial state information from other agents. During the centralized training phase, a shared value function evaluates the global environment state to optimize individual actor policies toward maximizing collective rewards. An architecture diagram of MA-AC-MPC is provided in Figure 2. 5 Results In this section, we evaluate the proposed MA-AC-MPC framework on two different multi-agent environments and compare the results to a standard multilayer perceptron actor. The first environment is a multi-agent pursuit evasion environment with two pursuers and two evaders. The evaders use MA-AC-MPC and the pursuers use a fixed pursuit strategy. The second environment is a drone and rover environment where the drone is tasked with landing on a rover and both agents have their own policy and MPC model. This allows us to compare MA-AC-MPC for both a homogeneous environment with a shared policy for each agent and a homogeneous environment with different policies for each agent. For training our models, we use SKRL (Serrano-Muñoz et al., 2023) with its in-built implementation of the MAPPO algorithm and a slight modification to include MPC initial states in the memory buffer. Table 1: Reward function components and coefficients. Component Expression Coefficient Sparse event rewards Capture rcap⋅∑i=1Ne[captureei]r_cap· _i=1^N_e 1[capture_e_i] −5.0-5.0 Evader collision ree⋅∑i=1Ne[collisionei]r_e· _i=1^N_e 1[collision_e_i] −5.0-5.0 Pursuer collision rpp⋅∑i=1Np[collisionpi]r_p· _i=1^N_p 1[collision_p_i] 30.030.0 P closing velocity crv⋅∑i≠jmax(−ij⊤^ij, 0)c_rv· _i≠ j (-v_ij r_ij,\;0) 1.01.0 Boundary violation rbnd⋅∑i=1Ne[geofenceei]r_bnd· _i=1^N_e 1[geofence_e_i] −5.0-5.0 Shaping rewards Pursuer proximity cprox⋅e−λprox∥p1−p2∥c_prox· e^- _prox\, _p_1-p_p_2 0.3,λ=2.50.3,\;λ=2.5 Penalties (subtracted) Attitude cΦ⋅∑i=1Ne∥ei∥c_ · _i=1^N_e _e_i 0.010.01 Velocity cv⋅∑i=1Ne∥ei∥2c_v· _i=1^N_e _e_i ^2 0.030.03 Hover thrust cu⋅∑i=1Ne(ueiT−uhoverT)2c_u· _i=1^N_e(u_e_i^T-u_hover^T)^2 0.50.5 Thrust smoothness cΔuT⋅∑i=1Ne(ΔueiT)2c_ u_T· _i=1^N_e( u_e_i^T)^2 5.05.0 Control smoothness cΔu⋅∑i=1Ne∥Δeirpy∥2c_ u· _i=1^N_e _e_i^rpy ^2 1.01.0 [⋅] 1[·] denotes the indicator function. All terms are summed over active agents and normalized by Npairs=2N_pairs=2. 5.1 Multi-Agent Pursuit Evasion 5.1.1 Problem Definition A multi-agent pursuit-evasion scenario was proposed by Llanes et al. (2026) where a team of evaders and a team of pursuers operate in a 3-dimensional environment. The pursuers use one of two fixed strategies to chase and capture evaders: pure pursuit or augmented proportional navigation (Zarchan, 2019). The evaders are tasked with evading the pursuers and are rewarded with reaching a goal with minimal velocity. This environment is a good application for MA-AC-MPC since evaders are assumed to be homogeneous agents with the same dynamics and overall must achieve a goal that requires cooperation. We apply this problem for a 2 pursuer versus 2 evader environment with some slight modifications to the original paper to demonstrate that MA-AC-MPC is more robust and sample efficient in learning for this problem. The first modification is to the objective such that there is no longer a goal and the main task of evaders is to make the pursuers collide with each other. However, an evader win condition now requires that all pursuers must be dead and at least one evader is alive. A pursuer win condition occurs when all evaders are dead even if all pursuers are dead as well. An evader agent dies if it is captured, crashes onto the ground, crashes with a teammate, or exits the boundary. A pursuer agent dies if it collides with teammates or captures an evader. We fix the pursuer strategy to use proportional navigation only without feedback of the evader acceleration. This was done to reduce the amount of information that was passed through the limited bandwidth radio hardware. However, we add an additional line-of-sight acceleration to the pursuers such that there is no need for switching to pure-pursuit below some closure velocity. This leads to a satisfactory pursuit policy that captures evaders with near 100%100\% success. Additionally, each evader and pursuer is spawned within its own box that is nominally spaced with its team member. A distance check is performed to ensure that the spawn location exceeds the team’s collision radius by a defined safety margin to ensure that members do not collide immediately on environment reset. We extend the evader team rewards for the environment with an exponential proximity for the pursuers, an extension to the reward for pursuer-on-pursuer collision by rewarding higher closing velocity on collision, an angle penalty for evader Euler angles Φe=[ϕ,θ,ψ] _e=[φ,θ,ψ] to deter large orientation maneuvers, a velocity penalty for flying too quickly, a penalty for thrust difference from hover, and a control smoothness penalty for variations in sequential control actions. Together, these extra rewards ensured a satisfactory training process and simulation-to-hardware. Table 2: Curriculum level parameters. Evader spawn is deterministic at nominal positions for levels 1–9 and randomized (Δxyz=0.1 _xyz=0.1 m) at level 10. Pursuer–pursuer and pursuer–evader collision tolerances are equal (dpp=dped_p=d_pe) at all levels. The advance threshold is a 70% evader win rate. Pursuer Spawn Collision Tol. Domain Rand. Disturbance Level Δxy _xy Δz _z deed_e dppd_p σm _m σJ _J σf _f στ _τ [m] [m] [m] [m] [g] [×10−6\!×\!10^-6] [mN] [μ m] 1 0.10 0.10 0.20 0.50 — — — — 2 0.25 0.25 0.30 0.40 — — — — 3 0.35 0.35 0.30 0.40 — — — — 4 0.40 0.40 0.30 0.40 1.0 0.5 — — 5 0.40 0.40 0.30 0.40 2.0 1.0 — — 6 0.40 0.40 0.30 0.40 2.0 1.0 5.0 50 7 0.50 0.40 0.30 0.35 2.0 1.0 5.0 50 8 0.50 0.40 0.30 0.30 2.0 1.0 5.0 50 9 0.50 0.40 0.20 0.20 2.0 1.0 5.0 50 10† 0.50 0.40 0.20 0.20 2.0 1.0 5.0 50 †Evader spawn randomized: Δxyz=0.1 _xyz=0.1 m. The rewards are more clearly defined in Table 1. 5.1.2 Model and Training Configuration The dynamics in (Llanes et al., 2026) use a simplified multirotor drone model with an approximation of the rotational dynamics as a first order system for training. The authors demonstrated successful sim-to-real, but the dynamics can be improved to transfer policies with more agile maneuvers. This is why we chose to use crazyflow (Schuck et al., 2026), a fast parallelizable Crazyflie drone swarm simulator using JAX. For the environment simulation, we use the first_principles model, which is a high-fidelity full rigid body simulation of the drone with rotor dynamics, aerodynamic drag, and blade gyroscopic effects. The Mellinger controller in crazyflow is used because it contains the same parameters as the Crazyflie firmware. For the MPC model, we use the so_rpy model which is faster with a tradeoff of lower complexity. It requires computing system identification coefficients for the rotational dynamics. For all of our experiments, we use the provided system identification parameters for the Crazyflie thrust upgrade model cf2x_T350. Table 3: A comparison of training times, inference times, and neural network sizes for MA-AC-MPC with N=2N=2 and MA-AC-MLP methods. All MA-AC-MLP training times are for 4M steps and MA-AC-MPC training times are for 2M steps. Inference times are per agent. Method Net Size (A / C) Train Time Inference Time MPC [256×2 / 256×2] 162.2h 1.052 ±0.474ms MLP [256×2 / 256×2] 18.0h 0.120 ±0.063ms MLP [512×2 / 512×2] 26.8h 0.140 ±0.077ms MLP [512×3 / 512×2] 30.0h 0.197 ±0.175ms Figure 3: Training curves for mean total reward, evader win rate, and curriculum level for various MA-AC-MLP actor and critic neural network sizes and the proposed MA-AC-MPC. Curriculum level advancement occurs when the evader win rate reaches a threshold of 70%70\%. The MA-AC-MPC model uses a single policy shared by both evader agents. The observation to the policy is the agent’s position, velocity, rotation matrix flattened, body rates, and a one-hot ID for the agent using the policy. Additionally, the observation includes teammate and pursuer states, such as position and velocity, which are masked based on whether the agent is alive, along with an explicit alive flag. Finally, the target of the pursuer is provided as one-hot vectors. The shared state provided to the critic is similar but includes rotation matrix and body rates for all evaders and pursuers. We trained MA-AC-MPC and different MA-AC-MLP model sizes on a 16-core AMD Ryzen 9 5950x desktop processor and tabulate the training time and inference time in Table 3. One downside of the acados framework is that the code generation tools are built and optimized for CPU operations and cannot be run on the GPU. The MPC step is Δt=20 ms t=20 ms which results in a control frequency of 50 Hz50 Hz. The horizon for the MPC model N=2N=2 was chosen based on the results of (Romero et al., 2025). Additionally, we tested N=5N=5 and did not achieve better training or simulation results. We provide training curves in Figure 3 for MA-AC-MPC and the various MA-AC-MLP model sizes. Although the MA-AC-MPC architecture requires longer training times per iteration compared to the MA-AC-MLP models, it achieves superior win rates in significantly fewer steps. This suggests enhanced representational efficiency; by offloading low-level dynamics to the MPC, the neural network is not forced to encode complex model information. Consequently, the network functions as a high-level task planner rather than a black box, end-to-end controller. We initially attempted to train the evader versus pursuer environment with totally random positions, but noticed that the evaders struggled to learn to cooperate to lead the pursuers into a collision. This led us to design a set of curriculum levels with increasingly difficult environment parameters such as larger spawn distance between pursuers, tighter collision tolerances, and more disturbances. These parameters are tabulated in Table 2 and the progression of levels throughout training is plotted in Figure 3. Figure 4: Evader win rate for independent evader mass variations for both MA-AC-MPC [256×2 / 256×2] and MA-AC-MLP [512×2 / 512×2]. Both models are trained with a nominal mass of 40.6. The mass is only changed within the environment and not the MPC dynamics which uses the nominal mass. The third plot is the difference in win rate between both methods where darker red implies MA-AC-MPC win rate is much higher than MA-AC-MLP. This plot is generated for 1000 episodes of each mass variation for each method. Figure 5: Evader and pursuer position plots for hardware, Crazyflow, and CrazySim. See Extension 1 for Crazyflow simulation video and Extension 2 for hardware video. 5.1.3 Simulation and Hardware Experiments We tested MA-AC-MPC trained (N=2)(N=2) with [Actor size / Critic size] [256×2 / 256 × 2] against the MA-AC-MLP model size with the highest evader win rate which is [512×2 / 512 × 2] in a mass sweep robustness test shown in Figure 4 where the mass is only changed within the environment and not the MPC dynamics. The mass sweep test is performed between 30.6 and 50.6 in intervals of 2 with a nominal trained mass of 40.6. The left plot shows the contour color-coded with evader win rate for the MA-AC-MPC. The middle plot is the evader win rate contour for MA-AC-MLP. The right-most plot is difference in win rate between both models where red implies MA-AC-MPC has higher win rate than MA-AC-MLP. There is no blue shading in the right-most plot implying that MA-AC-MPC outperformed MA-AC-MLP in the mass sweep robustness test. We deployed the trained MA-AC-MPC model to a ROS 2 framework using Crazyswarm2 (Preiss* et al., 2017) for hardware testing. However, before hardware, we tested the ROS 2 nodes with the actual Crazyflie firmware in the loop using CrazySim (Llanes et al., 2024) with the MuJoCo simulator that models rotor dynamics, aerodynamic effects, and rotor gyroscopic precession in CrazySim. We compared Crazyflow which is the environment used for training, CrazySim, and hardware trajectory data and plotted the results in Figure 5. The position trajectories for the evaders and pursuers are plotted in the top curves and the 3-dimensional position plot is shown in the bottom. The curves show a similar trend in the trajectory even with hardware sensor noise, disturbances, estimator state uncertainty, and any model mismatch. Another trajectory is shown in Figure 1 for different initial positions where the evaders cooperate to lead the pursuers into a head-on collision. 5.2 Drone Landing on Ground Robot Table 4: Reward function components and coefficients for the drone and rover landing task. Component Expression Coefficient Team rewards XY progress cprog(dxyt−1−dxyt)c_prog(d_xy^t-1-d_xy^t) 10.010.0 Z progress cprogσc(dzt−1−dzt)c_prog _c(d_z^t-1-d_z^t) 10.010.0 Landing bonus [landed](rland−clv∥rel∥ 1[landed](r_land-c_lv _rel +cprecmax(1−dxy/rlz,0))+\,c_prec (1-d_xy/r_lz,0)) 100.0, 2.0, 10.0100.0,\;2.0,\;10.0 Drone-only rewards and penalties Crash rcrash[crash]r_crash 1[crash] −50.0-50.0 Boundary violation rbnd[OOB]r_bnd 1[OOB] −50.0-50.0 Descent speed −cdescσcmax(vdesc−vmax,0)2-c_desc _c (v_desc-v_max,0)^2 2.02.0 Altitude hold −calt(1−σc)(zcruise−zd)2-c_alt(1- _c)(z_cruise-z_d)^2 0.30.3 Velocity −cvmax(∥d∥−vmaxd,0)2-c_v ( _d -v_max^d,0)^2 2.02.0 XY corridor velocity −cxyσc∥d,xy∥2-c_xy _c _d,xy ^2 5.05.0 Attitude −cΦ(ϕ2+θ2)-c_ (φ^2+θ^2) 0.010.01 Thrust smoothness −cΔT(ΔudT)2-c_ T( u_d^T)^2 3.03.0 RPY smoothness −cΔrpy∥Δdrpy∥2-c_ _d^rpy ^2 5.05.0 Rover-only penalties Stillness −cstillσcvr2-c_still _cv_r^2 0.50.5 Yaw rate −cωσcωr2-c_ω _c _r^2 0.50.5 Lateral/backward −clat(vy,r2+max(−vx,r,0)2)-c_lat(v_y,r^2+ (-v_x,r,0)^2) 0.20.2 vxv_x smoothness −cΔvx(Δurvx)2-c_ v_x( u_r^v_x)^2 0.030.03 vyv_y smoothness −cΔvy(Δurvy)2-c_ v_y( u_r^v_y)^2 0.030.03 ωz _z smoothness −cΔω(Δurωz)2-c_ ω( u_r _z)^2 0.0010.001 Rover boundary −crb[rover at edge]-c_rb 1[rover at edge] 1.01.0 Here dxy=‖[xd−xr,yd−yr]⊤‖2d_xy=\|[x_d-x_r,\;y_d-y_r] \|_2 is the horizontal drone-rover distance and dz=|zd−hr|d_z=|z_d-h_r| is the vertical distance to the rover landing-pad height hrh_r. The corridor gate is σc=σ((rcorr−dxy)/w) _c=σ((r_corr-d_xy)/w), with rcorr=0.3mr_corr=0.3\,m and w=0.1mw=0.1\,m. The relative touchdown velocity is rel=[x˙d−vr,xw,y˙d−vr,yw,z˙d]⊤v_rel=[ x_d-v_r,x^w, y_d-v_r,y^w, z_d] , vdesc=max(−z˙d,0)v_desc= (- z_d,0), and d,xy=[x˙d,y˙d]⊤v_d,xy=[ x_d, y_d] . The rover speed is vr=vx,r2+vy,r2v_r= v_x,r^2+v_y,r^2, where vx,rv_x,r and vy,rv_y,r are body-frame velocities, and ωr _r is the rover yaw rate. The symbols ϕφ and θ denote drone roll and pitch. The terms ΔudT u_d^T, Δdrpy _d^rpy, and Δur(⋅) u_r^(·) denote changes in thrust, drone attitude commands, and rover velocity commands between consecutive time steps. The constants are rlz=0.07mr_lz=0.07\,m, vmax=0.3m/sv_max=0.3\,m/s, vmaxd=0.6m/sv_max^d=0.6\,m/s, and zcruise=1.0mz_cruise=1.0\,m. The indicator functions denote successful landing, crash/contact failure, drone out-of-bounds (OOB), and rover contact with the arena edge. The drone receives rteam+rdroner_team+r_drone and the rover receives rteam+rroverr_team+r_rover. Table 5: Curriculum levels and their configurations. Drone Spawn Rover Spawn Domain Rand. Level Δxy _xy Δz _z vmaxv_max Stationary σm _m σJ _J [m] [m] [m/s] [g] [×10−6\!×\!10^-6] 1 0.50 0.5–1.0 — ✓ — — 2 1.00 0.5–1.5 0.3 — — 3 1.50 0.5–1.5 0.6 — — 4 2.50 0.5–1.5 0.9 2.0 1.0 5 3.50 0.5–1.5 0.9 2.0 1.0 6 4.50 0.5–2.0 1.0 2.0 1.0 5.2.1 Problem Definition To further evaluate MA-AC-MPC, we construct a heterogeneous multi-agent environment where a drone is tasked with landing on a moving ground robot. Each agent receives an observation vector tailored to its role in the cooperative landing task. We denote the drone position and velocity by d=[xd,yd,zd]⊤,d=[x˙d,y˙d,z˙d]⊤.p_d=[x_d,y_d,z_d] , _d=[ x_d, y_d, z_d] . (11) The relative drone-rover position is dr=d−[xr,yr,0]⊤,p_dr=p_d-[x_r,y_r,0] , (12) with horizontal and vertical distances dxy=‖[xd−xr,yd−yr]⊤‖2,dz=|zd−hr|,d_xy=\|[x_d-x_r,\;y_d-y_r] \|_2, d_z=|z_d-h_r|, (13) where hrh_r is the rover landing-pad height. For the rover, the observation contains the rover state, relative drone position, and drone velocity: r=[r⊤dr⊤d⊤‖d‖2‖dr‖2]⊤∈ℝ15,o_r= bmatrixx_r &p_dr &v_d &\|v_d\|_2&\|p_dr\|_2 bmatrix ^15, (14) where the rover state is r=[xryrcosθrsinθrvx,rvy,rωr]⊤∈ℝ7.x_r= bmatrixx_r&y_r& _r& _r&v_x,r&v_y,r& _r bmatrix ^7. (15) Here, vx,rv_x,r and vy,rv_y,r are rover body-frame velocities, and ωr _r is the rover yaw rate. The drone observation contains the drone state, rover motion, and relative position: d=[d⊤d⊤vec(d)⊤d⊤r⊤dr⊤]⊤∈ℝ29,o_d= bmatrixp_d &v_d &vec(R_d) & ω_d &z_r &p_dr bmatrix ^29, (16) where d∈SO(3)R_d∈ SO(3) is the drone rotation matrix, d ω_d is the drone angular velocity, and r=[xryr(rw)⊤sinθrcosθrvrvy,r]⊤∈ℝ8.z_r= bmatrixx_r&y_r&(v_r^w) & _r& _r&v_r&v_y,r bmatrix ^8. (17) The rover speed used in the reward table is vr=vx,r2+vy,r2,v_r= v_x,r^2+v_y,r^2, (18) and its world-frame velocity is rw=[vx,rcosθr−vy,rsinθrvx,rsinθr+vy,rcosθr].v_r^w= bmatrixv_x,r _r-v_y,r _r\\ v_x,r _r+v_y,r _r bmatrix. (19) Finally, the relative velocity used in the landing bonus is rel=[x˙d−vr,xwy˙d−vr,ywz˙d]⊤,v_rel= bmatrix x_d-v_r,x^w& y_d-v_r,y^w& z_d bmatrix , (20) and d,xy=[x˙d,y˙d]⊤v_d,xy=[ x_d, y_d] . The two agents must cooperate to maximize the cumulative team and individual rewards summarized in Table 4. The drone model uses the first_principles model for the physics of the environment and so_rpy for the MPC model. For the ground robot, we select the Yahboom ROSMASTER X3 shown in Figure 6. The X3 is a mecanum-wheeled omnidirectional platform that allows independent control of forward, lateral, and rotational velocities. The ground robot uses the following dynamics x˙ x =vxcosθ−vysinθ =v_x θ-v_y θ (21) y˙ y =vxsinθ+vycosθ =v_x θ+v_y θ θ˙ θ =ωz = _z ω˙i ω_i =ω¯i−ωiτ,i∈FL,FR,BL,BR = ω_i- _iτ, i∈\FL,FR,BL,BR\ Figure 6: Close-up view of the Crazyflie brushless on top of the ROSMASTER X3 rover. where x,yx,y are the world-frame position, θ is the heading, vxv_x and vyv_y are body-frame forward and lateral velocities, ωz _z is the yaw rate, ωi _i are the individual wheel angular velocities, and τ is the first-order motor time constant modeling the PID loop response. The body velocities are related to the wheel speeds through the forward kinematics. The target wheel speeds ω¯i ω_i are obtained via the mecanum inverse kinematics ω¯FL ω_FL =1r(vx,cmd−vy,cmd−Kωz,cmd) = 1r (v_x,cmd-v_y,cmd-K _z,cmd ) (22) ω¯FR ω_FR =1r(vx,cmd+vy,cmd+Kωz,cmd) = 1r (v_x,cmd+v_y,cmd+K _z,cmd ) ω¯BL ω_BL =1r(vx,cmd+vy,cmd−Kωz,cmd) = 1r (v_x,cmd+v_y,cmd-K _z,cmd ) ω¯BR ω_BR =1r(vx,cmd−vy,cmd+Kωz,cmd) = 1r (v_x,cmd-v_y,cmd+K _z,cmd ) with each wheel clipped to [−ωmax,ωmax][- _ , _ ], where K=l+dK=l+d is the combined kinematic parameter from the standard 45° roller geometry. The MPC uses smooth body-level first-order velocity dynamics with state r=[xr,yr,θr,vx,r,vy,r,ωr]⊤x_r=[x_r,y_r, _r,v_x,r,v_y,r, _r] and control r=[vx,cmd,vy,cmd,ωcmd]⊤u_r=[v_x,cmd,v_y,cmd, _cmd] is defined as x˙r x_r =vx,rcosθr−vy,rsinθr, =v_x,r _r-v_y,r _r, (23) y˙r y_r =vx,rsinθr+vy,rcosθr, =v_x,r _r+v_y,r _r, θ˙r θ_r =ωr, = _r, v˙x,r v_x,r =vx,cmd−vx,rτ, = v_x,cmd-v_x,rτ, v˙y,r v_y,r =vy,cmd−vy,rτ, = v_y,cmd-v_y,rτ, ω˙r ω_r =ωcmd−ωrτ. = _cmd- _rτ. This body-velocity input model matches the X3 command interface, which accepts linear and angular velocity commands rather than direct wheel-speed commands. The model is discretized with RK4 inside the MPC. The wheel limits are enforced as linear inequality constraints on the commanded body velocity: −ωmax≤Dwheelr≤ωmax,- _ 1≤ D_wheelu_r≤ _ 1, (24) where Dwheel=1rw[1−1−K11K11−K1−1K],K=l+d.D_wheel= 1r_w bmatrix1&-1&-K\\ 1&1&K\\ 1&1&-K\\ 1&-1&K bmatrix, K=l+d. (25) Parameters are taken from the X3 URDF and ROS2 controller configuration: wheel radius rw=0.0325mr_w=0.0325\,m, half-wheelbase l=0.08ml=0.08\,m, half-track width d=0.0845md=0.0845\,m, K=0.1645mK=0.1645\,m, motor time constant τ=0.1sτ=0.1\,s, and maximum wheel speed ωmax=34.9rad/s _ =34.9\,rad/s. Curriculum learning is also used in this multi-agent environment to stabilize learning through a sequence of increasingly difficult parameter configurations shown in Table 5. Figure 7: Landing rate for MA-AC-MPC and MA-AC-MLP during training. Solid curves show the smoothed landing rate, faint curves show the raw logged values, and dashed vertical lines indicate curriculum level advancements. 5.2.2 Simulation and Hardware We train both the MA-AC-MPC and MA-AC-MLP with critic and actor network size of [256,256][256,256] using ReLU activations. Both methods are trained with MAPPO for 1.2×1061.2× 10^6 environment steps using 128128 parallel environments. The PPO rollout length is 256256, with 44 learning epochs and 44 mini-batches per update. We use a learning rate of 3×10−43× 10^-4, discount factor γ=0.99γ=0.99, GAE parameter λ=0.95λ=0.95, clipping ratio ϵ=0.2ε=0.2, entropy coefficient 0.010.01, value loss coefficient 1.01.0, and gradient norm clipping at 0.50.5. The MPC horizon for both drone and rover is N=2N=2. We provide plots for the landing rate during training in Figure 7. See Extension 3 for a simulation video. In this particular environment, we noticed that MA-AC-MLP outperforms MA-AC-MPC in sample efficiency during training. However, we tested two trained policies for both the Crazyflie thrust upgrade model cf2x_T350 and the brushless Crazyflie model in hardware experiments and noticed that MA-AC-MPC consistently landed on the X3 rover every single time with repeatable trajectories while MA-AC-MLP overshot the landing pad and only landed approximately 60%60\% of the time compared to the 100%100\% success rate of MA-AC-MPC. We tabulate results for various hardware trials for both models in Table 6. See Extension 4 and 5 for hardware videos of these trials. We also tested other trials and found consistent results with MA-AC-MPC landing every single time. We also show the trajectories of each trial for both MA-AC-MPC and MA-AC-MLP in Figure 8. This demonstrates that for this environment MA-AC-MPC is more robust and successful at transferring trained policies to hardware than MA-AC-MLP. Table 6: Landing performance comparison between MA-AC-MPC and MA-AC-MLP. MA-AC-MPC (Success Rate: 5/5, 100%, Mean Error: 0.055 m) Trial Result Δx x [m] Δy y [m] d [m] T1 Success 0.054 0.025 0.060 T2 Success 0.073 0.015 0.075 T3 Success 0.047 0.011 0.049 T4 Success 0.048 0.023 0.053 T5 Success 0.038 0.012 0.040 MA-AC-MLP (Success Rate: 3/5, 60%, Mean Error: 0.240 m) Trial Result Δx x [m] Δy y [m] d [m] T1 Failure 0.284 -0.507 0.581 T2 Success 0.021 -0.032 0.038 T3 Success 0.065 -0.036 0.074 T4 Success 0.038 -0.025 0.046 T5 Failure 0.280 -0.366 0.460 Figure 8: Three-dimensional landing trajectories for the two policies across five hardware trials, with MA-AC-MPC shown on top and MA-AC-MLP on the bottom. In each plot, the drone trajectory is drawn with solid lines and the rover trajectory with dashed lines, while endpoint markers indicate landing outcome. MA-AC-MPC exhibits consistently tight convergence to the rover landing pad across all trials, whereas MA-AC-MLP shows larger variability and two clear failure cases with substantial terminal error. Other runs not shown showed similar behavior for both policies. 6 Conclusions We propose a framework for combining MARL with model predictive control by extending an existing AC-MPC framework for MARL. Specifically, the MA-AC-MPC framework contains a shared parameter cost network used for the model predictive controller on each agent. In our results, we deploy the framework on a multi-agent pursuit-evasion problem and a drone landing on a rover scenario. We successfully demonstrate higher evader win rates using MA-AC-MPC compared to MA-AC-MLP by sweeping the mass at execution while only being trained on a smaller mass distribution. We also demonstrate improved sample efficiency in training and higher evader win rates during training for MA-AC-MPC while even compared to a larger actor network and critic for MA-AC-MLP. The drone landing on a rover scenario consistently demonstrated a success rate of 100%100\% for MA-AC-MPC in hardware compared to 60%60\% for MA-AC-MLP. ethical There are no human participants in this article and informed consent is not required. contrib Christian Llanes: Conceptualization, investigation, methodology, software, writing - original draft. Spencer W. Jensen: Funding acquisition, Supervision. Samuel Coogan: Funding acquisition, project administration, writing – review and editing acks This work received contributions from AI assisted tools for the development of code for simulation and hardware testing of both multi-agent environments using Claude Opus 4.5-4.6. Additionally, Claude Opus was used for generating Figure 2, code for visualization of data in Figure 4, and assisting in developing Table 1,2,4, 5, and 6 based on the code and tests from the simulation and hardware. Claude Opus was also used for generating equations 11-25 and sentences to describe the rover dynamics from the environment code. Gemini Pro and Flash 3.1 was used for assisting in grammar checks and improvement in sentence wording. Gemini Pro was also minimally used in pointing to literature articles which were verified by the corresponding author to be relevant to this article. funding This work is supported in part by the NASA University Leadership Initiative (ULI) under grant number 80NSSC20M0161. This paper describes objective technical results and analysis. Any subjective views or opinions that might be expressed in the paper do not necessarily represent the views of the U.S. Department of Energy or the United States Government. Sandia National Laboratories is a multimission laboratory managed and operated by National Technology & Engineering Solutions of Sandia, LLC, a wholly owned subsidiary of Honeywell International Inc., for the U.S. Department of Energy’s National Nuclear Security Administration under contract DE-NA0003525. SAND#0000-X dci The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article. sm The code used to run the simulation and hardware experiments for both multi-agent environments can be found publically available in the links below: https://github.com/llanesc/crazyflie-mape-crazyflow https://github.com/llanesc/crazyflie-rover-landing. References E. Adabag, M. Greiff, J. Subosits, and T. Lew (2025) Differentiable model predictive control on the gpu. External Links: 2510.06179, Link Cited by: §3.2. C. Amato (2024) An introduction to centralized training for decentralized execution in cooperative multi-agent reinforcement learning. External Links: 2409.03052, Link Cited by: §3.1. B. Amos, I. D. J. Rodriguez, J. Sacks, B. Boots, and J. Z. Kolter (2018) Differentiable mpc for end-to-end planning and control. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS’18, Red Hook, NY, USA, p. 8299–8310. Cited by: §2, §3.2. A. S. Anand, D. Reinhardt, S. Sawant, J. T. Gravdahl, and S. Gros (2023) A painless deterministic policy gradient method for learning-based mpc. In 2023 European Control Conference (ECC), Vol. , p. 1–7. External Links: Document Cited by: §2. O. Bounou, J. Ponce, and J. Carpentier (2023) Leveraging proximal optimization for differentiating optimal control solvers. In 2023 62nd IEEE Conference on Decision and Control (CDC), Vol. , p. 6313–6320. External Links: Document Cited by: §3.2. L. Busoniu, R. Babuska, and B. De Schutter (2008) A comprehensive survey of multiagent reinforcement learning. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews) 38 (2), p. 156–172. External Links: Document Cited by: §1. T. Dinev, C. Mastalli, V. Ivan, S. Tonneau, and S. Vijayakumar (2022) Differentiable optimal control via differential dynamic programming. External Links: 2209.01117, Link Cited by: §3.2. L. Fichtner, dirkpr, JasperHoffmann, F. Airaldi, J. Frey, J. K. Hromatko, K. Baumgaertner, M. Amria, RudolfReiter, and S. Sawant (2025) Leap-c/leap-c: v0.1.0-alpha External Links: Document, Link Cited by: item 3, §3.2. J. N. Foerster, Y. M. Assael, N. de Freitas, and S. Whiteson (2016) Learning to communicate with deep multi-agent reinforcement learning. In Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS’16, Red Hook, NY, USA, p. 2145–2153. External Links: ISBN 9781510838819 Cited by: §3.1. J. Frey, K. Baumgärtner, G. Frison, D. Reinhardt, J. Hoffmann, L. Fichtner, S. Gros, and M. Diehl (2025) Differentiable nonlinear model predictive control. External Links: 2505.01353, Link Cited by: §3.2. J. Frey, J. De Schutter, and M. Diehl (2023) Fast integrators with sensitivity propagation for use in CasADi. In ecc, Cited by: §3.2. A. Ghezzi, J. Hoffman, J. Frey, J. Boedecker, and M. Diehl (2023) Imitation learning from nonlinear mpc via the exact q-loss and its gauss-newton approximation. In 2023 62nd IEEE Conference on Decision and Control (CDC), Vol. , p. 4766–4771. External Links: Document Cited by: §2. S. Gronauer and K. Diepold (2022) Multi-agent deep reinforcement learning: a survey. Artificial Intelligence Review 55 (2), p. 895–943. External Links: ISSN 1573-7462, Document, Link Cited by: §3.1, §3.1. J. K. Gupta, M. Egorov, and M. Kochenderfer (2017) Cooperative multi-agent control using deep reinforcement learning. In Autonomous Agents and Multiagent Systems, G. Sukthankar and J. A. Rodriguez-Aguilar (Eds.), Cham, p. 66–83. External Links: ISBN 978-3-319-71682-4 Cited by: §3.1, §3.1. J. Hoffmann, D. F. Clausen, J. Brosseit, J. Bernhard, K. Esterle, M. Werling, M. Karg, and J. J. Bödecker (2024) PlanNetX: Learning an efficient neural network planner from MPC for longitudinal control. In Proceedings of the 6th Annual Learning for Dynamics & Control Conference, A. Abate, M. Cannon, K. Margellos, and A. Papachristodoulou (Eds.), Proceedings of Machine Learning Research, Vol. 242, p. 1214–1227. External Links: Link Cited by: §2. J. Jiang, K. Su, and Z. Lu (2024) Fully decentralized cooperative multi-agent reinforcement learning: a survey. External Links: 2401.04934, Link Cited by: §3.1, §3.1. W. Jin, Z. Wang, Z. Yang, and S. Mou (2021) Pontryagin differentiable programming: an end-to-end learning and control framework. External Links: 1912.12970, Link Cited by: §3.2. N. O. Lambert, D. S. Drew, J. Yaconelli, S. Levine, R. Calandra, and K. S. J. Pister (2019) Low-level control of a quadrotor with deep model-based reinforcement learning. IEEE Robotics and Automation Letters 4 (4), p. 4224–4230. External Links: Document Cited by: §2. C. Llanes, Z. Kakish, K. Williams, and S. Coogan (2024) CrazySim: a software-in-the-loop simulator for the crazyflie nano quadrotor. In 2024 IEEE International Conference on Robotics and Automation (ICRA), Vol. , p. 12248–12254. External Links: Document Cited by: §5.1.3. C. Llanes, K. A. Williams, S. W. Jensen, and S. Coogan (2026) Learning cooperative strategies for drone swarms using multi-agent reinforcement learning. In To appear in Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), Vienna, Austria. Cited by: §1, §5.1.1, §5.1.2. S. Mallick, F. Airaldi, A. Dabiri, and B. D. Schutter (2024) Multi-agent reinforcement learning via distributed MPC as a function approximator. Automatica 167, p. 111803. External Links: Document Cited by: §2. F. Moreno-Mora, L. Beckenbach, and S. Streif (2023) Predictive control with learning-based terminal costs using approximate value iteration. IFAC-PapersOnLine 56 (2), p. 3874–3879. Note: 22nd IFAC World Congress External Links: ISSN 2405-8963, Document, Link Cited by: §2. R. R. Negenborn, B. D. Schutter, and J. Hellendoorn (2009) Multi-agent model predictive control: a survey. External Links: 0908.1076, Link Cited by: §4.2. F. A. Oliehoek and C. Amato (2016) A concise introduction to decentralized pomdps. 1st edition, Springer Publishing Company, Incorporated. External Links: ISBN 3319289276 Cited by: §3.1. A. Oshin, H. Almubarak, and E. Theodorou (2024) Differentiable Robust Model Predictive Control. In Proceedings of Robotics: Science and Systems, Delft, Netherlands. External Links: Document Cited by: §2. J. A. Preiss*, W. Hönig*, G. S. Sukhatme, and N. Ayanian (2017) Crazyswarm: A large nano-quadcopter swarm. In IEEE International Conference on Robotics and Automation (ICRA), p. 3299–3304. Note: Software available at https://github.com/USC-ACTLab/crazyswarm External Links: Link, Document Cited by: §5.1.3. R. Reiter, A. Ghezzi, K. Baumgärtner, J. Hoffmann, R. D. McAllister, and M. Diehl (2025a) AC4MPC: actor-critic reinforcement learning for guiding model predictive control. IEEE Transactions on Control Systems Technology (), p. 1–16. External Links: Document Cited by: §2. R. Reiter, J. Hoffmann, D. Reinhardt, F. Messerer, K. Baumgärtner, S. Sawant, J. Boedecker, M. Diehl, and S. Gros (2025b) Synthesis of model predictive control and reinforcement learning: survey and classification. External Links: 2502.02133, Link Cited by: §2. A. Romero, E. Aljalbout, Y. Song, and D. Scaramuzza (2025) Actor-critic model predictive control: differentiable optimization meets reinforcement learning for agile flight. IEEE Transactions on Robotics. Cited by: item 3, §1, §2, §5.1.2. M. Samvelyan, T. Rashid, C. Schroeder de Witt, G. Farquhar, N. Nardelli, T. G. J. Rudner, C. Hung, P. H. S. Torr, J. Foerster, and S. Whiteson (2019) The starcraft multi-agent challenge. In Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems, AAMAS ’19, Richland, SC, p. 2186–2188. External Links: ISBN 9781450363099 Cited by: §1. P. P. Santos, D. S. Carvalho, M. Vasco, A. Sardinha, P. A. Santos, A. Paiva, and F. S. Melo (2025) Centralized training with hybrid execution in multi-agent reinforcement learning via predictive observation imputation. Artificial Intelligence 348, p. 104404. External Links: ISSN 0004-3702, Document, Link Cited by: §3.1. M. Schuck, M. P. Rath, Y. Hua, A. Goudar, S. Zhou, and A. P. Schoellig (2026) Crazyflow: an accurate, gpu-accelerated differentiable drone simulator in jax. Note: Preprint Cited by: §5.1.2. J. Schulman, P. Moritz, S. Levine, M. Jordan, and P. Abbeel (2018) High-dimensional continuous control using generalized advantage estimation. External Links: 1506.02438 Cited by: §4.1. F. Schulz, J. Hoffmann, Y. Zhang, and J. Boedecker (2024) Learning when to trust the expert for guided exploration in RL. In ICML 2024 Workshop: Foundations of Reinforcement Learning and Control – Connections and Perspectives, External Links: Link Cited by: §2. A. Serrano-Muñoz, D. Chrysostomou, S. Bøgh, and N. Arana-Arexolaleiba (2023) Skrl: modular and flexible library for reinforcement learning. Journal of Machine Learning Research 24 (254), p. 1–9. External Links: Link Cited by: §5. Y. Song and D. Scaramuzza (2022) Policy search for model predictive control with application to agile drone flight. IEEE Transactions on Robotics 38 (4), p. 2114–2130. External Links: Document Cited by: §2. S. Sukhbaatar, A. Szlam, and R. Fergus (2016) Learning multiagent communication with backpropagation. In Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS’16, Red Hook, NY, USA, p. 2252–2260. External Links: ISBN 9781510838819 Cited by: §3.1. R. Tao, S. Cheng, X. Wang, S. Wang, and N. Hovakimyan (2024) DiffTune-mpc: closed-loop learning for model predictive control. IEEE Robotics and Automation Letters 9 (8), p. 7294–7301. External Links: Document Cited by: §2. R. Verschueren, G. Frison, D. Kouzoupis, J. Frey, N. van Duijkeren, A. Zanelli, B. Novoselnik, T. Albin, R. Quirynen, and M. Diehl (2021) Acados – a modular open-source framework for fast embedded optimal control. Mathematical Programming Computation. Cited by: item 3, §3.2. C. Yu, A. Velu, E. Vinitsky, J. Gao, Y. Wang, A. Bayen, and Y. Wu (2022) The surprising effectiveness of ppo in cooperative multi-agent games. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS ’22, Red Hook, NY, USA. External Links: ISBN 9781713871088 Cited by: §4.1, §4.1. P. Zarchan (2019) Tactical and strategic missile guidance: an introduction. 7 edition, American Institute of Aeronautics and Astronautics. Cited by: §5.1.1. Appendix A: Index to multimedia extensions Table 7: Index of multimedia extensions. Extension Media type Description 1 Video Simulation of multi-agent pursuit-evasion environment using MA-AC-MPC for evaders. 2 Video Hardware demonstratin of multi-agent pursuit-evasion environment using MA-AC-MPC for evaders. 3 Video Drone and rover multi-agent environment simulation where both agents use MA-AC-MPC. 4 Video Hardware demonstration tests 1 to 6 for drone landing on rover environment using MA-AC-MLP. 5 Video Hardware demonstration tests 1 to 5 for drone landing on rover environment using MA-AC-MPC.