Paper deep dive
Sampling-Based Coordination-Informed Multi-Objective Multi-Robot Reinforcement Learning
Antonio Marino, Esteban Restrepo, Soon-jo Chung, Paolo Robuffo Giordano, Claudio Pacchierotti
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 96%
Last extracted: 7/5/2026, 1:32:36 AM
Summary
The paper introduces the Coordination-Informed Multi-Objective Multi-Robot Reinforcement Learning (CIMORL) framework, designed to optimize competing objectives in multi-robot systems while maintaining coordinated behavior. The framework addresses the limitations of fixed or centralized coordination by integrating a distributed weight prediction mechanism, a privileged expert training strategy (using MCTS and MPPI), and theoretical guarantees for Pareto-optimality. The authors prove that while global weight synchronization is necessary for team-wide Pareto optimality, local synchronization within coalitions is sufficient when agents specialize in disjoint subsets of objectives. The method is validated using Crazyflie drones in resource allocation and multi-attacker/multi-defender scenarios, demonstrating a 21.2% hypervolume improvement over baselines.
Entities (8)
Relation Signals (6)
CIMORL → includesvariant → CIMORL-TS
confidence 100% · We present the base CIMORL method alongside two sampling-based variants, CIMORL-TS (Tree Search) and CIMORL-MPPI (MPPI)
CIMORL → includesvariant → CIMORL-MPPI
confidence 100% · We present the base CIMORL method alongside two sampling-based variants, CIMORL-TS (Tree Search) and CIMORL-MPPI (MPPI)
CIMORL → uses → Tchebycheff Scalarization
confidence 100% · integrates the predicted weights into a Tchebycheff problem formulation
CIMORL → validatedon → Crazyflie
confidence 100% · Real-world experiments with Crazyflie drones further validate the framework’s robustness
CIMORL-MPPI → uses → Model Predictive Path Integral
confidence 90% · and Model Predictive Path Integral control (MPPI)
CIMORL-TS → uses → Monte Carlo Tree Search
confidence 90% · employing multi-objective extensions of Monte Carlo Tree Search (MCTS)
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Multi-robot systems must simultaneously optimize competing objectives while maintaining coordinated behavior. Existing multi-agent reinforcement learning approaches often rely on fixed or centralized coordination, which limits adaptability and violates distributed constraints. This work introduces the Coordination-Informed Multi-Objective Reinforcement Learning (CIMORL) framework, integrating a distributed weight prediction mechanism, a privileged expert training strategy, and theoretical guarantees for Pareto-optimal solutions. We present the base CIMORL method alongside two sampling-based variants, CIMORL-TS (Tree Search) and CIMORL-MPPI (MPPI), which leverage privileged global information during training to enable fully decentralized deployment. Experimental validation in cooperative and adversarial scenarios demonstrates a $21.2\%$ hypervolume improvement and superior policy stability compared to state-of-the-art baselines. Real-world experiments with Crazyflie drones further validate the framework's robustness in resource allocation and multi-attacker multi-defend scenarios under partial observability.
Tags
Links
- Source: https://arxiv.org/abs/2606.30893v1
- Canonical: https://arxiv.org/abs/2606.30893v1
Trouble viewing inline? Open PDF directly →
Full Text
125,531 characters extracted from source content.
Expand or collapse full text
Sampling-Based Coordination-Informed Multi-Objective Multi-Robot Reinforcement Learning Antonio Marino, Esteban Restrepo, Soon-Jo Chung, Paolo Robuffo Giordano, Claudio Pacchierotti A. Marino is with University of Cambridge — Cambridge, United Kingdom. E-mail: antonio.marino@cl.cam.ac.ukE. Restrepo, C. Pacchierotti and P. Robuffo Giordano are with CNRS, Univ Rennes, Inria, IRISA — Rennes, France. E-mail: esteban.restrepo, claudio.pacchierotti, prg@irisa.frSoon-Jo Chung is with the Division of Engineering and Applied Science, California Institute of Technology — Pasadena, CA 91125 USA. E-mail: sjchung@caltech.eduThis work was supported by the ANR-20-CHIA-0017 project “MULTISHARED” Abstract Multi-robot systems must simultaneously optimize competing objectives while maintaining coordinated behavior. Existing multi-agent reinforcement learning approaches often rely on fixed or centralized coordination, which limits adaptability and violates distributed constraints. This work introduces the Coordination-Informed Multi-Objective Reinforcement Learning (CIMORL) framework, integrating a distributed weight prediction mechanism, a privileged expert training strategy, and theoretical guarantees for Pareto-optimal solutions. We present the base CIMORL method alongside two sampling-based variants, CIMORL-TS (Tree Search) and CIMORL-MPPI (MPPI), which leverage privileged global information during training to enable fully decentralized deployment. Experimental validation in cooperative and adversarial scenarios demonstrates a 21.2%21.2\% hypervolume improvement and superior policy stability compared to state-of-the-art baselines. Real-world experiments with Crazyflie drones further validate the framework’s robustness in resource allocation and multi-attacker multi-defend scenarios under partial observability. I Introduction Multi-robot systems operating in complex environments must simultaneously optimize multiple competing objectives while maintaining coordinated team behavior. Contemporary robotic applications, from heterogeneous drone swarms in search and rescue operations to collaborative manipulation tasks, require balancing conflicting objectives including task completion time, energy consumption, communication bandwidth, and collision avoidance. Multi-objective multi-agent systems (MOMAS) provide realistic models that capture the complexity of inter-agent interactions and the multi-dimensional nature of their objectives [29]. While multi-agent reinforcement learning (MARL) has emerged as a powerful paradigm for robotic team coordination [16], its extension to multi-objective settings introduces significant theoretical and practical complexities. The fundamental challenge lies in determining appropriate objective weightings that guide robot teams toward globally optimal solutions while respecting distributed decision-making processes, limited communication bandwidth, and heterogeneous sensor capabilities. Existing multi-objective multi-agent approaches fail to tackle these challenges properly as they predominantly rely on fixed trade-off schemes or centralized coordination mechanisms [15]. These methods lack adaptability to dynamic environmental conditions, violate distributed execution requirements and suffer from poor scalability. To address these limitations, we introduce the Coordination-Informed Multi-Objective Reinforcement Learning (CIMORL) framework. Our approach bridges the gap between theoretical optimality and practical deployment constraints by integrating three key innovations: (i) a fully distributed weight prediction mechanism that dynamically adapts objective preferences relying solely on local observations and neighbor communication; (i) a privileged expert training strategy employing multi-objective extensions of Monte Carlo Tree Search (MCTS) [9] and Model Predictive Path Integral control (MPPI) [42]; and (i) theoretical guarantees for Pareto-optimal solutions through proper weight synchronization among robot subgroups. Our sampling-based variants, CIMORL-TS and CIMORL-MPPI, leverage privileged global information during training, yet execute in a fully decentralized manner under realistic partial observability and communication constraints at deployment. Ultimately, our framework provides a principled, scalable solution for deploying adaptive, multi-objective robotic teams in complex environments. I Related Works Multi-objective reinforcement learning (MORL) extends classical RL to optimize multiple, often conflicting, objectives concurrently [31, 39]. Traditional reinforcement learning seeks to maximize a single scalar reward, an approach that suffices when task priorities reduce cleanly to one metric. However, collapsing multiple criteria into a fixed linear scalarization risks to mask key trade-offs and producing biased or brittle behavior [18, 40]. MORL addresses this limitation by representing the reward as a vector, preserving feedback for each objective and shifting the learning target to a set of mutually non-dominated, Pareto-optimal policies. Existing single-agent MORL methods generally follow two paradigms. Outer-loop methods treat each preference weight as a separate task, optimizing them sequentially. While algorithms like MORL/D [14], PGMORL [44], and GPI-LS [2] improve sample efficiency by sharing information across nearby weights or prioritizing promising preferences, they often require extensive retraining for new targets. Inner-loop methods instead condition a single policy on the weight, amortizing learning across weightings. Approaches such as CAPQL [22] and MOPDERL [37] achieve markedly higher sample efficiency for continuous robotic control tasks. Our solution falls in this latter category, finding a continuous weight-dependent policy manifold. Despite this progress, MORL research has largely focused on isolated agents, overlooking the non-stationarity, partial observability, and credit assignment issues that arise when multiple robots interact [45]. The intersection of these fields defines multi-objective multi-agent reinforcement learning (MOMARL), where agents interact in a shared environment under vector-valued rewards and must reconcile individual preferences with joint dynamics. This challenge is compounded by the need for coordination under partial observability and non-stationarity, together with optimization across multiple conflicting objectives [29]. Early approaches to MOMARL often sidestepped its full complexity by scalarizing vector rewards with fixed weights and applying existing MARL algorithms [23]. While tractable, this produced only a single policy per preference and relied on fully decentralized learning, leading to instability in the face of non-stationarity. Recent works have begun to explore multi-policy learning in multi-agent settings. For discrete-action domains, MO-MIX [20] utilizes a centralized training with decentralized execution (CTDE) architecture and mixing logic to tackle credit assignment in cooperative tasks. For continuous-action frameworks, algorithms such as MO-AIM [13] and MOAVOA-MADDPG [1] extend learning to asynchronous agents; however, their outer-loop nature requires training separate policies for fixed preferences, lacking a unified preference-conditioned solution capable of covering the Pareto front in a single training run. Furthermore, while recent continuous-domain approaches like MOMA-AC [10] use actor-critic training for centralized coordination, they fail to synchronize preference weights, a mechanism we demonstrate in Section I is essential for solving global team multi-objective problems. Similarly, the contemporary MOMAPPO framework [15] adapts multi-agent PPO via weighted-sum scalarization and weight sampling strategies (e.g., OLS [32], GPI-LS [2]). However, these static sampling methods fail to account for dynamic environmental conditions that fundamentally shift optimal preference trade-offs. Addressing these limitations requires algorithms that provide support for continuous state and action spaces, decentralized execution of preference-conditioned policies, and centralized coordination mechanisms that adaptively reshape weights in response to evolving tasks. Consequently, we introduce a dynamic weight prediction methodology based on agent observations and inter-agent communication [6]. Inspired by graph clustering and opinion dynamics [3, 5, 47], we design a Graph Neural ODE framework [24] that distinctly models objective preferences as evolving consensus states for distributed weight prediction. Furthermore, we establish a novel theoretical guarantee, proving that cluster-based synchronization is sufficient and necessary condition for team-wide Pareto optimality when agents specialize in specific sub-objectives. Complementarily, we propose a training paradigm enhanced by sampling-based planning, which provides high-quality supervisory signals to accelerate convergence toward comprehensive Pareto approximations. We leverage privileged information during training via established search methodologies like MCTS [41, 19] and MPPI [42, 4]. To overcome the established limitations of linear scalarization, our final architectural contribution integrates the predicted weights into a Tchebycheff problem formulation. This strategic formulation effectively captures non-convex Pareto fronts while fully preserving the decentralized execution required for scalable multi-robot autonomy. Collectively, these components form a unified framework that adapts weights via distributed opinion dynamics, exploits search-based supervision, and preserves the decentralized execution required for scalable multi-robot autonomy. I Problem Statement TABLE I: Summary of Key Notation Symbol Description sV_s, NsN_s Team of cooperative agents and its total size s∈s , u∈u Joint environment state and joint control action z∈z Local partial observation vector rwirw_i Multi-objective reward vector for agent i iπJ_i^π, JπJ^π Expected return for agent i and global team objective under policy π VπV^π, VϕV_φ Value function and its neural network approximator w, w¯ w Distributed predicted weights and static reference weights |⋅||·|, ∥⋅∥\|·\| Absolute value (element-wise) and Euclidean norm (⋅)|(·)_| Column-wise matrix vectorization operator ⊗ Kronacker product SoxSo_x Softmax attention coefficient on the variable x In this section, we formulate the multi-objective problem within the Markov games framework, following established reinforcement learning (RL) literature. We consider a convex environment Q⊂ℝnQ ^n where a fully cooperative multi-agent team sV_s composed of NsN_s agents operates. The system state s∈s represents the agents’ joint states and evolves according to the dynamics s˙=Ω(s,u):×Ns→ s= (s,u):S×U^N_s , where u∈Nsu ^N_s denotes the joint action of all agents. Each agent has access to partial observations z∈z , defined by its sensing set Si=x∈Q∣d(x,vi)≤ϱS_i=\x∈ Q d(x,v_i)≤ \ with d the euclidean distance, i.e., the environment confined within a sphere of radius ϱ centered at that agent. We assume that cooperative agents within the sensing set are allowed to communicate therefore forming a communication graph G. The team is tasked with a multi-objective problem, where each agent receives an individual reward vector rwi(s,u):×→ℝKrw_i(s,u):S×U ^K, whose K entries quantify performance across different objectives. The problem can be formally modeled as a Decentralized Partially Observable Markov Decision Process (Dec-POMDP) [27], represented by the tuple ⟨,,Ω,rw0,…,rwNs,,γ⟩ ,U, ,\rw_0,…,rw_N_s\,Z,γ , where γ∈[0,1)γ∈[0,1) is the discount factor. Given a policy distribution πi(zi) _i(z_i) for each agent, the expected discounted return over a finite horizon T is defined as i=s0∼P0,s′∼P,u∼π[∑t=0Tγtrwi(st,ut)],J=∑i=1Nsi.J_i=E_s_0 P_0,\,s P,\,u π [ _t=0^Tγ^trw_i(s_t,u_t) ], J= _i=1^N_sJ_i. (1) The objective is to find the optimal agent policies πi(zi) _i(z_i) that maximize the collective team performance J: maxπ<J1π,…,JKπ>. _π <J_1^π,…,J_K^π>. (2) where each JiπJ^π_i is an element of the objective JπJ^π in (1). This type of problem is typically solved by scalarizing the multiple objectives and consequently casting it as a single-objective problem. To this end, we employ the augmented weighted Tchebycheff scalarization [8]. Unlike simple linear scalarization, which fails to find solutions in non-convex regions of the Pareto front, the Tchebycheff approach formulates a min-max problem that can discover the entire Pareto frontier by evaluating solutions based on their worst-performing weighted objective. Therefore, we can reformulate the problem as follows: maxπminiw¯iJiπ+ρ∑i=1KJiπ=maxπU(w¯,Jπ) _π _i w_iJ_i^π+ρ _i=1^KJ_i^π= _πU( w,J^π) (3) for some weights w¯∈Δ w∈ satisfying ∑i=1Kw¯i=1 _i=1^K w_i=1 and a small ρ>0ρ>0. We call U(⋅)U(·) the team utility. The augmentation parameter ρ plays a critical role in this formulation. Standard Tchebycheff scalarization (i.e., when ρ=0ρ=0) can sometimes yield weakly Pareto optimal solutions due to flat regions in its objective contours. Adding the linear sum term, weighted by a strictly positive ρ, creates a monotonic slope along these contours. This prevents the selection of weakly dominated points and mathematically guarantees strict Pareto optimality for the team. To characterize the solutions to problem (3), we first provide the following definitions: Definition I.1. (Dominated solution): Consider a multi-objective problem and two joint policy solutions π,π′π,π . A policy π is said to dominate a policy π′π if Jkπ≥Jkπ′J_k^π≥ J_k^π for all k∈1,…,Kk∈\1,…,K\, and Jjπ>Jjπ′J_j^π>J_j^π holds for at least one objective with j∈1,…,Kj∈\1,…,K\. Definition I.2. (Pareto-front): Given a multi-objective problem, the set of Pareto-optimal solutions—called Pareto-front—is the subset of all solutions that are not dominated by another solution. All optimal solutions to the Tchebycheff problem are Pareto-optimal and also Pareto-complete, meaning they fully capture the Pareto front even when it is non-convex [8]. In contrast, the weighted sum formulation captures only convex portions of the Pareto front; therefore, there might be Pareto-optimal solutions that are not solutions of the weighted sum problem. However, Tchebycheff problem is NP-hard, therefore intractable. On the other hand, the team weights w¯ w are also crucial for obtaining high-quality solutions, as they define the multi-objective trade-off, i.e., how the team evaluates the relative importance of each objective for the specific problem. As we are assuming a dynamic environment, these weights should adapt to the environment with the agent’s objectives developing in the environment. Formally, we can represent the team weights as a function of the joint observations of all agents, w¯(z1,…,zN) w(z_1,…,z_N). However, under partial observability, joint weights cannot be directly computed and must instead be estimated locally by each agent using its own observation and information exchanged through team communication. In a cooperative multi-agent setting, this motivates redefining the problem with distributed weights: maxπ _π mini∑j=1Nswjijiπi+ρ∑i=1K∑j=1Nsjiπi=maxπU(w,π), _i _j=1^N_sw_jiJ_ji _i+ρ _i=1^K _j=1^N_sJ_ji _i= _πU(w,J^π), (4) where jπJ^π_j is defined in (1), the weights satisfy ∑i=1Kwji=1 _i=1^Kw_ji=1 and w represents the stack of agents’ weights. The team maximizes the utility of the worst-performing weighted objective. This incentivizes a balanced performance across all goals, promoting altruistic behaviors where agents support a lagging objective for the overall benefit of the team. The key challenge that arises is: How should these weights be derived in a distributed manner? To achieve Pareto-optimal solutions, the team weights must satisfy specific constraints in a cooperative setting. Consider a case where each agent can achieve a combination of the objectives, and each agent j is assigned a weight vector wjw_j. Let j∗J^*_j denote the solution of agent j on its local Pareto front for the given Tchebycheff problem. For simplicity, consider two agents with respective optimal solutions 1∗,2∗J^*_1,J^*_2. In general, there does not exist a common weight vector w such that u([w1iw2i],[1i∗2i∗])=U(w,1i∗+2i∗)u ( bmatrixw_1i\\ w_2i bmatrix, bmatrixJ^*_1i\\ J^*_2i bmatrix )=U(w,J^*_1i+J^*_2i) Indeed, the left-hand side does not correspond to the original problem (3) and may over- or under-estimate points on the Pareto front, as illustrated in Figure 1. Therefore, Pareto-optimal solutions for the team can only be achieved when the weights are shared across agents. For analysis purposes, let’s assume the theoretical lower bound of ijJ_ij is ij=0J_ij=0. We can formally derive the following Lemma: Lemma 1. When the agents can achieve any combination of the K objectives, i.e., they can attain any objective vector with strictly non-zero elements, Pareto-optimal solutions for the team sV_s can be obtained through problem (4) if and only if the agents’ weight vectors w are synchronized across the team. Proof. Let ji∗J^*_ji denote the optimal return for objective i∈1,…,Ki∈\1,…,K\ achieved by agent j∈1,…,Nsj∈\1,…,N_s\. The centralized team problem under the augmented Tchebycheff scalarization minimizes the worst-performing weighted objective over the sum of team returns. To achieve the exact same Pareto-optimal solutions for any arbitrary environment, the decentralized aggregation must perfectly recover this centralized formulation: mini∑j=1Nswjiji∗=miniwi∑j=1Nsji∗. _i _j=1^N_sw_jiJ^*_ji= _iw_i _j=1^N_sJ^*_ji. (5) If the agent weights are synchronized such that wji=wiw_ji=w_i for all j, the weight wiw_i becomes a constant with respect to the summation over the agents. It can be factored out, yielding mini(wi∑j=1Nsji∗) _i(w_i _j=1^N_sJ^*_ji). This exactly matches the centralized team objective, proving the synchronization to be a sufficient condition. Moreover, we can show that if the equality in (5) holds over the entire space of achievable returns, then wji=wiw_ji=w_i must be true. Because we assume the agents can achieve any combination of the K objectives, the variables ji∗J^*_ji are independent and can take arbitrary non-zero values. For the outputs of the two min operators to be identical across this entire independently spanning continuous domain, their respective inner arguments must be algebraically equivalent for every i. Therefore, we can focus on the linear combinations equality: ∑j=1Nswjiji∗=∑j=1Nswiji∗∀i. _j=1^N_sw_jiJ^*_ji= _j=1^N_sw_iJ^*_ji ∀ i. Rearranging the terms yields: ∑j=1Ns(wji−wi)ji∗=0∀i. _j=1^N_s(w_ji-w_i)J^*_ji=0 ∀ i. Since this equation must hold for any arbitrary combination of returns ji∗J^*_ji, the coefficients of these independent variables must be strictly zero. This requires wji−wi=0w_ji-w_i=0, which implies wji=wiw_ji=w_i for all agents j and objectives i. This proves the statement of the lemma. ∎ In scenarios where individual agents can only contribute to a specific subset of the K objectives (e.g., due to orthogonal task assignments or spatial constraints), the strict necessity of global weight synchronization established in Lemma 1 is relaxed. Instead, agents naturally form coalitions to satisfy different objective subsets. Formally, a coalition partition =C1,…,CNcC=\C_1,…,C_N_c\ divides the team NsN_s into non-empty, disjoint subsets (⋃q=1NcCq=Ns _q=1^N_cC_q=N_s and Cp∩Cq=∅C_p∩ C_q= for p≠qp≠ q) such that agents within the same coalition CqC_q tackle the same subset of objectives. Therefore, the global team problem can be treated as a set of independent sub-teams, leading to the following relaxation: Lemma 2. Assume the team NsN_s is partitioned into a set of coalitions C, where each coalition strictly influences a disjoint subset of the K objectives. Pareto-optimal solutions for the global team can be achieved if and only if agents synchronize their weight vectors locally within their respective coalitions. Consequently, different coalitions do not need to share the same weights, allowing the team’s weight vectors to diverge into distinct clusters. Proof. Let =C1,…,CNcC=\C_1,…,C_N_c\ be the coalition partition. Let ji∗J^*_ji be the optimal return for objective i∈1,…,Ki∈\1,…,K\ achieved by agent j∈1,…,Nsj∈\1,…,N_s\. Because the subset of objectives influenced by each coalition is disjoint (orthogonal), for any given objective i, there is exactly one corresponding coalition, denoted C(i)C_(i), capable of achieving it. For all agents outside this coalition (j∉C(i)j∉ C_(i)), the return is strictly zero (ji∗=0J^*_ji=0). Therefore, to perfectly recover the centralized Tchebycheff objective for the global team, we can restrict the same argument of Lemma 1 to each individual coalition: ∑j∈C(i)wjiji∗=∑j∈C(i)wiji∗∀i. _j∈ C_(i)w_jiJ^*_ji= _j∈ C_(i)w_iJ^*_ji ∀ i. Rearranging the terms yields: ∑j∈C(i)(wji−wi)ji∗=0∀i. _j∈ C_(i)(w_ji-w_i)J^*_ji=0 ∀ i. Because the agents within C(i)C_(i) can achieve arbitrary combinations of their specific active objectives, the variables ji∗J^*_ji are non-zero and independent. This requires (wji−wi)=0(w_ji-w_i)=0, strictly enforcing wji=wiw_ji=w_i for all agents j∈C(i)j∈ C_(i). Crucially, for any agent j′∉C(i)j ∉ C_(i), their weight wj′iw_j i is multiplied by zero. Therefore, its value has no impact on the algebraic equivalence of the team objective. This proves that weight synchronization is a necessary and sufficient condition only within the active coalition C(i)C_(i) for any given objective. Iterating this reasoning across all NcN_c coalitions and K objectives demonstrates that agents only need to synchronize weights locally within their coalition, proving the lemma. ∎ The lemma 2 reveals in a multi-agent team s=1,2,…,NsV_s=\1,2,…,N_s\ operating under partial observability constraints, where agents may form coalitions =C1,…,CNcC=\C_1,…,C_N_c\ to optimize different subsets of K objectives, the weights w have to cluster into NcN_c equal vectors corresponding to the NcN_c robot multi-objective coalitions. Therefore, in this paper we seek to: 1. Learn a distributed weight prediction model ψ:→ΔW_ψ:Z→ that maps agent observations to weight distributions over objectives, ensuring intra-coalition weight synchronization while allowing inter-coalition weight diversity. 2. Develop a reinforcement learning framework that approximates the NP-hard optimization problem (4) through sampling-based policy search, enabling uniform exploration of the Pareto front via the weight distribution W. 3. Ensure the learned policies satisfy the coalition constraints derived in Lemma 2, where agents within each coalition Cj∈C_j maintain synchronized weights wCjw_C_j while different coalitions may adopt distinct weight vectors. The solution we propose (CIMORL) addresses all these challenges and operates under partial observability constraints, enabling distributed execution. Figure 1: Pareto Front representations: Pareto Front representation for (a) two-objective and two-agent problem and (b) three-objective and a generic team. IV Multi-Objective Weight Prediction In this section, we develop a distributed dynamical system for predicting agent weight distributions that enables effective multi-objective coordination. The proposed weight prediction framework addresses two fundamental requirements: (i) promoting local consensus among neighboring agents with similar observations to form cohesive coalitions, and (i) allowing divergence between distinct groups to ensure comprehensive coverage of all objectives across the team. This design directly satisfies the synchronization conditions established in Lemma 2, where agents within coalitions must coordinate their weights to achieve Pareto-optimal solutions. When weights naturally cluster into subgroups corresponding to different objectives, the team achieves improved objective coverage while maintaining the distributed structure required by problem (4). First, we report the proposed opinion dynamics to predict the weights, a core component of our CIMORL approach. IV-A Weight Opinion Dynamics For simplicity, we present the model using the stack observation vector z, defined as the concatenation of individual agent observations in a shared feature space. However, all computations are implemented in a fully decentralized manner, with agents relying only on local observations and neighbor communication. To predict the weights, each agent first processes its local observation through a shared nonlinear mapping g: Ξ=g(z)∈ℝNs×K, =g(z) ^N_s× K, (6) where Ξ is used as an observation-dependent preference embedding that simultaneously governs agent clustering, convergence rates, and the final weight distribution. Rather than directly predicting the final weights, Ξ serves as a continuous input that influences how the weight distribution evolves over time based on the current environmental observations. The function g(⋅)g(·) learns to map observations to objective-specific driving signals, enabling the dynamical system to adapt the weight preferences as the environment changes. To ensure numerical stability and bounded dynamics, the output Ξ is normalized using either min-max normalization (as employed in our experiments) or a softmax transformation So(⋅)=Softmax(⋅)So(·)=Softmax(·). The weight evolution is modeled as an opinion formation process where the agents iteratively refine their weight estimates through neighbor interactions. Two agents influence each other if their Ξ are sufficiently similar, formally defined by the condition dist(Ξi,Ξj)<ϵdist( _i, _j)<ε for some distance metric dist(⋅,⋅)dist(·,·) and threshold ϵ>0ε>0. This similarity-based interaction mechanism naturally facilitates coalition formation among agents with compatible objective preferences. To implement this similarity criterion, we define a continuous similarity matrix that approaches unity for similar weight estimates and zero otherwise: σΞ=σ(WΞ(ΞΞ⊤‖Ξ‖2−bΞ)) _ =σ (W_ ( || ||^2-b_ ) ) where WΞW_ and bΞb_ are trainable parameters that define the similarity threshold and sensitivity. The bias term bΞb_ establishes a configurable basin of attraction between agents, allowing adaptation to different multi-objective task requirements. The sigmoid function σ ensures smooth transitions between independent signal and coalition attraction regimes. Let S∈ℝNs×NsS ^N_s× N_s represent the adjacency matrix of the communication graph, encoding which agents can directly exchange information. The effective similarity matrix is then constrained by the communication topology: SoΞ=S∘So(WσσΞ)So_ =S (W_σ _ ) where ∘ denotes the Hadamard product and WσW_σ provides additional learnable scaling. This formulation ensures that opinion exchange occurs only between agents within communication range, maintaining the distributed nature of the system. The complete weight dynamics are characterized by a two-state system comprising a primary weight state x2∈ℝNs×Kx_2 ^N_s× K and an auxiliary coordination state x1∈ℝNs×Kx_1 ^N_s× K. This second-order dynamics is strictly required to achieve exact clustering agreements; specifically, x1x_1 acts as an integrator that accumulates disagreement to compensate for the differences between dynamically changing neighboring signals, ensuring x2x_2 converges to a stable consensus. The final weight distribution for each agent follows a Dirichlet distribution with concentration parameters determined by the softmax transformation of x2x_2. Inspired by the Liquid Graph Time-Constant (LGTC) framework [24], we define the coupled opinion dynamics as: x˙1=−(IN−SoΞ)x2WBx˙2=−(τ+Ξ)∘x2−(IN−Sox)x2WA+(IN−SoΞ)x1WB+ΞBpw=So(x2). cases aligned x_1=&-(I_N-So_ )x_2W_B\\ x_2=&- (τ+ ) x_2-(I_N-So_x)x_2W_A\\ &+(I_N-So_ )x_1W_B+ B\\ p_w=&So(x_2) aligned cases. (7) Here, WBW_B and WAW_A are learnable weight matrices that act uniquely as coupling gains between agents to govern the internal state coordination and the network consensus dynamics, respectively. We emphasize that these matrices do not encode any inter-objective trade-offs; their sole purpose is to facilitate network synchronization. To ensure positive definiteness and stability, we impose W=W~⊤W~>0W= W W>0 for the system matrices. The bias parameters τ, bΞb_ , and B are implemented as 1N⊗b1_N b to ensure equal biases across all agents while maintaining learnable adaptation capabilities. The system also incorporates an attention mechanism for x2x_2 communication, defined as: Sox=S∘So(Wx(x2x2⊤‖x2‖2))So_x=S (W_x ( x_2x_2 ||x_2||^2 )) This attention matrix ensures subgroup constant row sum [43], which is essential for proving convergence. This is because the softmax, applied over communicating agents with similar states, effectively selects the agents with similar states leading to potential coalition, which is a key condition for stable clustering [43]. In this system, Ξ appears in multiple roles: it determines the similarity-based coupling structure through SoΞSo_ , provides damping in the x2x_2 dynamics through the (τ+Ξ)∘x2(τ+ ) x_2 term, and drives the system evolution via the ΞB B input term. This multivalent role allows the observation-dependent signal Ξ to simultaneously influence agent clustering, convergence rates, and the target weight distribution. Before stating the main theorem of this section, we provide the following definition of clustered trajectories: Definition IV.1. Given N agent state trajectories x¯c=[x1c,…,xNc] x^c=[x^c_1,…,x^c_N], agents are grouped into clusters if they follow the same time-evolving state. Formally, agents i and j belong to the same cluster CmC_m if and only if xic=xjc,∀i,j∈Cm,m∈1,…,M.x^c_i=x^c_j, ∀ i,j∈ C_m, m∈\1,…,M\. That is, the cluster set is defined as Cm=i,j∈|xic=xjc.C_m= \i,j \; |\;x^c_i=x^c_j \. The clustered team state can then be represented as x¯c=[xc1⊗C1,…,xcM⊗CM]⊤ x^c=[x_c_1 1_C_1,…,x_c_M 1_C_M] where xcmx_c_m is the representative state trajectory for cluster CmC_m. Considering the communication graph G, the following result is derived by making the following assumption on the agent connectivity: Assumption 1. We assume the agent communication graph G is connected throughout the task episode. We are now ready to state the theorem about the convergence properties of the system in (7). Theorem 1. The distributed system in (7) clusters in different stable equilibria x¯c x^c given by Definition IV.1 and dictated by the communication graph topology and observed environment. We reported the proof in the Appendix IX-A. Figure 2 illustrates an example of 20 agents weight concentration parameter trajectories for three objectives randomly initialized (a) and the resulting distributions. The trajectories follow agents’ periodic preference embeddings with a different phase per agent. The clustering trajectories are highlighted in different colors and result very stable and distinct. However the trajectories get noisier when the clusters are close as expected since SoΞSo_ and SoxSo_x will have similar value across the agents. We note that even if the assumption 1 is not satisfied, the convergence is guaranteed for the connected subgroups. While Theorem 1 guarantees that agents within a connected subgroup will reach a stable consensus on their concentration parameters pwp_w, the dynamical system relies on learnable components, such as the observation mapping g(⋅)g(·), to dictate where these equilibria form. For the team to effectively solve the underlying task, these parameters must be optimized so that the resulting clustered states translate into policies that adequately cover the objective space. Consequently, we must map the converged states to actionable agent weights and establish a training objective that guides the system toward optimal multi-objective performance. This is discussed in the next section. IV-B Weight Distribution Training The concentration parameters pwp_w define a Dirichlet distribution i(w|pw)W_i(w|p_w) from which agent weights are sampled during training and execution. To train the weight distribution W effectively, we develop a regret-based loss function that encourages sampling of high-quality weights while ensuring uniform exploration of the Pareto front [7]. During training, we estimate the Pareto front ℱPF achieved by the current joint policy π(z,w)π(z,w), which depends on the weight samples utilized during the rollouts. The set ℱPF is comprised of the optimal value functions Vπ,wV^π,w obtained throughout the training process. In the following, we will use VπV^π, to indicate a value function given w and the policy π. We reformulate the Tchebycheff problem as a minimization to accommodate mixed positive-negative reward scenarios: ℒw(V)=∑j=1Nsmaxiwji|Vjiπ−vi∗||vi∗|L_w(V)= _j=1^N_s _iw_ji |V^π_ji-v^*_i||v^*_i| (8) where vi∗v^*_i represents the ideal (utopian) value for objective i. Following common practice in empirical multi-objective reinforcement learning [31], this reference point is not fixed but continuously estimated online during training. Specifically, vi∗v^*_i is defined as the maximum return achieved for each objective across the historical estimated Pareto front ℱPF. To prevent optimization distortion from noisy value estimates early in training, we apply a monotonic update rule after each rollout iteration: vi∗←max(vi∗,maxV∈ℱVji)v^*_i← (v^*_i, _V V_ji ), ensuring the reference point only shifts when genuinely superior policies are discovered. A small constant ϵ=10−6ε=10^-6 is added to the denominator to prevent division by zero. Let w∗w^* denote a set of uniformly sampled weights available prior to training. We associate each weight w∗w^* with a value V∗∈ℱV^* as the argument that minimizes minV∈ℱℒw(V) _V L_w(V). In this manner, we define the optimal weight w∗w^* for the achievable value VπV^π. Let Vπ(z,w)V^π(z,w) be the value estimated through the rollout of the current policy for observation z. We project Vπ(z,w)V^π(z,w) onto the Pareto front to obtain V∗π=argminV∈ℱ‖V−Vπ(z,w)‖V^*π= _V ||V-V^π(z,w)||. The regret incurred by using a sampled weight w^∼(⋅|pw) w (·|p_w) instead of the optimal weight w∗w^* for the associated optimal value V∗πV^*π is defined as: r(w^|w∗)=minw∗[ℒw∗(V∗π)−ℒw^(V∗π)].r( w|w^*)= _w^* [L_w^*(V^*π)-L_ w(V^*π) ]. This choice encourages the distribution W to cover the entire Pareto front, as a sampled weight is only penalized for its distance to its nearest optimal counterpart, not to a single, arbitrarily chosen one. We therefore minimize the expected regret over the weight distribution W to encourage sampling of weights that yield low regret relative to the optimal Pareto front. Furthermore, since the weight distribution also influences the policy, we seek to maximize the expected reward obtained by the sampled weights. Given the rollouts dataset D and the agent advantage estimation per objective AijπiA _i_ij [34], we minimize the following loss according to objective (4): jmax j_max =argmaxj∑i=1Nswij|Vij(z,w^i)−v∗|v∗ = *arg\,max_j _i=1^N_sw_ij |V_ij(z, w_i)-v^*|v^* ℒw _w =(a,z,w^)∼[log(w^|z)⋅r(w^|w∗)] =E_(a,z, w) [ ( w|z)· r( w|w^*) ] −(u,z)∼[log(w^|pw)⋅∑i=1NsAijmaxπi] -E_(u,z) [ ( w|p_w)· _i=1^N_sA _i_ij_max ] The preceding loss function employs the log-likelihood trick for computational convenience. Note that this loss formulation is compatible with the clipped update mechanism utilized in PPO [35]. This loss function balances two goals: minimizing regret to ensure comprehensive exploration of the Pareto front, and maximizing the policy’s advantage to exploit high-performing weight regions. Remark 1. The weight distribution loss can be extended to incorporate human preferences for specific objectives. This can be achieved by adding a cross-entropy term that encourages W to align with desired objective priorities, enabling human-guided multi-objective optimization. (a) Concentration parameters trajectories (b) Final Dirichlet distributions Figure 2: Weight clustering trajectories: (a) 20 agents concentration parameters trajectories for 33 objectives starting from random state and clustering in three subgroups while subjected to a periodic preference embedding with a different phase for each agent. (b) Dirichlet distribution for three objectives at the end of clustering trajectories. V Policy Training In this section, we describe our approach for training agent policies using local observations and predicted weights to address the multi-objective optimization problem in (4). We introduce two sets of parameters: θ encoding the policy approximator πθ _θ, and ϕφ encoding the value function approximator VϕV_φ. The base CIMORL method trains these networks directly using proximal policy optimization. However, the training of the weight distribution depends critically on both the exploration of Pareto-optimal solutions and the quality of the learned policy. To enhance policy optimality and create our advanced variants, CIMORL-TS and CIMORL-MPPI, we employ a privileged expert framework [11] during training. Our approach constructs a high-quality dataset using an expert sampling methodology that generates approximately optimal policy distributions for collection in an expert dataset. The privileged expert identifies optimal policies for the entire team by leveraging full system dynamics and environmental state information over a finite planning horizon. We implement two complementary sample-based expert algorithms to drive these variants: Multi-Objective Neural Tree Search (MONTS) to generate the CIMORL-TS policy, and Multi-Objective Model Predictive Path Integral (MOMPPI) control to generate the CIMORL-MPPI policy. MONTS extends Neural Tree Expansion (NTE) search [30] by incorporating predicted weights to scalarize the multi-objective value function. For a given node n in the search tree with accumulated team value V(n)V(n), we define the scalarized node value as: Vmax(n)←maxi∑j=1Nswij|Vij(n)−vi∗|vi∗V_max(n)← _i _j=1^N_sw_ij |V_ij(n)-v_i^*|v_i^* (9) Here, Vmax(n)V_max(n) represents the worst-case weighted deviation from the ideal performance across all objectives, acting as the scalarized heuristic value for node n. The term Vij(n)V_ij(n) denotes the accumulated value for agent j on objective i at that specific node, while wijw_ij is the corresponding predicted weight. The parameter vi∗v_i^* represents the ideal, utopian reference value (estimated during training), which normalizes the deviations. By minimizing this maximum normalized deviation, the formulation strictly aligns with the Tchebycheff scalarization defined in (4). This design preserves standard tree search exploration rules while enabling branches to evaluate gains across all objectives simultaneously. Algorithm 1 details this implementation, which crucially returns a new optimal Gaussian action distribution over the root’s child nodes rather than selecting a single best action. In contrast, MOMPPI extends Model Predictive Path Integral (MPPI) control [42] by integrating the learned neural policy for action sampling during rollout simulations. Given the team’s accumulated rewards V across T parallel rollout trajectories, we apply the vectorized Tchebycheff utility function: Vmax←maxi∑j=1Nswij|Vij−vi∗|vi∗∈ℝTV_max← _i _j=1^N_sw_ij |V_ij-v_i^*|v_i^* ^T (10) Importance weights are computed using a softmax transformation with temperature λ: IS←exp(−Vmax/λ)∈ℝTIS← (-V_max/λ) ^T. Unlike traditional MPPI that iteratively updates noisy reference actions, our approach leverages the pre-trained policy πθ _θ to directly yield near-optimal action distributions, eliminating the need for iterative refinement. Both algorithms operate as privileged experts during training by accessing global states and utilizing substantial, identical computational resources to perform deep searches, ensuring a fair algorithmic baseline free from hardware limitations. At runtime, agents execute a localized, computationally-bounded version of these strategies using only partial observations. We introduce both methods because their fundamental mechanisms present complementary trade-offs. MONTS prioritizes the deep expansion of the most promising search paths through selective node evaluation. Consequently, MONTS can achieve faster theoretical convergence, with approximation errors decreasing as O(1/T1−c3/c2)O(1/T^1-c_3/c_2) [36] compared to MOMPPI’s O(1/T)O(1/ T) [42], but incurs higher sequential computation costs. Conversely, MOMPPI performs broad, uniform exploration across the action space, trading sample efficiency for massive parallelization that scales exceptionally well on modern GPU hardware. As illustrated in Figure 3, the optimal choice between these experts depends entirely on the environment’s complexity and the target system’s parallelization capabilities, making a simple tabular comparison inadequate. Given a weight sample w∼ψ(z)w _ψ(z) from the agent’s predicted distribution, we generate multiple trajectory rollouts by sampling actions from the policy πθ _θ. Using these trajectories, we compute the optimal policy distribution through one of our sampling search (S) methods: π∗←S(z,πθ,Vϕ,w,T,D)π^* (z, _θ,V_φ,w,T,D) (11) where T denotes the number of rollout samples and D represents the planning horizon depth. The complete dataset D stores the expert-derived optimal policy distribution π∗π^*, the current joint policy πθ _θ, rollout observations z, joint actions u, value function estimates VϕV_φ, and the corresponding weights w. At the k-th training iteration, we assume Gaussian policy distributions where parameters θ encode both the mean μθ _θ and standard deviation σθ _θ. The policy training for agent i minimizes the following supervised learning objective: Jπi∗(θ)=(u,z)∼[ J_π^*_i(θ)=E_(u,z) [ ||μi∗−μiθ(z)||2+||σi∗−σiθ(z)||2] ||μ^*_i- _i_θ(z)||^2+||σ^*_i- _i_θ(z)||^2 ] −αℋℋ(πiθ) - _HH( _iθ) where ℋ(πiθ)H( _iθ) represents the policy entropy and αℋ _H is a temperature parameter controlling policy stochasticity to maintain adequate action space exploration. To accelerate initial convergence, we incorporate an auxiliary loss term based on the scalarized advantage function: jmax j_max =argmaxj∑i=1Nswij|Vij(z,wi,ϕ)−v∗|v∗ = *arg\,max_j _i=1^N_sw_ij |V_ij(z,w_i,φ)-v^*|v^* Jπi(θ) J_ _i(θ) =−(u,z)∼[πi(zi,θ)⋅Aijmaxπi(⋅,θk)] =-E_(u,z) [ _i(z_i,θ)· A_ij_max _i(·, _k) ] The final loss for policy training combines both objectives: Jπi(θ)=(1−α∗)Jπi∗(θ)+α∗Jπi(θ)J_ _i(θ)=(1-α^*)J_π^*_i(θ)+α^*J_ _i(θ) (12) with α∗α^* decaying from 1 to 0 over the training process. This supervised expert loss provides a strong implicit regularization effect. The learned policy is trained to predict the expert coordination strategies at deployment while relying strictly on local, partial observations without requiring the expert’s heavy computational overhead. The value function network Vi(z,wi,ϕ)V_i(z,w_i,φ), which outputs the vector of objective values, is trained by minimizing the squared ℒ2L_2 norm of the temporal difference error across all objectives: JVi(ϕ)= J_V_i(φ)= (u,z)∼[‖(Vi(z,wi,ϕ)−(Aiπi(ui,θk)+Vi(z,wi,ϕk)))‖2] _(u,z) [ \| (V_i(z,w_i,φ)- (A_i _i(u_i, _k)+V_i(z,w_i, _k) ) ) \|^2 ] Data: zs,θ,ϕ,wz_s,θ,φ,w, T samples, D horizon Result: Find the initializer. 1 create root node n0n_0 with state zsz_s 2 store the initial value prediction v0←Vϕ(zs)v_0← V_φ(z_s) 3 ni←n0n_i← n_0 4 for t=1…Tt=1… T do 5 rw←rw← [ ] 6 for i=1…Di=1… D do 7 ni,rwi←TreePolicy(ni)n_i,rw_i← TreePolicy(n_i) 8 stores rwirw_i in the list rwrw 9 end for 10 append the terminal state value Vϕ(zs(ni))V_φ(z_s(n_i)) to rwrw 11 Backup (nin_i, rwrw) 12 13 end for 14return π∗←GetDistribution(n0)π^*← GetDistribution(n_0) 15 16function TreePolicy(n) 17 if n not fully expanded then 18 return Expand(n) Expand(n) 19 20else 21 n←argmaxn′∈Child(n)[Vmax(n′)N(n′)+c1N(n)c3N(n′)c2] n← _n (n) [ V_max(n )N(n )+c_1 N(n)^c_3N(n )^c_2 ] 22 return Expand (n) 23 24 end if 25 26function Expand(n,w) 27 get observation vector zn←nz_n← n 28 sample an action u∼πθ(zn,w)u _θ(z_n,w) 29 create a new child node n′←f(zn′|zn,u)n ← f(z_n |z_n,u) 30 compute the reward rw(zn′)rw(z_n ) 31 return n′n , rw(zn′)rw(z_n ) 32 33function Backup(n,rwn,rw) 34 while n≠nulln do 35 d←Depth(n)d← Depth(n) 36 for i=d…Di=d… D do 37 V(n)←V(n)+γdrw[d]V(n)← V(n)+γ^drw[d] 38 39 end for 40 N(n)←N(n)+1N(n)← N(n)+1 41 n←Parent(n)n← Parent(n) 42 43 end while 44 45function GetDistribution(n) 46 47 48N←∑c∈Child(n)N(c)N← _c∈ Child(n)N(c) 49 μ←1N∑c∈Child(n)N(c)⋅W(c)μ← 1N _c∈ Child(n)N(c)· W(c) 50 σ2←1N∑c∈Child(n)N(c)⋅(W(c)−μ)2σ^2← 1N _c∈ Child(n)N(c)·(W(c)-μ)^2 51 return (μ,σ2)(μ, σ^2) Algorithm 1 MONTS Data: zs,θ,ϕ,wz_s,θ,φ,w, T samples, D horizon Result: Optimal policy ϕ∗φ^* 1 Sample T control actions u0∼πθ(zs,w)u_0 _θ(z_s,w) 2 Reconstruct the initial state s0s_0 from zsz_s 3 Initialize cumulative reward V=0V=0 4 for i=1i=1 to D do 5 Get new state from environment simulation and reward si,rwi←f(si−1,ui−1)s_i,rw_i← f(s_i-1,u_i-1) 6 Get new observation zsi←h(si)z_s_i← h(s_i) 7 Accumulate reward V←V+γirwiV← V+γ^irw_i 8 Get action ui∼πθ(zsi,w)u_i _θ(z_s_i,w) 9 10 end for 11Add terminal reward V←V+γdVϕ(zsD)V← V+γ^dV_φ(z_s_D) 12 Compute MPPI importance weights IS←exp(−Vmax/λ)∈ℝTIS← (-V_max/λ) ^T 13 return π∗←GetDistribution(u0,IS)π^*← GetDistribution(u_0,IS) 14 function GetDistribution(u0u_0, ISIS) 15 W←∑i=1TISiW← _i=1^TIS_i 16 μ←1W∑i=1TISi⋅u0,iμ← 1W _i=1^TIS_i· u_0,i 17 σ2←1W∑i=1TISi⋅(u0,i−μ)2σ^2← 1W _i=1^TIS_i·(u_0,i-μ)^2 18 return (μ,σ2)(μ, σ^2) Algorithm 2 MOMPPI Figure 3: Comparison of MPPI and NTS: on the left, the selection strategy of the Neural Tree Search (NTS); on the right, the Monte-Carlo sampling of Model Predictive Path Integral control. VI Closed-Loop Stability In this section, we analyze the closed-loop stability of the overall system when the multi-agent team employs the learned policy πθ(z,w) _θ(z,w). Since the observations are part of the environment state and the team state, we can focus only on the overall team dynamics, i.e. where the agent interactions happen. We will make two further assumptions to provide closed-loop stability insights. Assumption 2. Given the joint team observation z, The joint policy is described by πθ(z,w)=μθ(z,w)+σθ(z,w)ϵ _θ(z,w)= _θ(z,w)+ _θ(z,w)ε with ϵ∼(0,I)ε (0,I). Moreover, supz,w‖σθ‖≤σg _z,w|| _θ||≤ _g. Assumption 3. We assume that the robots’ observation transition dynamics are deterministic, which is a common simplification for tractable stability analysis, and thus attribute all stochasticity to the control policy. The system dynamics under control π can therefore be described by the ito^it o stochastic differential equation: =f(z,t)+μθ(z,w)+σθ(z,w)ϵdz=f(z,t)dt+ _θ(z,w)dt+ _θ(z,w)ε (13) Note that this system model also captures the state transition experienced in the sampling search. Since the weights w serve as objective selectors that focus the observation vector on the objectives while preserving the underlying processing functions, we can analyze the closed-loop stability by considering each weight-induced subsystem individually. This approach leverages contraction theory [21] for switching systems [33], where each ’subsystem’ corresponds to the agent dynamics (13) under a fixed weight vector w. The overall system’s stability is then analyzed as these weights evolve over time. For the stability analysis, we require a contraction matrix M(z,t)≻0M(z,t) 0 that is Lipschitz continuous with constant LML_M and satisfies m¯I⪯M⪯m¯I mI M mI for positive scalars m¯ m and m¯ m. The matrix M(z,t)M(z,t) defines a state-dependent Riemannian metric. Unlike standard Lyapunov theory which measures the energy or distance of a single trajectory to a fixed equilibrium, contraction analysis utilizes M(z,t)M(z,t) to continuously measure the differential distance between any two neighboring system trajectories. If the vector field of the system ensures that this differential distance is strictly shrinking with respect to the metric defined by M(z,t)M(z,t), all trajectories will exponentially converge toward each other, regardless of their initial conditions. Under this contraction matrix, the agent dynamics with policy πθ _θ satisfies the contraction condition: M˙+MT∂(f+μθ)∂z+∂(f+μθ)∂z⊤M⪯−αM−αsI M+M^T ∂(f+ _θ)∂ z+ ∂(f+ _θ)∂ z M -α M- _sI where α>0α>0 represents the contraction rate, αs=LM(2σg2)/(αg+0.5) _s=L_M(2 _g^2)/( _g+0.5) accounts for stochastic perturbations, and αg>0 _g>0 is a design parameter. Under these conditions, results in stochastic contraction theory [38] guarantee exponential convergence in expectation between any two system trajectories ξ1 _1 and ξ2 _2: [‖ξ2(t)−ξ1(t)‖]≤[∫ξ1|(0)ξ2|(0)δz⊤Mδz]m¯e−2αt+C2αm¯m¯E [\| _2(t)- _1(t)\| ]≤ E [ _ _1|(0) _2|(0)δ z Mδ z ] me^-2α t+ C2α m m where C=(2σg2)/(2αg−1+1)C=(2 _g^2)/(2 _g^-1+1) bounds the steady-state error due to stochastic noise. The overall system’s contraction rate is given by a=miniαia= _i _i across all weight-induced subsystems. Additionally, let LwL_w denote the Lipschitz constant of system (13) with respect to the weight parameters w. To establish bounded behavior of the weight dynamics, we first present the following result: Lemma 3. The weight state x2x_2 of system (7) remains bounded in the invariant set [−B,B]N×K[-B,B]^N× K for all t≥0t≥ 0 if initialized within this set. Furthermore, the auxiliary state x1x_1 evolves within a bounded set 1X_1. The proof is reported in the Appendix IX-B. Given the Lipschitz constant LZL_Z of the function g in (6), the closed loop system is contracting under the following Theorem: Theorem 2. Under the joint control policy πθ _θ and weight dynamics (7), the closed-loop multi-agent system is contracting if the small-gain condition λmax(F22)⋅a>LG⋅Lw _ (F_22)· a>L_G· L_w (14) is satisfied, where LG=2LZ(‖W‖∞+1+116‖WB‖∞)L_G=2L_Z (\|W\|_∞+1+ 116\|W_B\|_∞ ) (15) represents the coupling strength between the weight and observation dynamics. We provide a proof in the appendix IX-C Remark 2. The Lipschitz constant LZL_Z can be readily computed for neural networks g employing standard activation functions such as ReLU or Tanh. Furthermore, LZL_Z has a well-defined analytical form when g is spectrally-normalized [26], providing practical computational advantages for stability verification. Alternatively, we can apply the same stability analysis framework to the weight selection dynamics under w. This leads to the following equivalent characterization: Corollary 1. The stability condition in Theorem 2 is equivalent to ensuring that each subsystem of (13) induced by a fixed weight selection w exhibits contractive behavior, along with the weight dynamics system (7) having its own contraction rate. The reverse implication also holds. Proof. This equivalence follows directly from Theorem 2 by considering each subsystem individually with Lw=0L_w=0, which decouples the weight dynamics from the observation dynamics. When Lw=0L_w=0, the small-gain condition reduces to verifying contraction of each subsystem independently. ∎ The conditions stated in Theorem 2 and the agent dynamics contraction can be enforced during training by adding a regularization term to the policy and weight-predictor loss functions that penalizes the spectral norms of the network parameters. This helps control their Lipschitz constants and satisfy the small-gain condition. Remark 3. Assumptions 2 is a standard policy exploration assumptions derived from the learning framework while Assumption 3 is a direct controlled consequence of our learning formulation. However, the deterministic observation dynamics might be restrictive even if our contraction analysis inherently guarantees robustness against bounded additive disturbances [38]. We reserve to future works the extension of the formal analysis to encompass general stochastic dynamics and observation. VII Evaluations (a) Multi-resource assignment scenario (b) Multi-team attacker-defender scenario Figure 4: Experimental scenarios: (a) Agents must deliver resources to consumer locations (red) while avoiding collisions and maintaining connectivity. (b) Multiple teams compete against each other to infiltrate multiple opponent safe areas while defending their own territory against opposing teams. We evaluate our base method, CIMORL, and its sampling-augmented variants (CIMORL-TS and CIMORL-MPPI) on two multi-agent domains: multi-resource assignment and multi-team attacker-defender. Detailed hardware, training, and architectural setups are deferred to the Appendix. We compare against state-of-the-art multi-policy baselines MOMAPPO [15] and MO-MIX [20], which generate discrete sets of policies. To ensure a fair algorithmic comparison against our continuous conditioned approach, we evaluate all methods over a uniformly sampled batch of evaluation preference weights w∗w^*. For the multi-policy baselines, for each weight vector in the evaluation batch, we select the specific policy from their generated set that maximizes the expected scalarized utility under that weight. Furthermore, we introduce MOMAPPO-TS, an on-policy baseline augmented with Tree Search that incorporates privileged expert data via the loss defined in Eq. (12). This further baselines serves as further validation of our method when the baseline is trained under the same privileged expert conditions. To evaluate the quality, coverage, and distribution of the solutions, we report the hypervolume indicator [17], spacing, and diversity metrics [20]. Policy adaptability and performance are assessed via the Expected Scalarized Return (ESR) formulation of expected utility (4). Because exhaustive empirical rollouts in continuous multi-objective spaces are intractable and suffer from high variance, we efficiently approximate the Pareto front by evaluating the trained value function VϕV_φ over identical, uniformly sampled weights w∗w^* consistently across all methods [15]. The final front is then refined using a pruning K-means clustering method [28]. Furthermore, to empirically validate Lemma 1 and Lemma 2, we evaluate two ablation baselines derived from our method: CIMORL w/o Sync (which bypasses consensus, directly mapping the embedding Ξ to the weight distribution) and CIMORL w/o Clust (which sets SoΞ=SSo_ =S and Sox=SSo_x=S in Eq. (7) to force a single global consensus rather than distinct coalitions). The following sections detail each scenario, presenting both the simulated training results and the real-world experimental validation using a team of Crazyflie 2.1 drones. VII-A Multi-Resource Assignment We consider a multi-resource assignment scenario involving a fully cooperative multi-agent team sV_s comprising NsN_s agents. The task requires delivering r distinct resource types to K consumer locations situated within a confined operational space Q, where each consumer may demand a specific subset of the available resources. Each consumer is associated with an interaction region of radius RkR_k, within which an agent may release its carried resources to satisfy the consumer’s demand, as illustrated in Figure 4. Furthermore, agents experience collisions when their inter-agent distance falls below a threshold RclR_cl. The information structure enforces partial observability constraints, where each agent i has access to limited observations zi∈z_i comprising: (i) its current position pip_i, velocity viv_i, and carried resources soi∈ℝrso_i ^r; (i) the locations of all target areas dskds_k and their corresponding demanded resources dek∈ℝrde_k ^r; and (i) a restricted set of teammate positions s=pj∣j∈iO_s=\p_j j _i\ within the agent’s sensing range. To ensure the existence of a unique resource allocation solution, we assume that the aggregate available supply ∑i=1Nssoi _i=1^N_sso_i does not exceed the total demand ∑k=1Kdek _k=1^Kde_k. The objective of this scenario is to determine a collaborative control policy that enables the agents to optimally satisfy the resource demands within the environment while maintaining network connectivity and avoiding inter-agent collisions. These objectives inherently conflict; optimizing delivery speed requires spatial dispersion that compromises connectivity, while aggressive movement increases spatial density and collision risks. Each agent receives a vector reward rwi(s,u):×→ℝK+2rw_i(s,u):S×U ^K+2 where the first K components reward the successful fulfillment of the K consumer demands, and the final two components incentivize connectivity maintenance and collision avoidance, respectively. Note that the reward rwirw_i depends on the joint state and actions of all agents, reflecting the inherently cooperative nature of the task. Specific details regarding the mathematical formulation of these rewards and the neural network architectures are deferred to the Appendix. VII-A1 Training Results Figure 5: Multi-resource assignment training comparison: Hypervolume indicator and expected utility results for all methods across multiple training runs. Figure 6: Normalized return for resource-allocation task for number of agents in the team N=[5,10,20]N=[5,10,20] and 66 objectives (44 demanding locations, obstacle avoidance and connectivity) reported as bar plots with their standard deviation as black lines. Figure 7: 3D trajectories in multi-resources allocation experiment: 3D trajectory comparison of the four methods. Figure 8: Demanding resource evolutions in multi-resources allocation experiment TABLE I: Disconnection rate in multi-resource allocation experiment MOMAPPO CIMORL CIMORL-MPPI CIMORL-TS 0.0%0.0\% 0.0%0.0\% 0.2%0.2\% 2.8%2.8\% For the multi-resource assignment scenario, we conducted training with 55 agents, 33 resource types, and 44 target locations. All sampling-based algorithms were trained with identical simulation budgets and planning horizons to ensure fair comparison. The learning rate was scheduled to decrease during training to promote convergence stability. Figure 5 presents the training comparison across all methods using four key metrics: Pareto Front hypervolume, sparsity and diversity, and expected utility. The results demonstrate clear performance distinctions between the proposed methods and baseline approaches. The sampling-based variants of our approach, CIMORL-MPPI and CIMORL-TS, achieved superior performance with statistically equivalent results. Both methods attained hypervolume values of approximately 8.0×104±4.0×1038.0× 10^4± 4.0× 10^3, indicating excellent Pareto front coverage, sparsity and diversity. Their expected utility performance was similarly strong, reaching 12.0±0.512.0± 0.5, demonstrating consistent policy quality across multiple evaluation runs. CIMORL-MPPI and CIMORL-TS perform comparably, proving that both MPPI and Monte Carlo Tree Search offer effective expert guidance. With 20002000 simulation samples and a horizon of 2525, there is no statistically significant difference between the two approaches. The base CIMORL method, while employing the same weight prediction framework, achieved moderately lower performance with a hypervolume of 6.4×104±6.0×1036.4× 10^4± 6.0× 10^3 and expected utility of 11.0±0.311.0± 0.3. This performance gap of approximately 20%20\% in hypervolume compared to the sampling-based variants highlights the importance of privileged expert guidance during training. Nevertheless, CIMORL still outperformed the baseline methods, demonstrating the effectiveness of the proposed weight prediction mechanism. All these methods reach similar levels of Pareto Front and Sparsity, showing high diversity of the weight that span very different points on the Pareto Front. Among the baseline methods, MOMAPPO-TS showed improved performance over standard MOMAPPO, achieving a hypervolume of 6.6×104±8.0×1036.6× 10^4± 8.0× 10^3 compared to MOMAPPO’s 5.8×104±5.0×1035.8× 10^4± 5.0× 10^3. This 13.8%13.8\% improvement demonstrates that tree search integration provides meaningful benefits even for fixed-weight approaches. Moreover, the improvement of 21.2%21.2\% in hypervolume achieved by CIMORL-MPPI and CIMORL-TS over MOMAPPO-TS shows the impact of the weight prediction model. However, all baseline methods exhibited substantially lower expected utility performance, with MOMAPPO achieving 8.5±0.58.5± 0.5, similar to MO-MIX and MOMAPPO-TS reaching 10.5±1.010.5± 1.0. The significantly higher variance in MOMAPPO-TS’s expected utility (σ=1.0σ=1.0) indicates inconsistent policy performance, likely due to the fixed weight selection strategy’s inability to adapt to varying environmental conditions. Without synchronization, our framework’s Pareto Front converges to a low, stable hypervolume alongside a marked decline in sparsity and diversity. This plateau demonstrates that lacking explicit coordination yields stable, albeit lower-quality solutions, as reflected by the expected utilities. Conversely, the dynamic synchronization in CIMORL w/o Clust effectively generates adaptive, high-value solutions. This configuration achieves a hypervolume of 6.0±0.5×1046.0± 0.5× 10^4 and expected utilities of approximately 9.0±0.49.0± 0.4, outperforming MOMAPPO while remaining subordinate to the complete CIMORL model. Furthermore, this variant maintains sparsity and diversity levels comparable to both CIMORL and its sampling-based counterpart. Ultimately, while this method produces an evenly distributed Pareto Front, its solutions remain strictly dominated by those discovered by the full CIMORL framework. In Figure 6 , we present the results of scaling the team size from 55 to 2020 agents. To facilitate comparison, returns were shifted and normalized to [0,1][0,1] based on the maximum return achieved for each objective across all 1010 random seeds. As expected, overall returns decrease with larger teams. However, connectivity (Objective 66) improves, as the denser environment naturally makes this objective easier to satisfy. In general, CIMORL-TS and CIMORL-MPPI maintain returns above 0.90.9 across all objectives, whereas performance drops to approximately 0.8 under the base CIMORL policy. This suggests that the sampling search achieves broader coordination than what is observed during training. Furthermore, performance remains remarkably consistent across the different objectives, demonstrating that roles are evenly assigned and the clustering mechanism effectively distributes tasks among agents as the team size grows. VII-A2 Experimental Comparison We performed real lab experiments to evaluate our approach. The experimental setup utilized 50 simulations for the three search-based methods and 1000 simulations for the MPPI-based approach with a horizon length of 10 steps to maintain computation within the identical hardware time budget of 5050 ms. This discrepancy is not an artificial setting but the natural outcome of the methods’ inherent mechanisms: MONTS requires sequential node expansion that bottlenecks parallelization, whereas MOMPPI evaluates independent trajectories, allowing massive GPU batching within the same latency constraint. Figure 7 reports the 3D trajectories of the experiments while the resource allocation dynamics are illustrated in Figure 8, for the four methods, MOMAPPO, CIMORL, CIMORL-TS, and CIMORL-MPPI. The results reveal significant behavioral differences among the methods. MOMAPPO demonstrated the most severe limitations, with agents colliding almost immediately upon mission initiation, preventing any meaningful resource allocation. CIMORL exhibited marginally better performance, allowing agents to operate for a brief period before collision occurred, yet still failing to complete the resource allocation task. In contrast, CIMORL-TS successfully avoided inter-agent collisions throughout the experiment, demonstrating that the sampling-based search mechanism effectively adapts to real drone dynamics. However, despite collision avoidance, CIMORL-TS failed to achieve complete resource allocation even during extended flight operations. CIMORL-MPPI emerged as the only method capable of both collision avoidance and successful resource allocation, validating the effectiveness of the proposed coordination-informed approach. This shows the compromise of CIMORL-TS in real-time applications as we had to drastically reduce the number of simulations to stay within the time budget. Consequently, the asymmetric search tree is computationally starved under finite sampling; it lacks the iterative depth necessary to reliably balance exploration and exploitation, systematically leading to brittle decisions based on unrefined leaf evaluations. Conversely, the 1000 uniform parallel samples achieved by CIMORL-MPPI generate a robust, smoothed approximation of the objective landscape, inherently producing safer, average-case optimal decisions under the identical time constraint. Table I provides insights into the connectivity objective. MOMAPPO and CIMORL both exhibit 0%0\% disconnection rates, as the flights terminate so rapidly due to collisions that insufficient time elapses for agents disconnection to manifest. CIMORL-MPPI achieves the most favorable disconnection performance with only 0.2%0.2\% disconnection rate, demonstrating robust coordination capabilities. CIMORL-TS experiences a higher disconnection rate of 2.8%2.8\%. Because agents take longer to fly and spread out further to allocate the remaining resources, the increased distance between them leads to more frequent link failures. The experimental results validate several key aspects of our approach: (i) the proposed weight prediction framework consistently outperforms fixed-weight baselines across both metrics; (i) privileged expert guidance through sampling-based algorithms provides substantial performance improvements; and (i) both MPPI and MCTS variants achieve comparable results, offering flexibility in computational resource allocation based on specific deployment constraints. VII-B Multi-team Attackers-Defenders Figure 9: Normalized returns for the attacker-defender task for varying team sizes of 55, 1010, and 2020 agents, evaluated across 44 teams ( therefore 44 objectives). Results are reported as bar plots with their standard deviation as black lines. We consider a complex multi-team adversarial scenario featuring asymmetric attack and defense objectives. The environment consists of a fully cooperative attacking team sV_s comprising NsN_s agents that must simultaneously accomplish the following objectives: penetrate K distinct opponent safe areas ℛkR_k, each defended by teams kV_k of size NkN_k, while protecting their own territorial zone ℛsR_s located at position dsd_s. The complete set of opposing teams is denoted as =1,…,KK=\1,…,K\. Each agent possesses a circular interaction zone of radius R within which it can eliminate an opponent agent through direct engagement, resulting in mutual destruction of both participants. Figure 4 illustrates this multi-objective adversarial configuration. The information structure enforces partial observability constraints, where each agent i has access to limited observations zi∈z_i comprising: (i) its current position pip_i and velocity viv_i; (i) the locations of all safe areas dskds_k; (i) a restricted set of teammate positions s=pj∣j∈iO_s=\p_j j _i\ within sensing range; and (iv) a limited collection of opponent positions k=pok∣k∈k,∀k∈O_k=\p_o_k k _k,∀ k \ across all opposing teams. The primary objective is to derive a collaborative control policy that enables team sV_s to successfully capture all adversarial safe areas while maintaining control of their own territory under partial observability constraints. Each agent receives a multi-dimensional reward signal rwi(s,u):×→ℝK+1rw_i(s,u):S×U ^K+1 that evaluates performance across K attack objectives (corresponding to opponent safe areas) and one defense objective (protecting the home territory). The reward structure must carefully balance individual agent contributions with collective team achievements, incorporating coordination incentives for collaborative attack and defense strategies while enforcing operational constraints such as collision avoidance. Full architectural parameters and reward function aggregations are detailed in the Appendix. VII-B1 Training Results Figure 10: Multi-team attacker-defender performance comparison Figure 11: 3D trajectories in multi-team attacker-defender experiments. TABLE I: Multi-Objective Scores with Alternating Policies. The assigned policy rotates across the Blue, Red, and Green teams for each scenario. Scenario Red Team Green Team Blue Team a CIMORL MOMAPPO CIMORL-MPPI (−694,992,379)(-694,992,379) (−2499,0,0)(-2499,0,0) (−379,694,1507)(-379,694,1507) b CIMORL-MPPI CIMORL MOMAPPO (0,727,0)(0,727,0) (−727,0,0)(-727,0,0) (0,0,0)(0,0,0) c MOMAPPO CIMORL-MPPI CIMORL (0,0,0)(0,0,0) (−418,0,0)(-418,0,0) (0,0,418)(0,0,418) For the multi-team attacker-defender scenario, we conducted experiments with 2020 agents distributed equally across 44 teams, creating a complex adversarial environment with multiple competing objectives. The sampling-based algorithms maintained identical simulation budgets and planning horizons as in the resource assignment experiments, with the same decreasing learning rate schedule to ensure training stability. Figure 10 presents the performance comparison for this challenging scenario, revealing how different algorithms navigate the trade-offs between hypervolume, utility, sparsity, and diversity in competitive dynamics. CIMORL-MPPI achieved the strongest overall performance with a hypervolume of 7.25×103±0.25×1037.25× 10^3± 0.25× 10^3 and expected utility of 11.0±0.4511.0± 0.45. The relatively low variance in both metrics indicates consistent performance across evaluation runs, suggesting that the MPPI-based approach provides stable policy learning in adversarial settings. This stability can be attributed to MPPI’s uniform exploration strategy, which systematically samples the action space without making overly optimistic assumptions about opponent behavior. CIMORL-TS achieved comparable hypervolume performance (7.25×103±0.33×1037.25× 10^3± 0.33× 10^3) but exhibited slightly lower expected utility (10.7±0.510.7± 0.5) and higher variance during training. This performance profile suggests that Monte Carlo Tree Search’s optimistic selection strategy drives excellent exploration, establishing the highest upward trajectories in both Pareto sparsity (reaching near 450450) and diversity (peaking above 110110), but leaves the policy more sensitive to unpredictable adversarial shifts, as evidenced by the higher variance (σ=0.33σ=0.33 vs. 0.250.25 for hypervolume, σ=0.5σ=0.5 vs. 0.450.45 for utility).The base CIMORL method demonstrated strong performance with a hypervolume of 7.0×103±0.5×1037.0× 10^3± 0.5× 10^3 and expected utility of 9.5±0.39.5± 0.3. Notably, the performance gap between CIMORL and the sampling-based variants is smaller in this adversarial scenario compared to the cooperative resource assignment task. Its sparsity and diversity metrics closely mirror those of CIMORL-MPPI, maintaining a strong middle-to-high position throughout training. Among the baseline methods, MOMAPPO-TS achieved superior performance compared to standard MOMAPPO, with hypervolume values of 6.9×103±0.4×1036.9× 10^3± 0.4× 10^3 versus 6.7×103±0.3×1036.7× 10^3± 0.3× 10^3, representing a 3%3\% improvement. MOMAPPO-TS exhibited significantly higher variance in expected utility (9.6±1.09.6± 1.0 vs. 8.4±0.258.4± 0.25), indicating that while tree search integration improves average performance, it also introduces inconsistency in policy execution. MO-MIX is subjected more to the non-stationarity in adversarial games, as it is an off-policy method. Therefore, it reaches the worst performances in all the metrics. Consequently, these baselines exhibit notably lower sparsity, plateauing early in the 250250 to 300300 range, and cluster in the bottom tier for diversity. Additionally, all metrics display wide, heavily overlapping confidence intervals across these algorithms, reflecting the substantial variance inherent in competitive multi-agent environments. Both CIMORL w/o Sync and CIMORL w/o Clust converge to a similar hypervolume of 6.65±0.2×1036.65± 0.2× 10^3. However, consistent with the previous scenario, the w/o Sync variant yields lower expected utilities compared to the other baselines and suffers a sharp, early plateau in sparsity. Furthermore, the expected utilities produced by CIMORL w/o Clust clearly illustrate the performance gap between full synchronization and isolated clustering dynamics, amounting to an average difference of approximately 0.80.8. Without clustering, the model also remains confined to the lower tier of Pareto diversity, further underscoring the necessity of the proposed coordination mechanisms. We also evaluated the proposed approach on larger teams, with results presented in Figure 9. As with the resource-allocation task, we normalized the returns to a [0,1][0,1] scale. We observed that the baseline CIMORL policy struggles to maintain performance across objectives as the team size increases, regardless of whether the agents are assigned attacker or defender roles. In contrast, both MPPI and tree search successfully sustain their performance. This confirms that the proposed multi-objective online search effectively extends coordination capabilities to larger teams than those encountered during training. VII-B2 Experimental Comparison We conducted experiments with three competing teams, each composed of three drones, rotating the policies among them in three different experiments. Again due to time budget, MPPI uses 1000 simulations with an horizon length of 10 steps for computational efficiency. Figure 11 reports the 3D trajectory across the three distinct experimental configurations, each showcasing different team-policy assignments. In experiment (a), the blue team employs CIMORL-MPPI, the green team uses MOMAPPO, and the red team implements CIMORL. In experiment (b), policy assignments rotate with the red team utilizing CIMORL-MPPI, blue team employing MOMAPPO, and green team implementing CIMORL. Experiment (c) completes the rotation with the green team using CIMORL-MPPI, red team employing MOMAPPO, and blue team implementing CIMORL. Table I quantifies team performance through a comprehensive scoring system that awards at every time step +1 points for each agent successfully infiltrating opponent areas and -1 points for allowing enemy agents into their own territory for the corresponding objectives. In experiment (a), the blue team (CIMORL-MPPI) demonstrates superior strategic coordination, achieving scores of (-379, 649, 1507) through a circular patrol pattern that simultaneously enables area protection and coordinated attacks. This was an emergent behavior from the policy not pre-programmed or searched during training. This behavior exemplifies the adaptive weight dynamics’ capability to balance competing objectives dynamically. The red team (CIMORL) exhibits a more conservative strategy with scores of (-649, 992, 379), distributing drones uniformly among safe areas, which demonstrates effective weight distribution across objectives but lacks a robust coordination pattern. The green team (MOMAPPO) experiences catastrophic failure due to inter-agent collisions, resulting in severely negative performance with a score of (-2499, 0, 0). This failure mode is characteristic of fixed-weight architectures: lacking the ability to dynamically down-weight aggressive territorial infiltration in favor of immediate spatial de-confliction, the agents exhibit rigid, brittle behavior that easily collapses into physical gridlock when facing heterogeneous opponents. Experiment (b) reveals distinct strategic adaptations, where the red team (CIMORL-MPPI) with scores of (0, 727, 0) develops an advanced enclosure strategy, effectively surrounding and neutralizing green team drones protecting their territory. The blue team (MOMAPPO) suffers immediate collision-induced failure, achieving null scores (0, 0, 0), while the green team (CIMORL), with scores of (-727, 0, 0), demonstrates improved collision avoidance compared to MOMAPPO but fails to mount effective defensive strategies, allowing complete territorial infiltration. Crucially, CIMORL’s failure mode is distinct from MOMAPPO’s immediate physical gridlock. Instead of colliding, the green team’s trajectories exhibit severe spatial oscillation (as seen in Figure 11.b). This oscillatory hesitation occurs because the dynamic weight consensus mechanism is attempting to adapt to the adversarial pressure, but without expert demonstrations, the model has found an oscillatory strategy. The agents constantly shift their priority weights between attacking and defending, delaying decisive cooperative actions. This behavior is consistent with our theoretical analysis in Sec. VI, illustrating a system that is provably contracting, but where the contraction rate is bottlenecked by the baseline training parameters. The sampling-based search in CIMORL-MPPI overcomes this by utilizing privileged expert examples to rapidly smooth the objective landscape, inducing a much higher contraction rate that translates to decisive, rather than hesitant, spatial maneuvers. In experiment (c), coordinated aggression from red and blue teams against the green team creates a unique defensive scenario. The green team (CIMORL-MPPI) adopts a sacrifice-based defensive strategy, concentrating all resources on territorial protection. The blue team achieves modest success with scores of (0, 0, 418) through effective resource allocation, while the red team’s escape attempts ultimately fail, resulting in null scores (0, 0, 0). The green team’s defensive sacrifice yields scores of (-418, 0, 0), demonstrating CIMORL-MPPI’s ability to adapt strategy based on overwhelming adversarial pressure. A critical observation from these real-world experiments is the substantial behavioral differences compared to training scenarios. During training, policies are developed through self-play, where agents learn to coordinate against opponents employing identical strategies and behavioral patterns. However, in the heterogeneous real-world experimental setup, each team employs fundamentally different policies with distinct coordination mechanisms and strategic approaches. Transitioning from homogeneous self-play in training to heterogeneous open-play in reality creates a significant deployment gap. This gap exposes the limitations of traditional reinforcement learning and proves that online adaptation is crucial. Specifically, it reveals two distinct suboptimal failure modes in the baselines: (i) the rigid brittleness of fixed-weight approaches (MOMAPPO), which leads to unresolvable spatial conflicts and collisions, particularly in tight resource-assignment scenarios, and (i) the oscillatory hesitation of unaugmented dynamic approaches (base CIMORL), which successfully avoid collisions but fail to commit to a decisive strategy due to slow weight consensus. The sampling-based search integration in CIMORL-MPPI demonstrates remarkable online adaptation abilities, enabling real-time strategy adjustment when confronted with unfamiliar opponent behaviors that were not encountered during training. This adaptive capacity explains CIMORL-MPPI’s consistent superior performance across all experimental configurations, as it can dynamically recalibrate its coordination patterns, weight distributions, and tactical decisions based on observed opponent strategies. In contrast, fixed-weight approaches like MOMAPPO and weight-distribution methods like CIMORL struggle to adapt beyond their training distributions, leading to suboptimal performance or catastrophic failures when facing novel adversarial strategies. VIII Conclusion This work introduced a coordination-informed multi-objective reinforcement learning (CIMORL) framework addressing fundamental challenges in balancing competing objectives within multi-robot systems. The approach integrates three key innovations: distributed weight prediction through opinion dynamics, privileged expert training via sampling-based search methods, and theoretical guarantees for Pareto-optimal coordination through coalition formation. The experimental validation demonstrated significant improvements over state-of-the-art baselines, with our sampling-based variants demonstrating strong performance in complex real-world drone experiments where baseline methods failed, highlighting the critical role of online adaptation. Future research directions will include improving the convergence speed of the proposed weight model, which currently converges slowly to the equilibria, by imposing a higher contraction rate. Investigating methods to enforce this higher contraction rate while maintaining the numerical stability of the integration presents a challenging but important direction. Second, we aim to evaluate the implications of the hierarchical scheme in the proposed weight-policy model and explore more efficient training methods to minimize policy variance. Additionally, we plan to conduct a formal theoretical and empirical analysis of the weight adaptation mechanism’s robustness against explicit communication delays and network noise. Finally, while our current formulation successfully scales to four simultaneous objectives—including inter-agent collision avoidance—extending this coordination framework to explicitly handle advanced path-planning in cluttered layouts with static and dynamic environmental obstacles remains an exciting avenue for our future research. IX Appendix IX-A Proof to Theorem 1 In this section, we report the proof for Theorem 1 Proof. The proof strategy is twofold: first, we show that a clustered state, where agents in a coalition share the same trajectory, is a valid equilibrium solution for the system’s dynamics; second, we prove that the system is contracting, which guarantees that any initial state will globally converge to such a stable, clustered trajectory. Consider M<NsM<N_s clusters of team sV_s, where each cluster crc_r contains agents satisfying dist(Ξi,Ξj)<ϵdist( _i, _j)<ε for i,j∈cri,j∈ c_r and sufficiently small ϵ>0ε>0. Let the joint state be x=[x1,x2]x=[x_1,x_2], and denote two state trajectories as ξ=[ξ1,ξ2]ξ=[ _1, _2] and ξ¯=[ξ¯1,ξ¯2] ξ=[ ξ_1, ξ_2]. We define ξ¯2 ξ_2 such that it follows the clustered structure: ξ¯2=[1c1⊗xc1,…,1cM⊗xcM] ξ_2=[1_c_1 x_c_1,…,1_c_M x_c_M] and ξ¯1 ξ_1 to make ξ¯2 ξ_2 an equilibrium trajectory for x2x_2. Using the vectorization operator X|=vec(X)X_|=vec(X), we have x|=[x1|,x2|]∈ℝNs⋅2Kx_|=[x_1|,x_2|] ^N_s· 2K. Define the virtual state x=ξ+μ(ξ¯−ξ)x=ξ+μ( ξ-ξ) with x(μ=0)=ξx(μ=0)=ξ and x(μ=1)=ξ¯x(μ=1)= ξ. We now show that ξ¯ ξ is a particular solution of the system described by Eq. (7). For agent i and its driving signal Ξi _i, its equilibrium trajectory is given by: ξ¯˙i1 ξ_i1 =(SoΞiξ¯2WB−ξ¯i2WB) =(So_ _i ξ_2W_B- ξ_i2W_B) (16) ξ¯˙i2 ξ_i2 =−(τ+Ξi)∘ξ¯i2+Soξ¯iξ¯2WA−ξ¯i2WA =- (τ+ _i ) ξ_i2+So_ ξ_i ξ_2W_A- ξ_i2W_A −SoΞiξ¯i1WB+ξ¯i1WB+ΞiB -So_ _i ξ_i1W_B+ ξ_i1W_B+ _iB where SoΞi,Soξ¯iSo_ _i,So_ ξ_i are the i-th rows of the matrices. Next, we select agent j from the same cluster as i and consider the difference in their dynamics, ξ¯˙i−ξ¯˙j ξ_i- ξ_j, which is: ξ¯˙1i−ξ¯˙1j=SoΞiξ¯2WB−SoΞjξ¯2WBξ¯˙2i−ξ¯˙2j=−(Ξi∘ξ¯2i−Ξj∘ξ¯2j)+(Soξ¯iξ¯2WA−Soξ¯jξ¯2WA)−(SoΞiξ¯1WB−SoΞjξ¯1WB)+(Ξi−Ξj)B. aligned ξ_1i- ξ_1j=&So_ _i ξ_2W_B-So_ _j ξ_2W_B\\ ξ_2i- ξ_2j=&-( _i ξ_2i- _j ξ_2j)+(So_ ξ_i ξ_2W_A-So_ ξ_j ξ_2W_A)\\ &-(So_ _i ξ_1W_B-So_ _j ξ_1W_B)+( _i- _j)B aligned. (17) The difference in eq(17) is equal to the null vector as long as SoΞ,SoξSo_ ,So_ξ have constant row sum in the block matrices among the clusters, as demonstrated in [43]. Given our definition for these two matrices, this property is structurally satisfied. Moreover, the differences in eq(17) are zero when ξ¯1 ξ_1 is such that (SoΞi−SoΞj)ξ¯1WB=(Ξi−Ξj)B−(Ξi∘ξ¯2i−Ξj∘ξ¯2j)(So_ i-So_ j) ξ_1W_B=( _i- _j)B-( _i ξ_2i- _j ξ_2j). We now prove that the system is contracting for the two trajectories ξ and ξ¯ ξ. The dynamic of the virtual displacement δx|δ x_| is defined by δx|˙=Fδx|+Gδz|δ x_|=Fδ x_|+Gδ z_| (18) with F F =[0WB⊤⊗(IN−SoΞ)WB⊤⊗(IN−SoΞ)−diag(τ+Ξ)+WA⊤⊗(IN−Sox)] = bmatrix0&W_B (I_N-So_ )\\ W_B (I_N-So_ )&- *diag(τ+ )+W_A (I_N-So_x) bmatrix (19) G G =[WB⊤⊗(∂SoΞ∂Ξ∂g∂z)x2|diag(∂g∂z)x2|+diag(∂g∂z)B|−WB⊤⊗(∂SoΞ∂Ξ∂g∂z)x1|] = bmatrixW_B ( _ ∂ ∂ g∂ z)x_2|\\ *diag( ∂ g∂ z)x_2|+ *diag( ∂ g∂ z)B_|-W_B ( _ ∂ ∂ g∂ z)x_1| bmatrix To establish convergence to the clustered equilibrium ξ¯ ξ, we analyze the contraction properties of the unforced differential dynamics with respect to the state x, given by δx˙|=Fδx|δ x_|=Fδ x_|. We define the differential Lyapunov function P representing the Riemannian energy of the virtual displacement: P=∫01δx|⊤δx|dμP= _0^1δ x_| δ x_|dμ (20) Taking the time derivative of P along the trajectories of the system yields: P˙=∫01δx|⊤(F+F⊤)δx|dμ P= _0^1δ x_| (F+F )δ x_|dμ (21) The stability of the system is governed by the symmetric part of the Jacobian matrix, Fs=12(F+F⊤)F_s= 12(F+F ). Analyzing the block structure of F, the top-left block corresponding to the auxiliary state x1x_1 is strictly zero. However, the bottom-right block governing the primary weight state x2x_2 is defined by F22=−diag(τ+Ξ)+WA⊤⊗(IN−Sox)F_22=-diag(τ+ )+W_A (I_N-So_x). By design, the bias τ>0τ>0, the activation Ξ≥0 ≥ 0, and the attention mechanism (IN−Sox)(I_N-So_x) acts as a network Laplacian. Consequently, F22F_22 is strictly Hurwitz, and its symmetric part is negative definite. While the presence of the zero block in F implies that the full Jacobian is only negative semi-definite, the system exhibits weak (or partial) contraction. If we consider the projected state transformation y^=[0,I]x|⊤=x2| y=[0,I]x_| =x_2|, its differential dynamic is strongly contracting with a rate bounded by the maximum eigenvalue of the symmetric part of F22F_22, such that λmax(F22+F22⊤)<0 _ (F_22+F_22 )<0. As shown in standard partial contraction theory [12], because the x2x_2 subsystem is strictly contracting, all trajectories of x2x_2 exponentially converge to a single unique trajectory, regardless of their initial conditions. This rigorous contraction of the x2x_2 state forms the direct logical connection to our clustering definition. We have already established in Eq. (17) that the clustered trajectory ξ¯ ξ, where agents in a coalition share the exact same state, is a valid equilibrium solution of the system. Because the system is contracting with respect to x2x_2, the actual network state x2(t)x_2(t) must asymptotically converge to this clustered equilibrium ξ¯2(t) ξ_2(t). Thus, the agents inherently form the clusters specified in Definition IV.1. ∎ IX-B Proof to Lemma 3 In this section, we report the proof to Lemma 3 Proof. We establish the boundedness of the system using the quadratic Lyapunov function P1=12x1|⊤x1|+12x2|⊤x2|P_1= 12x_1| x_1|+ 12x_2| x_2| where the notation (⋅)|=vec(⋅)(·)_|=vec(·) denotes the standard column-wise vectorization operator. This specific choice of P1P_1 is standard in nonlinear systems analysis for establishing L2L_2 stability and proving the existence of bounded invariant sets by evaluating the generalized energy of the states. Focusing on the x2x_2 dynamics, the time derivative of P1P_1 along the system trajectories yields: P˙1 P_1 ≤−x2|⊤diag(τ2|)x2|−x2|⊤diag(Ξ|)x2| ≤-x_2| diag( _2|)x_2|-x_2| diag( _|)x_2| −x2|⊤(WA⊤⊗(IN−SoΞ))x2|+x2|⊤diag(Ξ|)B| -x_2| (W_A (I_N-So_ ))x_2|+x_2| diag( _|)B_| Assuming the structural matrices associated with x2|x_2| and the network coupling WA⊤⊗(IN−SoΞ)W_A (I_N-So_ ) are positive semi-definite, their corresponding quadratic forms are non-positive. To prove that [−B,B]N×K[-B,B]^N× K is a positively invariant set, we analyze the remaining terms on the boundary by expanding them element-wise for each index i∈1,…,NKi∈\1,…,NK\: −x2|⊤diag(Ξ|)x2|+x2|⊤diag(Ξ|)B|=∑i=1NKΞ|,i(−x2|,i2+B|,ix2|,i)-x_2| diag( _|)x_2|+x_2| diag( _|)B_|= _i=1^NK _|,i (-x_2|,i^2+B_|,ix_2|,i ) Consider the case where the state x2x_2 attempts to leave the region [−B,B]N×K[-B,B]^N× K. This implies that for at least one element index i, the state exceeds the boundary such that |x2|,i|>B|,i|x_2|,i|>B_|,i. Since B|,i>0B_|,i>0 by definition, the squared term strictly dominates the linear term: x2|,i2>B|,i|x2|,i|≥B|,ix2|,ix_2|,i^2>B_|,i|x_2|,i|≥ B_|,ix_2|,i. Given that the activation functions yield Ξ|,i≥0 _|,i≥ 0, we can explicitly bound the contribution of the i-th element outside the boundary: Ξ|,i(−x2|,i2+B|,ix2|,i)<Ξ|,i(−B|,i|x2|,i|+B|,i|x2|,i|)=0 _|,i (-x_2|,i^2+B_|,ix_2|,i )< _|,i (-B_|,i|x_2|,i|+B_|,i|x_2|,i| )=0 Because the negative quadratic term strictly dominates the linear input term for any |x2|,i|>B|,i|x_2|,i|>B_|,i, it ensures that P˙1<0 P_1<0 strictly outside the defined set. This continuous dissipation of energy on the exterior of [−B,B]N×K[-B,B]^N× K forces the trajectories to remain bounded within it. Since x2|x_2| remains bounded and contractive, the coupling integral is finite:Lx1=‖WB‖∞∫0t‖(I−SoΞ(s))x2|(s)‖∞s<‖WB‖∞‖B‖∞λmax(F22)<∞L_x_1=\|W_B\|_∞ _0^t\|(I-So_ (s))x_2|(s)\|_∞ds<\|W_B\|_∞ \|B\|_∞ _ (F_22)<∞ Therefore, x1|(t)x_1|(t) remains bounded in 1X_1 when initialized within this set. ∎ IX-C Proof to Theorem 2 In this section, we report the proof to Theorem 2 Proof. To formally prove the closed-loop contraction of the multi-agent system, we analyze the overall architecture as an interconnected system comprised of two subsystems: the observation dynamics (13) and the weight prediction dynamics (7). The stochastic small-gain theorem [25, 33] dictates that an interconnection of two contracting subsystems remains globally contracting if the product of their individual contraction rates is strictly greater than the product of their cross-coupling Lipschitz constants. Let the individual contraction rate of the weight dynamics be bounded by λmax(F22) _ (F_22), and let the contraction rate of the observation dynamics be a=miniαia= _i _i. We denote the Lipschitz constant of the observation dynamics with respect to the weights w as LwL_w. The remaining requirement is to explicitly derive LGL_G, which bounds the coupling strength of the weight dynamics with respect to the observation z. Consider the coupling function G(x)G(x) derived from the differential dynamics of the system. We must evaluate its Jacobian with respect to the observation vector z. Applying the chain rule through the nonlinear mapping Ξ=g(z) =g(z), the partial derivatives of the effective similarity matrix SoΞSo_ and the primary state dynamics yield the following block matrix structure for the Jacobian J=∂G∂xJ= ∂ G∂ x: J=[0WB⊤⊗(∂SoΞ∂Ξ∂g∂z)0−WB⊤⊗(∂SoΞ∂Ξ∂g∂z)diag(∂g∂z)diag(∂g∂z)][x1|x2|B|]J= bmatrix0&W_B ( _ ∂ ∂ g∂ z )&0\\ -W_B ( _ ∂ ∂ g∂ z )& *diag ( ∂ g∂ z )& *diag ( ∂ g∂ z ) bmatrix bmatrixx_1|\\ x_2|\\ B_| bmatrix To establish the upper bound LGL_G, we evaluate the infinity norm of this block matrix, ‖J‖∞ J _∞. We leverage the known analytical bounds of the specific nonlinearities employed in the architecture. Specifically, the gradients of both the standard sigmoid function σ(⋅)σ(·) and the softmax operation So(⋅)So(·) are strictly bounded such that their maximum singular values never exceed 14 14. Additionally, we denote the maximum Lipschitz constant of the observation mapping network g(z)g(z) as LZL_Z, meaning ‖∂g∂z‖∞≤LZ ∂ g∂ z _∞≤ L_Z. Applying the triangle inequality and these known bounds to the block components, we can bound the supremum of the coupling function’s norm across all states x bounded by B: supx‖G(x)‖∞ _x G(x) _∞ ≤max(B,‖WB‖∞‖B‖∞λmax(F22)) ≤ (B,\|W_B\|_∞ \|B\|_∞ _max(F_22) ) ×LZ(‖W‖∞+1+116‖WB‖∞) × L_Z ( W _∞+1+ 116 W_B _∞ ) The factor of 116 116 emerges directly from the sequential application of the chain rule through both the sigmoid and softmax functions (14×14 14× 14). Simplifying the expression relative to the coupling weights yields the final bound: LG=2LZ(‖W‖∞+1+116‖WB‖∞)L_G=2L_Z (\|W\|_∞+1+ 116\|W_B\|_∞ ) With LGL_G strictly bounding the sensitivity of the weight dynamics to the observations, and LwL_w bounding the sensitivity of the observations to the weights, the closed-loop system is verified to be stable and contracting if the small-gain condition is satisfied: λmax(F22)⋅a>LG⋅Lw _ (F_22)· a>L_G· L_w This concludes the proof, demonstrating that as long as the internal consensus forces and local observation contractions dominate the cross-coupling perturbations, the entire multi-agent team will reliably converge to stable weight clusters. ∎ IX-D Hardware Setup and Training Parameters The agents are modeled as double integrators operating at 20 Hz, stabilized by a proportional-derivative (PD) policy with stiffness kp=7k_p=7 and damping kd=5k_d=5. This yields the closed-loop matrix A=[0I−kpI−kdI]A= [ smallmatrix0&I\\ -k_pI&-k_dI smallmatrix ]. To satisfy Theorem 2 directly in the physical observation space, we introduce a cross-coupling term ϵ=0.05ε=0.05 to define the non-diagonal contraction metric M=[(kp+ϵkd)IϵIϵI]M= [ smallmatrix(k_p+ε k_d)I&ε I\\ ε I&I smallmatrix ]. This specific choice makes the symmetric generalized Jacobian strictly negative definite by perfectly canceling the off-diagonal cross-terms: ATM+MA=[−2ϵkpI00−2(kd−ϵ)I]A^TM+MA= [ smallmatrix-2ε k_pI&0\\ 0&-2(k_d-ε)I smallmatrix ]. Consequently, the single-agent subsystem is proven to be strictly contracting with a rate of α=min(ϵkp,kd−ϵ)α= (ε k_p,k_d-ε). At the same sampling rate, the agents communicate with their neighbors and generate actions from the learned policy. All training and evaluations were conducted on a machine running Ubuntu 22.04 with an Intel Core i7-9750H @ 2.60GHz CPU, Nvidia RTX 2080Ti, and 32GB RAM. The observation vectors were normalized to the range [−1,1][-1,1], and a communication range of 11 was considered in the normalized space. A full breakdown of the hyperparameters utilized across algorithms is detailed in Table IV. We chose the training hyperparameters and the coefficients c1,c2,c3c_1,c_2,c_3 leading to the best performances in out configuration by performing parameters sweeps. TABLE IV: Training parameters Parameters CIMORL—w/o Clust—w/o Sync MOMAPPO MO-MIX CIMORL-TS—CIMORL-MPPI MOMAPPO-TS γ 0.99 0.99 0.99 0.99 0.99 Episode length 250 250 250 250 250 λGAE _GAE 0.95 0.95 - 0.95 0.95 Batch size 200 200 256 200 200 Epochs 6 6 1 6 6 rollout buffer 2000 2000 - 1000 1000 replay buffer - - 150×103150×10^3 - - Entropy coefficient 0.02 0.02 0.05 0.01 0.01 Clip ϵε 0.3 0.3 - 0.3 0.3 Max gradient norm 0.5 0.5 0.5 0.5 0.5 Adam learning rate 2×10−32×10^-3 2×10−32×10^-3· 0.9×10−30.9×10^-3 2×10−32×10^-3 2×10−32×10^-3 Sampled w∗w^* 500 500 500 500 500 Training steps per weight - 150×103150×10^3 150×103150×10^3 - 150×103150×10^3 Search horizon - - - 25 25 Search simulations - - - 2000 2000 c1,c2,c3c_1,c_2,c_3 - - - [1.5, 0.5, 0.75] [1.5, 0.5, 0.75] IX-E Network Architectures and Reward Structures Multi-Resource Assignment The reward function is designed to incentivize agents to approach consumer areas when they possess resources that match the consumers’ requirements. Specifically, for each consumer demand k, we assign the following reward: rwik=β(‖pit−1−dsk‖−‖pit−dsk‖)if ‖pit−dsk‖≥Rkand soi⋅dek>0RWikotherwiserw_ik= casesβ(\|p_i^t-1-ds_k\|-\|p_i^t-ds_k\|)&if \|p^t_i-ds_k\|≥ R_k\\ &and so_i·de_k>0\\ RW_ik&otherwise cases where β>0β>0 is a scaling factor and RWik>0RW_ik>0 represents a positive reward assigned to agent i upon successfully delivering a resource to consumer area k. The collision avoidance reward is defined as rwi(K+1)=rw_i(K+1)= −RWclif ∃j≠i,‖pi−pj‖<Rclγcl(minj‖pi−pj‖−3⋅Rcl)if minj‖pi−pj‖<3⋅Rcl0otherwise cases-RW_cl&if ∃ j≠ i,\|p_i-p_j\|<R_cl\\ _cl( _j\|p_i-p_j\|-3· R_cl)&if _j\|p_i-p_j\|<3· R_cl\\ 0&otherwise cases where RWcl>0RW_cl>0 is a fixed penalty for collisions and γcl>0 _cl>0 is a scaling factor. The connectivity reward is defined as rwi(K+2)=−RWcorw_i(K+2)=-RW_co if the i-th row of the communication adjacency matrix S contains only zeros, and 0 otherwise, where RWco>0RW_co>0 represents the connectivity penalty. We have experimented with RWco=0.4RW_co=0.4, RWcl=15RW_cl=15, Rcl=0.05R_cl=0.05, γcl=0.1 _cl=0.1, β=0.1β=0.1, RWik=15RW_ik=15 and Rk=0.2R_k=0.2 Let sdik=soi⋅dek‖soi‖⋅‖dek‖+ϵsd_ik= so_i·de_k\|so_i\|·\|de_k\|+ε indicating whether the agent resource or demand resource are zero for a small ϵ>0ε>0. The agent observation vector ziz_i is structured as follows: zi=[vi,pi−dsk,sdik⋅(soi+dek)k∈,pi−pjpj∈s]z_i= [v_i,\p_i-ds_k,sd_ik·(so_i+de_k)\_k ,\p_i-p_j\_p_j _s ] We employ a DeepSets [46] architecture to achieve permutation-invariant aggregation across each observation set. For teammate relative positions, we compute: Φs=ϕ2(∑pj∈sϕ1(pi−pj)) _s= _2 ( _p_j _s _1(p_i-p_j) ) (22) where ϕ1(⋅) _1(·) and ϕ2(⋅) _2(·) represent MLPs mapping inputs to an F-dimensional feature space. Similarly, we define aggregation functions Φk _k for each resource set k. The resulting features from different sets are concatenated and processed in an objective-specific manner. For the K resource-specific objectives, each objective k concatenates the relative distance to the consumer location pi−dskp_i-ds_k with the aggregated resource features Φk(sdik⋅(soi+dek)) _k(sd_ik·(so_i+de_k)). The collision avoidance and connectivity maintenance objectives utilize the teammate aggregation features Φs _s. All objectives incorporate velocity information to enable dynamic motion planning and coordination. The concatenated features are processed through a final MLP g(⋅)g(·) as defined in equation (6). This modular observation processing architecture enables flexible objective scaling between training and deployment phases, which may involve different numbers of consumer locations, resource types, and team sizes. Alternatively, a unified MLP g(⋅)g(·) can directly extract fixed-dimensional objective features from the complete observation vector ziz_i. The value function VϕV_φ utilizes an identical architecture to estimate objective-specific episode values. The policy network integrates the sampled weight vector w with the extracted objective features into a unified representation, which is then processed by a final MLP to generate action distributions. All MLPs have [32,32][32,32] features and tanh activation function for which is possible to compute the Lipschitz constant easily. All the methods in the following use the same networks with the same dimensions. Multi-team Attackers-Defenders The reward function is formalized through a team-level structure by defining RWs∈ℝK+1RW_s ^K+1 as the aggregated reward vector for team sV_s at time t, treating the collective as a unified strategic entity. The problem structure naturally leads to a generalized zero-sum multi-player game formulation when viewed from the team perspective. For each attack objective k, we assign a positive reward RWsk>0RW_sk>0 to team sV_s upon successful penetration of opponent region ℛkR_k by any team member. Correspondingly, the defending team kV_k receives RWkk=−RWskRW_k=-RW_sk, establishing the adversarial reward structure. For the defense objective s, the roles are reversed: team sV_s seeks to minimize intrusions while opposing teams attempt to maximize their territorial gains. The team’s strategic goal is to simultaneously optimize all K+1K+1 objectives, contingent upon sufficient agent allocation and coordination capabilities. The transition from team-level to individual agent rewards requires careful consideration of the partial observability constraints. To keep consistency with our observability constraints, we implement a localized reward structure where agent i receives rwik>0rw_ik>0 upon successfully infiltrating opponent region ℛkR_k, and rwis<0rw_is<0 when detecting enemy penetration of the home territory ℛsR_s within its observation range. Additionally, agents incur a substantial penalty rwi≪0rw_i 0 for teammate collisions across all objective rewards to promote collision-free coordination. We assigned rwi=−20rw_i=-20 and −rwis=rwik=1-rw_is=rw_ik=1 The agent observation vector ziz_i is structured as follows: zi=[vi,pi−dskk∈∪s,pi−pjpj∈s,pi−pkpk∈k]z_i= [v_i,\p_i-ds_k\_k ∪\s\,\p_i-p_j\_p_j _s,\p_i-p_k\_p_k _k ] This representation encodes relative positional information across three distinct categories: distances to all safe areas, relative positions of observable teammates, and locations of detected opponents. These categorical groups undergo separate processing to extract objective-specific features. For attack objectives, we combine relative positions to target safe areas with corresponding opponent locations to form offensive feature representations. The defensive objective aggregates the home safe area position with all observable opponent locations. Teammate positions and agent velocity are incorporated across all objectives to enable coordination. This design ensures that each attack objective focuses exclusively on its designated opponent team, treating other teams as neutral entities that may serve as obstacles or potential allies. As in Section VII-A, we employ DeepSets to process the position differences with permutation-invariant aggregation. The resulting aggregated features are concatenated and processed through the MLP g(⋅)g(·) to create objective-specific features. For this scenario, the value function VϕV_φ shares the same architecture, and the policy network integrates the sampled weight vector w with the extracted objective features before processing them with a final MLP to generate action distributions. All MLPs have [32,32][32,32] features and tanh activation function. The same neural networks are used in all the methods compared. As this environment involves adversarial agents, the max value in equation (9) and equation (10) could fail to capture the worst-case opponent behavior and the policy could loose robustness resulting too optimistic. Normally, this problem is addressed in tree search methods by making the agent playing in turns, with a turn corresponding to one node expansion. However, this strategy works only for discrete games with a full separation of agents turns, like chess or go. In our case, the agents are moving simultaneously and the tree search explores simultaneously the actions of all agents. Moreover, applying a strict minimax formulation often leads to overly pessimistic, degenerate policies in such continuous environments, effectively paralyzing the agents. Finally, MPPI in general is not designed to handle adversarial problems so there is no standard way to address this problem. Therefore, rather than assuming a strict worst-case opponent, we introduce a risk-sensitive regularized optimization. We slightly modify the max value characterization in both the sampling-based search to penalize deviations of the opponent team values from a nominal value v0v_0, which represents an expected-case, boundedly rational opponent. Given the state of the observed adversarial agents reconstructed by the observation vector, the nominal value v0v_0 is computed as the expected value of the opponent observed teams at the beginning of the search. Moreover, to assign the objective preference to the opponent teams, we consider the average weight vector w w between agent i solving the tree search and neighboring teammates, computed by agent i from the communicated state of the weight predictor. Let’s denote all the opponents agents observed by agent i as N−sN_-s, we report the modified max value for the tree search: w w =1Ns∑j=1Nswj = 1N_s _j=1^N_sw_j Vmax(n) V_ (n) = = maxi∑j=1Nswij|Vij(n)−v∗|v∗+p∑j=1N−s|w^i⊤(Vij(n)−v0)| _i _j=1^N_sw_ij |V_ij(n)-v^*|v^*+p _j=1^N_-s | w _i (V_ij(n)-v_0 ) | where p acts as a conservatism coefficient that dictates the algorithm’s risk aversion. A value of p=0.5p=0.5 was chosen empirically to balance naive optimism against strict minimax pessimism. We note that a comprehensive game-theoretic analysis of this adversarial adaptation and an ablation over p∈[0,1]p∈[0,1] are scoped as future work, as the primary focus of this paper remains the cooperative multi-objective framework. References [1] M. S. Abid, H. J. Apon, S. Hossain, A. Ahmed, R. Ahshan, and M. H. Lipu (2024) A novel multi-objective optimization based multi-agent deep reinforcement learning approach for microgrid resources planning. Applied Energy 353, p. 122029. Cited by: §I. [2] L. N. Alegre, A. L. Bazzan, D. M. Roijers, A. Nowé, and B. C. da Silva (2023) Sample-efficient multi-objective learning via generalized policy improvement prioritization. arXiv preprint arXiv:2301.07784. Cited by: §I, §I. [3] Z. Aminzare, B. Dey, E. N. Davison, and N. E. Leonard (2020) Cluster synchronization of diffusively coupled nonlinear systems: a contraction-based approach. Journal of Nonlinear Science 30, p. 2235–2257. Cited by: §I. [4] R. Ariizumi, H. Sago, T. Asai, and S. Azuma (2023) Multi-objective reinforcement learning with path integral policy improvement. In 2023 62nd Annual Conference of the Society of Instrument and Control Engineers (SICE), p. 1418–1423. Cited by: §I. [5] A. Bizyaeva, A. Franci, and N. E. Leonard (2022) Nonlinear opinion dynamics with tunable sensitivity. IEEE Transactions on Automatic Control 68 (3), p. 1415–1430. Cited by: §I. [6] M. J. Blondin and M. Hale (2020) An algorithm for multi-objective multi-agent optimization. In 2020 American Control Conference (ACC), p. 1489–1494. Cited by: §I. [7] A. Botros, N. Wilde, A. Sadeghi, J. Alonso-Mora, and S. L. Smith (2024) Regret-based sampling of pareto fronts for multi-objective robot planning problems. IEEE Transactions on Robotics. Cited by: §IV-B. [8] J. Branke (2008) Multiobjective optimization: interactive and evolutionary approaches. Vol. 5252, Springer Science & Business Media. Cited by: §I, §I. [9] C. B. Browne, E. Powley, D. Whitehouse, S. M. Lucas, P. I. Cowling, P. Rohlfshagen, S. Tavener, D. Perez, S. Samothrakis, and S. Colton (2012) A survey of monte carlo tree search methods. IEEE Transactions on Computational Intelligence and AI in games 4 (1), p. 1–43. Cited by: §I. [10] A. Callaghan, K. Mason, and P. Mannion (2025) MOMA-ac: a preference-driven actor-critic framework for continuous multi-objective multi-agent reinforcement learning. Neurocomputing, p. 132032. Cited by: §I. [11] D. Chen, B. Zhou, V. Koltun, and P. Krähenbühl (2020) Learning by cheating. In Conference on Robot Learning, p. 66–75. Cited by: §V. [12] A. P. Dani, S. Chung, and S. Hutchinson (2014) Observer design for stochastic nonlinear systems via contraction-based incremental stability. IEEE Transactions on Automatic Control 60 (3), p. 700–714. Cited by: §IX-A. [13] G. Dixit and K. Tumer (2023) Learning synergies for multi-objective optimization in asymmetric multiagent systems. In Proceedings of the Genetic and Evolutionary Computation Conference, p. 447–455. Cited by: §I. [14] F. Felten, E. Talbi, and G. Danoy (2024) Multi-objective reinforcement learning based on decomposition: a taxonomy and framework. Journal of Artificial Intelligence Research 79, p. 679–723. Cited by: §I. [15] F. Felten, U. Ucak, H. Azmani, G. Peng, W. Röpke, H. Baier, P. Mannion, D. M. Roijers, J. K. Terry, E. Talbi, et al. (2024) MOMAland: a set of benchmarks for multi-objective multi-agent reinforcement learning. arXiv preprint arXiv:2407.16312. Cited by: §I, §I, §VII, §VII. [16] S. Gronauer and K. Diepold (2022) Multi-agent deep reinforcement learning: a survey. Artificial Intelligence Review 55 (2), p. 895–943. Cited by: §I. [17] A. P. Guerreiro, C. M. Fonseca, and L. Paquete (2021) The hypervolume indicator: computational problems and algorithms. ACM Computing Surveys (CSUR) 54 (6), p. 1–42. Cited by: §VII. [18] C. F. Hayes, R. Rădulescu, E. Bargiacchi, J. Källström, M. Macfarlane, M. Reymond, T. Verstraeten, L. M. Zintgraf, R. Dazeley, F. Heintz, et al. (2021) A practical guide to multi-objective reinforcement learning and planning. arXiv preprint arXiv:2103.09568. Cited by: §I. [19] C. F. Hayes, M. Reymond, D. M. Roijers, E. Howley, and P. Mannion (2023) Monte carlo tree search algorithms for risk-aware and multi-objective reinforcement learning. Autonomous Agents and Multi-Agent Systems 37 (2), p. 26. Cited by: §I. [20] T. Hu, B. Luo, C. Yang, and T. Huang (2023) MO-mix: multi-objective multi-agent cooperative decision-making with deep reinforcement learning. IEEE Transactions on Pattern Analysis and Machine Intelligence 45 (10), p. 12098–12112. Cited by: §I, §VII, §VII. [21] W. Lohmiller and J. E. Slotine (1998) On contraction analysis for non-linear systems. Automatica 34 (6), p. 683–696. Cited by: §VI. [22] H. Lu, D. Herman, and Y. Yu (2023) Multi-objective reinforcement learning: convexity, stationarity and pareto optimality. In The Eleventh International Conference on Learning Representations, Cited by: §I. [23] P. Mannion, K. Mason, S. Devlin, J. Duggan, and E. Howley (2016) Multi-objective dynamic dispatch optimisation using multi-agent reinforcement learning. In Proceedings of the 2016 International Conference on Autonomous Agents & Multiagent Systems, p. 1345–1346. Cited by: §I. [24] A. Marino, C. Pacchierotti, and P. Robuffo Giordano (2024-12) Liquid-Graph Time-Constant Network for Multi-Agent Systems Control. In CDC 2024 - 63rd IEEE Conference on Decision and Control, Milan (Italie), Italy. Cited by: §I, §IV-A. [25] A. Mironchenko and H. Ito (2015) Construction of lyapunov functions for interconnected parabolic systems: an iiss approach. SIAM Journal on Control and Optimization 53 (6), p. 3364–3382. Cited by: §IX-C. [26] T. Miyato, T. Kataoka, M. Koyama, and Y. Yoshida (2018) Spectral normalization for generative adversarial networks. arXiv preprint arXiv:1802.05957. Cited by: Remark 2. [27] F. A. Oliehoek, C. Amato, et al. (2016) A concise introduction to decentralized pomdps. Vol. 1, Springer. Cited by: §I. [28] S. Petchrompo, D. W. Coit, A. Brintrup, A. Wannakrairot, and A. K. Parlikad (2022) A review of pareto pruning methods for multi-objective optimization. Computers & Industrial Engineering 167, p. 108022. Cited by: §VII. [29] R. Radulescu (2024) The world is a multi-objective multi-agent system: now what?. In 27th European Conference on Artificial Intelligence, p. 32–38. Cited by: §I, §I. [30] B. Riviere, W. Hönig, M. Anderson, and S. Chung (2021) Neural tree expansion for multi-robot planning in non-cooperative environments. IEEE Robotics and Automation Letters 6 (4), p. 6868–6875. Cited by: §V. [31] D. M. Roijers, P. Vamplew, S. Whiteson, and R. Dazeley (2013) A survey of multi-objective sequential decision-making. Journal of Artificial Intelligence Research 48, p. 67–113. Cited by: §I, §IV-B. [32] D. M. Roijers, S. Whiteson, R. Brachman, and P. Stone (2017) Multi-objective decision making. Springer. Cited by: §I. [33] G. Russo, M. Di Bernardo, and E. D. Sontag (2012) A contraction approach to the hierarchical analysis and design of networked systems. IEEE Transactions on Automatic Control 58 (5), p. 1328–1331. Cited by: §VI, §IX-C. [34] J. Schulman, P. Moritz, S. Levine, M. Jordan, and P. Abbeel (2015) High-dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02438. Cited by: §IV-B. [35] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov (2017) Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. Cited by: §IV-B. [36] D. Shah, Q. Xie, and Z. Xu (2020) Non-asymptotic analysis of monte carlo tree search. In Abstracts of the 2020 SIGMETRICS/Performance Joint International Conference on Measurement and Modeling of Computer Systems, p. 31–32. Cited by: §V. [37] H. Tran, L. Doan, N. H. Luong, and H. T. T. Binh (2023) A two-stage multi-objective evolutionary reinforcement learning framework for continuous robot control. In Proceedings of the Genetic and Evolutionary Computation Conference, p. 577–585. Cited by: §I. [38] H. Tsukamoto, S. Chung, and J. E. Slotine (2021) Contraction theory for nonlinear stability analysis and learning-based control: a tutorial overview. Annual Reviews in Control 52, p. 135–169. Cited by: §VI, Remark 3. [39] P. Vamplew, R. Dazeley, A. Berry, R. Issabekov, and E. Dekker (2011) Empirical evaluation methods for multiobjective reinforcement learning algorithms. Machine learning 84 (1), p. 51–80. Cited by: §I. [40] K. Van Moffaert and A. Nowé (2014) Multi-objective reinforcement learning using sets of pareto dominating policies. The Journal of Machine Learning Research 15 (1), p. 3483–3512. Cited by: §I. [41] W. Wang and M. Sebag (2012) Multi-objective monte-carlo tree search. In Asian conference on machine learning, p. 507–522. Cited by: §I. [42] G. Williams, P. Drews, B. Goldfain, J. M. Rehg, and E. A. Theodorou (2018) Information-theoretic model predictive control: theory and applications to autonomous driving. IEEE Transactions on Robotics 34 (6), p. 1603–1622. Cited by: §I, §I, §V, §V. [43] W. Xia and M. Cao (2011) Clustering in diffusively coupled networks. Automatica 47 (11), p. 2395–2405. Cited by: §IV-A, §IX-A. [44] J. Xu, Y. Tian, P. Ma, D. Rus, S. Sueda, and W. Matusik (2020) Prediction-guided multi-objective reinforcement learning for continuous robot control. In International conference on machine learning, p. 10607–10616. Cited by: §I. [45] L. Yliniemi and K. Tumer (2016) Multi-objective multiagent credit assignment in reinforcement learning and nsga-i. Soft Computing 20 (10), p. 3869–3887. Cited by: §I. [46] M. Zaheer, S. Kottur, S. Ravanbakhsh, B. Poczos, R. R. Salakhutdinov, and A. J. Smola (2017) Deep sets. Advances in neural information processing systems 30. Cited by: §IX-E. [47] Z. Zhang, S. Al-Abri, and F. Zhang (2025) A generalized kuramoto model for opinion dynamics on the unit sphere. Automatica 171, p. 111957. Cited by: §I.