Paper deep dive
Diffusing to Coordinate: Efficient Online Multi-Agent Diffusion Policies
Zhuoran Li, Hai Zhong, Xun Wang, Qingxin Xia, Lihua Zhang, Longbo Huang
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 90%
Last extracted: 7/20/2026, 10:15:28 PM
Summary
The paper introduces OMAD (Online off-policy MARL framework using Diffusion policies), a novel framework for online multi-agent reinforcement learning that utilizes diffusion-based generative models to enhance policy expressiveness and handle multimodal coordination strategies. It addresses the challenge of intractable likelihoods in diffusion models by proposing a relaxed policy objective that maximizes scaled joint entropy, enabling effective exploration without relying on tractable likelihoods. Within the Centralized Training with Decentralized Execution (CTDE) paradigm, OMAD employs a joint distributional value function to optimize decentralized diffusion policies using tractable entropy-augmented targets, ensuring stable coordination. The method demonstrates state-of-the-art performance across 10 diverse tasks in MPE and MAMuJoCo benchmarks, achieving significant improvements in sample efficiency.
Entities (8)
Relation Signals (7)
OMAD → uses → Diffusion Policies
confidence 95% · we propose among the first Online off-policy MARL framework using Diffusion policies (OMAD)
OMAD → operatesunder → CTDE
confidence 92% · within the centralized training with decentralized execution (CTDE) paradigm, we employ a joint distributional value function
OMAD → optimizes → Scaled Joint Entropy
confidence 90% · Our key innovation is a relaxed policy objective that maximizes scaled joint entropy
OMAD → evaluatedon → MPE
confidence 88% · Extensive evaluations on MPE and MAMuJoCo establish our method as the new state-of-the-art
OMAD → evaluatedon → MAMuJoCo
confidence 88% · Extensive evaluations on MPE and MAMuJoCo establish our method as the new state-of-the-art
OMAD → employs → Joint Distributional Value Function
confidence 85% · we employ a joint distributional value function to optimize decentralized diffusion policies.
Diffusion Policies → challenges → MARL
confidence 80% · A major obstacle is that the intractable likelihoods of diffusion models impede entropy-based exploration and coordination.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Online Multi-Agent Reinforcement Learning (MARL) is a prominent framework for efficient agent coordination. Crucially, enhancing policy expressiveness is pivotal for achieving superior performance. Diffusion-based generative models are well-positioned to meet this demand, having demonstrated remarkable expressiveness and multimodal representation in image generation and offline settings. Yet, their potential in online MARL remains largely under-explored. A major obstacle is that the intractable likelihoods of diffusion models impede entropy-based exploration and coordination. To tackle this challenge, we propose among the first \underline{O}nline off-policy \underline{MA}RL framework using \underline{D}iffusion policies (\textbf{OMAD}) to orchestrate coordination. Our key innovation is a relaxed policy objective that maximizes scaled joint entropy, facilitating effective exploration without relying on tractable likelihood. Complementing this, within the centralized training with decentralized execution (CTDE) paradigm, we employ a joint distributional value function to optimize decentralized diffusion policies. It leverages tractable entropy-augmented targets to guide the simultaneous updates of diffusion policies, thereby ensuring stable coordination. Extensive evaluations on MPE and MAMuJoCo establish our method as the new state-of-the-art across $10$ diverse tasks, demonstrating a remarkable $2.5\times$ to $5\times$ improvement in sample efficiency.
Tags
Links
- Source: https://arxiv.org/abs/2602.18291v2
- Canonical: https://arxiv.org/abs/2602.18291v2
Trouble viewing inline? Open PDF directly →
Full Text
101,396 characters extracted from source content.
Expand or collapse full text
Diffusing to Coordinate: Efficient Online Multi-Agent Diffusion Policies Zhuoran Li Hai Zhong Xun Wang Qingxin Xia Lihua Zhang Longbo Huang* Abstract Online Multi-Agent Reinforcement Learning (MARL) is a prominent framework for efficient agent coordination. Crucially, enhancing policy expressiveness is pivotal for achieving superior performance. Diffusion-based generative models are well-positioned to meet this demand, having demonstrated remarkable expressiveness and multimodal representation in image generation and offline settings. Yet, their potential in online MARL remains largely under-explored. A major obstacle is that the intractable likelihoods of diffusion models impede entropy-based exploration and coordination. To tackle this challenge, we propose among the first Online off-policy MARL framework using Diffusion policies (OMAD) to orchestrate coordination. Our key innovation is a relaxed policy objective that maximizes scaled joint entropy, facilitating effective exploration without relying on tractable likelihood. Complementing this, within the centralized training with decentralized execution (CTDE) paradigm, we employ a joint distributional value function to optimize decentralized diffusion policies. It leverages tractable entropy-augmented targets to guide the simultaneous updates of diffusion policies, thereby ensuring stable coordination. Extensive evaluations on MPE and MAMuJoCo establish our method as the new state-of-the-art across 1010 diverse tasks, demonstrating a remarkable 2.5×2.5× to 5×5× improvement in sample efficiency. Machine Learning, ICML 1 Introduction Multi-agent reinforcement learning (MARL (Oliehoek et al., 2008)) provides a robust framework for decision-making in complex systems, ranging from autonomous driving (Zhou et al., 2021) to robot swarms (Chen et al., 2025a). A central challenge in these domains is agent coordination under severe non-stationarity (Tan, 1993). To mitigate this issue, Centralized Training with Decentralized Execution (CTDE) (Bernstein et al., 2002; Rashid et al., 2018) has emerged as the predominant paradigm and has achieved remarkable success. However, the prevailing use of unimodal policy distributions (e.g., Gaussian) struggles to represent the highly complex and multimodal coordination strategies (Wang et al., 2024; Ma et al., 2025), which is essential for multi-agent interactions. In order to address the limitations of policy expressiveness, diffusion-based generative models (Ho et al., 2020) have emerged as a powerful alternative. Through progressive denoising, diffusion models demonstrate exceptional capability in modeling highly multimodal distributions (Song et al., 2021). This strong expressiveness naturally makes them particularly appealing for sequential decision-making (Janner et al., 2022; Wang et al., 2023), where optimal policies often involve diverse strategies. Indeed, diffusion policies have demonstrated strong performance in offline single- and multi-agent settings with expert demonstrations, achieving substantial gains in manipulation and locomotion tasks (Li et al., 2023b; Chi et al., 2023; Huang et al., 2024). However, extending diffusion models to online MARL remains challenging, particularly in achieving efficient training from scratch without relying on offline demonstrations. Specifically, the intractable likelihoods (Ho et al., 2020) inherent in diffusion models hinder efficient exploration. This intractability precludes conventional entropy regularization (Haarnoja et al., 2017). Such incompatibility is severely exacerbated in multi-agent settings where coordinated exploration is essential to navigate non-stationarity and discover cooperative strategies (Liu et al., 2024). While single-agent adaptations exist (Ma et al., 2025; Celik et al., 2025), extending these frameworks to off-policy MARL remains a non-trivial endeavor. We tackle these challenges by proposing the first Online off-policy MARL framework tailored for Diffusion policies called OMAD. Our algorithm is a novel tractable maximum entropy paradigm for multi-agent diffusion policies. We break the computational barriers of prior generative MARL approaches by introducing a tractable relaxed joint policy objective. Under this unified framework, we orchestrate a holistic optimization process: training a centralized distributional critic to capture aleatoric uncertainty, guiding per-agent diffusion policies via synchronized updates, and auto-tuning temperature constraints. Facilitating the high expressiveness of diffusion policies, our framework empowers agents to achieve consistent coordination, setting a robust and scalable standard for online generative MARL. We summarize our key contributions as follows: • We propose the first online off-policy diffusion framework for MARL. To overcome the intractability of likelihood computation, we introduce a relaxed objective based on scaled joint entropy. This formulation extends tractable exploration to the multi-agent domain, enabling agents to effectively coordinate their exploration in high-dimensional joint action spaces. • We develop a unified off-policy learning mechanism within the CTDE paradigm. By leveraging a joint distributional value function, we optimize distinct policies for all agents simultaneously under a single unified objective. This synchronous update strategy effectively mitigates non-stationarity, fostering efficient collaboration and stable convergence. • We evaluate our OMAD algorithm on standard continuous control benchmarks, including Multi-Agent Particle Environments (MPE) and Multi-Agent MuJoCo (MAMuJoCo). Our method establishes a new state-of-the-art across 1010 diverse scenarios, consistently surpassing baselines with a substantial 2.5×2.5× to 5×5× gain in sample efficiency. 2 Related Works Due to space constraints, we highlight key works on online MARL and diffusion policies in RL. A comprehensive discussion is provided in Appendix A. 2.1 Online Multi-Agent Reinforcement Learning Online multi-agent reinforcement learning (MARL) has made substantial progress under the centralized training with decentralized execution (CTDE) paradigm (Oliehoek et al., 2008; Rashid et al., 2018). Representative methods such as MADDPG (Lowe et al., 2017), MAPPO (Yu et al., 2022), COMA (Foerster et al., 2018), and value decomposition approaches including VDN (Sunehag et al., 2018), QMIX (Rashid et al., 2018), and QPLEX (Wang et al., 2021) significantly improve coordination and learning stability in cooperative tasks. More recent works focus on addressing non-stationarity, exploration efficiency, and agent heterogeneity, including RES-QMIX (Pan et al., 2021), SMPE (Kontogiannis et al., 2025) and RCO (Li et al., 2025b). In particular, heterogeneity-aware frameworks such as HARL and its variants (Zhong et al., 2024; Liu et al., 2024) represent the state of the art by explicitly coordinating heterogeneous agents. Despite these advances, existing online MARL methods predominantly rely on unimodal policy representations, limiting their ability to model complex, multi-modal coordination behaviors. 2.2 Diffusion Policies in Reinforcement Learning Diffusion models are powerful generative models with strong capacity for representing expressive and multi-modal distributions (Ho et al., 2020; Song et al., 2021), and have been widely adopted in sequential decision-making, particularly in imitation learning (Xie et al., 2025) and offline reinforcement learning (Wang et al., 2023). Representative approaches such as Diffuser (Janner et al., 2022), Decision Diffuser (Ajay et al., 2023), and Diff-QL (Wang et al., 2023) achieve significant gains by modeling complex action or trajectory distributions, with extensions to offline multi-agent settings improving policy diversity (Li et al., 2023b) and data efficiency (Zhu et al., 2024; Li et al., 2025a). In contrast, extending diffusion policies to online reinforcement learning remains challenging due to intractable likelihoods and inefficient exploration. Recent methods mitigate these issues via value-guided (Psenka et al., 2024; Wang et al., 2024) or entropy-regularized objectives (Celik et al., 2025; Ma et al., 2025), but are largely limited to single-agent settings and do not scale to multi-agent systems with coordinated exploration and non-stationarity. Our work bridges this gap by proposing among the first online off-policy diffusion frameworks tailored for MARL. 3 Preliminary 3.1 MARL and Efficient Value Estimation We study cooperative multi-agent reinforcement learning in the framework of decentralized partially observable Markov decision processes (Dec-POMDPs) (Oliehoek and Amato, 2016). An N-agent cooperative task is represented by the tuple G=⟨ℐ,,,,Π,,ℛ,N,γ⟩G= ,S,O,A, ,P,R,N,γ . Here, ℐ=1,…,NI=\1,…,N\ denotes the agents, S is the global state space, =(1,…,N)O=(O_1,…,O_N) represents the local observations, and =(1,…,N)A=(A_1,…,A_N) means the joint action space. Each agent i selects ai∈ia_i _i according to its policy πi∈Πi _i∈ _i. The environment evolves via (s′∣s,a)P(s s,a), and agents receive rewards from the shared function ℛ:×→ℝR:S×A . The goal of standard MARL is to learn a joint policy =(π1,…,πN) π=( _1,…, _N) maximizing the expected discounted team return [∑t=0∞γtrt]E_ π\! [ _t=0^∞γ^tr_t ], given the discounted factor γ∈[0,1)γ∈[0,1) and the sum of rewards for each agent rt=∑i=1Nritr_t= _i=1^Nr_i^t. To prevent premature convergence to sub-optimal deterministic policies in complex multi-agent interactions (Liu et al., 2024), Maximum Entropy RL framework (Ziebart et al., 2008; Haarnoja et al., 2018) augments the standard return with policy entropy to promote exploration. In multi-agent settings, this translates to maximizing the expected return alongside the joint policy entropy ℋ((⋅|st))H( π(·|s_t)) that: [∑t=0∞γtrt+αℋ((⋅|st))]E_ π [ _t=0^∞γ^tr_t+ ( π(·|s_t)) ], where α regulates the exploration-exploitation trade-off. For decentralized policies (|s)=∏iπi(ai|s) π(a|s)= _i _i(a_i|s) (Zhong et al., 2024), the framework yields a softened Bellman operator, where the soft Q-function is optimized via the Bellman residual: ℒQ(θ) _Q(θ) =(s,a,r,s′)∼[(Qθ(s,a)−r =E_(s,a,r,s ) [(Q_θ(s,a)-r (1) −γ(Qπ(s′,a′)−α∑i=1Nlogπi(ai′|s′)))2], -γ(Q^π(s ,a )-α _i=1^N _i(a _i|s )))^2], and the associated soft policy iteration admits the closed-form policy improvement update: πk+1 _k+1 =argminπDKL(π(⋅|s)∥exp(1αQπk(s,⋅))Z(s)), = _πD_KL\! (π(·|s)\, \|\, \! ( 1αQ _k(s,·) )Z(s) ), (2) where Z(s)=∫exp(1αQπk(s,a′))da′Z(s)= \! ( 1αQ _k(s,a ) )da is the normalizing factor that does not affect policy optimization. To ensure reliable value estimation, we list two complementary techniques: CrossQ and Distributional Q-learning. CrossQ (Bhatt et al., 2024) eliminates target networks by applying Batch Normalization to concatenated state-action pairs and applying a stop-gradient to the next-state estimates for high update-to-data ratios. Distributional Q-learning (Bellemare et al., 2017) models the full return distribution Zϕ(s,a)Z_φ(s,a) rather than its expectation Qϕ(s,a)=[Zϕ(s,a)]Q_φ(s,a)=E[Z_φ(s,a)] as higher-order return statistics to provide richer signals, mitigating the high variance of value functions. 3.2 Diffusion policy Diffusion policy (Ma et al., 2025; Dong et al., 2025; Wang et al., 2025b; Celik et al., 2025) parameterizes the policy as the terminal distribution of a state-conditioned process, governed by continuous-time Ornstein–Uhlenbeck (Särkkä and Solin, 2019) (OU) dynamics over state s and t∈[0,T]t∈[0,T], dat=−βtatdt+η2βtdBt,a0∼π0(⋅|s). _t=- _ta_tdt+η 2 _t\,dB_t,a_0 _0(·|s). (3) Here, βt _t and η are diffusion and drift coefficients, BtB_t is standard Brownian motion, and π0 _0 is the target policy. Let πt _t denote the marginal at time t; with a proper schedule, this distribution converges to πT≈(0,η2I) _T (0,η^2I). Denoising diffusion policy is defined by the reverse dynamics (Eq. 4) starting from aT∼(0,η2I)a_T (0,η^2I), where a network fθf_θ approximates the score ∇atlogπt _a_t _t for denoising, dat=(−βtat−2η2βtfθ(at,s,t))dt+η2βtdB~t. _t=(- _ta_t-2η^2 _tf_θ(a_t,s,t))dt+η 2 _t\,d B_t. (4) This yields samples a0∼πθa_0 _θ. Theoretically, accurate score estimation fθ≈∇logπtf_θ≈∇ _t ensures πθ _θ recovers the target π0 _0. Applying Euler-Maruyama discretization (Särkkä and Solin, 2019), the discrete forward and reverse dynamics are: ah+1 a_h+1 =ah−βhahδ+ϵh, =a_h- _ha_h\,δ+ _h, (5) ah−1 a_h-1 =ah+(βhah+2η2βhfθ(ah,s,hT/H))δ+ξh. =a_h+( _ha_h+2η^2 _hf_θ(a_h,s,hT/H))\,δ+ _h. (6) Here, ϵh,ξh∼(0,2η2βhδI) _h, _h (0,2η^2 _hδ I), step size δ=T/Hδ=T/H, and ah≡ath/Ta_h≡ a_th/T. Under EM discretization, the noising and denoising processes exhibit the following joint distributions: π(a0:H|s) π(a_0:H|s) =π(a0|s)∏h=0H−1π(ah+1|ah,s), =π(a_0|s) _h=0^H-1π(a_h+1|a_h,s), (7) πθ(a0:H|s) _θ(a_0:H|s) =πθ(aH|s)∏h=1Hπθ(ah−1|ah,s). = _θ(a_H|s) _h=1^H _θ(a_h-1|a_h,s). (8) The output a=a0a=a_0 constitutes the action, providing expressiveness for complex, multimodal distributions. This helps expose inherent computational bottlenecks in entropy estimation, motivating our proposed online MARL framework. 4 Bridge to OMAD and Theoretical Insight Despite the generative prowess of diffusion models, their integration into online MARL is obstructed by three inherent mechanism mismatches as follows. First, entropy-based exploration is hindered by intractable likelihoods. In contrast to the Gaussian policies, diffusion models lack tractable likelihoods (Ho et al., 2020). This prohibits exact joint entropy computation, blocking maximum entropy objectives (Ma et al., 2025) crucial for preventing premature convergence in online MARL. Second, an architectural conflict with CTDE complicates deployment. Standard monolithic backbones violate (Zhu et al., 2024) the independence required for decentralized execution. Conversely, naive independent models fail to capture complex joint dependencies, necessitating a factorized design with decentralization to facilitate data collection. Finally, a misalignment in coordination optimization persists as a significant open challenge. The policy’s iterative denoising nature inherently amplifies the difficulty of optimizing for sustained step-wise inter-agent coordination (Pan et al., 2021; Li et al., 2025b). Formulating a loss that effectively guides this multi-step generation towards global cooperation remains a significant algorithmic challenge. To overcome these obstacles, we first establish a theoretical foundation for tractable entropy estimation. Assuming the joint policy factorizes as (a|s)=∏i=1Nπθi(ai|s) _θ(a|s)= _i=1^N _ _i(a^i|s), we leverage variational properties to derive a tractable lower bound, formalized as follows: Theorem 4.1. (Entropy Lower Bound for Decentralized Diffusion Policies) Given that the joint policy is factorized into independent diffusion processes, the entropy of the joint distribution ℋ((a|s))H( _θ(a|s)) is lower-bounded by the sum of individual variational bounds: ℋ((a|s)) ( _θ(a|s)) ≥∑i=1Nlπθi(ai|s), ≥ _i=1^Nl_ _ _i(a^i|s), (9) where lπθi(ai|s)=πθi[logπi(a1:Hi|a0i,s)πθi(a0:Hi|s)]l_ _ _i(a^i|s)=E_ _ _i [ _i(a^i_1:H|a^i_0,s) _ _i(a^i_0:H|s) ] is the evidence lower bound for each agent. The expectation is taken with respect to the agent’s diffusion policy πθi _ _i, representing the entire sequential trajectory generation process. Proof. (Sketch) The proof exploits the factorized structure of our multi-agent policy, which allows the intractable joint entropy to decompose into a summation of individual marginal entropies. We then derive a tractable evidence lower bound (ELBO) for each agent’s diffusion trajectory via variational inference to complete the proof. Detailed derivations are provided in Appendix B. ∎ Since the exact computation of the joint policy entropy is intractable, the derived lower bound serves as a tractable surrogate for approximating the Maximum Entropy RL objective (Haarnoja et al., 2017). Facilitating this helps us effectively incentivize exploration while harnessing the high expressiveness of diffusion policies. Under the CTDE framework, this formulation transforms joint entropy into a computable metric, providing unified global exploration bonuses that actively incentivize robust exploration while unlocking the full expressiveness of diffusion policies. Crucially, this theoretical foundation drives our proposed OMAD algorithm, which goes far beyond a simple policy factorization. Rather, OMAD is a holistic, system-level architecture meticulously designed for efficient multi-agent collaboration. By integrating this tailored entropy bound with our Centralized Joint Distributional Critic, OMAD actively resolves the severe non-stationarity and compounded variance of multi-agent interactions. This comprehensive design ensures agents do not merely act independently, but achieve highly stable, synchronized off-policy learning for state-of-the-art cooperative performance. Figure 1: The CTDE framework of OMAD. The left panel illustrates Decentralized Execution, where agents independently sample actions via a denoising diffusion process. The right panel depicts Centralized Training, where a shared Distributional Critic provides unified Value Guidance to jointly optimize policies, stabilized by adaptive regularization for the entropy evidence lower bound. 5 The OMAD Method This section introduces our proposed diffusion policy algorithm for online multi-agent reinforcement learning, with a systemic overview depicted in Figure 1. To tackle the challenges of applying diffusion models in online MARL, our framework integrates two strategic designs. (i) First, we construct a specialized network architecture that seamlessly unifies policy learning with iterative denoising action generation. (i) Second, to strictly adhere to the CTDE paradigm, we introduce a centralized distributional critic. This critic plays a pivotal role in guiding synchronized policy training by maximizing a scaled joint entropy Evidence Lower Bound (ELBO). We detail the formulation and implementation of these components below. 5.1 Decentralized Diffusion Policy Formulation To scale diffusion-based control to multi-agent domains, we employ a factorized policy architecture (a|s)=∏i=1Nπθi(ai|s) _θ(a|s)= _i=1^N _ _i(a^i|s) (Zhong et al., 2024). This architecture, corresponding to the decentralized execution phase (blue region in Figure 1), eschews joint space modeling to ensure efficient data collection while retaining policy expressiveness. We instantiate a reverse-time SDE introduced in Sec. 3.2 for each agent i, parameterizing the drift with a specific score network fθi(ati,s,t)≈∇atilogπi,t(ati|s)f_ _i(a_t^i,s,t)≈ _a_t^i _i,t(a_t^i|s). This decoupling facilitates efficient parallel sampling via the Euler-Maruyama solver, where the synchronized generation over H steps is governed by: ah−1i a_h-1^i =ahi+(βhahi+2η2βhfθi(ahi,s,th))⏟Drift (Score Guidance)δ+ξhi, =a_h^i+ ( _ha_h^i+2η^2 _hf_ _i(a_h^i,s,t_h))_Drift (Score Guidance)\,δ+ _h^i, (10) where th=hT/Ht_h=hT/H and ξhi _h^i follows Eq. (4). The action a0ia^i_0 is obtained by iteratively denoising aHi∼(0,η2I)a^i_H (0,η^2I). Such scheme effectively captures complex multi-modal joint distributions through the composition of individual policies. 5.2 Online Centralized Training the Diffusion Policy With the tractable entropy lower bound established in Theorem 4.1, we can now operationalize the Maximum Entropy principle within our multi-agent framework. Specifically, we substitute the computationally intractable exact entropy with the derived ELBO, utilizing it as a practical surrogate to guide exploration. This substitution enables stable and effective policy optimization even in high-dimensional joint action spaces. In this subsection, we formulate the overall objective for online centralized training, defining how the diffusion policy is jointly optimized with the scaled entropy regularization to balance reward maximization and exploration. This process corresponds to the orange-shaded phase on the right side of Figure 1. Tractable Maximum Entropy MARL Objective. Leveraging the derived entropy evidence lower bound in Theorem 4.1, we formulate a tractable surrogate objective for Maximum Entropy MARL. In contrast to prior works that eschew entropy regularization (Ma et al., 2025) due to the intractable likelihoods of implicit policies, we explicitly incorporate the variational lower bound lπθil_ _ _i as an intrinsic exploration bonus. The resulting joint objective is to maximize the cumulative discounted sum of the reward and the scaled entropy surrogate: J((⋅|s)):=[∑t=0∞∑i=1Nγt(rti+αlπθi(ati,st))]. J( π_ θ(·|s))=E_ _θ [ _t=0^∞ _i=1^Nγ^t(r_t^i+α l_ _ _i(a^i_t,s_t)) ]. (11) Here, α>0α>0 is the temperature parameter regulating the stochasticity of the policy, consistent with the maximum entropy reinforcement learning framework (Haarnoja et al., 2018). This objective fundamentally transforms the optimization landscape: by maximizing Eq. (11), agents are driven to maximize expected returns while simultaneously maintaining high stochasticity in their diffusion policies, thereby preventing premature convergence. To effectively optimize this tractable entropy-regularized objective, we adopt the CTDE paradigm. Crucially, we introduce a centralized distributional critic to model the value distribution of the joint state-action pairs, providing a more robust signal to guide the diffusion policy optimization. Centralized Joint Distributional Critic Learning. To tackle the inherent multimodality of multi-agent diffusion, we propose a Joint Distributional Critic Zϕ(s,a)Z_φ(s,a). Departing from standard CTDE methods that compress returns into the expected joint return Qϕ(s,a)=[Zϕ(s,a)]Q_φ(s,a)=E[Z_φ(s,a)], we model the full value distribution (Bellemare et al., 2017). Crucially, this disentangles true coordination signals from the compounded stochasticity of interacting diffusion policies for robust supervision that stabilizes joint optimization. We implement the critic (located in the top-right of Figure 1) using the CrossQ architecture (Bhatt et al., 2024) adapted for the multi-agent domain. We define the distributional Bellman operator applied to the joint return distribution. Specifically, the target distribution is constructed by augmenting shared rewards with the derived sum of individual entropy lower bounds, thereby enforcing a unified objective: Zϕ(s,a):=∑i=1Nri+γ(Zϕ(s′,a′)+α∑i=1Nlπθi′((a′)i|s′)), _φ(s,a)= _i=1^Nr^i+γ(Z_φ(s ,a )+α _i=1^Nl_ _θ _i((a )^i|s )), (12) where a′=((a′)1,…,(a′)N)a =((a )^1,…,(a )^N) denotes the joint action sampled from the target diffusion policies πθi _ _i given the next state s′s . Here, the equality holds in distribution. This formulation ensures that the critic evaluates the global quality of joint actions while accounting for the exploration bonuses of all agents, thereby aligning individual incentives with the global objective, which is written as: ℒ(ϕ)=[KL(Zϕ(s,a),sg(Zϕ(s,a)))]+ξℋ(Zϕ(s,a)). (φ)=E[KL(Z_φ(s,a),sg(TZ_φ(s,a)))]+ (Z_φ(s,a)). (13) To efficiently optimize this target, we instantiate a multi-agent CrossQ (Bhatt et al., 2024) mechanism by combining stop-gradients sg(⋅)sg(·) with input Batch Normalization, enabling high data efficiency without divergence. Furthermore, our entropy-regularized distributional critic (ZϕZ_φ with coefficient ξ) captures the granular uncertainty of joint interactions, surpassing expectation-based baselines. Synchronized Diffusion Policy Optimization. We formulate the multi-agent policy optimization as an iterative projection problem, aiming to align the joint policy π_ θ with the distribution derived by the learned critic, i.e., (a|s)∝exp(1αQϕ(s,a)) π_ θ(a|s) ( 1αQ_φ(s,a)) corresponding to Eq. (2) (which is located in the bottom-right of Figure 1). This corresponds to minimizing the KL divergence term, expressed as: DKL((⋅|s)∥1Zexp(1αQϕ(s,⋅)))D_KL( π_ θ(·|s)\| 1Z ( 1αQ_φ(s,·))). However, direct optimization is intractable due to the iterative diffusion process. We thus derive a tractable surrogate loss by expanding the divergence over latent trajectories, yielding a unified synchronized objective: ℒ(θ)=[∑i=1Nlogπθi(aHi|s)−1αQϕ(s,a0) (θ)=E [ _i=1^N _ _i(a_H^i|s)- 1αQ_φ(s,a_0) (14) +∑i=1N∑h=1Hlogπθi(ah−1i|ahi,s)πi(ahi|ah−1i,s)]+logZ(s). + _i=1^N _h=1^H _ _i(a_h-1^i|a_h^i,s) _i(a_h^i|a_h-1^i,s) ]+ Z(s). Here, a0a_0 denotes the terminal joint action synthesized through the synchronized reverse diffusion trajectories aHi,…,a0ii=1N\a_H^i,…,a_0^i\_i=1^N governed by Eq. (10), πi(ahi|ah−1i,s) _i(a_h^i|a_h-1^i,s) represents the forward noising distribution constructed with 1Z(s)exp(1αQϕ(s,⋅)) 1Z(s) ( 1αQ_φ(s,·)) as the target policy, and Qϕ(s,a0)=[Zϕ(s,a0)]Q_φ(s,a_0)=E[Z_φ(s,a_0)] denotes the expectation of Zϕ(s,a0)Z_φ(s,a_0). Crucially, this formulation establishes a shared optimization landscape where agents perform synchronized updates via a holistic loss. In stark contrast to decoupled approaches as HARL (Zhong et al., 2024; Liu et al., 2024) that rely on fragmented local losses, our method orchestrates a unified update guided by the global value function, ensuring superior coordination stability and efficiency. Detailed derivations of the objective function are provided in the Appendix C. Variational Surrogate for Temperature Auto-Tuning. To dynamically regulate the exploration-exploitation trade-off without manual tuning, we formulate the temperature α adjustment as a dual constrained optimization problem (shaded green in Figure 1). Uniquely, we adapt the maximum entropy principle to the diffusion setting by enforcing a constraint on the joint variational lower bound rather than the intractable exact entropy. Specifically, we optimize α to maintain the aggregate ELBO above a target threshold: ℒ(α)=α(ℋtarget−∑i=1Nlπθi(ai|s)).L(α)=α (H_target- _i=1^Nl_ _ _i(a^i|s) ). (15) Here, ℋtargetH_target defines the minimum exploration budget. Minimizing this objective automatically scales α to maintain the aggregate ELBO at the target level. OMAD establishes a tractable paradigm by deriving a decomposable entropy lower bound aligned with diffusion denoising. Our framework integrates a centralized distributional critic, synchronized policy updates, and auto-tuned temperature. This design ensures robust high-entropy coordination while preserving decentralized execution, providing a scalable foundation for diffusion-based online MARL. 5.3 Algorithm and Discussion Algorithm 1 Online Multi-Agent Diffusion Policy ( OMAD) 1: Initialize: Distributional state-action function Zϕ(s,a)Z_φ(s,a), diffusion policy and the target term πθi,πθi′ _ _i, _θ _i for i=1,2,…,Ni=1,2,...,N agents, temperature α, policy delay dld_l, target entropy ℋtargetH_target, learning rate η, replay buffer D and the threshold buffer size LinitL_init. // Initialization 2: for m=1m=1 to M episodes do 3: Sample trajectory (s0,a0,r0,s1,a1,…,sT,aT,rT)(s_0,a_0,r_0,s_1,a_1,...,s_T,a_T,r_T) for T timesteps using πθi′ _θ _i and insert them into D. // Trajectory Generation 4: if Buffer length L>LinitL_D>L_init then 5: Sample B=(s,a,r,s′)B=\(s,a,r,s )\ from D. // Sampling 6: Sample actions a′=((a′)1,…,(a′)N)a =((a )^1,...,(a )^N) using πθi′ _θ _i, calculate ℒ(ϕ)L(φ) using Eq. (13) and optimize ϕφ. // Critic Optimization 7: if m mod dl=0m mod d_l=0 then 8: Sample actions a=(a1,…,aN)a=(a^1,...,a^N) using πθi _ _i, calculate ℒ(θ)L(θ) using Eq. (14) and jointly optimize θii=1N\ _i\_i=1^N. // Diffusion Policy Optimization 9: end if 10: Calculate lπθil_ _ _i and optimize the temperature α via minimizing Eq. (15). // Temperature Optimization 11: Update target networks θi′←ρθi′+(1−ρ)θiθ _i←ρθ _i+(1-ρ) _i. // Target Network Optimization 12: end if 13: end for The training procedure of the proposed online multi-agent diffusion policy is summarized in Algorithm 1. Line 1 initializes the distributional critic, diffusion policies for all agents, target networks, and the replay buffer. Line 3 collects trajectories by executing the target diffusion policies and stores the resulting transitions for off-policy learning. When the replay buffer is sufficiently populated, Line 5 samples mini-batches for training. Line 6 updates the distributional state–action value function using actions sampled from the target diffusion policies. Line 8 jointly optimizes the diffusion policies by minimizing the diffusion-based policy objective guided by the learned distributional critic. Line 9 adapts the entropy temperature to enforce the target entropy, and Line 10 softly updates the target policy networks to stabilize training. OMAD is a novel online diffusion-based MARL algorithm, departing from restrictive Gaussian policies used in methods, including HARL (Zhong et al., 2024; Liu et al., 2024). By modeling policies as generative diffusion processes, OMAD captures complex, multi-modal joint action distributions essential for sophisticated coordination. Distinct from discrete-time (Wang et al., 2024; Ma et al., 2025) or deterministic ODE-based approaches (Dong et al., 2025), OMAD employs an SDE formulation to ensure sustained stochasticity. Advancing beyond DIME (Celik et al., 2025), we derive a factorized entropy bound with a centralized distributional critic to address non-stationarity, offering a sample-efficient off-policy alternative to HADQ (Lin and Lee, 2026). Figure 2: Learning curves comparing OMAD with state-of-the-art online MARL baselines (HATD3 and HASAC) and two representative extensions of the diffusion policies (MADPMD and MASDAC) on MPE and MAMuJoCo benchmarks. The plots report the average episode return over training steps, averaged across 5 random seeds, with shaded regions indicating one standard deviation. Results demonstrate that OMAD consistently achieves faster convergence and superior final performance across both low-dimensional MPE tasks and high-dimensional continuous control environments. Overall, OMAD establishes a centralized distributional critic with synchronized diffusion policies within a tractable variational framework, providing a robust off-policy paradigm for scalable, sample-efficient coordination. This formulation hinges on two complementary pillars: a tractable ELBO variant that enables practical maximum entropy exploration, and an entropy-augmented joint critic that robustly synchronizes decentralized policy updates. We next empirically validate how OMAD’s high expressiveness manifests as efficient exploration and superior performance. 6 Experiments We empirically validate the efficacy of OMAD across diverse continuous-control multi-agent benchmarks. In this section, we detail our experimental setup and present a rigorous comparative analysis against state-of-the-art baselines. Notably, we visually demonstrate how diffusion expressiveness unlocks superior state exploration. Finally, we conduct ablation studies to isolate the contribution of each key algorithmic component, empirically justifying the rationality of our proposed framework.111Code is available at: https://github.com/lizr16/OMAD 6.1 Experiment Setup Environments. We evaluate our method on two widely adopted multi-agent benchmarks: the Multi-Agent Particle Environments (MPE) (Lowe et al., 2017) and the challenging, high-dimensional Multi-Agent MuJoCo (MAMuJoCo) tasks (Peng et al., 2021). We evaluate on representative tasks from MPE (Physical Deception, Cooperative Navigation) involving particle cooperation, and MAMuJoCo (HalfCheetah, Ant, Walker, Swimmer) requiring coordinated locomotion. Details are provided in Appendix D.1. Baselines: We compare our algorithm with the following state-of-the-art baseline online MARL algorithms: HATD3 (Zhong et al., 2024) and HASAC (Liu et al., 2024). Besides, we compare our algorithm with the extension of the single-agent diffusion-based policy as MADPMD and MASDAC (Ma et al., 2025). Each algorithm is executed for 55 random seeds and the mean performance and the standard deviation for 1010 episodes are presented. A detailed description of hyperparameters, neural network structures, and setup can be found in Appendix D.2. 6.2 Experiment Results The experimental results in different tasks are shown in Figure 2. Our method consistently outperforms all compared baselines across a wide range of cooperative and competitive tasks, including MPE and challenging MAMuJoCo benchmarks. In low-dimensional MPE tasks, OMAD achieves faster convergence and higher final returns, reducing training steps to reach baseline peaks by up to 5×5×. This indicates improved coordination efficiency and training stability compared to both value-based and diffusion-based baselines. Notably, while MADPMD and MASDAC benefit from diffusion modeling, they exhibit slower convergence or inferior asymptotic performance, suggesting that directly extending diffusion policies to multi-agent settings is insufficient without effective centralized training and value guidance. In high-dimensional continuous control environments such as Ant (2×42× 4, 2×42× 4d, 4×24× 2), HalfCheetah (2×32× 3, 6×16× 1), Walker2d (2×32× 3), and Swimmer (2×12× 1), OMAD demonstrates substantial performance gains with lower variance and exhibits a consistent 2.5∼5×2.5 5× speedup in sample efficiency compared to the strongest baselines across these complex dynamics. These results highlight the scalability of our approach and its robustness in complex dynamics with strong inter-agent coupling. Overall, the empirical results validate that integrating diffusion-based policy generation with centralized value modeling leads to more stable learning and superior performance in both low- and high-dimensional multi-agent environments. Figure 3: State coverage comparison on representative dimensions (11 and 2121) at 250250k steps. We visualize the state occupancy within the replay buffers for HATD3, HASAC, and OMAD. Colored regions (red, green, blue and orange) indicate visited states. OMAD achieves the broadest coverage, where orange regions are uniquely explored by OMAD, demonstrating superior exploration. To empirically validate the exploration benefits conferred by the high expressiveness of our diffusion policies, we analyze the state distributions accumulated in the replay buffers after the first 250250k training steps under the Ant 2×42× 4 task. We discretize the 2D state space (dimensions 11 and 2121 corresponding to the agent’s spatial navigation) over the range [−19,18]×[−16,19][-19,18]×[-16,19] with a grid interval of 1.01.0. Specifically, the space is partitioned using half-unit boundaries around integer centers, resulting in 38×36=136838× 36=1368 bins in total. Under this metric, OMAD achieves the broadest coverage of 68.3%68.3\% (both blue and orange areas), representing a significant relative improvement of 41%41\% and 24%24\% over HATD3 (48.4%48.4\%) and HASAC (55.0%55.0\%), respectively. The regions highlighted in orange represent state spaces uniquely explored by OMAD, providing compelling evidence that the superior expressiveness of our entropy-regularized diffusion policy empowers the agent to escape local optima and cover a substantially wider solution space. 6.3 Ablation Study To investigate the sensitivity of OMAD, we perform a detailed ablation study on the Ant 2×42× 4 task (10610^6 steps) to isolate the impact of three key components: (1) distributional hyperparameters (VmaxV_ and atom count); (2) diffusion denoising steps; and (3) the scaled entropy coefficient α. For a more comprehensive ablation study, additional results on the HalfCheetah 6×16× 1 task are included in the Appendix D.3.2. Impact of Distributional Hyperparameters. Figure 4 (Left) highlights the sensitivity to the value support upper bound, VmaxV_ . Low thresholds (e.g., 200200) severely hamper performance due to distribution truncation, whereas performance stabilizes around Vmax=1000V_ =1000, which effectively covers the full support of expected returns. Regarding distribution granularity (Figure 4, Right), while generally robust, we identify 100100 intervals (101101 discrete support points, or atoms). Lower atom resolutions fail to capture distribution complexity, while higher ones yield diminishing returns. We thus adopt Vmax=1200V_ =1200 and 101101 atoms as our default configuration under this task. Figure 4: Ablation study on Distributional Q-function hyperparameters. Left: The sensitivity of the agent’s performance to the support upper bound VmaxV_ . Right: The impact of the discretization resolution on performance for different number of atoms. Impact of Denoising Steps. We analyze the performance-efficiency trade-off by varying denoising steps (Figure 5). Performance saturates at 88 steps (≈6000≈ 6000 return), matching 1212 and 1616-step models, whereas 22 and 44-step variants underperform (<4500<4500). Conversely, computational costs increase linearly: raising steps from 88 to 1616 increases training time from 2222h to 2626h and latency by ∼3 3s. Consequently, we select 88 steps as the optimal balance, achieving asymptotic performance with minimal overhead. Figure 5: Ablation study on the number of denoising steps. Left: Episode return curves during training for varying denoising steps. Right: The trade-off between computational cost (training/inference time) and the number of steps. Impact of Adaptive Entropy Regularization. As illustrated in Figure 6, training OMAD algorithm exhibits high sensitivity to the entropy coefficient α. A large fixed coefficient (α=0.1α=0.1) injects excessive stochasticity, destabilizing the denoising process and preventing convergence. Conversely, while manually tuned smaller values (α∈0.001,0.01α∈\0.001,0.01\) yield high returns, they lack adaptability. Crucially, our Auto-Tuning mechanism outperforms this rigid paradigm. By dynamically modulating α, it flexibly balances exploration and exploitation, matching the peak performance (≈6000≈ 6000) of the best fixed settings. This adaptive capability eliminates the need for exhaustive hyperparameter search, facilitating the efficient and robust training of expressive diffusion policies. Figure 6: Efficacy of Auto-Tuning vs. Fixed Entropy Coefficients. 7 Conclusion We present OMAD, a novel algorithm that unlocks online diffusion-based MARL via a tractable variational lower bound for joint entropy. By synergizing this principled exploration with a centralized distributional critic, OMAD robustly mitigates non-stationarity and ensures stable coordination. Empirical results across MPE and MAMuJoCo confirm that OMAD establishes a new state-of-the-art, significantly outperforming both strong value-based baselines and naive diffusion extensions in sample efficiency and asymptotic performance. Acknowledgement This work was supported by the National Natural Science Foundation of China Grant 52494974. Impact Statement This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here. References J. Ackermann, V. Gabler, T. Osa, and M. Sugiyama (2019) Reducing overestimation bias in multi-agent domains using double centralized critics. Advances in Neural Information Processing Systems: Deep Reinforcement Learning Workshop. Cited by: §A.1. A. Ajay, Y. Du, A. Gupta, J. B. Tenenbaum, T. S. Jaakkola, and P. Agrawal (2023) Is conditional generative modeling all you need for decision making?. In The Eleventh International Conference on Learning Representations, Cited by: §A.2, §2.2. M. G. Bellemare, W. Dabney, and R. Munos (2017) A distributional perspective on reinforcement learning. In International conference on machine learning, p. 449–458. Cited by: §3.1, §5.2. D. S. Bernstein, R. Givan, N. Immerman, and S. Zilberstein (2002) The complexity of decentralized control of markov decision processes. Mathematics of operations research 27 (4), p. 819–840. Cited by: §1. A. Bhatt, D. Palenicek, B. Belousov, M. Argus, A. Amiranashvili, T. Brox, and J. Peters (2024) Crossq: batch normalization in deep reinforcement learning for greater sample efficiency and simplicity. In The Twelfth International Conference on Learning Representations, Vol. 2024, p. 55293–55311. Cited by: §D.2, §3.1, §5.2, §5.2. T. Brooks, B. Peebles, C. Holmes, W. DePue, Y. Guo, L. Jing, D. Schnurr, J. Taylor, T. Luhman, E. Luhman, C. Ng, R. Wang, and A. Ramesh (2024) Video generation models as world simulators. OpenAI Blog. External Links: Link Cited by: §A.2. O. Celik, Z. Li, D. Blessing, G. Li, D. Palenicek, J. Peters, G. Chalvatzaki, and G. Neumann (2025) DIME: diffusion-based maximum entropy reinforcement learning. In Forty-second International Conference on Machine Learning, Cited by: §A.3, §A.3, Appendix B, Appendix C, Appendix C, Appendix C, §1, §2.2, §3.2, §5.3. H. Chen, C. Lu, C. Ying, H. Su, and J. Zhu (2023) Offline reinforcement learning via high-fidelity generative behavior modeling. In The Eleventh International Conference on Learning Representations, Cited by: §A.2. J. Chen, Z. Yang, H. G. Xu, D. Zhang, and G. Mylonas (2025a) Multi-agent systems for robotic autonomy with llms. In Proceedings of the Computer Vision and Pattern Recognition Conference, p. 4194–4204. Cited by: §1. Y. Chen, K. Yang, J. Tao, and J. Lyu (2025b) Novelty-guided data reuse for efficient and diversified multi-agent reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 39, p. 15930–15938. Cited by: §A.1. C. Chi, S. Feng, Y. Du, Z. Xu, E. Cousineau, B. Burchfiel, and S. Song (2023) Diffusion policy: visuomotor policy learning via action diffusion. In Proceedings of Robotics: Science and Systems (RSS), Cited by: §A.2, §1. C. S. de Witt, T. Gupta, D. Makoviichuk, V. Makoviychuk, P. H. Torr, M. Sun, and S. Whiteson (2020) Is independent learning all you need in the starcraft multi-agent challenge?. arXiv preprint arXiv:2011.09533. Cited by: §A.1. S. Ding, K. Hu, Z. Zhang, K. Ren, W. Zhang, J. Yu, J. Wang, and Y. Shi (2024) Diffusion-based reinforcement learning via q-weighted variational policy optimization. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, Cited by: §A.3. X. Dong, J. Cheng, and X. S. Zhang (2025) Maximum entropy reinforcement learning with diffusion policy. In Forty-second International Conference on Machine Learning, Cited by: §A.3, §A.3, §A.3, §3.2, §5.3. L. Fang, R. Liu, J. Zhang, W. Wang, and B. Jing (2024) Diffusion actor-critic: formulating constrained policy iteration as diffusion noise regression for offline reinforcement learning. In The Thirteenth International Conference on Learning Representations, Cited by: §A.2. J. Foerster, G. Farquhar, T. Afouras, N. Nardelli, and S. Whiteson (2018) Counterfactual multi-agent policy gradients. In Proceedings of the AAAI conference on artificial intelligence, Vol. 32. Cited by: §A.1, §2.1. Y. Fu, Y. Zhu, J. Zhao, J. Chai, and D. Zhao (2025) INS: interaction-aware synthesis to enhance offline multi-agent reinforcement learning. In The Thirteenth International Conference on Learning Representations, External Links: Link Cited by: §A.2. T. Haarnoja, H. Tang, P. Abbeel, and S. Levine (2017) Reinforcement learning with deep energy-based policies. In International conference on machine learning, p. 1352–1361. Cited by: §1, §4. T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine (2018) Soft actor-critic: off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, p. 1861–1870. Cited by: §3.1, §5.2. P. Hansen-Estruch, I. Kostrikov, M. Janner, J. G. Kuba, and S. Levine (2023) Idql: implicit q-learning as an actor-critic method with diffusion policies. arXiv preprint arXiv:2304.10573. Cited by: §A.2. J. Ho, A. Jain, and P. Abbeel (2020) Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems 33, p. 6840–6851. Cited by: §A.2, §1, §1, §2.2, §4. X. Hu, F. Wang, T. Zhang, and Z. Cui (2025) Value diffusion reinforcement learning. Advances in Neural Information Processing Systems 38, p. 72674–72698. Cited by: §A.2. C. Hua, J. Gu, and Y. Tang (2025) Continuous q-score matching: diffusion guided reinforcement learning for continuous-time control. Advances in Neural Information Processing Systems 38, p. 12573–12602. Cited by: §A.3. X. Huang, Y. Chi, R. Wang, Z. Li, X. B. Peng, S. Shao, B. Nikolic, and K. Sreenath (2024) DiffuseLoco: real-time legged locomotion control with diffusion from offline datasets. In 8th Annual Conference on Robot Learning, External Links: Link Cited by: §1. X. Huang, Y. Chi, R. Wang, Z. Li, X. B. Peng, S. Shao, B. Nikolic, and K. Sreenath (2025) DiffuseLoco: real-time legged locomotion control with diffusion from offline datasets. In Conference on Robot Learning, p. 1567–1589. Cited by: §A.2. M. Janner, Y. Du, J. Tenenbaum, and S. Levine (2022) Planning with diffusion for flexible behavior synthesis. In International Conference on Machine Learning, Cited by: §A.2, §1, §2.2. W. Jung, S. Hong, D. Yoon, K. Lee, and W. Lim (2025) Agent-centric actor-critic for asynchronous multi-agent reinforcement learning. In Forty-second International Conference on Machine Learning, Cited by: §A.1. B. Kang, X. Ma, C. Du, T. Pang, and S. Yan (2023) Efficient diffusion policies for offline reinforcement learning. Advances in Neural Information Processing Systems 36, p. 67195–67212. Cited by: §A.2. A. Kontogiannis, K. Papathanasiou, Y. Shen, G. Stamou, M. M. Zavlanos, and G. Vouros (2025) Enhancing cooperative multi-agent reinforcement learning with state modelling and adversarial exploration. In Forty-second International Conference on Machine Learning, Cited by: §A.1, §2.1. P. Langley (2000) Crafting papers on machine learning. In Proceedings of the 17th International Conference on Machine Learning (ICML 2000), P. Langley (Ed.), Stanford, CA, p. 1207–1216. Cited by: §D.3.4. J. Lee, S. Yun, T. Yun, and J. Park (2024) Gta: generative trajectory augmentation with guidance for offline reinforcement learning. Advances in Neural Information Processing Systems 37, p. 56766–56801. Cited by: §A.2. C. Li, Z. Deng, C. Lin, W. Chen, Y. Fu, W. Liu, C. Wen, C. Wang, and S. Shen (2025a) DoF: a diffusion factorization framework for offline multi-agent decision making. In International Conference on Learning Representations, Cited by: §A.2, §A.3, §2.2. P. Li, J. Hao, H. Tang, Y. Zheng, and X. Fu (2023a) Race: improve multi-agent reinforcement learning with representation asymmetry and collaborative evolution. In International Conference on Machine Learning, p. 19490–19503. Cited by: §A.1. Y. Li, G. Xie, and Z. Lu (2025b) Revisiting cooperative off-policy multi-agent reinforcement learning. In Forty-second International Conference on Machine Learning, Cited by: §A.1, §2.1, §4. Z. Li, L. Pan, and L. Huang (2023b) Beyond conservatism: diffusion policies in offline multi-agent reinforcement learning. External Links: 2307.01472, Link Cited by: §A.2, §1, §2.2. B. Lin and C. Lee (2026) Heterogeneous agent q-weighted policy optimization. In The Fourteenth International Conference on Learning Representations, External Links: Link Cited by: §A.3, §A.3, §5.3. J. Liu, Y. Zhong, S. Hu, H. Fu, Q. FU, X. Chang, and Y. Yang (2024) Maximum entropy heterogeneous-agent reinforcement learning. In The Twelfth International Conference on Learning Representations, Cited by: §A.1, §A.3, §1, §2.1, §3.1, §5.2, §5.3, §6.1. R. Lowe, Y. I. Wu, A. Tamar, J. Harb, O. Pieter Abbeel, and I. Mordatch (2017) Multi-agent actor-critic for mixed cooperative-competitive environments. Advances in neural information processing systems 30. Cited by: §A.1, §D.1, §2.1, §6.1. C. Lu, H. Chen, J. Chen, H. Su, C. Li, and J. Zhu (2023a) Contrastive energy prediction for exact energy-guided diffusion sampling in offline reinforcement learning. In International Conference on Machine Learning, p. 22825–22855. Cited by: §A.2. C. Lu, Y. Zhou, F. Bao, J. Chen, C. Li, and J. Zhu (2022) Dpm-solver: a fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances in neural information processing systems 35, p. 5775–5787. Cited by: §A.2. C. Lu, P. Ball, Y. W. Teh, and J. Parker-Holder (2023b) Synthetic experience replay. Advances in Neural Information Processing Systems 36, p. 46323–46344. Cited by: §A.2. H. Ma, T. Chen, K. Wang, N. Li, and B. Dai (2025) Efficient online reinforcement learning for diffusion policy. In Forty-second International Conference on Machine Learning, Cited by: §A.3, §A.3, Appendix C, §1, §1, §2.2, §3.2, §4, §5.2, §5.3, §6.1. L. Matignon, L. Jeanpierre, and A. Mouaddib (2012) Coordinated multi-robot exploration under communication constraints using decentralized markov decision processes. In Twenty-sixth AAAI conference on artificial intelligence, Cited by: §A.1. F. A. Oliehoek and C. Amato (2016) A concise introduction to decentralized pomdps. Springer. Cited by: §3.1. F. A. Oliehoek, M. T. Spaan, and N. Vlassis (2008) Optimal and approximate q-value functions for decentralized pomdps. Journal of Artificial Intelligence Research 32, p. 289–353. Cited by: §A.1, §1, §2.1. L. Pan, T. Rashid, B. Peng, L. Huang, and S. Whiteson (2021) Regularized softmax deep multi-agent q-learning. Advances in Neural Information Processing Systems 34, p. 1365–1377. Cited by: §A.1, §2.1, §4. T. Pearce, T. Rashid, A. Kanervisto, D. Bignell, M. Sun, R. Georgescu, S. V. Macua, S. Z. Tan, I. Momennejad, K. Hofmann, et al. (2024) Imitating human behaviour with diffusion models. In The Eleventh International Conference on Learning Representations, Cited by: §A.2. B. Peng, T. Rashid, C. Schroeder de Witt, P. Kamienny, P. Torr, W. Böhmer, and S. Whiteson (2021) Facmac: factored multi-agent centralised policy gradients. Advances in Neural Information Processing Systems 34, p. 12208–12221. Cited by: §D.1, §6.1. M. Psenka, A. Escontrela, P. Abbeel, and Y. Ma (2024) Learning a diffusion model policy from rewards via q-score matching. In International Conference on Machine Learning, p. 41163–41182. Cited by: §A.3, §2.2. D. Qiao, W. Li, S. Yang, H. Zha, and B. Wang (2025) Offline multi-agent reinforcement learning via score decomposition. External Links: 2505.05968, Link Cited by: §A.2. T. Rashid, M. Samvelyan, C. Schroeder, G. Farquhar, J. Foerster, and S. Whiteson (2018) Qmix: monotonic value function factorisation for deep multi-agent reinforcement learning. In International conference on machine learning, p. 4295–4304. Cited by: §A.1, §1, §2.1. A. Z. Ren, J. Lidard, L. L. Ankile, A. Simeonov, P. Agrawal, A. Majumdar, B. Burchfiel, H. Dai, and M. Simchowitz (2024) Diffusion policy policy optimization. In CoRL 2024 Workshop on Mastering Robot Manipulation in a World of Abundant Data, Cited by: §A.3. S. Särkkä and A. Solin (2019) Applied stochastic differential equations. Vol. 10, Cambridge University Press. Cited by: §3.2, §3.2. J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli (2015) Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning, p. 2256–2265. Cited by: §A.2. K. Son, D. Kim, W. J. Kang, D. E. Hostallero, and Y. Yi (2019) Qtran: learning to factorize with transformation for cooperative multi-agent reinforcement learning. In International conference on machine learning, p. 5887–5896. Cited by: §A.1. Y. Song and S. Ermon (2019) Generative modeling by estimating gradients of the data distribution. Advances in Neural Information Processing Systems 32. Cited by: §A.2. Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole (2021) Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, External Links: Link Cited by: §A.2, §1, §2.2. P. Sunehag, G. Lever, A. Gruslys, W. M. Czarnecki, V. Zambaldi, M. Jaderberg, M. Lanctot, N. Sonnerat, J. Z. Leibo, K. Tuyls, et al. (2018) Value-decomposition networks for cooperative multi-agent learning based on team reward. In Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, p. 2085–2087. Cited by: §A.1, §2.1. A. Tampuu, T. Matiisen, D. Kodelja, I. Kuzovkin, K. Korjus, J. Aru, J. Aru, and R. Vicente (2017) Multiagent cooperation and competition with deep reinforcement learning. PloS one 12 (4), p. e0172395. Cited by: §A.1. M. Tan (1993) Multi-agent reinforcement learning: independent vs. cooperative agents. In Proceedings of the tenth international conference on machine learning, p. 330–337. Cited by: §1. J. Wang, Z. Ren, T. Liu, Y. Yu, and C. Zhang (2021) QPLEX: duplex dueling multi-agent q-learning. In International Conference on Learning Representations, Cited by: §A.1, §2.1. T. Wang, H. Dong, Y. Jiang, D. C. Parkes, and M. Tambe (2025a) On diffusion models for multi-agent partial observability: shared attractors, error bounds, and composite flow. In Proceedings of the 24th International Conference on Autonomous Agents and Multiagent Systems, p. 2143–2152. Cited by: §A.3. Y. Wang, L. Wang, Y. Jiang, W. Zou, T. Liu, X. Song, W. Wang, L. Xiao, J. Wu, J. Duan, et al. (2024) Diffusion actor-critic with entropy regulator. In Advances in Neural Information Processing Systems, Cited by: §A.3, §A.3, §A.3, §1, §2.2, §5.3. Y. Wang, L. Wang, M. Tan, W. Zou, X. Song, W. Wang, T. Liu, G. Zhan, T. Zhu, S. Liu, et al. (2025b) Enhanced dacer algorithm with high diffusion efficiency. arXiv preprint arXiv:2505.23426. Cited by: §A.3, §3.2. Z. Wang, J. J. Hunt, and M. Zhou (2023) Diffusion policies as an expressive policy class for offline reinforcement learning. In The Eleventh International Conference on Learning Representations, External Links: Link Cited by: §A.2, §1, §2.2. Z. Wang, J. Liu, and L. Pan (2025c) Learning intractable multimodal policies with reparameterization and diversity regularization. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, External Links: Link Cited by: §A.3. A. Xie, O. Rybkin, D. Sadigh, and C. Finn (2025) Latent diffusion planning for imitation learning. In Forty-second International Conference on Machine Learning, Cited by: §A.2, §2.2. Z. Xu, H. Mao, N. Zhang, X. Xin, P. Ren, D. Li, B. Zhang, G. Fan, Z. Chen, C. Wang, et al. (2024) Beyond local views: global state inference with diffusion models for cooperative multi-agent reinforcement learning. arXiv preprint arXiv:2408.09501. Cited by: §A.3. G. Yang, J. Qiao, T. Yang, Y. Wu, J. Huo, X. Chen, and Y. Gao (2025a) Multi-agent reinforcement learning with communication-constrained priors. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, Cited by: §A.1. L. Yang, Z. Huang, F. Lei, Y. Zhong, Y. Yang, C. Fang, S. Wen, B. Zhou, and Z. Lin (2023) Policy representation via diffusion probability model for reinforcement learning. arXiv preprint arXiv:2305.13122. Cited by: §A.3. N. Yang, J. Gao, F. Gao, Y. Wu, and C. Yu (2025b) Fine-tuning diffusion policies with backpropagation through diffusion timesteps. External Links: 2505.10482, Link Cited by: §A.3. Y. Yang, X. Yang, Y. Jiang, N. Mu, H. Hu, R. Xie, Z. Zhang, S. Li, Y. Ni, Q. Zhao, and B. XU (2026) GlobeDiff: state diffusion process for partial observability in multi-agent system. In The Fourteenth International Conference on Learning Representations, External Links: Link Cited by: §A.3. C. Yu, A. Velu, E. Vinitsky, J. Gao, Y. Wang, A. Bayen, and Y. Wu (2022) The surprising effectiveness of ppo in cooperative multi-agent games. Advances in neural information processing systems 35, p. 24611–24624. Cited by: §A.1, §2.1. C. Zhang, Z. Wan, F. Chen, X. Yu, I. Tsang, and B. An (2025a) GoRL: an algorithm-agnostic framework for online reinforcement learning with generative policies. arXiv preprint arXiv:2512.02581. Cited by: §A.3. J. Zhang, K. Zheng, K. Jiang, H. Wang, I. Stoica, J. E. Gonzalez, J. Chen, and J. Zhu (2025b) TurboDiffusion: accelerating video diffusion models by 100-200 times. arXiv preprint arXiv:2512.16093. Cited by: §A.2. L. Zhang, A. Rao, and M. Agrawala (2023) Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF international conference on computer vision, p. 3836–3847. Cited by: §A.2. L. Zhang, A. Rao, and M. Agrawala (2025c) Scaling in-the-wild training for diffusion-based illumination harmonization and editing by imposing consistent light transport. In The Thirteenth International Conference on Learning Representations, Cited by: §A.2. Y. Zhang, X. Li, J. Ye, S. Qiu, D. Qu, X. Li, C. Zhang, and C. Bai (2025d) Revisiting multi-agent world modeling from a diffusion-inspired perspective. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, External Links: Link Cited by: §A.3. Y. Zhong, J. G. Kuba, X. Feng, S. Hu, J. Ji, and Y. Yang (2024) Heterogeneous-agent reinforcement learning. Journal of Machine Learning Research 25 (32), p. 1–67. Cited by: §A.1, §A.3, §2.1, §3.1, §5.1, §5.2, §5.3, §6.1. M. Zhou, J. Luo, J. Villella, Y. Yang, D. Rusu, J. Miao, W. Zhang, M. Alban, I. Fadakar, Z. Chen, et al. (2021) Smarts: an open-source scalable multi-agent rl training school for autonomous driving. In Conference on robot learning, p. 264–285. Cited by: §1. Z. Zhu, M. Liu, L. Mao, B. Kang, M. Xu, Y. Yu, S. Ermon, and W. Zhang (2024) Madiff: offline multi-agent learning with diffusion models. Advances in Neural Information Processing Systems 37, p. 4177–4206. Cited by: §A.2, §A.3, §2.2, §4. B. D. Ziebart, A. L. Maas, J. A. Bagnell, A. K. Dey, et al. (2008) Maximum entropy inverse reinforcement learning.. In Aaai, Vol. 8, p. 1433–1438. Cited by: §3.1. Appendix A Detailed Related Works Here we propose our detailed related works about online MARL, diffusion policies in offline RL and single-agent online RL scenarios. We also discuss online MARL challenges and compare OMAD against baselines. A.1 Online MARL Online multi-agent reinforcement learning (MARL) has achieved substantial progress in recent years. Under the centralized training with decentralized execution (CTDE) paradigm (Oliehoek et al., 2008; Matignon et al., 2012), a series of influential algorithms, including MADDPG (Lowe et al., 2017), MAPPO (Yu et al., 2022), VDN (Sunehag et al., 2018), COMA (Foerster et al., 2018), QTRAN (Son et al., 2019) QMIX (Rashid et al., 2018), and QPLEX (Wang et al., 2021) have significantly improved coordination and stability in cooperative tasks. In parallel, fully decentralized approaches such as IQL (Tampuu et al., 2017), IPPO (de Witt et al., 2020), and MATD3 (Ackermann et al., 2019) achieve competitive performance without centralized critics, underscoring the scalability of online MARL. Recent advances further enhance robustness, exploration efficiency, and scalability, including RES-QMIX (Pan et al., 2021), RACE (Li et al., 2023a), RCO (Li et al., 2025b), ACAC (Jung et al., 2025), MANGER (Chen et al., 2025b), and SMPE (Kontogiannis et al., 2025), as well as studies on communication-constrained MARL (Yang et al., 2025a). More recently, heterogeneity-aware frameworks such as HARL and its variants (HATRPO, HAPPO, HATD3, HASAC) (Zhong et al., 2024; Liu et al., 2024), provide principled solutions for coordinating heterogeneous agents and mitigating non-stationarity, representing the state of the art in online MARL. A.2 Diffusion Policy in Offline RL Diffusion-based generative models have achieved remarkable success in representing complex probability distributions with superior expressiveness and multimodality, demonstrating outstanding performance in image and video generation (Sohl-Dickstein et al., 2015; Song and Ermon, 2019; Ho et al., 2020; Song et al., 2021; Lu et al., 2022; Zhang et al., 2023; Brooks et al., 2024; Zhang et al., 2025c, b). These advantages naturally benefit sequential decision-making problems, leading to the introduction of diffusion policies in imitation learning (Pearce et al., 2024; Xie et al., 2025), where expert datasets are available. In offline RL (Wang et al., 2023), diffusion models have been explored for trajectory generation, e.g., Diffuser (Janner et al., 2022) and Decision Diffuser (Ajay et al., 2023), expressive policy representations including Diff-QL (Wang et al., 2023), SfBC (Chen et al., 2023), EDP (Kang et al., 2023), IDQL (Hansen-Estruch et al., 2023), CEP (Lu et al., 2023a), DAC (Fang et al., 2024), value representations (VDQL (Hu et al., 2025)), data augmentation, for instance, SynthER (Lu et al., 2023b) and GTA (Lee et al., 2024) and robot learning (Chi et al., 2023; Huang et al., 2025). Recent offline MARL methods leverage diffusion models for multi-modality and data scarcity, including trajectory modeling (MADiff (Zhu et al., 2024), DoF (Li et al., 2025a)), data synthesis (INS (Fu et al., 2025)), diverse action generation (DOM2 (Li et al., 2023b)), and score decomposition (OMSD (Qiao et al., 2025)). However, relying strictly on fixed, pre-collected datasets, offline MARL creates a fundamental gap from online settings, as it circumvents the critical need for active, coordinated exploration in non-stationary environments essential to tackle the severe non-stationary environments. A.3 Diffusion Policy in Online RL In online RL, the optimal policy cannot be directly sampled and must be learned via return- or value-based optimization, posing a fundamental challenge. To address this challenge, recent online diffusion-based RL methods adopt value-guided training, including gradient-updated behavior cloning (DIPO (Yang et al., 2023), DPPO (Ren et al., 2024) and DRAC (Wang et al., 2025c)), Q-gradient score matching (QSM (Psenka et al., 2024), CQSM (Hua et al., 2025)), entropy-controlled backpropagation through diffusion chains (DACER (Wang et al., 2024), DACER-v2 (Wang et al., 2025b), DIME (Celik et al., 2025) and NCDPO (Yang et al., 2025b)), score approximation (MaxEntDP (Dong et al., 2025)) and Q-weighted diffusion objectives with uniform exploration (QVPO (Ding et al., 2024), HAQO (Lin and Lee, 2026) and RSM (Ma et al., 2025)). In spite of policy-centric approaches, DIMA (Zhang et al., 2025d) uses diffusion models as environment dynamics to boost data efficiency. Beyond single-agent dynamics, recent advancements leverage diffusion models to infer global states from local observations in partially observable multi-agent cooperative environments (Xu et al., 2024; Wang et al., 2025a; Yang et al., 2026). Furthermore, diffusion models have been successfully extended to online reinforcement learning as algorithm-agnostic generative policies, effectively decoupling policy optimization from action generation to improve training stability (Zhang et al., 2025a). However, extending these methods to online MARL faces three hurdles: computational inefficiency, as gradient propagation through diffusion chains (Wang et al., 2024) scales poorly with the number of agents; ineffective exploration, where simple heuristics (Dong et al., 2025) fail in high-dimensional joint spaces; and lack of coordination, as single-agent designs ignore non-stationarity inherent in multi-agent systems. To the best of our knowledge, our work is among the first to propose an efficient, off-policy diffusion framework specifically designed to overcome these hurdles, achieving superior training efficiency and coordination in online MARL. To the best of our knowledge, OMAD is among the first algorithm to achieve the integration of diffusion-based policies into the online, off-policy multi-agent reinforcement learning (MARL). This represents a fundamental paradigm shift from existing value-based MARL architectures (e.g., HARL (Zhong et al., 2024; Liu et al., 2024)). While these conventional methods predominantly rely on restrictive parametric representations, such as unimodal Gaussian policies, that fundamentally limit agent behavior, OMAD explicitly models the policy as a generative diffusion process. This innovation unlocks the capacity to capture the highly complex, multimodal joint action distributions essential for sophisticated continuous control. Furthermore, OMAD distinguishes itself from recent offline diffusion MARL methods (e.g., MADIFF (Zhu et al., 2024), DoF (Li et al., 2025a)), which circumvent the active exploration problem by relying entirely on fixed, pre-collected expert datasets. Instead, OMAD operates directly in an online setting from scratch, employing a tractable maximum entropy paradigm via our derived ELBO to efficiently drive active exploration without requiring prior data. Compared to single-agent online diffusion baselines (e.g., DACER (Wang et al., 2024), DIME (Celik et al., 2025)) and their naive multi-agent extensions (e.g., MADPMD, MASDAC), OMAD represents a significant leap in overcoming severe non-stationarity and the lack of unified value guidance. Naive multi-agent adaptations inherently struggle to coordinate stably; OMAD systematically overcomes this computational barrier by coupling a novel factorized joint entropy bound with a synchronized centralized distributional critic. Moreover, while prior works such as DPMD and SDAC (Ma et al., 2025) rely on discrete-time formulations, and MaxEntRL (Dong et al., 2025) employs deterministic ODE sampling, OMAD leverages an efficient SDE-based formulation to ensure sustained stochasticity for continuous-time denoising. Ultimately, by explicitly synchronizing decentralized updates through our novel CTDE mechanism and joint distributional critic, OMAD yields a highly robust off-policy framework. This ensures stable multi-agent coordination and, despite the existence of on-policy methods like HADQ (Lin and Lee, 2026), significantly enhances sample efficiency and data reuse. Appendix B Proof of Theorem 4.1 Proof: Notice that the definition of the entropy about the joint action distribution a=(a1,a2…,aN)∼(a|s)a=(a^1,a^2...,a^N) π(a|s) can be written as: ℋ((a|s)) ( π(a|s)) =∫a−(a|s)log((a|s))da = _a- π(a|s) ( π(a|s))da (16) =∫a−∏j=1nπj(aj|s)log(∏i=1nπi(ai|s))da = _a- _j=1^n _j(a^j|s) ( _i=1^n _i(a^i|s))da =∫a−∏j=1nπj(aj|s)∑i=1nlog(πi(ai|s))da = _a- _j=1^n _j(a^j|s) _i=1^n ( _i(a^i|s))da =∑i=1n∫a1,⋯,aN−∏j=1nπj(aj|s)log(πi(ai|s))da1⋯daN = _i=1^n _a^1,·s,a^N- _j=1^n _j(a^j|s) ( _i(a^i|s))da^1·sda^N =∑i=1n∫ai−πi(ai|s)log(πi(ai|s))dai = _i=1^n _a^i- _i(a^i|s) ( _i(a^i|s))da^i =∑i=1nℋ(πi(ai|s)). = _i=1^nH( _i(a^i|s)). It means that the entropy of the joint policy can be split as the summation of the independent policy. Moreover, we calculate the evidence lower bound of a single-agent policy, which is the same as (Celik et al., 2025) that (here we renotate that ai=a0ia^i=a^i_0 and notate the policy of agent i as πθi _ _i to involve the terminal timestep for the reverse diffusion process) and the corresponding forward process as π: ℋ(πθi(a0i|s)) ( _ _i(a^i_0|s)) =∫a0i−πθi(a0i|s)log(πθi(a0i|s))da0i=πθi[−log(πθi(a0i|s))] = _a^i_0- _ _i(a^i_0|s) ( _ _i(a^i_0|s))da^i_0=E_ _ _i[- ( _ _i(a^i_0|s))] (17) Notice that πθi(a0:Hi|s)=πθi(a0i|s)πθi(a1:Hi|a0i,s) _ _i(a^i_0:H|s)= _ _i(a^i_0|s) _ _i(a^i_1:H|a^i_0,s), so we replace πθi(a0i|s)=πθi(a0:Hi|s)πθi(a1:Hi|a0i,s) _ _i(a^i_0|s)= _ _i(a^i_0:H|s) _ _i(a^i_1:H|a^i_0,s) such that: πθi[−log(πθi(a0i|s))] _ _ _i[- ( _ _i(a^i_0|s))] =πθi[−log(πθi(a0:Hi|s)πθi(a1:Hi|a0i,s))] =E_ _ _i [- ( _ _i(a^i_0:H|s) _ _i(a^i_1:H|a^i_0,s) ) ] (18) =πθi[log(πθi(a1:Hi|a0i,s)πθi(a0:Hi|s))] =E_ _ _i [ ( _ _i(a^i_1:H|a^i_0,s) _ _i(a^i_0:H|s) ) ] ≥πθi[log(πi(a1:Hi|a0i,s)πθi(a0:Hi|s))]=lπθi(a0i|s). _ _ _i [ ( _i(a^i_1:H|a^i_0,s) _ _i(a^i_0:H|s) ) ]=l_ _ _i(a_0^i|s). The final inequality is due to the fact that: πθi[log(πθi(a1:Hi|a0i,s))] _ _ _i[ ( _ _i(a^i_1:H|a^i_0,s))] =πθi[log(πθi(a1:Hi|a0i,s))−log(πi(a1:Hi|a0i,s))+log(πi(a1:Hi|a0i,s))] =E_ _ _i[ ( _ _i(a^i_1:H|a^i_0,s))- ( _i(a^i_1:H|a^i_0,s))+ ( _i(a^i_1:H|a^i_0,s))] (19) =(πθi(a1:Hi|a0i,s)∥πi(a1:Hi|a0i,s))+πθi[log(πi(a1:Hi|a0i,s))] =KL( _ _i(a^i_1:H|a^i_0,s)\| _i(a^i_1:H|a^i_0,s))+E_ _ _i[ ( _i(a^i_1:H|a^i_0,s))] ≥πθi[log(πi(a1:Hi|a0i,s))]. _ _ _i[ ( _i(a^i_1:H|a^i_0,s))]. It means that πθi[log(πθi(a1:Hi|a0i,s))]≥πθi[log(πi(a1:Hi|a0i,s))]E_ _ _i[ ( _ _i(a^i_1:H|a^i_0,s))] _ _ _i[ ( _i(a^i_1:H|a^i_0,s))] due to the non-negativity of the KL divergence. Combine Eq. (16), Eq. (18) and Eq. (19), we complete the proof. The expectation is taken with respect to the agent’s diffusion policy πθi _ _i, representing the entire sequential trajectory generation process. In practice, a batch of actions is sampled by initializing from standard Gaussian noise and executing a learned iterative denoising sequence.While the forward diffusion process analytically adds Gaussian noise with predetermined schedules, the reverse denoising process is governed by the policy network πθi _ _i. Consequently, given the noisy intermediate actions at each timestep, the exact likelihood of the reverse Gaussian transitions is fully determined by the network, allowing for tractable optimization over the sampled batch. Appendix C Derivation of the Synchronized Policy Objective In this section, we present the background and motivation for our synchronized policy objective. We define our objective function as: J((⋅|s)):=[∑t=0∞∑i=1Nγt(rti+αlπθi(ati,st))].J( π_ θ(·|s)):=E_ _θ [ _t=0^∞ _i=1^Nγ^t(r_t^i+α l_ _ _i(a^i_t,s_t)) ]. Incorporating a scaled maximum entropy regularizer, we define the corresponding value function for a joint state-action pair as follows: Qπ(st,at0)=∑i=1Nrti+∑l=1∞γlρπ[∑i=1N(rt+li+αlπi(at+li,st+l))]. Q^π(s_t,a_t^0)= _i=1^Nr^i_t+ _l=1^∞γ^lE_ _π [ _i=1^N(r^i_t+l+α l_ _i(a^i_t+l,s_t+l)) ]. (20) This definition aligns with Eq. (17) in (Celik et al., 2025) for joint states and actions. Given a concrete policy π, the iterative policy optimization aims to minimize the KL divergence between the policy and the exponential soft Q-values: πk+1=argminπDKL(π(⋅|s)∥exp(1αQπk(s,⋅))Z(s)), π^k+1= _πD_KL\! (π(·|s)\, \|\, \! ( 1αQ _k(s,·) )Z(s) ), (21) which corresponds to Eq. (5) in (Ma et al., 2025). Since the exact likelihood of the diffusion policy is intractable, we upper bound the KL divergence as (which builds upon Eq. (20) in (Celik et al., 2025)): DKL(πθ(⋅|s)∥exp(1αQϕ(s,⋅))Z(s))≤DKL(πθ(a0:H|s)∥π(a0:H|s)). D_KL\! ( _θ(·|s)\, \|\, \! ( 1αQ_φ(s,·) )Z(s) )≤ D_KL( _θ(a_0:H|s)\|π(a_0:H|s)). (22) The trajectory distributions are factorized as: πθ(a0:H|s) _θ(a_0:H|s) =∏i=1Nπθi(a0:Hi|s)=∏i=1Nπθi(aHi|s)∏h=1Hπθi(ah−1i|ahi,s), = _i=1^N _ _i(a^i_0:H|s)= _i=1^N _ _i(a^i_H|s) _h=1^H _ _i(a^i_h-1|a^i_h,s), (23) π(a0:H|s) π(a_0:H|s) =π(a0|s)∏h=0H−1π(ah+1|ah,s)=π(a0|s)∏h=0H−1∏i=1Nπ(ah+1i|ahi,s). =π(a_0|s) _h=0^H-1π(a_h+1|a_h,s)=π(a_0|s) _h=0^H-1 _i=1^Nπ(a_h+1^i|a_h^i,s). Here, πθ(a0:H|s) _θ(a_0:H|s) represents the probability that the action is sampled via the reverse diffusion process for each individual agent, corresponding to Eq. (13) and Eq. (14) in (Celik et al., 2025). The target policy is defined as a forward diffusion process, where π(a0|s)=1Z(s)exp(1αQϕ(s,a0))π(a_0|s)= 1Z(s) \! ( 1αQ_φ(s,a_0) ) is the target distribution and π(ah+1i|ahi,s)π(a_h+1^i|a_h^i,s) corresponds to the forward diffusion term. By substituting these terms into the KL divergence objective, we obtain: ℒ(θ) (θ) =DKL(πθ(a0:H|s)∥π(a0:H|s)) =D_KL( _θ(a_0:H|s)\|π(a_0:H|s)) (24) =[log(πθ(a0:H|s)π(a0:H|s))] =E [ ( _θ(a_0:H|s)π(a_0:H|s) ) ] =[log(∏i=1Nπθi(aHi|s)∏h=1Hπθi(ah−1i|ahi,s)π(a0|s)∏h=0H−1∏i=1Nπ(ah+1i|ahi,s))] =E [ ( _i=1^N _ _i(a^i_H|s) _h=1^H _ _i(a^i_h-1|a^i_h,s)π(a_0|s) _h=0^H-1 _i=1^Nπ(a_h+1^i|a_h^i,s) ) ] =[∑i=1Nlogπθi(aHi|s)−1αQϕ(s,a0)+∑i=1N∑h=1Hlogπθi(ah−1i|ahi,s)πi(ahi|ah−1i,s)]+logZ(s). =E [ _i=1^N _ _i(a_H^i|s)- 1αQ_φ(s,a_0)+ _i=1^N _h=1^H _ _i(a_h-1^i|a_h^i,s) _i(a_h^i|a_h-1^i,s) ]+ Z(s). Note that the target policy is defined based on the joint distributional value Qϕ(s,a)=[Zϕ(s,a)]Q_φ(s,a)=E[Z_φ(s,a)], implying that the loss function is shared across all agents. Crucially, this design choice is pivotal for coordination: while we employ a factorized entropy lower bound for computational tractability in Eq. (16), the multi-agent coordination is strictly enforced by the global guidance of this centralized distributional critic Zϕ(s,a)Z_φ(s,a). In contrast to independent learning methods, the joint critic explicitly models the complex, multi-modal correlations of agent interactions and backpropagates a unified gradient signal, thereby synchronizing the decentralized diffusion processes towards a coherent joint equilibrium. Given this strong coordination signal, while applying value factorization techniques could theoretically enable distributed policy optimization by assigning appropriate credits to different agents, our empirical results suggest that these techniques are insufficient to guide the policy toward optimal behaviors in this setting compared to our joint critic approach. Furthermore, leveraging an off-policy paradigm, our framework allows for repeated sampling from the replay buffer, thereby maximizing data utility and significantly outperforming on-policy counterparts in sample efficiency. Appendix D Details about the Experiments D.1 Experimental Setup: Environments (a) (b) (c) (d) (e) (f) Figure 7: Multi-agent particle environments (MPE) and Multi-agent HalfCheetah task in MuJoCo Environment (MAMuJoCo). We evaluate our algorithm and baselines on two standard benchmarks: the Multi-Agent Particle Environments (MPE) (Lowe et al., 2017)222https://pettingzoo.farama.org/environments/mpe/ and Multi-Agent MuJoCo (MAMuJoCo) (Peng et al., 2021)333https://robotics.farama.org/envs/MaMuJoCo/. Within the MPE domain, we select Cooperative Navigation and Physical Deception as representative scenarios. In Cooperative Navigation shown as Figure 7(a), agents must coordinate to occupy landmarks while avoiding collisions. In Physical Deception shown as Figure 7(b), agents collaborate to reach a target landmark while concealing its identity from an adversary. In the Multi-Agent MuJoCo (MAMuJoCo) domain, we evaluate our method across a diverse set of locomotion scenarios, including Ant in Figure 7(c), HalfCheetah in Figure 7(d), Walker2d in Figure 7(e), and Swimmer in Figure 7(f). In these tasks, the original high-dimensional action space of a single robot is partitioned among multiple decentralized agents. This partitioning necessitates complex coordination, as agents must infer the intentions of others and align their local policies to achieve stable and rapid global locomotion. The specific characteristics of each environment are detailed below: Ant (2×42× 4, 2×42× 4d, 4×24× 2) A 3D quadrupedal robot with high-dimensional state-action spaces. Different from planar tasks, the Ant requires agents to maintain balance in three dimensions while coordinating four multi-joint legs. The challenge lies in synchronizing independent limb movements to generate forward velocity without toppling the robot. HalfCheetah (2×32× 3, 6×16× 1) A planar biped designed for high-speed running. In multi-agent configurations, the primary difficulty is synthesizing a coherent, rhythmic gait from partitioned actuation (e.g., separate agents for thighs and feet). Agents must maximize forward momentum through emergent cooperation without the strict balance constraints. Walker2d A bipedal walker that introduces significant stability constraints. In contrast to the HalfCheetah, the Walker2d must maintain an upright torso to avoid falling. Agents controlling different leg segments face a difficult credit assignment problem, as they must balance the trade-off between generating forward thrust and actively stabilizing the robot’s pitch. Swimmer Locomotion in a viscous fluid governed by fluid dynamics rather than ground reaction forces. Agents must coordinate to produce undulatory, snake-like motion. The task requires precise temporal synchronization along the kinematic chain to leverage fluid friction effectively, as uncoordinated actions result in negligible displacement. The fundamental distinction between the standard MuJoCo benchmark and MAMuJoCo lies in the partitioning of the control authority. In the single-agent setting, a monolithic policy governs the entire kinematic chain, implicitly handling the coordination between joints through centralized optimization. In contrast, MAMuJoCo transforms these tasks into decentralized coordination problems where the action space is split among independent agents. This introduces significant challenges: (1) Physical Coupling: due to the rigid-body dynamics, an action taken by one agent instantaneously alters the state dynamics experienced by others, creating a highly non-stationary environment; (2) Implicit Coordination: agents must achieve global synchronization (e.g., a stable gait) purely through local observations and rewards, without explicit communication channels. Success in this domain thus demonstrates an algorithm’s capability to solve complex credit assignment problems in high-dimensional continuous control tasks. Table 1: Hyperparameters in the MPE and MAMuJoCo environment for all tasks. Name of the Hyperparameter Value Warmup Timesteps 6000060000 Timesteps to Start Learning LinitL_init 50005000 Policy Update ρ 0.00.0 Discount Factor γ 0.990.99 Policy Delay dld_l 33 Batch Size 256256 Buffer Size of D 10000001000000 Initial Entropy Coefficient Term α 1.01.0 Target Entropy ℋtargetH_target 4dim()4dim(A) Number of Atoms for Distributional Q-Learning NatomsN_atoms 100100 Coefficient Term for Distributional Q-Learning Regularization ξ 0.0050.005 Batch Normalization Momentum 0.990.99 Batch Normalization Warmup Timesteps 100000100000 Optimizer Adam Adam β1 _1 0.50.5 Adam β2 _2 0.9990.999 Gradient Clip Norm 1.01.0 Diffusion Denoise Steps H 88 Table 2: Hyperparameters in the MPE and MAMuJoCo environment for all tasks including the learning rate for the actor and critic, and the maximum value VmaxV_max for distribution Q-Learning. Task Learning Rate Maximum Value VmaxV_max Cooperative Navigation 33 Agents 3.5×10−53.5× 10^-5 200200 Cooperative Navigation 44 Agents 2.5×10−52.5× 10^-5 200200 Physical Deception 22 Agents 5.0×10−65.0× 10^-6 200200 Ant 2×42× 4 7.0×10−67.0× 10^-6 12001200 Ant 2×42× 4d 2.0×10−52.0× 10^-5 50005000 Ant 4×24× 2 5.0×10−65.0× 10^-6 30003000 HalfCheetah 2×32× 3 1.0×10−41.0× 10^-4 2000020000 HalfCheetah 6×16× 1 1.0×10−31.0× 10^-3 30003000 Walker2d 2×32× 3 5.0×10−65.0× 10^-6 5000050000 Swimmer 2×12× 1 2.0×10−52.0× 10^-5 400400 D.2 Experimental Setup: Network Structures and Hyperparameters For the critic, we employ a Multi-Layer Perceptron (MLP) to model the distributional Q-function. Following the CrossQ technique (Bhatt et al., 2024), we concatenate the state-action pairs and normalize them via a Batch Normalization layer before feeding them into the MLP. The network consists of two hidden layers with 20482048 units each, utilizing ReLU activations and Batch Normalization. The final output is processed via a softmax function to generate the probability distribution over the atoms. The support of the distribution is defined over [Vmin,Vmax][V_min,V_max], where Vmin=−VmaxV_min=-V_max for all tasks. For the actor, we utilize a diffusion-based policy parameterized by a noise prediction network fθi(ati,s,t)f_ _i(a_t^i,s,t). This network is implemented as a 33-layer MLP with GeLU activations and a hidden dimension of 256256. The input consists of the concatenated state, noisy action, and a 256-dimensional Fourier timestep embedding. We employ a cosine noise schedule with βmin=10−3 _min=10^-3 and βmax=0.9999 _max=0.9999. The number of diffusion steps H is set to 88 for both training and inference. Similar to the critic, the input vector undergoes Batch Normalization prior to entering the MLP layers. We emphasize that while our centralized distributional critic adopts the CrossQ architecture to forego the target Q-network, we explicitly retain the target policy network. This design ensures training stability during data collection and stabilizes the joint optimization of the critic and actor. Crucially, sampling next-step actions from a slowly updating target policy mitigates the high variance arising from the compounded stochasticity of interacting diffusion agents, thereby preventing divergence in value estimation. To facilitate full reproducibility, we provide a detailed breakdown of the experimental configurations. Table 1 summarizes the common hyperparameters applied uniformly across all tasks, such as network architecture and optimizer settings. In contrast, Table 2 details the task-specific hyperparameters, including learning rates and distributional value limits, which are tailored to each environment. All reported experiments were conducted using 55 independent random seeds to ensure statistical reliability. We clarify that setting ρ=0ρ=0 reduces the target network synchronization to a standard hard update. D.3 Additional Experimental Results and Visualization D.3.1 Quantitative Comparison To rigorously evaluate the asymptotic performance and robustness of our approach, we conduct extensive experiments using high-performance computing resources. To highlight OMAD’s efficiency, the training horizon for baselines is extended to 1×1071× 10^7 steps to guarantee convergence. In contrast, OMAD requires only within just 3×1063× 10^6 steps (except for HalfCheetah 6×16× 1, which uses 1×1061× 10^6). As detailed in Table 3, OMAD consistently outperforms all baselines despite their significantly longer training horizons. We report the maximum average episode returns over 55 random seeds, highlighting the optimal performance in bold. These results show that OMAD achieves state-of-the-art performance across nearly all MPE and MAMuJoCo tasks with low variance. This comprehensive evaluation underscores that OMAD not only excels in sample efficiency but also delivers superior final performance, validating its effectiveness for long-horizon multi-agent control. Table 3: Quantitative comparison of asymptotic performance on MPE and MAMuJoCo benchmarks. We report the maximum average episode returns and the standard deviation over 55 random seeds. Notably, OMAD achieves state-of-the-art results despite being trained for significantly fewer timesteps (3×1063× 10^6) compared to the extended horizon of baselines (1×1071× 10^7), demonstrating exceptional sample efficiency. An exception is HalfCheetah 6×16× 1, where diffusion models were evaluated at 1×1061× 10^6 steps. Bold indicates optimal performance (within 1%1\% gap). Task HATD3 HASAC MADPMD MASDAC OMAD(Ours) Cooperative Navigation N=3N=3 −26.2±3.5-26.2± 3.5 −25.1±1.3-25.1± 1.3 −29.4±1.6-29.4± 1.6 −41.0±1.1-41.0± 1.1 −23.9±1.1 -23.9± 1.1 Cooperative Navigation N=4N=4 −63.0±2.2-63.0± 2.2 −65.6±3.5-65.6± 3.5 −85.0±3.8-85.0± 3.8 −84.6±3.1-84.6± 3.1 −57.9±0.8 -57.9± 0.8 Physical Deception N=2N=2 43.0±3.643.0± 3.6 45.4±3.1 45.4± 3.1 36.9±4.536.9± 4.5 33.1±5.133.1± 5.1 45.1±4.1 45.1± 4.1 Ant 2×42× 4 6151.9±408.96151.9± 408.9 6980.4±458.56980.4± 458.5 7042.7±379.87042.7± 379.8 7266.2±431.57266.2± 431.5 7517.0±279.2 7517.0± 279.2 Ant 2×42× 4d 4992.1±595.24992.1± 595.2 4941.9±320.04941.9± 320.0 6672.5±467.56672.5± 467.5 6936.2±309.76936.2± 309.7 7449.3±169.3 7449.3± 169.3 Ant 4×24× 2 5911.2±254.05911.2± 254.0 6597.1±418.76597.1± 418.7 7139.9±462.17139.9± 462.1 6198.5±1012.16198.5± 1012.1 7403.7±41.7 7403.7± 41.7 HalfCheetah 2×32× 3 8732.6±413.68732.6± 413.6 8835.7±356.38835.7± 356.3 10305.5±735.910305.5± 735.9 10540.7±963.810540.7± 963.8 14368.5±1166.0 14368.5± 1166.0 HalfCheetah 6×16× 1 8453.7±503.48453.7± 503.4 8714.3±424.68714.3± 424.6 10402.8±622.410402.8± 622.4 10049.9±751.410049.9± 751.4 11044.3±249.6 11044.3± 249.6 Walker2d 2×32× 3 6248.6±622.36248.6± 622.3 6753.5±436.26753.5± 436.2 6112.8±269.36112.8± 269.3 8154.4±322.5 8154.4± 322.5 8180.4±803.5 8180.4± 803.5 Swimmer 2×12× 1 98.5±52.398.5± 52.3 134.2±23.3134.2± 23.3 136.8±7.6136.8± 7.6 149.3±12.6149.3± 12.6 162.0±22.5 162.0± 22.5 These comparative results offer implicit insights into the structural determinants of multi-agent coordination. Notably, the diffusion-based baselines (MADPMD and MASDAC) were implemented using a centralized training and centralized execution paradigm. While this approach theoretically maximizes coordination by circumventing the limitations of distributed policies, their suboptimal performance underscores that merely incorporating expressive diffusion policies is insufficient without a robust value estimation mechanism and efficient design of the diffusion policy architecture. We argue that an advanced distributional critic alone is also insufficient, as unimodal Gaussian policies inevitably suffer from sub-optimal mean-seeking behavior in multi-modal multi-agent landscapes. To empirically validate this, our evaluation on the 3-agent Cooperative Navigation environment demonstrates that a standard Gaussian policy only achieves a score of −26.1±2.9-26.1± 2.9, whereas our diffusion-based approach significantly improves coordination to reach −23.9±1.1-23.9± 1.1. OMAD’s superior performance stems from the synergy between the distributional critic and the expressive diffusion policy, which naturally models multi-modality to fully leverage the critic’s rich guidance. Regarding the policy structure, while OMAD employs an SDE-based formulation to ensure sustained stochasticity, identifying network architectures that maximize the trade-off between runtime efficiency and control performance remains vital. Specifically, although the iterative denoising process inherent in diffusion models increases per-step computational overhead, this drawback is heavily offset by OMAD’s superior sample efficiency. In practice, our method achieves convergence in just 3M steps, whereas traditional baselines require up to 10M steps, demonstrating a trade-off that strongly favors overall training speed and final performance. Consequently, while improving online sampling efficiency remains a promising future direction, the theoretical implications of the ELBO approximation error warrant further scrutiny. While the intractable likelihood of diffusion models precludes a precise quantification of the variational approximation gap during training, this error theoretically vanishes as the learned policy converges to the optimal target distribution. Deriving tighter lower bounds to mitigate potential bias thus remains a valuable direction for theoretical refinement. Moreover, in terms of computational cost, although hardware variations introduce some noise in wall-clock measurements, OMAD generally exhibits superior performance compared to HASAC and HATD3 within equivalent timeframes. Nevertheless, developing techniques to further optimize the wall-clock training efficiency of diffusion-based agents remains a vital direction for enhancing the scalability of this framework. It is worth noting that this work explicitly focuses on efficient online multi-agent coordination within the continuous control domain. To this end, our framework is validated on representative continuous benchmarks, including MPE and MAMuJoCo, which pose highly complex coordination challenges through continuous physical coupling. We deliberately exclude discrete-action benchmarks, e.g., SMAC, as our continuous-time diffusion framework is structurally tailored for continuous action spaces. Extending this framework to discrete environments (e.g., via discrete diffusion variants or categorical mappings to evaluate on tasks such as SMAC) presents a promising avenue for future research to further broaden its applicability. D.3.2 Additional Ablation Studies on HalfCheetah To rigorously validate the robustness of OMAD across diverse dynamic characteristics, we present additional ablation studies on the heavily coupled HalfCheetah 6×16× 1 task. While it is a common practice in computationally expensive multi-agent reinforcement learning (MARL) literature to conduct comprehensive ablations on a single, highly representative environment (e.g., HASAC), we extend our evaluation here to demonstrate OMAD’s stability under entirely different environment dynamics. Specifically, we investigate the sensitivity of three core components of our framework: (1) the number of atoms (the granularity of the distributional value function); (2) the value bounds (i.e., VmaxV_ , the range of the return distribution); and (3) the entropy coefficient (i.e., α, the scaling factor for policy exploration). As detailed in Tables 4, 5, and 6, our default hyperparameter configuration, which consists of 100100 atoms, Vmax=3000V_ =3000, and automatic entropy tuning (Auto), consistently yields the optimal episodic performance. These results underscore the stability and robustness of OMAD across challenging coordinate-dependent control tasks. Table 4: Ablation study on the number of atoms (HalfCheetah 6×16× 1). Number of Atoms 25 50 100 (Default) 150 200 Performance 11004.9±266.211004.9± 266.2 10786.9±165.710786.9± 165.7 11064.6±298.811064.6± 298.8 6346.0±283.46346.0± 283.4 6694.6±78.76694.6± 78.7 As detailed in Tables 4, 5, and 6, our default hyperparameter configuration—consisting of 100100 atoms, Vmax=3000V_ =3000, and automatic entropy tuning (Auto)—consistently yields the optimal episodic performance. These results underscore the stability and robustness of OMAD across challenging coordinate-dependent control tasks. Table 5: Ablation study on the value bound VmaxV_ (HalfCheetah 6×16× 1). VmaxV_ Value 2000 2500 3000 (Default) 3500 4000 Performance 5290.1±68.15290.1± 68.1 5361.2±173.45361.2± 173.4 11064.6±298.811064.6± 298.8 7696.5±383.07696.5± 383.0 8772.9±1031.08772.9± 1031.0 Table 6: Ablation study on the entropy coefficient α (HalfCheetah 6×16× 1). Entropy Coeff. (α) 0.001 0.01 0.025 0.1 Auto (Default) Performance 8829.2±1994.08829.2± 1994.0 8486.9±2735.28486.9± 2735.2 10635.2±478.510635.2± 478.5 9175.8±2074.39175.8± 2074.3 11064.6±298.811064.6± 298.8 D.3.3 Scalability Scaling multi-agent reinforcement learning (MARL) systems presents profound computational and theoretical challenges, particularly as the number of agents, N, increases. The complexity compounds across three primary dimensions: critic optimization, policy optimization, and inference. Specifically, the input dimension of the centralized value function grows linearly, while the joint action space expands exponentially, creating a combinatorial explosion that demands substantially larger network capacities and exacerbates optimization instability. Furthermore, because iterative denoising steps in diffusion policies scale linearly in either inference latency (if executed serially) or memory consumption (if executed in parallel), scaling to massive systems (e.g., N≫100N 100) remains a formidable open problem for the entire MARL community. Conducting standard ablation studies on the number of agents is also fundamentally impractical in continuous control benchmarks like MAMuJoCo; arbitrarily adding agents completely alters the underlying physical morphology and dynamics of the task, rendering direct performance comparisons meaningless. While prior works have explored mean-field approximations and dimensionality reduction techniques to alleviate these burdens, achieving optimal control in massive, continuous multi-agent systems remains an unresolved frontier. Table 7: Performance comparison with more agents in Cooperative Navigation tasks with 1010 agents. Algorithm HATD3 HASAC MADPMD MASDAC OMAD(Ours) Performance −464.3±16.8-464.3± 16.8 −471.3±14.0-471.3± 14.0 −489.3±33.5-489.3± 33.5 −483.5±21.3-483.5± 21.3 −445.1±3.3-445.1± 3.3 Despite these inherent, field-wide hurdles, OMAD exhibits robust scalability and exceptional coordination capabilities within and beyond standard continuous control bounds. Standard evaluation environments typically feature five or fewer agents to rigorously test coordination without overwhelming the state-action space; however, OMAD demonstrates significant advantages in more complex settings, such as the 6-agent HalfCheetah task. To further validate its scalability, we evaluated OMAD in a denser 10-agent Cooperative Navigation environment shown in Table 7. In this setting, OMAD successfully navigated the expanded joint action space to achieve state-of-the-art optimal performance (−445.1±3.3-445.1± 3.3), exhibiting both higher returns and significantly lower variance compared to strong baselines like HATD3, HASAC, and naive multi-agent diffusion extensions (e.g., MADPMD, MASDAC). While the primary computational overhead in our algorithm natively stems from the diffusion generation process, OMAD effectively manages this by balancing the trade-offs between denoising steps and policy performance. Ultimately, OMAD provides a highly stable, capable framework for continuous multi-agent coordination without succumbing to the compounded variance that typically plagues large-scale off-policy learning. D.3.4 Visualization (a) (b) (c) (d) Figure 8: Visualization of learned diffusion policies across four distinct MAMuJoCo tasks. We display snapshots of the agents at timesteps t∈1,100,250,500t∈\1,100,250,500\ with the instantaneous velocity, demonstrating the stable and coordinated behaviors achieved by our OMAD algorithm. To further validate the effectiveness of OMAD beyond numerical metrics, we provide a qualitative visualization of the learned behaviors in Figure 8. The figure displays the temporal evolution of agent policies across four distinct MAMuJoCo tasks (Ant 4×24× 2, HalfCheetah 6×16× 1, Walker2d 2×32× 3, and Swimmer 2×12× 1 as the representative tasks) at timesteps t∈1,100,250,500t∈\1,100,250,500\. As observed, the agents rapidly transition from initial states to stable, high-velocity locomotion. Notably, in complex scenarios such as Ant 4×24× 2 and HalfCheetah 6×16× 1, the physically decoupled agents exhibit remarkable inter-agent coordination, effectively synchronizing their joint movements to maintain balance and maximize forward momentum without conflicts. The high episode returns annotated in the figure (e.g., 12499.812499.8 for HalfCheetah and 7521.97521.9 for Ant) significantly surpass the performance of competing multi-agent baselines reported in Table 3. These visual results confirm that OMAD not only optimizes for scalar rewards but also successfully masters the intricate dynamics required for robust and cooperative multi-agent control.