Paper deep dive
Dreamer-CPC: Message Learning with World Models for Decentralized Multi-agent Reinforcement Learning
Taisuke Takayama, Naoto Yoshida, Tadahiro Taniguchi
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 93%
Last extracted: 7/23/2026, 1:57:58 AM
Summary
The paper introduces Dreamer-CPC, a decentralized model-based multi-agent reinforcement learning (MARL) method that integrates Collective Predictive Coding (CPC) message learning into the world model of DreamerV3. Unlike existing methods that rely solely on current observations, Dreamer-CPC allows agents to infer and exchange messages based on latent states reflecting the history of past observations and actions. Evaluated in the Observer and CatchApple environments, Dreamer-CPC outperforms baselines like IPPO-CPC, particularly in scenarios with missing observations, demonstrating that communication grounded in world model latent dynamics supports effective decentralized coordination.
Entities (8)
Relation Signals (7)
Dreamer-CPC → buildsupon → DreamerV3
confidence 95% · integrates message learning... into the world model of DreamerV3
Dreamer-CPC → evaluatedin → Observer
confidence 95% · We evaluated Dreamer-CPC in two environments: Observer
Dreamer-CPC → evaluatedin → CatchApple
confidence 95% · and CatchApple, a newly introduced task
Dreamer-CPC → integrates → Collective Predictive Coding
confidence 95% · Dreamer-CPC... integrates message learning based on Collective Predictive Coding (CPC) into the world model
CatchApple → characterizedby → missing observations
confidence 90% · task-relevant observations are temporarily missing
Dreamer-CPC → outperforms → IPPO-CPC
confidence 90% · Dreamer-CPC outperformed IPPO-CPC... as well as no-communication baselines.
Dreamer-CPC → uses → Recurrent State-Space Model
confidence 90% · each agent learns messages from the latent states of the recurrent state-space model (RSSM)
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:In multi-agent reinforcement learning (MARL), inter-agent communication is effective for improving performance under partial observability. Representation learning-based approaches enable decentralized agents to learn messages grounded in their own observations, but they rely only on current observations and cannot convey information accumulated over time. We propose Dreamer-CPC, a decentralized model-based MARL method that integrates message learning based on Collective Predictive Coding (CPC) into the world model of DreamerV3. Each agent independently maintains a world model and a message module, and infers and exchanges messages from the latent states of the world model that reflect the history of past observations and actions. We evaluated Dreamer-CPC in two environments: Observer, a non-cooperative information-sharing task, and CatchApple, a newly introduced task in which task-relevant observations are temporarily missing. In both environments, Dreamer-CPC outperformed IPPO-CPC, an existing CPC-based method that generates messages from current observations, as well as no-communication baselines. In particular, in CatchApple, Dreamer-CPC achieved 4 to 5 times the episode return of IPPO-CPC, demonstrating effective coordination where other methods fail due to missing observations. These results suggest that communication grounded in the latent dynamics of world models can support decentralized decision-making when current observations alone are insufficient.
Tags
Links
- Source: https://arxiv.org/abs/2607.19809v1
- Canonical: https://arxiv.org/abs/2607.19809v1
Trouble viewing inline? Open PDF directly →
Full Text
43,368 characters extracted from source content.
Expand or collapse full text
11institutetext: Graduate School of Informatics, Kyoto University, Kyoto, Japan 22institutetext: Research Organization of Science and Technology, Ritsumeikan University, Shiga, Japan 22email: takayama.taisuke.73z@st.kyoto-u.ac.jp, yoshida.naoto.8x@kyoto-u.ac.jp,taniguchi@i.kyoto-u.ac.jp Dreamer-CPC: Message Learning with World Models for Decentralized Multi-agent Reinforcement Learning Taisuke Takayama Naoto Yoshida Tadahiro Taniguchi Abstract In multi-agent reinforcement learning (MARL), inter-agent communication is effective for improving performance under partial observability. Representation learning-based approaches enable decentralized agents to learn messages grounded in their own observations, but they rely only on current observations and cannot convey information accumulated over time. We propose Dreamer-CPC, a decentralized model-based MARL method that integrates message learning based on Collective Predictive Coding (CPC) into the world model of DreamerV3. Each agent independently maintains a world model and a message module, and infers and exchanges messages from the latent states of the world model that reflect the history of past observations and actions. We evaluated Dreamer-CPC in two environments: Observer, a non-cooperative information-sharing task, and CatchApple, a newly introduced task in which task-relevant observations are temporarily missing. In both environments, Dreamer-CPC outperformed IPPO-CPC, an existing CPC-based method that generates messages from current observations, as well as no-communication baselines. In particular, in CatchApple, Dreamer-CPC achieved 4 to 5 times the episode return of IPPO-CPC, demonstrating effective coordination where other methods fail due to missing observations. These results suggest that communication grounded in the latent dynamics of world models can support decentralized decision-making when current observations alone are insufficient. 1 Introduction In partially observable and dynamic environments, an agent cannot always acquire sufficient information about task-relevant states from its current local observation alone [11, 13, 18]. This issue is central in decentralized multi-agent settings, where each agent acts based on its own local observation history and task-relevant information may be distributed across agents. Inter-agent communication can mitigate this problem by allowing agents to share information derived from their private observations and observation histories [1, 6, 30]. These settings motivate communication mechanisms that convey information accumulated and inferred over time, rather than information contained only in the current local observation. In multi-agent reinforcement learning (MARL), communication has been shown to enable information sharing among agents and improve performance under partial observability. Many existing methods adopt centralized training with decentralized execution (CTDE), coupling agents through a centralized optimization framework. This includes architectures such as RIAL, DIAL [5], and CommNet [20], as well as the use of global value functions [17, 21], which improve policy optimization through gradient propagation across agents [30]. Parameter sharing among agents is also widely employed [30]. In addition, many of these methods optimize messages through task rewards and generally assume cooperative settings in which agents learn under a shared reward. While such approaches contribute to engineering performance improvements, they often diverge from the decentralized learning observed in natural agents. In contrast to these centralized paradigms, representation learning-based approaches enable decentralized agents to learn communication grounded in their own observations. Lin et al. [16] proposed a method in which each agent learns a message representation from its own observation using an autoencoder and broadcasts it to other agents. Meanwhile, Collective Predictive Coding (CPC) [24, 22] from emergent communication research formulates symbol emergence as decentralized Bayesian inference in a joint generative model over agents’ observations, providing a theoretical foundation for representation learning-based approaches. Ebara et al. [3] integrated CPC-based message learning into multi-agent reinforcement learning using the Metropolis-Hastings naming game [23], demonstrating that discrete messages indicating agents’ states can emerge through communication between independent agents without centralized training. MARL-CPC [28] formulated autoencoder-based message learning within a variational inference framework and demonstrated that messages useful for coordination can emerge even in non-cooperative settings where rewards are not shared among agents. However, all of these methods generate messages based only on current observations, which is insufficient when task-relevant information is temporarily occluded or when agents must act based on predictions of future states. We propose Dreamer-CPC, which integrates CPC-based message learning into a world model [7] that learns latent dynamics from histories of observations and actions, thereby extending CPC-based message learning beyond current observations. Specifically, each agent learns messages from the latent states of the recurrent state-space model (RSSM) [8] in DreamerV3 [10], enabling communication grounded not only in current observations but also in the history of past observations and actions accumulated by the world model. World model-based MARL methods have been proposed primarily to improve sample efficiency [4, 26, 27, 29], but these methods are all based on the CTDE framework, and the decentralized setting in which each agent learns from its own observation and action history and communication messages alone remains largely unexplored. Nomura et al. [19] proposed a decentralized world model in which each agent maintains an RSSM-based local model and learns messages through InfoNCE loss [25]. However, their policy is learned through behavioral cloning from expert demonstrations and does not involve reinforcement learning in the imagined space of the world model. Dreamer-CPC integrates CPC-based message learning with an RSSM under a unified objective function and establishes a novel model-based MARL framework based on decentralized training with decentralized execution (DTDE) that optimizes policies through imagination rollouts. We evaluated Dreamer-CPC in two environments: Observer [28], a non-cooperative information-sharing task, and CatchApple, a newly introduced environment that requires predictive control under temporally occluded observations. The contributions of this paper are as follows. First, we propose Dreamer-CPC, a novel deep MARL method that integrates CPC-based communication learning into the latent dynamics of an RSSM, enabling communication learning grounded in the temporal context accumulated by the world model beyond current observations. Second, we show that world model learning and message learning can be jointly optimized under a unified objective function. Third, through experiments in Observer and the newly introduced CatchApple environment, we demonstrated that Dreamer-CPC outperforms IPPO-CPC [28], an existing MARL-CPC method, and no-communication baselines. The results show that communication grounded in the latent dynamics of an RSSM can support coordination when task-relevant observations are temporally unavailable. 2 Dreamer-CPC We propose Dreamer-CPC, a decentralized model-based MARL method. Fig. 1 provides an overview of Dreamer-CPC. To enable message learning grounded in temporal latent dynamics beyond current observations, Dreamer-CPC integrates CPC-based message learning into the world model and generates messages from its latent states. Each agent independently maintains a world model and a message module, and infers and exchanges messages based on the temporal latent dynamics accumulated by the RSSM. 2.1 Problem Settings Figure 1: Overview of Dreamer-CPC. Each agent maintains an RSSM-based world model and a message module. Messages are inferred from the latent states of the world model and exchanged between agents. We formulate the multi-agent reinforcement learning problem considered in this paper as a partially observable Markov game (POMG) [11]. The game is specified by =⟨ℐ,,kk∈ℐ,kk∈ℐ,μ0,ptr,pobs,ρkk∈ℐ,γ⟩G= ,S,\A^k\_k ,\X^k\_k , _0,p_tr,p_obs,\ρ^k\_k ,γ . Here, ℐ=1,…,KI=\1,…,K\ is the set of K agents, S is the state space, kA^k is the action space of agent k, and kX^k is the observation space of agent k. The joint action and observation spaces are denoted by =∏k∈ℐkA= _k A^k and =∏k∈ℐkX= _k X^k, respectively. The initial state is sampled as s0∼μ0s_0 _0, where μ0∈Δ() _0∈ (S) is the initial state distribution and Δ() (S) denotes the set of probability distributions over S. At each time step t, each agent k selects an action atk∈ka_t^k ^k according to its policy πkπ^k, using only its available local history. Given the joint action at=(at1,…,atK)a_t=(a_t^1,…,a_t^K), the environment state evolves as st+1∼ptr(⋅∣st,at)s_t+1 p_tr(· s_t,a_t), and the next joint observation xt+1=(xt+11,…,xt+1K)x_t+1=(x_t+1^1,…,x_t+1^K) is generated as xt+1∼pobs(⋅∣st+1,at)x_t+1 p_obs(· s_t+1,a_t). Each agent has its own reward function ρk:×→ℝρ^k:S×A , and its immediate reward at time step t is rtk=ρk(st,at)r_t^k=ρ^k(s_t,a_t). Let π=(π1,…,πK)π=(π^1,…,π^K) denote the joint policy, and let π−kπ^-k denote the policies of all agents except agent k. The discounted return of agent k is Gtk=∑ℓ=0∞γℓrt+ℓkG_t^k= _ =0^∞γ r_t+ ^k, where γ∈[0,1)γ∈[0,1) is the discount factor. The objective of agent k is to maximize Jk(πk;π−k)=μ0,ptr,pobs,π[G0k], J^k(π^k;π^-k)=E_ _0,p_tr,p_obs,π [G_0^k ], (1) where the expectation is taken over trajectories induced by the initial state distribution, the transition kernel, the observation kernel, and the joint policy. 2.2 World Model World models learn the dynamics of the environment from the agent’s experience and enable prediction of future observations and rewards given actions. They map observations into a latent space and predict future state transitions within this space. This enables policy learning through imagined trajectories in the latent space without additional interaction with the real environment. In Dreamer-CPC, each agent k∈ℐk maintains its own world model, which learns the temporal dynamics of the environment from its local observation and action history as well as messages received from other agents. Each agent’s world model is implemented as an RSSM, following DreamerV3. The RSSM has a deterministic recurrent state htkh^k_t and a stochastic latent state ztkz^k_t. At each time step, htkh^k_t is first computed from the previous latent state zt−1kz^k_t-1 and action at−1ka^k_t-1. A prior prediction z^tk z^k_t is then generated from htkh^k_t alone, and ztkz^k_t is inferred from htkh^k_t and the observation xtkx^k_t. From htkh^k_t and ztkz^k_t, the observation xtkx^k_t, reward rtkr^k_t, and continuation flag ct∈0,1c_t∈\0,1\ are predicted, where ct=1c_t=1 indicates that the episode continues beyond time step t. The RSSM of agent k is formulated as follows, where ϕkφ^k denotes its parameters: htk h^k_t =fϕk(ht−1k,zt−1k,at−1k) =f_φ^k(h^k_t-1,z^k_t-1,a^k_t-1) (2) z^tk z^k_t ∼pϕk(⋅∣htk) p_φ^k(· h^k_t) (3) ztk z^k_t ∼qϕk(⋅∣htk,xtk) q_φ^k(· h^k_t,x^k_t) (4) x^tk x^k_t ∼pϕk(⋅∣htk,ztk) p_φ^k(· h^k_t,z^k_t) (5) r^tk r^k_t ∼pϕk(⋅∣htk,ztk) p_φ^k(· h^k_t,z^k_t) (6) c^tk c^k_t ∼pϕk(⋅∣htk,ztk). p_φ^k(· h^k_t,z^k_t). (7) 2.3 World Model and Message Learning Dreamer-CPC integrates the learning of each agent’s RSSM with message learning. In addition to the RSSM, each agent k maintains a recurrent message module. The message module has a message recurrent state ξtkξ^k_t, which summarizes the history of previously inferred messages. At each time step, ξtkξ^k_t is updated from the messages t−1m_t-1 at the previous step. The message mtkm^k_t is modeled stochastically with a prior distribution (message prior) generated from ξtkξ^k_t alone and a posterior distribution (message posterior) conditioned on ξtkξ^k_t together with the RSSM states htkh^k_t and ztkz^k_t. Denoting the message module parameters by ψkψ^k, the message module is formulated as follows: ξtk ξ^k_t =gψk(ξt−1k,t−1) =g_ψ^k(ξ^k_t-1,m_t-1) (8) m^tk m^k_t ∼pψk(⋅∣ξtk) p_ψ^k(· ξ^k_t) (9) mtk m^k_t ∼qψk(⋅∣ξtk,htk,ztk). q_ψ^k(· ξ^k_t,h^k_t,z^k_t). (10) Here, ^t=(m^t1,…,m^tK) m_t=( m^1_t,…, m^K_t) denotes the joint predicted message collecting the messages sampled from each agent’s message prior, and t=(mt1,…,mtK)m_t=(m^1_t,…,m^K_t) denotes the joint inferred message collecting the messages sampled from each agent’s message posterior. Messages are incorporated into two components of the RSSM. First, the joint predicted message ^t m_t is fed into the transition of the recurrent state htkh^k_t. Second, the joint inferred message tm_t is added as a conditioning variable for the observation prediction x^tk x^k_t. Accordingly, Eq. (2) and Eq. (5) are extended as follows: htk h^k_t =fϕk(ht−1k,zt−1k,at−1k,^t) =f_φ^k(h^k_t-1,z^k_t-1,a^k_t-1, m_t) (11) x^tk x^k_t ∼pϕk(⋅∣htk,ztk,t). p_φ^k(· h^k_t,z^k_t,m_t). (12) Eq. (3), Eq. (4), Eq. (6), and Eq. (7) remain unchanged from Section 2.2. Fig. 2 provides an overview of the learning procedure for this message-augmented world model. To realize decentralized learning, Dreamer-CPC applies stop gradient to messages generated by other agents. This allows each agent to use other agents’ messages for prediction without propagating gradients through other agents’ parameters. Figure 2: Overview of world-model learning in Dreamer-CPC. Each agent learns an RSSM state from its local observation history and action history. The message module predicts a message from its recurrent state and infers a message from the RSSM state. The learned model reconstructs observations using the inferred message and received messages, and predicts rewards and continuation from the RSSM state. Messages from other agents are treated with stop gradient, so that communication can be used for prediction without backpropagating through other agents. The world model and the message module are learned jointly by minimizing a unified loss function. We write pϕk(ztk∣htk)p_φ^k(z_t^k h_t^k) and pψk(mtk∣ξtk)p_ψ^k(m_t^k _t^k) for the prior densities corresponding to the distributions from which z^tk z_t^k and m^tk m_t^k are sampled. For compact notation, we denote the prediction targets of agent k at time step t by ytk=(xtk,rtk,ct)y_t^k=(x_t^k,r_t^k,c_t). The prediction likelihood for agent k is factorized as pϕk(ytk∣htk,ztk,t) p_φ^k(y_t^k h_t^k,z_t^k,m_t) =pϕk(xtk∣htk,ztk,t)pϕk(rtk∣htk,ztk)pϕk(ct∣htk,ztk). =p_φ^k(x_t^k h_t^k,z_t^k,m_t)p_φ^k(r_t^k h_t^k,z_t^k)p_φ^k(c_t h_t^k,z_t^k). (13) We then define the variational posterior over the stochastic latent state and the message of agent k as qϕk,ψk(z1:Tk,m1:Tk) q_φ^k,ψ^k(z_1:T^k,m_1:T^k) =∏t=1Tqϕk(ztk∣htk,xtk)qψk(mtk∣ξtk,htk,ztk). = _t=1^Tq_φ^k(z_t^k h_t^k,x_t^k)q_ψ^k(m_t^k _t^k,h_t^k,z_t^k). (14) Assuming fixed initial states, we omit the initial-state terms, and the marginal log-likelihood of the prediction targets can be rewritten as logpϕk,ψk(y1:Tk∣a0:T−1k) p_φ^k,ψ^k(y_1:T^k a_0:T-1^k) =log∬∏t=1T[pϕk(ytk∣htk,ztk,t)pϕk(ztk∣htk)pψk(mtk∣ξtk)]dz1:Tkdm1:Tk \>= _t=1^T [p_φ^k(y_t^k h_t^k,z_t^k,m_t)p_φ^k(z_t^k h_t^k)p_ψ^k(m_t^k _t^k) ]\,dz_1:T^kdm_1:T^k (15) =logqϕk,ψk[∏t=1T[pϕk(ytk∣htk,ztk,t)pϕk(ztk∣htk)qϕk(ztk∣htk,xtk)pψk(mtk∣ξtk)qψk(mtk∣ξtk,htk,ztk)]]. \>= _q_φ^k,ψ^k [ _t=1^T [p_φ^k(y_t^k h_t^k,z_t^k,m_t) p_φ^k(z_t^k h_t^k)q_φ^k(z_t^k h_t^k,x_t^k) p_ψ^k(m_t^k _t^k)q_ψ^k(m_t^k _t^k,h_t^k,z_t^k) ] ]. (16) Applying Jensen’s inequality [12, 14] yields the following variational lower bound: ℰk=∑t=1Tqϕk,ψk[logpϕk(ytk∣htk,ztk,t)]−∑t=1Tqϕk,ψk[DKL(qϕk(ztk∣htk,xtk)∥pϕk(ztk∣htk))]−∑t=1Tqϕk,ψk[DKL(qψk(mtk∣ξtk,htk,ztk)∥pψk(mtk∣ξtk))], aligned E^k&= _t=1^TE_q_φ^k,ψ^k [ p_φ^k(y_t^k h_t^k,z_t^k,m_t) ]\\ & - _t=1^TE_q_φ^k,ψ^k [D_KL (q_φ^k(z_t^k h_t^k,x_t^k)\,\|\,p_φ^k(z_t^k h_t^k) ) ]\\ & - _t=1^TE_q_φ^k,ψ^k [D_KL (q_ψ^k(m_t^k _t^k,h_t^k,z_t^k)\,\|\,p_ψ^k(m_t^k _t^k) ) ], aligned (17) which satisfies logpϕk,ψk(y1:Tk∣a0:T−1k) p_φ^k,ψ^k(y_1:T^k a_0:T-1^k) ≥ℰk. ^k. (18) In practice, we optimize a stabilized surrogate of the variational objective that balances predictive accuracy with the predictability of latent states and messages. Following DreamerV3, we replace the KL regularizers in Eq. (17) with KL-balanced losses [9] with free bits [15]. Here, βpred _pred, βdyn _dyn, βrep _rep, βm-dyn _m -dyn, and βm-rep _m -rep are weighting coefficients for the corresponding loss terms. We set βpred=βdyn=1 _pred= _dyn=1, βrep=0.1 _rep=0.1, βm-dyn=1×10−3 _m -dyn=1× 10^-3, and βm-rep=1×10−4 _m -rep=1× 10^-4. The resulting objective for agent k is ℒwm-msgk ^k_wm -msg =∑t=1T[βpredℒpredk+βdynℒdynk+βrepℒrepk+βm-dynℒm-dynk+βm-repℒm-repk], \>= _t=1^TE [ _predL_pred^k+ _dynL_dyn^k+ _repL_rep^k+ _m -dynL_m -dyn^k+ _m -repL_m -rep^k ], (19) where ℒpredk _pred^k =−logpϕk(ytk∣htk,ztk,t) =- p_φ^k (y_t^k h_t^k,z_t^k,m_t ) (20) ℒdynk _dyn^k =max(1,DKL[sg(qϕk(ztk∣htk,xtk))∥pϕk(ztk∣htk)]) = (1,D_KL [sg (q_φ^k(z_t^k h_t^k,x_t^k) )\,\|\,p_φ^k(z_t^k h_t^k) ] ) (21) ℒrepk _rep^k =max(1,DKL[qϕk(ztk∣htk,xtk)∥sg(pϕk(ztk∣htk))]) = (1,D_KL [q_φ^k(z_t^k h_t^k,x_t^k)\,\|\,sg (p_φ^k(z_t^k h_t^k) ) ] ) (22) ℒm-dynk _m -dyn^k =max(0.5,DKL[sg(qψk(mtk∣ξtk,htk,ztk))∥pψk(mtk∣ξtk)]) = (0.5,D_KL [sg (q_ψ^k(m_t^k _t^k,h_t^k,z_t^k) )\,\|\,p_ψ^k(m_t^k _t^k) ] ) (23) ℒm-repk _m -rep^k =max(0.5,DKL[qψk(mtk∣ξtk,htk,ztk)∥sg(pψk(mtk∣ξtk))]). = (0.5,D_KL [q_ψ^k(m_t^k _t^k,h_t^k,z_t^k)\,\|\,sg (p_ψ^k(m_t^k _t^k) ) ] ). (24) 2.4 Actor–Critic Learning Dreamer-CPC follows the same actor–critic learning procedure as DreamerV3. The actor πθk _θ^k of each agent k∈ℐk parameterizes a distribution over atka^k_t conditioned on htkh^k_t and ztkz^k_t, and the critic vηkv_η^k parameterizes a distribution that estimates the discounted return Gtk=∑ℓ=0∞γℓrt+ℓkG^k_t= _ =0^∞γ r^k_t+ under the same conditioning, where θkθ^k and ηkη^k denote the actor and critic parameters, respectively. Both are trained with the same objective functions as DreamerV3. The key difference from DreamerV3 is that imagined trajectories are generated with inter-agent message exchange. Fig. 3 illustrates this procedure. At each imagination step, every agent infers a message from its world-model state and exchanges it with other agents. The received messages are then incorporated into the subsequent RSSM transition. Algorithm 2 details this procedure. Figure 3: Overview of actor and critic learning with message exchange during imagination. At each imagined step, agents synchronously apply the message generation and inference procedure, share messages, and update their RSSM states using the exchanged messages. Actor and critic losses are then computed on the resulting imagination rollout. Stop gradient labels indicate that received messages are detached during optimization. 3 Experiments We evaluated our proposed method in two MARL environments where each agent receives an individual reward, while information sharing through communication can affect overall performance. In the Observer [28] environment, the global state of the environment can be accurately inferred at each time step by aggregating the observations of all agents. In contrast, we constructed the CatchApple environment, in which agents’ observations are temporarily missing at certain time steps. As a result, there are time steps at which the global state cannot be accurately inferred even by aggregating the observations of all agents. Therefore, each agent must infer the current state from its own observation history and the information received from other agents, and act while predicting future states. We compared Dreamer-CPC with IPPO-CPC [28], IPPO [2], independent DreamerV3 agents without a message channel (no-comm), and DreamerV3 agents that share observations with each other (obs-shared). For IPPO-CPC and Dreamer-CPC, which include a message module, we evaluated two message-channel configurations: a single 20-way one-hot message (20×120× 1 message) and four independent 5-way one-hot messages (5×45× 4 messages). Each agent’s RSSM and actor-critic were implemented as approximately 3M-parameter models based on the architecture of DreamerV3. The message module uses a BlockGRU (hidden==64, blocks==8), and both the message prior and posterior networks are 2-layer MLPs (hidden==128) with GELU activation. 3.1 Observer: Information Sharing without Shared Rewards Figure 4: Overview of the Observer environment. Agent 1 observes the reward location, shown as the colored cell, but cannot move, whereas Agent 2 observes only its own grid position and must locate the reward through communication. This environment is a non-cooperative MARL environment in which multiple agents receive asymmetric observations, the complete state information of the environment can be reconstructed by integrating their observations, and rewards are not shared, so each agent acts according to its own individual objective. The environment contains two agents. Agent 1 remains stationary and receives no reward, observes a 16-dimensional one-hot vector indicating the reward location, and has a single no-op action, included only to define its action space. In contrast, Agent 2 acts on a 4×44× 4 grid and observes a one-hot vector indicating its own grid position, but cannot observe the location of the reward. Agent 2 selects one action from the six actions: 2=,,,,,A_2=\ up, down, left, right, still, dig\. It can obtain a reward of +1+1 by selecting the dig action on the cell containing the hidden reward, whereas all other actions incur a penalty of −0.01-0.01. Episodes were run with a fixed horizon of H=200H=200 steps. Performance is reported as Agent 2’s episode return, G2=∑t=1Hrt2G^2= _t=1^Hr_t^2. Since Agent 1 receives no reward, this environment tests whether message grounding based on representation learning alone can transmit task-relevant information available to another agent. 3.2 CatchApple: Information Sharing with Temporarily Missing Observations Figure 5: Overview of CatchApple environment. (A) The environment contains an 8×88× 8 falling object area and one ground row. Agent 1 observes the falling object trajectory and receives no reward, whereas Agent 2 controls the catcher on the ground row and observes its own column. The hatched region indicates an occluded region in which the object position is replaced by a hidden flag. (B) The object moves downward, possibly with diagonal motion and boundary reflection, and capture is judged when the object reaches the lowest row of the falling object area. This experiment evaluates whether agents can predict the underlying environmental state and act accordingly in an environment where there are time steps at which the full state cannot be reconstructed even by aggregating the observations of all agents. Fig. 5 provides an overview of the environment. The environment contains two agents. Agent 1 remains stationary, receives no reward, observes the trajectory of the falling object, and has only a single no-op action. In contrast, Agent 2 controls a catcher located on the ground row and observes its own column, but cannot directly observe the future landing column of the object. Since Agent 2 observes the object’s column only after the capture decision has already been made, it cannot capture the current object by reacting to its own observation. Therefore, Agent 2 must predict the object’s future landing column from the message received from Agent 1 and move to that column in advance. The field consists of a falling object area with 8 columns and 8 rows, and a ground row. Agent 1 observes a 65-dimensional one-hot vector representing either the position of the object on the 8×88× 8 falling object area or a hidden flag indicating that the object is in the occluded row. Agent 1 has only a single no-op action. Agent 2 observes an 8-dimensional one-hot vector indicating its own column and a 9-dimensional one-hot vector indicating either the object’s column or a hidden flag. Agent 2 selects one action from three actions, 2=,,A_2=\ still, left, right\. For each object, the initial column is sampled uniformly at random, and the trajectory type is sampled from three possible types: vertical, diagonally left, and diagonally right. At each step, the object moves downward by one row. For diagonal trajectories, the object additionally moves horizontally by one cell every two rows. When the object reaches a field boundary, its horizontal motion is reflected. A capture is successful if the object’s column on the lowest row of the falling object area matches the column of Agent 2. When a capture succeeds, Agent 2 receives a reward of +1+1; at all other time steps, it receives a penalty of −0.01-0.01. Agent 1 receives no reward. Immediately after the capture judgment, a new object is generated. Episodes were run with a fixed horizon of H=500H=500 steps. Performance is reported as Agent 2’s episode return, G2=∑t=1Hrt2G^2= _t=1^Hr_t^2. As in Observer, Agent 1 receives no reward. This environment further tests whether such message grounding supports coordination under temporally occluded observations. Figure 6: Learning curves in Observer and CatchApple. Curves show the IQM of episode return over 10 independent runs. Shaded regions show 95% confidence intervals estimated by paired nonparametric bootstrap over runs. The left and right panels show Observer and CatchApple, respectively. 3.3 Results Fig. 6 shows the learning curves in Observer and CatchApple. Each curve represents the interquartile mean (IQM) of the episode return over 10 independent runs, and the shaded regions represent the 95% confidence intervals estimated by bootstrap resampling of runs with replacement using 2,000 bootstrap samples. In Observer, the obs-shared condition, in which agents share observations, reached an IQM of 53.63. Among the settings without direct observation sharing, Dreamer-CPC achieved the highest performance. The 5×45× 4 message configuration reached a final-step IQM of 39.85 (95% CI [34.22, 40.82]), and the 20×120× 1 message configuration reached 19.72 (95% CI [8.48, 25.35]). In contrast, IPPO-CPC reached only 7.31 with the 20×120× 1 message configuration and 6.81 with the 5×45× 4 message configuration. IPPO, which has no communication channel, and the no-comm DreamerV3 condition remained near zero, with IQMs of 1.74 and 0.48, respectively. Observer is an environment in which the full state can be reconstructed by aggregating the observations of all agents at each time step. These results suggest that Dreamer-CPC can learn information transmission that substantially outperforms the baseline methods, even in non-cooperative settings with severely restricted observability. In CatchApple, the obs-shared condition again achieved the highest return (IQM 38.21). Dreamer-CPC attained an IQM of 28.53 (95% CI [23.87, 31.56]) with 5×45× 4 messages and 20.47 (95% CI [19.04, 22.45]) with 20×120× 1 message, substantially outperforming IPPO-CPC (5×45× 4: 5.52, 20×120× 1: 4.98) and the no-comm DreamerV3 condition (5.93). IPPO-CPC, on the other hand, remained at a level comparable to IPPO (4.66), which has no communication channel, and the no-comm condition. In CatchApple, there exist time steps at which observations are temporarily missing, so Agent 2 must act before directly observing the landing column of the object. These results suggest that message learning grounded in the latent dynamics of a world model can contribute to acquiring coordinated behavior in environments where observations are temporarily missing. 4 Conclusion We proposed Dreamer-CPC, a decentralized model-based MARL method that integrates CPC-based message learning into an RSSM-based world model. In Dreamer-CPC, each agent learns stochastic messages through a recurrent message module, and the received messages are used for predicting observations, rewards, and continuation signals. By treating messages from other agents as stop-gradient inputs, the method achieves decentralized learning without introducing direct gradient paths between agents. Experiments in Observer and CatchApple showed that Dreamer-CPC achieves higher performance than IPPO-CPC and no-communication baselines in non-cooperative settings. In particular, the results on CatchApple suggest that communication learned through a world model can support effective coordination even when task-relevant observations are temporarily unavailable. This work has several limitations. First, we did not analyze what information the learned messages carry. Qualitative and quantitative evaluation of the content and structure of messages is necessary for a deeper understanding of the proposed method. Second, our experiments are limited to two-agent settings, and validation with a larger number of agents is needed. Third, both Observer and CatchApple are environments designed in this paper, and evaluating the effectiveness of Dreamer-CPC on more general MARL benchmarks remains as future work. Acknowledgments We would like to thank Prof. Masatoshi Nagano for extensive and valuable advice on the writing of this paper. AI Usage Declaration We used generative AI tools to assist in the implementation of the proposed method and in the preparation of this manuscript. All generated outputs, including source code and text, were reviewed, verified, and approved by the authors, who take full responsibility for the content of this work. Appendix 0.A Algorithms Algorithm 1 World Model and Message Module Learning 1:Initialize h0k,z0k,ξ0k,m0kh^k_0,z^k_0,ξ^k_0,m^k_0 for all k∈ℐk 2:Sample trajectories xtk,atk,rtk,ctt=1T\x^k_t,a^k_t,r^k_t,c_t\_t=1^T from replay buffer for all k∈ℐk 3:for t=1t=1 to sequence length T do 4: for each agent k∈ℐk do 5: ξtk←gψk(ξt−1k,(mt−1k,sg(mt−1−k)))ξ^k_t← g_ψ^k\! (ξ^k_t-1,\,(m^k_t-1,\,sg(m^-k_t-1)) ) 6: m^tk∼pψk(⋅∣ξtk) m^k_t p_ψ^k(· ξ^k_t) 7: end for 8: for each agent k∈ℐk do 9: htk←fϕk(ht−1k,zt−1k,at−1k,(m^tk,sg(m^t−k)))h^k_t← f_φ^k\! (h^k_t-1,\,z^k_t-1,\,a^k_t-1,\,( m^k_t,\,sg( m^-k_t)) ) 10: z^tk∼pϕk(⋅∣htk) z^k_t p_φ^k(· h^k_t) 11: ztk∼qϕk(⋅∣htk,xtk)z^k_t q_φ^k(· h^k_t,x^k_t) 12: mtk∼qψk(⋅∣ξtk,htk,ztk)m^k_t q_ψ^k(· ξ^k_t,h^k_t,z^k_t) 13: end for 14: for each agent k∈ℐk do 15: x^tk∼pϕk(⋅∣htk,ztk,(mtk,sg(mt−k))) x^k_t p_φ^k(· h^k_t,z^k_t,(m^k_t,\,sg(m^-k_t))) 16: r^tk∼pϕk(⋅∣htk,ztk) r^k_t p_φ^k(· h^k_t,z^k_t) 17: c^tk∼pϕk(⋅∣htk,ztk) c^k_t p_φ^k(· h^k_t,z^k_t) 18: end for 19:end for 20:Compute ℒwm-msgkL^k_wm -msg and update ϕk,ψkφ^k,ψ^k for each agent k∈ℐk Algorithm 2 Actor–Critic Learning with Imagination Rollouts 1:Initialize h0k,z^0k,ξ0k,a0k,m0kh^k_0, z^k_0,ξ^k_0,a^k_0,m^k_0 for all k∈ℐk 2:for t=1t=1 to imagination horizon H do 3: for each agent k∈ℐk do 4: ξtk←gψk(ξt−1k,(mt−1k,sg(mt−1−k)))ξ^k_t← g_ψ^k\! (ξ^k_t-1,\,(m^k_t-1,\,sg(m^-k_t-1)) ) 5: m^tk∼pψk(⋅∣ξtk) m^k_t p_ψ^k(· ξ^k_t) 6: end for 7: for each agent k∈ℐk do 8: htk←fϕk(ht−1k,z^t−1k,at−1k,(m^tk,sg(m^t−k)))h^k_t← f_φ^k\! (h^k_t-1,\, z^k_t-1,\,a^k_t-1,\,( m^k_t,\,sg( m^-k_t)) ) 9: z^tk∼pϕk(⋅∣htk) z^k_t p_φ^k(· h^k_t) 10: mtk∼qψk(⋅∣ξtk,htk,z^tk)m^k_t q_ψ^k(· ξ^k_t,h^k_t, z^k_t) 11: atk∼πθk(⋅∣htk,z^tk)a^k_t _θ^k(· h^k_t, z^k_t) 12: r^tk∼pϕk(⋅∣htk,z^tk) r^k_t p_φ^k(· h^k_t, z^k_t) 13: c^tk∼pϕk(⋅∣htk,z^tk) c^k_t p_φ^k(· h^k_t, z^k_t) 14: end for 15:end for 16:Update actor and critic for each agent following DreamerV3 References [1] S. V. Albrecht, F. Christianos, and L. Schäfer (2024) Multi-agent reinforcement learning: foundations and modern approaches. MIT Press. Cited by: §1. [2] C. S. de Witt, T. Gupta, D. Makoviichuk, V. Makoviychuk, P. H. S. Torr, M. Sun, and S. Whiteson (2020) Is independent learning all you need in the StarCraft multi-agent challenge?. arXiv preprint arXiv:2011.09533. Cited by: §3. [3] H. Ebara, T. Nakamura, A. Taniguchi, and T. Taniguchi (2023) Multi-agent reinforcement learning with emergent communication using discrete and indifferentiable message. In 2023 15th International Congress on Advanced Applied Informatics Winter (IIAI-AAI-Winter), p. 366–371. Cited by: §1. [4] V. Egorov and A. Shpilman (2022) Scalable multi-agent model-based reinforcement learning. In 21st International Conference on Autonomous Agents and Multiagent Systems, AAMAS 2022, Auckland, New Zealand, May 9-13, 2022, p. 381–390. Cited by: §1. [5] J. N. Foerster, Y. M. Assael, N. de Freitas, and S. Whiteson (2016) Learning to communicate with deep multi-agent reinforcement learning. In Advances in Neural Information Processing Systems, Vol. 29, p. 2137–2145. Cited by: §1. [6] S. Gronauer and K. Diepold (2022) Multi-agent deep reinforcement learning: a survey. Artificial Intelligence Review 55 (2), p. 895–943. Cited by: §1. [7] D. Ha and J. Schmidhuber (2018) Recurrent world models facilitate policy evolution. In Advances in Neural Information Processing Systems, Cited by: §1. [8] D. Hafner, T. Lillicrap, I. Fischer, R. Villegas, D. Ha, H. Lee, and J. Davidson (2019) Learning latent dynamics for planning from pixels. In Proceedings of the 36th International Conference on Machine Learning, p. 2555–2565. Cited by: §1. [9] D. Hafner, T. P. Lillicrap, M. Norouzi, and J. Ba (2021) Mastering Atari with discrete world models. In International Conference on Learning Representations, Cited by: §2.3. [10] D. Hafner, J. Pasukonis, J. Ba, and T. Lillicrap (2025) Mastering diverse control tasks through world models. Nature 640 (8059), p. 647–653. Cited by: §1. [11] E. A. Hansen, D. S. Bernstein, and S. Zilberstein (2004) Dynamic programming for partially observable stochastic games. In Proceedings of the Nineteenth National Conference on Artificial Intelligence, p. 709–715. Cited by: §1, §2.1. [12] J. L. W. V. Jensen (1906) Sur les fonctions convexes et les inégalités entre les valeurs moyennes. Acta Mathematica 30 (1), p. 175–193. Cited by: §2.3. [13] L. P. Kaelbling, M. L. Littman, and A. R. Cassandra (1998) Planning and acting in partially observable stochastic domains. Artificial Intelligence 101 (1–2), p. 99–134. Cited by: §1. [14] D. P. Kingma and M. Welling (2014) Auto-encoding variational bayes. In 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings, Cited by: §2.3. [15] D. P. Kingma, T. Salimans, R. Jozefowicz, X. Chen, I. Sutskever, and M. Welling (2016) Improved variational inference with inverse autoregressive flow. In Advances in Neural Information Processing Systems, Cited by: §2.3. [16] T. Lin, J. Huh, C. Stauffer, S. Lim, and P. Isola (2021) Learning to ground multi-agent communication with autoencoders. In Advances in Neural Information Processing Systems, p. 15230–15242. Cited by: §1. [17] R. Lowe, Y. Wu, A. Tamar, J. Harb, P. Abbeel, and I. Mordatch (2017) Multi-agent actor-critic for mixed cooperative-competitive environments. In Advances in Neural Information Processing Systems, Vol. 30. Cited by: §1. [18] A. Nayyar, A. Mahajan, and D. Teneketzis (2013) Decentralized stochastic control with partial history sharing: a common information approach. IEEE Transactions on Automatic Control 58 (7), p. 1644–1658. Cited by: §1. [19] K. Nomura, T. Aoki, T. Taniguchi, and T. Horii (2025) Decentralized collective world model for emergent communication and coordination. In 2025 IEEE International Conference on Development and Learning (ICDL), p. 1–8. Cited by: §1. [20] S. Sukhbaatar, A. Szlam, and R. Fergus (2016) Learning multiagent communication with backpropagation. In Advances in Neural Information Processing Systems, Cited by: §1. [21] P. Sunehag, G. Lever, A. Gruslys, W. M. Czarnecki, V. Zambaldi, M. Jaderberg, M. Lanctot, N. Sonnerat, J. Z. Leibo, K. Tuyls, and T. Graepel (2018) Value-decomposition networks for cooperative multi-agent learning based on team reward. In Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, p. 2085–2087. Cited by: §1. [22] T. Taniguchi, Y. Hirai, M. Suzuki, S. Murata, T. Horii, and K. Tanaka (2025) System 0/1/2/3: quad-process theory for multitimescale embodied collective cognitive systems. Artificial Life 31 (4), p. 465–496. Cited by: §1. [23] T. Taniguchi, Y. Yoshida, Y. Matsui, N. L. Hoang, A. Taniguchi, and Y. Hagiwara (2023) Emergent communication through metropolis-hastings naming game with deep generative models. Advanced Robotics 37 (19), p. 1266–1282. Cited by: §1. [24] T. Taniguchi (2024) Collective predictive coding hypothesis: symbol emergence as decentralized bayesian inference. Frontiers in Robotics and AI 11. Cited by: §1. [25] A. van den Oord, Y. Li, and O. Vinyals (2018) Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748. Cited by: §1. [26] Z. Wu, C. Yu, C. Chen, J. Hao, and H. H. Zhuo (2023) Models as agents: optimizing multi-step predictions of interactive local models in model-based multi-agent reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, Cited by: §1. [27] Z. Xu, D. Li, B. Zhang, Y. Zhan, Y. Bai, and G. Fan (2022) Mingling foresight with imagination: model-based cooperative multi-agent reinforcement learning. In Advances in Neural Information Processing Systems, Cited by: §1. [28] N. Yoshida and T. Taniguchi (2026) Reward-independent messaging for decentralized multi-agent reinforcement learning. In Neural Information Processing, Lecture Notes in Computer Science, Vol. 16309, p. 367–382. Cited by: §1, §1, §1, §3, §3. [29] Y. Zhang, C. Bai, B. Zhao, J. Yan, X. Li, and X. Li (2025) Decentralized Transformers with Centralized Aggregation Are Sample-Efficient Multi-Agent World Models. Transactions on Machine Learning Research. Cited by: §1. [30] C. Zhu, M. Dastani, and S. Wang (2024) A survey of multi-agent deep reinforcement learning with communication. Autonomous Agents and Multi-Agent Systems 38 (1). Cited by: §1, §1.