Paper deep dive
Decoupling Communication from Policy: Robust MARL under Bandwidth Constraints
Alexi Canesse, Benoît Goupil, Jesse Read, Sonia Vanier
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%
Last extracted: 7/8/2026, 6:17:03 PM
Summary
The paper addresses severe bandwidth constraints in multi-agent reinforcement learning (MARL) by introducing SLIM, an architecture that explicitly decouples the communication pathway from the policy's latent representation. It proposes a normalised per-agent bandwidth budget ($\beta$) to unify sparsity, communication rounds, and message dimension into a single constraint. SLIM utilizes a dedicated communication module with a message history cache and transformer-based attention to aggregate information, enabling robust performance under limited bandwidth while preserving high policy capacity. Evaluated on several partially-observable MARL benchmarks, the method achieves state-of-the-art results and demonstrates scalability and robustness as bandwidth decreases.
Entities (8)
Relation Signals (8)
SLIM → decouples → Policy Latent Representation
confidence 96% · SLIM explicitly decouples communication from the policy input via a dedicated communication module, allowing the message dimension to be drastically reduced without restricting the policy’s high-dimensional latent representation.
beta → unifies → Sparsity, Rounds, and Message Dimension
confidence 95% · By unifying message size, transmission rate, and graph sparsity into one constraint, we provide a framework for systematic benchmarking of communication strategies under identical physical bandwidth limitations.
SLIM → evaluatedon → Partially-observable MARL Benchmarks
confidence 94% · We evaluate our method on several partially-observable MARL benchmarks, where communication is essential.
SLIM → follows → CTDE
confidence 93% · Our training framework follows this paradigm [CTDE], allowing agents to be trained with access to global information while executing policies based only on local observations.
SLIM → uses → Message History Cache
confidence 92% · To address the partial observability of the environment, each agent maintains a message cache... stores the history of all exchanged messages
SLIM → benchmarksagainst → CommNet, IC3Net, TarMAC, CommFormer
confidence 91% · We compare our SLIM architecture against a diverse set of established MARL baselines... CommNet... IC3Net... TarMAC... CommFormer
SLIM → employs → Transformer-based Attention
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Communication enables coordination in multi-agent reinforcement learning (MARL), but many real-world applications, e.g., search-and-rescue with drone swarms, operate under severe bandwidth constraints. Many communication architectures still expose a coupled bottleneck in which a shared latent representation is used for both policy execution and inter-agent communication. Consequently, reducing message size directly limits the policy's latent space, often leading to significant performance degradation. We address this with two contributions. First, we introduce $\beta$, a normalised per-agent bandwidth budget that unifies sparsity, rounds, and message dimension into a single comparable constraint. Second, we provide SLIM, a minimal architecture that decouples the communication pathway from the policy's latent representation, allowing us to isolate the effect of bandwidth from the effect of policy capacity while benefiting from in-step communication. We evaluate our method on several partially-observable MARL benchmarks, where communication is essential. Our approach achieves state-of-the-art performance and exhibits scalability and robustness under limited communication, with only marginal degradation as bandwidth is reduced.
Tags
Links
- Source: https://arxiv.org/abs/2605.21085v1
- Canonical: https://arxiv.org/abs/2605.21085v1
Trouble viewing inline? Open PDF directly →
Full Text
52,605 characters extracted from source content.
Expand or collapse full text
Decoupling Communication from Policy: Robust MARL under Bandwidth Constraints Alexi Canesse, Benoît Goupil, Jesse Read, Sonia Vanier École polytechnique (LIX), CNRS, Institut Polytechnique de Paris, Palaiseau, France alexi.canesse@polytechnique.edu Abstract Communication enables coordination in multi-agent reinforcement learning (MARL), but many real-world applications, e.g., search-and-rescue with drone swarms, operate under severe bandwidth constraints. Many communication archi- tectures still expose a coupled bottleneck in which a shared latent representation is used for both policy execution and inter-agent communication. Consequently, reducing message size directly limits the policy’s latent space, often leading to sig- nificant performance degradation. We address this with two contributions. First, we introduceβ, a normalised per-agent bandwidth budget that unifies sparsity, rounds, and message dimension into a single comparable constraint. Second, we provide SLIM, a minimal architecture that decouples the communication pathway from the policy’s latent representation, allowing us to isolate the effect of bandwidth from the effect of policy capacity while benefiting from in-step communication. We evaluate our method on several partially-observable MARL benchmarks, where communication is essential. Our approach achieves state-of-the-art performance and exhibits scalability and robustness under limited communication, with only marginal degradation as bandwidth is reduced. 1 Introduction Multi-agent reinforcement learning (MARL) has seen significant advancements in recent years, enabling agents to learn cooperative behaviours in complex environments [37,4,44]. One of the main difficulties is effective cooperation between agents, especially in partially-observable settings where each agent has access to only limited information about the environment and other agents. Some methods rely on a central controller, effectively transforming the multi-agent problem into a single-agent one [3,40]. However, such centralised approaches are often impractical in real-world applications due not only to scalability limitations (with the state and action spaces scaling with the number of agents) but also because such centralisation is simply not feasible; agents may be required to operate in a decentralised fashion. Centralised training with decentralised execution (CTDE) [23,19] has emerged as a popular paradigm, allowing agents to be trained with access to global information while executing policies based only on local observations. Consequently, communication becomes essential to compensate for limited local observations. Communication is a mechanism that enables agents to get broader information about the environment and other agents. It enables agents to share information about their local observations, intentions, or learnt knowledge to improve cooperation and overall team performance. However, many real- world applications, such as search-and-rescue missions with drone swarms, autonomous vehicle fleets coordinating in traffic, or underwater exploration robots, involve severe bandwidth constraints. Reducing the size of communicated messages is essential for MARL systems to be deployed in such settings. Preprint. arXiv:2605.21085v1 [cs.MA] 20 May 2026 o t 0 o t 1 . . . o t n o t 0 o t 1 . . . o t n Observations Encoder ̃o t 0 ̃o t 1 . . . ̃o t n ̃o t 0 ̃o t 1 . . . ̃o t n Comm. Encoder m t−1 0 m t−1 1 . . . m t−1 n m t 0 m t 1 . . . m t n m t−1 0 m t−1 1 . . . m t−1 n . . . ˆo 0 0 m t−1 0 m t−1 1 . . . m t−1 n . . . m 0 0 Cache L Linear Positional embeddings K Q V Multi-head Attention Feed Forward L ̃m t 0 ̃m t 1 . . . ̃m t n ̃m t 0 ̃m t 1 . . . ̃m t n Communication Module ̃o t 0 , ̃m t 0 ̃o t 1 , ̃m t 1 . . . ̃o t n , ̃m t n ̃o t 0 , ̃m t 0 ̃o t 1 , ̃m t 1 . . . ̃o t n , ̃m t n Resnet Policy Policy Module a t 0 a t 1 . . . a t n a t 0 a t 1 . . . a t n Actions Figure 1: Architecture of the method. Observations(o t i ) i,t of agentsi 1,...,n are encoded to reduce their dimension and increase their expressiveness before being passed to the Communication Module and the Policy Module. Crucially, encoded observations bypass the communication to reach the policy, allowing the communication module to reduce the dimension of the communication, through messages (m t i ) i,t , without loss of information for the policy. A 1 A 2 A 3 A 4 A 5 A 6 A 3 A 4 A 3 A 4 Obs.Obs. Act.Act. A 3 A 4 ObservationObservation ActionAction d in < d out Figure 2: Problem setting. Agents inter- act; receive local observations, exchange messages, and take actions to maximise rewards. We designed a method to re- duce the size of the communicated mes- sages with limited loss in performance. However, reducing communication without compromis- ing on performance is a significant challenge. Machine learning models typically rely on high-dimensional fea- ture spaces, and forcing them to operate in very low- dimensional spaces can lead to significant performance degradation [46]. In MARL, many existing communi- cation methods use a single shared latent representation both for the agent’s policy and for the message it trans- mits [37,35,13]. In that regime, shrinking the message dimension imposes a double penalty: it removes infor- mation from transmitted messages and simultaneously re- duces the representational capacity available to the policy itself, degrading overall performance. Some methods [4] already use a dedicated communication pathway but they communicate between timesteps, which introduces a lag in the communication. This issue is often solved using multi- round communication, but that is not always possible and increases the communication cost. When deploying these systems in the real world, wireless networks impose two primary constraints: contention for medium access and bandwidth limitations [29]. The former requires sparsity in the communication graph (i.e., reducing the frequency or number of connections) while the latter can be addressed by employing either sparsity or smaller messages. The main focus of this paper is on the second constraint (as displayed in Figure 2). We posit that optimising the size of individual messages is a more effective strategy for preserving coordination under bandwidth constraints. To address this, we introduce Subdivided Lightweight Inter-agent Messaging (SLIM), a method that explicitly decouples communication from the policy input via a dedicated communication module. This separation allows the message dimension to be drastically reduced without restricting the policy’s high-dimensional latent representation. We evaluate our approach across several partially observable MARL environments where communication is essential. Our method achieves state-of- the-art performance in high-bandwidth settings, while maintaining robust performance even as the message dimension is constrained to minimal values. Concretely, we provide the following contributions: • We establish a standardised evaluation protocol for communication-efficient MARL by defining a normalised bandwidth limitβ. By unifying message size, transmission rate, and graph sparsity into one constraint, we provide a framework for systematic benchmarking of communication strategies under identical physical bandwidth limitations. • We propose a communication architecture tailored to within-timestep MARL communication that isolates message compression within a dedicated module, allowing message dimension 2 to be reduced without constraining the higher-dimensional latent representation used for action selection. • We provide an empirical study across four partially observable MARL benchmarks and sev- eral established baselines, showing that SLIM is consistently competitive at high bandwidth, notably more robust as bandwidth decreases, and that its cache is useful in environments that are not jointly fully observable. 2 Related Work Multi-agent Reinforcement Learning.A straightforward approach to MARL treats each agent as an independent learner (IQL), applying standard reinforcement learning algorithms to each agent while treating others as part of the environment dynamics [39,25]. This often leads to instability because the effective environment becomes non-stationary as other agents’ policies change. To address this, the centralised training with decentralised execution (CTDE) paradigm has emerged [23,8]. It allows the training process to take advantage of global information, typically through a centralised critic or joint value function[44] or value-factorisation methods [30], while ensuring agents rely only on local observations during decentralised execution. Our training framework follows this paradigm, using a PPO [32] actor–critic objective with a centralised value function to compute advantages. Communication in MARL.Many real-world applications of MARL involve partial observability, making inter-agent communication crucial for effective coordination and decision-making. Several works focus on discrete or interpretable communication [7,20,26,36] to address bandwidth limi- tations or accommodate low-power hardware. In contrast, our approach belongs to the category of continuous and differentiable communication. Within this domain, other methods have been pro- posed, such as different pooling techniques [37,35] and graph neural networks [14,21,33,16,5,17] to model agent interactions. Currently, attention mechanisms have emerged as the standard for MARL communication, whether integrated directly into the policy [13] or implemented as a separate module [4]. CommNet [37] introduced continuous communication in MARL by averaging agent hidden states to communicate between time steps. IC3Net [35] builds upon this using individual rewards to mitigate credit assignment challenges [8]. They also introduced a gating mechanism that enables the agents to learn when to communicate, further reducing communication throughput. However, in fully cooperative settings, this mechanism provides limited incentives to suppress unnecessary communication. Gated-ACML [24] addresses this limitation by using Q-value differences to decide when to communicate, facilitating the pruning of uninformative messages. Similarly to our approach, TarMAC [4] has a dedicated communication module separated from the policy’s latent space. They also use an attention mechanism for agent interaction. However, their communication protocol is executed between time steps, which introduces a one-step temporal delay in information exchange. While this can be mitigated through multi-round communication, such a strategy incurs a linear increase in bandwidth consumption relative to the number of rounds. In contrast to dense communication paradigms, Commformer [13] introduces sparsity in the commu- nication graph. Several works extend this by making the graph dynamic, allowing agents to selectively identify recipients or adapt content [15,6,43,42,28,19,22]. QLBT [9] goes further by taking into account network settings such as latency and saturation to adapt the communication graph between agents. Beyond topological adjustments, Sun et al. [38] adapt message dimensionality to available bandwidth, while SchedNet [18] frames agent selection as a scheduling problem, where only a subset of agents is permitted to transmit at each time step. Event-triggered methods [47,12,11] reduce transmission frequency by only communicating during critical state transitions. While effective, these approaches primarily focus on mitigating channel contention rather than reducing the dimensionality of individual messages, which is the central focus of our work. We consider these objectives to be complementary; combining temporal or topological sparsity with our proposed compression framework could further minimise overall communication overhead. Information Theory.Bandwidth limitations in MARL communication have also been studied from information-theoretic perspectives [41,5,45]. Such approaches typically aim to produce compact and informative messages, often motivated by classic results of transmission-rate constraints and 3 principles of source coding [34]. Methods from this literature operate at the level of message encoding (in contrast to our architectural considerations at the representation and policy level); while not directly within the scope of our work, they could potentially be integrated together in a multi-agent system, for greater communication efficiency. 3 Subdivided Lightweight Inter-agent Messaging (SLIM) In this section we introduce our SLIM architecture. We first present the notation and preliminaries for MARL with communication. We then describe the SLIM architecture in detail, followed by the training procedure. 3.1 Notation and Preliminaries We consider multi-agent reinforcement learning (MARL) problems modelled as decentralised partially observable Markov decision processes (Dec-POMDPs) [27], defined by the tuple (S,A,P,R,Z,O,n,γ).This tuple consists of a global state spaceS, a joint action space A =A 1 × ...×A n fornagents, state-transition dynamicsP, reward functionR :S×A→ R n , and a discount factorγ ∈ [0, 1]. Partial observability of these processes is modeled by the joint observation space Z = Z 1 × ...× Z n and the observation function O. At discrete timet, each agentireceives a local observationo i t ∼ O(· | s t ,a t−1 ) (whereo i t ∈ Z i ) and selects an actiona i t ∼ π i (·| τ i t ) under its policyπ i conditioned on its local action-observation historyτ i t = (o i 0 ,a i 0 ,...,a i t−1 ,o i t ) . The MARL objective is to learn a joint policy that maximises the expected cumulative discounted reward (return) J(π) = E " ∞ X t=0 γ t r t # . A Dec-POMDP is said to be jointly fully observable if the joint observation of all agents at all timet,z t = (o 1 t ,...,o n t ) , uniquely determines the true states t of the environment. A jointly fully observable Dec-POMDP is a decentralised Markov decision process (Dec-MDP) [27]. Communication is allowed in these settings to mitigate partial observability. Some methods consider communication as part of the action space while others treat it as a separate mechanism. In order to generalise over different communication protocols, we decompose the agent’s decision process into two distinct stages: message generation and transmission, followed by action selection. At each time stept, each agentifirst generates a messagem i t based on its local history via a generation function μ i . This message is then transmitted to other agents. The agent then selects its actiona i t conditioned on both its history and the set of messages received from the other agents. m i t = μ i (τ i t ); a i t ∼ π i (·| τ i t ,m j t j̸=i ). This formulation unifies various communication protocols, where the received message set can repre- sent direct signals, an aggregated mean field, or the output of a graph neural network. Furthermore, this process can be applied iteratively: steps 1 and 2 can be repeatedktimes within a single time step to allow for multi-round consensus or intent-based communication. 3.2 The SLIM Architecture Our proposed SLIM architecture is illustrated in Figure 1. It implements the two-stage decision process using three main components: an observation encoder, a communication module, and a policy network. Non-jointly fully observable settings are specifically addressed through a message history cache integrated within the communication module. Observation Encoding.The encoderEprocesses the local observationo i t of agentiat time stept to produce a latent representation ̃o i t = E(o i t ) . This encoding step reduces the dimensionality of the observation while preserving relevant information for the policy. 4 Message Generation and Transmission. The representation ̃o i t is passed to a communication encoderE c , which projects it into a compact message vector suitable for transmission defined by m i t = E c ( ̃o i t ).These messages are then broadcasted to the other agents. Simultaneously, the agent receives the set of current messagesm j t j̸=i from its peers. Message History Cache.To address the partial observability of the environment, where the current observation alone is insufficient to determine the true state, each agent maintains a message cacheC i t potentially containing knowledge about the environment that could be useful later. This buffer stores the history of all exchanged messages in the system up to time t: C i t =C i t−1 ∪m j t 1≤j≤n =m j t ′ 1≤t ′ ≤t 1≤j≤n . Because the cache stores only transmitted messages rather than full observation embeddings, its memory footprint scales linearly with the communication dimension d. Temporal Attention Aggregation.To synthesise the available information, we apply a transformer- based attention block over the full content of the cacheC i t , which contains both the historical log and the messages just received at the current step. This mechanism dynamically weights the importance of each entry, enabling the agent to jointly reason over past and current information to construct a context vector ̃m i t . Concretely, the transformer input is the sequence of cached messages augmented with two learnt positional embeddings: a temporal embedding that identifies the communication step and an agent embedding that identifies the sender. The current-step messages and the cached messages are processed by the same attention block, so ̃m i t is an attention-weighted summary over who sent what and when, rather than a recurrent hidden state that privileges recent messages by construction. Action Selection.Finally, the policy networkπ i selects an action by conditioning on both the local representation and the context vector using a i t ∼ π i ( ̃o i t , ̃m i t ). While the message history cache is critical for Dec-POMDPs, our architecture allows it to be optionally disabled. In Dec-MDP settings, where the current observation is sufficient for optimal decision-making, the cache mechanism can be deactivated. In this configuration, the attention block operates only on the current set of received messages. 3.3 Centralised Training We follow the centralised training with decentralised execution paradigm. During training, we use a shared value function that receives the concatenated inputs from all agents. However, during execution, each agent only has access to its local observation and the messages received from other agents. We use the standard MAPPO algorithm [44] to train our agents. Specifically, we apply the PPO objective [32] to each agent individually irrespective of parameter sharing. For an agentihaving policy parametersθ k with candidate policy parametersθ, an agentiwith parametersθ k is updated to new parameters θ by optimising the objectiveL p i (θ). ρ θ k s,a = π i θ (a|s) π i θ k (a|s) ; L p i (θ) =− E s,a∼π θ min ρ θ k s,a A i π θ k (s,a), clip ρ θ k s,a , 1− ε, 1 + ε A i π θ k (s,a) − α E s∼S X a∈A i π i θ (a|s) logπ i θ (a|s) whereA · π · (·,·)is the advantage function estimated using Generalised Advantage Estimation (GAE) [31],εis a hyperparameter controlling the size of the policy update,αis a temperature controlling the entropy bonus [10] that encourages exploration. A value estimation is necessary to compute the advantage estimates, and we train the value functionV i θ (·)for each agentiby minimising the loss L v i (θ) = E s,r∼π θ V i θ (s t )− ˆ R i t 2 . 5 • • • • × (a) Predator-Prey (medium). (b) Traffic Junction (hard). (c) VMAS: Navigation (10 agents) (d) SHAPES Figure 3: Environments used in our experiments. (3a) Predator-Prey:npredators cooperate to capture a fixed prey on a grid, each observing only a window (grey cells). (3b) Traffic Junction: cars navigate an intersection without vision, relying on communication to avoid collisions; green cells are spawn points, red cells goals, dashed arrows possible routes. Cars spawn with probabilityp, up to a cap ofn. (3c) Navigation:nagents in a bounded continuous space must reach individually assigned goals (matching colour) without seeing peers. (3d) SHAPES: agents spawn on a random image of coloured shapes and must reach a target shape while observing only a local patch (red squares). Since we are using centralised training, the states t used in the value function is the joint observation rather than the local observation of agenti. To this end, our value function takes as input the concatenation of all agents’ encoded observations and outputs a value estimate for each agent. We can then combine those two losses to obtain the overall loss function per agent and take the mean over all agents to get the final loss function L(θ,θ k ) = 1 n n X i=1 (L p i (θ) +L v i (θ)). 4 Experiments In this section, we use multi-agent environments to evaluate the performance of our proposed method, the SLIM architecture, against several baselines. Our experiments aim to answer the following research questions:(i)Under a high communication bandwidthβ, does SLIM match or outperform representative baselines on multi-agent tasks?(i)How does SLIM’s performance evolve as the communication bandwidth budget is reduced over a broad range, and is it more robust than baseline methods in partially observable environments?(i)Does the cache help in improving performance in non-jointly observable environments? 4.1 Normalised Agent Bandwidth We define the normalised agent bandwidthβas the maximum transmission capacity allocated to each agent (in floating-point scalars) per time step divided by the population. This value makes comparison easier between environments of varying number of agents. To ensure equivalent conditions, each model is subjected to the same normalised agent bandwidth. This constraint accounts for the message dimensiond, the message frequencyk(corresponding to the number of communication rounds per time step as defined in Section 3.1), and the sparsity of the communication graphσ, defined as the fraction of the total population a given agent transmits to, into a single scalar limit. Formally, this bandwidth constraint is defined as σ× k× d≤ β.(1) This standardisation allows us to compare different models at equal bandwidth limits, regardless of their specific communication strategy (e.g., trading sparsity in the communication graph for higher message resolution). Details on the computation of the bandwidth constraints for each model configuration are given in Section A.1. 4.2 Baselines We compare our SLIM architecture against a diverse set of established MARL baselines to cover a spectrum of communication strategies. We select CommNet [37] to represent a foundational continu- ous communication approach. We also include IC3Net [35] which improves upon this architecture 6 by using individual rewards and a gating mechanism. However, consistent with observations in the original paper, we verified that in our fully cooperative scenarios, this gating mechanism remains effectively open, causing IC3Net to operate with a fully dense communication graph. We further include TarMAC [4] because it is the closest prior architecture that already separates communica- tion from the policy pathway. Finally, we benchmark against CommFormer [13], a state-of-the-art transformer-based architecture. We evaluate both the dense variant (i.e., using a complete commu- nication graph) to serve as a high-performance upper bound, and the sparse variant, which learns a fixed sparse communication graph, to assess the effectiveness of communication graph sparsification techniques as a bandwidth accommodation strategy. 4.3 Environments We evaluate on four standard partially-observable MARL benchmarks (illustrated in Figure 3), with full descriptions in Section B. Predator-Prey [35] is a grid world where predators with local vision cooperate to locate a stationary prey; since the prey does not move, past observations carry state information absent from the current joint observation, making the environment non-jointly- observable and motivating the message history cache (ablated in Section 4.7). Traffic Junction [35] requires vision-less agents to cross an intersection along pre-assigned routes; concatenating all agents’ positions fully determines the global state, so the environment is a Dec-MDP and the cache is disabled. Navigation [2] places agents in a bounded continuous space where they must reach individual goals using acceleration actions under momentum, observing only their own position, velocity, and relative goal distance. SHAPES [1,4] spawns agents on an image of coloured shapes that must each reach a target colour while observing only a local patch; differing per-agent goals mean cached peer messages can carry goal-relevant information, providing a second testbed for the cache ablation. 4.4 Experimental Setup Reproducibility and Compute All experiments are conducted across four seeds1, 2, 3, 4to ensure statistical robustness. We report mean performance alongside the standard error of the mean (std/ √ n seeds )for all graphical representations and alongside the standard deviation for tabular results. The total computational budget for these experiments exceeded 5,000 GPU-hours on NVIDIA H100 GPUs. This is mostly due to the results necessary for Section 4.6. While this number can seem high, it does not reflect inefficiencies, it is driven by the depth of our experiments, which required about 600 trainings (5 algorithm, 5 environments, 7 β values and 4 training seeds). Hyperparameter ProtocolFor all baselines, we used the hyperparameters specified in their original publications after verifying their performance are unchanged atβ = 64. For CommFormer, which was not originally evaluated on TrafficJunction, we performed a grid search over the PPO epoch parameter in5, 10, 15and the PPO clip value in0.05, 0.2. These ranges were selected based on the values used in the original CommFormer experiments on other benchmarks. For SLIM, we performed a grid search, in high bandwidth, over the PPO epoch parameter in1, 5, 10. The final values are provided in Table 2 in the Appendix. 4.5 Performance in High-Throughput Regimes We first establish a performance ceiling for all evaluated architectures by analysing them at the upper bandwidth bound (β = 2 6 ). This regime represents a permissive setting where the commu- nication channel is sufficiently wide, allowing us to assess the maximum expressive power of each model’s communication protocol. As detailed in Table 1, SLIM consistently achieves state-of-the-art performance across all benchmarks, either outperforming or maintaining statistical parity with estab- lished baselines. This verifies that SLIM’s architectural design, while optimised for low-resource environments, does not sacrifice absolute performance when bandwidth is abundant. 4.6 Robustness Under Various Bandwidth Constraints To evaluate the robustness of each architecture, we swept the normalised agent bandwidthβacross a logarithmic scale ranging from2 0 to2 6 . For each value, we trained every model on 4 distinct random seeds, resulting in a total of 28 training runs per model. Figure 4 and Figure 5 illustrate the evolution of performance as a function ofβ, with shaded regions indicating the standard error. 7 Table 1: Results on the different baselines using a bandwidth limit ofβ = 2 6 . Performance on Predator-Prey is measured in average steps to capture the prey (lower is better), while performance on Traffic Junction is measured in success rate (higher is better) and in reward for Navigation (higher is better). (†) Results for CommFormer are missing on the Navigation environment because it appears that the method is not able to converge. EnvironmentDifficultyCommNetIC3NetTarMACCommFormerSLIM Predator-Prey Easy5.68±0.486.10±0.466.18±0.505.00±0.204.97±0.04 Medium24.78±0.9920.14±2.5217.86±3.4614.06±1.5412.57±0.15 Traffic Junction Easy31.2±19.285.8±10.065.5±29.684.5±13.299.3±0.30 Medium77.0±5.3480.6±7.4271.0±2.8796.0±4.3197.2±0.84 Navigation0.49±0.060.28±0.240.64±0.24 †0.81±0.05 2 0 2 1 2 2 2 3 2 4 2 5 2 6 5 10 15 20 Normalised Agent Bandwidth (β) Mean episode length Commformer (1.0) Commformer (0.5) IC3Net CommNet TarMAC SLIM (a) Predator-Prey easy 2 0 2 1 2 2 2 3 2 4 2 5 2 6 10 15 20 25 30 35 40 Normalised Agent Bandwidth (β) Mean episode length CommFormer (1.0) CommFormer (0.5) IC3Net CommNet TarMAC SLIM (b) Predator-Prey medium 2 0 2 1 2 2 2 3 2 4 2 5 2 6 20 40 60 80 100 Normalised Agent Bandwidth (β) Success rate (%) Commformer (1.) IC3Net CommNet TarMAC SLIM (c) Traffic Junction easy 2 0 2 1 2 2 2 3 2 4 2 5 2 6 60 70 80 90 100 Normalised Agent Bandwidth (β) Success rate (%) Commformer (1.) IC3Net CommNet TarMAC SLIM (d) Traffic Junction medium Figure 4: Performance of SLIM and baselines across a logarithmic range of normalised agent bandwidth valuesβ(from2 0 to2 6 ). Top: mean episode length for the Predator-Prey environment (lower values indicate better performance). Bottom: success rate for the Traffic Junction environment (higher is better). Shaded regions denote the standard error of the mean across 4 seeds. One can note that data points for the dense variant of CommFormer are absent in the lowest bandwidth regimes (β = 2 0 ) because the architectural constraints of the model prevent it from satisfying such strict bandwidth limits. See details in Section A.1. 2 0 2 1 2 2 2 3 2 4 2 5 2 6 −3 −2 −1 0 1 Normalised Agent Bandwidth (β) Average reward IC3Net CommNet TarMAC SLIM Figure 5: Performance on a logarith- mic range of normalised agent band- width valuesβin the Navigation en- vironment. Shaded regions denote the standard error. SLIM achieves the best performance in high bandwidth settings and is more resilient to the reduction of the bandwidth. CommFormer achieves competitive performance in high- bandwidth regimes; however, this performance degrades significantly as the bandwidth constraint tightens. The sparse variant (σ = 0.5) exhibits higher variability and lower results. We excluded this variant from Traffic Junc- tion because a fixed sparse communication graph is un- suitable for tasks where specific and dynamic interactions are required (e.g., cars at the same intersection). Similarly, IC3Net and CommNet are less robust, showing either consistently lower returns or quick collapse under lim- ited bandwidth. We observe similar trends for TarMAC, except in Navigation where it maintains a more stable performance. In contrast, SLIM demonstrates robustness under bandwidth constraints. It matches or surpasses the peak performance of the strongest baseline at high band- width while maintaining high rewards even under strict constraints where others fail. While a minor performance drop occurs in Traffic Junction under severe bandwidth limitations, SLIM consistently offers robust performances. It validates its ability to learn robust policies without relying on excessive transmission volume. 4.7 Ablation Study: Impact of Message History Cache To validate the hypothesis that the historical context is beneficial for environments with partial observability, we conducted an ablation study comparing the full SLIM architecture against the variant with the cache deactivated. The results, presented in Figure 6, demonstrate the clear advantage 8 010002000 5 5.5 6 Epoch Steps w/o Cache w/ Cache (a) Low bandwidth 010002000 5 5.5 6 Epoch Steps w/o Cache w/ Cache (b) High bandwidth 010002000 -0.2 -0.1 0 Epoch Reward w/o Cache w/ Cache (c) Low bandwidth 010002000 -0.2 -0.1 0 Epoch Reward w/o Cache w/ Cache (d) High bandwidth Figure 6: Ablation study on the effect of the cache on an non jointly observable environment. We compare the performance of SLIM with and without the temporal cache mechanism in the Predator-Prey easy environment ((6a) and (6b)) and the SHAPES environment ((6c) and (6d)) under two different communication bandwidths:2 3 ( 6a) and (6c)) and2 6 ((6b) and (6d)). The results demonstrate that the cache significantly improves results along the training process while increasing stability; for non-jointly fully observable environments. The line reported is the mean over 4 seeds, with the shaded area representing the standard error. of the memory mechanism. Additional detailed results are available in Table 4. In the Predator-Prey and the SHAPES environments, where tracking agent trajectories over time is advantageous, the cache-enabled model consistently outperforms the cache-disabled baseline, achieving the objective in fewer steps in Predator-Prey and higher rewards in SHAPES. The only exception occurs at very low bandwidths in Predator-Prey, where we hypothesise that the limited representational capacity of the latent space prevents the effective disentanglement of spatial features, temporal dependencies, and agent identities. 5 Limitations Our study targets algorithmic communication efficiency and does not constitute a complete model of real wireless deployment. The normalised bandwidth budgetβprovides a controlled proxy unifying message dimension, communication rounds, and graph sparsity, but it abstracts away packet headers, quantisation, latency, routing overhead, packet loss, and medium contention. Extendingβ to capture some of these factors, and evaluating SLIM under more realistic network conditions, is left for future work. We also do not empirically combine SLIM with information-theoretic message compression, which we hypothesise to be complementary but leave to future work. The message history cache had negligible overhead in our experiments, but its memory cost grows linearly with episode length; windowing strategies or compressed memory representations could be considered for very long-horizon tasks. 6 Conclusion We proposed SLIM, a simple and modular approach to communication in multi-agent reinforcement learning that explicitly decouples the architecture responsible for inter-agent messaging encoding, from that dedicated to policy execution (responsible for actions). We introduced an evaluation protocol based on a normalised agent bandwidth metric (β). This unified measure integrates message size, transmission frequency, and graph sparsity into a single constraint, enabling a systematic benchmarking of diverse communication strategies under identical physical limitations. Our empirical results across multiple partially observable benchmarks demonstrate that SLIM not only matches state-of-the-art methods in high-bandwidth regimes but exhibits clear robustness as constraints tighten. While baseline performance degrade rapidly under severe constraints, SLIM maintains effective coordination even with lower bandwidth limits. Finally, ablation studies confirmed the benefit of our proposed message history cache for improving learning in non-jointly observable environments. Future work will further explore connections between modern communication-efficient MARL settings and complementary developments in network science, where transmission rate and information-theoretical views have been widely considered; perhaps allowing further calibration of communication efficiency in a more general sense. 9 Acknowledgements This work received financial support from Crédit Agricole SA through the research chair Trustworthy and Responsible AI at École Polytechnique. This work was granted access to the HPC resources of IDRIS under the allocation 2025-AD011017102 made by GENCI. Finally, we would like to thank Mathis Le Bail, Clément Elliker and Mahammed Elsharkawy for their help and insights throughout the project. References [1]Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. Neural module networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 39–48, 2016. [2]Matteo Bettini, Ryan Kortvelesy, Jan Blumenkamp, and Amanda Prorok. Vmas: A vectorized multi-agent simulator for collective robot learning. The 16th International Symposium on Distributed Autonomous Robotic Systems, 2022. [3]Caroline Claus and Craig Boutilier. The dynamics of reinforcement learning in cooperative multiagent systems. AAAI/IAAI, 1998(746-752):2, 1998. [4] Abhishek Das, Théophile Gervet, Joshua Romoff, Dhruv Batra, Devi Parikh, Mike Rabbat, and Joelle Pineau. Tarmac: Targeted multi-agent communication. In International Conference on Machine Learning, pages 1538–1546. PMLR, 2019. [5]Shifei Ding, Wei Du, Ling Ding, Jian Zhang, Lili Guo, and Bo An. Robust multi-agent communication with graph information bottleneck optimization. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(5):3096–3107, 2023. [6]Ziluo Ding, Tiejun Huang, and Zongqing Lu. Learning individually inferred communication for multi-agent cooperation. Advances in neural information processing systems, 33:22069–22079, 2020. [7]Jakob Foerster, Ioannis Alexandros Assael, Nando De Freitas, and Shimon Whiteson. Learning to communicate with deep multi-agent reinforcement learning. Advances in neural information processing systems, 29, 2016. [8]Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018. [9]Ziyang Guo, Zhenyu Chen, Peng Liu, Jianjun Luo, Xun Yang, and Xinghua Sun. Multi-agent reinforcement learning-based distributed channel access for next generation wireless networks. IEEE Journal on Selected Areas in Communications, 40(5):1587–1599, 2022. [10]Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pages 1861–1870. Pmlr, 2018. [11]Shuai Han, Mehdi Dastani, and Shihan Wang. Model-based sparse communication in multi- agent reinforcement learning. In Proceedings of the 2023 international conference on au- tonomous agents and multiagent systems, pages 439–447. International Foundation for Au- tonomous Agents and Multiagent Systems (IFAAMAS), 2023. [12]Guangzheng Hu, Yuanheng Zhu, Dongbin Zhao, Mengchen Zhao, and Jianye Hao. Event- triggered communication network with limited-bandwidth constraint for multi-agent reinforce- ment learning. IEEE Transactions on Neural Networks and Learning Systems, 34(8):3966–3978, 2021. 10 [13]Shengchao Hu, Li Shen, Ya Zhang, and Dacheng Tao. Learning multi-agent communication from graph modeling perspective. In International Conference on Learning Representations, 2024. [14] Jiechuan Jiang, Chen Dun, Tiejun Huang, and Zongqing Lu. Graph convolutional reinforcement learning. 2020. [15]Jiechuan Jiang and Zongqing Lu. Learning attentional communication for multi-agent coopera- tion. Advances in neural information processing systems, 31, 2018. [16]Rui Jiang, Xuetao Zhang, Yisha Liu, Yi Xu, Xuebo Zhang, and Yan Zhuang. Multi-agent cooperative strategy with explicit teammate modeling and targeted informative communication. Neurocomput., 586(C), June 2024. [17]Rui Jiang, Xuetao Zhang, Yisha Liu, Yi Xu, Xuebo Zhang, and Yan Zhuang. Multi-agent cooperative strategy with explicit teammate modeling and targeted informative communication. Neurocomputing, 586:127638, 2024. [18] Daewoo Kim, Sangwoo Moon, David Hostallero, Wan Ju Kang, Taeyoung Lee, Kyunghwan Son, and Yung Yi. Learning to schedule communication in multi-agent reinforcement learning. In International Conference on Learning Representations, 2019. [19] Jakub Grudzien Kuba, Ruiqing Chen, Muning Wen, Ying Wen, Fanglei Sun, Jun Wang, and Yaodong Yang. Trust region policy optimisation in multi-agent reinforcement learning. In International Conference on Learning Representations, 2022. [20] Angeliki Lazaridou, Alexander Peysakhovich, and Marco Baroni. Multi-agent cooperation and the emergence of (natural) language. In International Conference on Learning Representations, 2017. [21]Sheng Li, Jayesh K Gupta, Peter Morales, Ross Allen, and Mykel J Kochenderfer. Deep implicit coordination graphs for multi-agent reinforcement learning. In Proceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems, pages 764–772, 2021. [22]Xinran Li and Jun Zhang. Context-aware communication for multi-agent reinforcement learning. In Proceedings of the 23rd International Conference on Autonomous Agents and Multiagent Systems, pages 1156–1164, 2024. [23]Ryan Lowe, Yi I Wu, Aviv Tamar, Jean Harb, OpenAI Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-competitive environments. Advances in neural information processing systems, 30, 2017. [24]Hangyu Mao, Zhengchao Zhang, Zhen Xiao, Zhibo Gong, and Yan Ni. Learning agent communication under limited bandwidth by message pruning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 5142–5149, 2020. [25] Laetitia Matignon, Guillaume J Laurent, and Nadine Le Fort-Piat. Independent reinforce- ment learners in cooperative markov games: a survey regarding coordination problems. The Knowledge Engineering Review, 27(1):1–31, 2012. [26] Igor Mordatch and Pieter Abbeel. Emergence of grounded compositional language in multi- agent populations. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018. [27]Frans A Oliehoek, Christopher Amato, et al. A concise introduction to decentralized POMDPs, volume 1. Springer, 2016. [28]Murtaza Rangwala and Ryan Williams. Learning multi-agent communication through structured attentive reasoning. Advances in Neural Information Processing Systems, 33:10088–10098, 2020. [29]Theodore S Rappaport. Wireless communications: Principles and practice, 2/E. Pearson Education India, 2010. 11 [30]Tabish Rashid, Mikayel Samvelyan, Christian Schroeder De Witt, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Monotonic value function factorisation for deep multi-agent reinforcement learning. Journal of Machine Learning Research, 21(178):1–51, 2020. [31] John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. In International Conference on Learning Representations, 2016. [32]John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. [33]Esmaeil Seraj, Zheyuan Wang, Rohan Paleja, Daniel Martin, Matthew Sklar, Anirudh Patel, and Matthew Gombolay. Learning efficient diverse communication for cooperative heterogeneous teaming. In Proceedings of the 21st International Conference on Autonomous Agents and Multiagent Systems, pages 1173–1182, 2022. [34]Claude E Shannon. A mathematical theory of communication. The Bell system technical journal, 27(3):379–423, 1948. [35] Amanpreet Singh, Tushar Jain, and Sainbayar Sukhbaatar. Learning when to communicate at scale in multiagent cooperative and competitive tasks. In International Conference on Learning Representations, 2019. [36] Yang Su, Yali Du, and Yansha Deng. Goal-oriented semantic communication in bandwidth- constrained marl. In 2025 IEEE International Conference on Communications Workshops (ICC Workshops), pages 1274–1279. IEEE, 2025. [37] Sainbayar Sukhbaatar, Rob Fergus, et al. Learning multiagent communication with backpropa- gation. Advances in neural information processing systems, 29, 2016. [38]Qingshuang Sun, Denis Steckelmacher, Yuan Yao, Ann Nowe, and Raphael Avalos. Dynamic size message scheduling for multi-agent communication under limited bandwidth. IEEE Transactions on Mobile Computing, 2024. [39] Ming Tan. Multi-agent reinforcement learning: Independent vs. cooperative agents. In Proceed- ings of the tenth international conference on Machine Learning, pages 330–337, 1993. [40]Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Jun- young Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft i using multi-agent reinforcement learning. nature, 575(7782):350–354, 2019. [41]Rundong Wang, Xu He, Runsheng Yu, Wei Qiu, Bo An, and Zinovi Rabinovich. Learning efficient multi-agent communication: An information bottleneck approach. In Hal Daumé I and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 9908–9918. PMLR, 13–18 Jul 2020. [42] Tonghan Wang, Jianhao Wang, Chongyi Zheng, and Chongjie Zhang. Learning nearly de- composable value functions via communication minimization. In International Conference on Learning Representations, 2020. [43] Yuanfei Wang, Fangwei Zhong, Jing Xu, and Yizhou Wang. Tom2c: Target-oriented multi-agent communication and cooperation with theory of mind. arXiv preprint arXiv:2111.09189, 2021. [44]Chao Yu, Akash Velu, Eugene Vinitsky, Jiaxuan Gao, Yu Wang, Alexandre Bayen, and Yi Wu. The surprising effectiveness of ppo in cooperative multi-agent games. Advances in neural information processing systems, 35:24611–24624, 2022. [45]Lebin Yu, Qiexiang Wang, Yunbo Qiu, Jian Wang, Xudong Zhang, and Zhu Han. Effective multi-agent communication under limited bandwidth. IEEE Transactions on Mobile Computing, 23(7):7771–7784, 2024. [46]Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. arXiv preprint arXiv:1611.03530, 2016. 12 [47]Sai Qian Zhang, Qi Zhang, and Jieyu Lin. Efficient communication in multi-agent reinforcement learning via variance based control. Advances in neural information processing systems, 32, 2019. 13 A Experiment Details Table 2: Hyperparameter configuration for the SLIM architecture across all benchmarks. These values were optimised using a fixed message dimensionality ofd = 64(2 6 ) to ensure that the architecture remains robust across varying bandwidths without overfitting to specific constraints. Predator-PreyTraffic JunctionNavigationSHAPES EasyMediumEasyMedium// Clip param. ε0.20.20.20.20.20.2 Entropy coeff. σ0.020.020.020.020.020.02 Episodes/epoch500500500500100100 Number of epochs20002000600400032002000 PPO Epochs551111 Hidden size 128128128128128128 Message dim.2 6 2 6 2 6 2 6 2 6 2 6 Using cacheTrue TrueFalse FalseFalseTrue Discount γ0.990.990.990.990.990.99 Learning rate 5×10 −4 5×10 −4 5×10 −4 5×10 −4 5×10 −4 5×10 −4 Replay buffer (# ep.)10001000100010002505000 GAE λ0.950.950.950.950.950.95 A.1 Detailed Communication Parameters Table 3: Communication parameters for the normalized agent bandwidth configurations. For each bandwidth budgetβ, we report the graph densityσ, communication roundsk, and largest feasible message dimensiondsatisfyingσ× k× d≤ β. TarMAC is evaluated in the dense one-pass setting used in our sweeps. Dense CommFormer cannot satisfyβ = 2 0 with integerd≥ 1. En-dashes (–) denote inaccessible configurations. β = 2 0 β = 2 1 β = 2 2 β = 2 3 β = 2 4 β = 2 5 β = 2 6 Modelσ k d σ k d σ k d σ k d σ k d σ k d σ k d CommFormer– – – 1.0 2 2 0 1.0 2 2 1 1.0 2 2 2 1.0 2 2 3 1.0 2 2 4 1.0 2 2 5 CommFormer (sparse) 0.5 2 2 0 0.5 2 2 1 0.5 2 2 2 0.5 2 2 3 0.5 2 2 4 0.5 2 2 5 0.5 2 2 6 CommNet1.0 1 2 0 1.0 1 2 1 1.0 1 2 2 1.0 1 2 3 1.0 1 2 4 1.0 1 2 5 1.0 1 2 6 IC3Net1.0 1 2 0 1.0 1 2 1 1.0 1 2 2 1.0 1 2 3 1.0 1 2 4 1.0 1 2 5 1.0 1 2 6 TarMAC1.0 1 2 0 1.0 1 2 1 1.0 1 2 2 1.0 1 2 3 1.0 1 2 4 1.0 1 2 5 1.0 1 2 6 SLIM1.0 1 2 0 1.0 1 2 1 1.0 1 2 2 1.0 1 2 3 1.0 1 2 4 1.0 1 2 5 1.0 1 2 6 B Additional Results NavigationIn order to illustrate that the method could scale to more agents, we also illustrate SLIM using 4, 20, 24, 32 and 42 agents (anonymised links). These illustrations show agents successfully reaching their individual goals while avoiding collisions despite not seeing each other. Due to computational constraints, we have only benchmarked SLIM against TarMAC (the strongest baseline when using 4 agents) in the 20 agents setting. In this setting, TarMAC gets a reward of 0.02± 0.01 while SLIM gets 0.95± 0.003. Reproducibility We publicly released the code to reproduce all our experiments on GitHub. All experiments can be run using a simple script. Environment files are available with all the libraries version used to be able to exactly reproduce our setting. 14 Table 4: Results of the ablation study on the impact of the cache in non-jointly observable environments.We report mean episode length across varying communication dimensions for Predator-Prey and rewards for SHAPES. Lower values are better for episode length; higher values are better for rewards. SettingCacheβ 1248163264 Pred.-Prey Easy w/6.61±.29 7.10±1.2 5.19±.18 5.01±.03 4.99±.03 4.99±.04 4.97±.04 w/o6.65±.34 6.49±.1 5.16±.06 5.22±.04 5.21±.05 5.12±.02 5.26±.04 Pred.-Prey Medium w/27.2±0.6 23.3±1.5 17.9±2.5 13.1±0.23 12.8±0.1 12.6±0.2 12.6±0.2 w/o26.8±0.34 27.2±2.9 15.8±4.5 14.3±1.5 13.3±0.4 13.1±0.2 13.3±0.4 SHAPES×10 2 w/-5.3±.20 -4.8±.82 -4.7±.30 -3.9±.64 -3.1±.18 -3.0±.32 -4.6±.22 w/o-7.0±.40 -7.2±.55 -6.8±.79 -5.9±.76 -6.4±.44 -7.0±.43 -7.2±.44 Licenses We release our implementation under the MIT licence. We do not redistribute the code of the baselines. CommFormer is available through the Apache License 2.0, IC3Net and TarMAC through the MIT License, and CommNet through the BSD license. The Predator-Prey and Traffic Junction environments are available using the MIT License, VMAS through the GPL3 license and we reimplemented the SHAPES environment based on the description provided in the original paper, thus we release it under the MIT License. Environments Details Predator-Prey.The Predator-Prey environment is a multi-agent grid world where agents (predators) must cooperate to locate a prey with limited vision (local observability) centred on them. Each agent gets a negative reward until it reaches the prey. Agents can move in four directions (up, down, left, right) or stay in place. They benefit from a strong communication since knowing where the prey is not gives information about its location. Since the prey cannot move, the environment is not jointly fully observable: information about previously seen locations is information on the state, that could be missing from the current joint observation. To address this, we enable the message history cache for this environment and provide an ablation study in Section 4.7 to assess its impact. Traffic Junction. The Traffic Junction environment simulates intersections that agents without any vision need to cross. Agents enter the environment stochastically and navigate along randomly assigned routes. The lack of vision makes communication essential to avoid collisions, making this environment a suitable benchmark for communication MARL. In this environment, agents do not have to navigate: they can either go forward along their route or wait. The reward strongly incentivise collision avoidance while lightly encouraging efficiency: agents receive an increasingly negative reward for each time step they are active in the environment, and a large negative reward if they collide with another car. Since the concatenation of all agents’ local observations (cars position) fully characterises the global configuration of the system, the environment is categorised as a decentralised Markov decision process (Dec-MDP). Consequently, the message history cache is deactivated for this environment. Navigation The navigation environment [2] implements a bounded continuous world wheren randomly spawned agents must navigate to individual randomly spawned goals without colliding with other agents, without seeing each other. Each agent only sees its position, speed and relative distance to its goal. Actions are acceleration in different directions and the environment simulates momentum, making fine control complicated. Agents receive a reward proportional to the distance gained to their goal at each time step, and a large negative reward if they collide with another agent. SHAPES. In order to verify the effectiveness of the message history cache in Section 4.7, we introduced a variation of the simple SHAPES environment [1,4]. Images containing coloured shapes 15 are generated. Agents spawn at random positions on a random image and must find a (possibly different) random colour each whilst only being able to observe their surroundings. This environment is not partially jointly fully observable. Since agents have different goals, information on what other agents have seen in the past can contain information about another agent’s goal. Each agent receives a reward of 10 −2 until they reach their target, at which point they receive a reward of 0. 16