Paper deep dive
Probing Dec-POMDP Reasoning in Cooperative MARL
Kale-ab Tessera, Leonard Hinckeldey, Riccardo Zamboni, David Abel, Amos Storkey
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 7/20/2026, 1:57:12 PM
Summary
This paper introduces a diagnostic suite to audit whether cooperative Multi-Agent Reinforcement Learning (MARL) benchmarks genuinely require Dec-POMDP reasoning (history-based inference and decentralized coordination) or allow success via simpler reactive strategies. Using information-theoretic probes (History-Action Relevance, Private Information Flow, Action-Action Coupling, Directed Action Information) on 37 scenarios across MPE, SMAX, Overcooked, Hanabi, and MaBrax, the authors find that reactive policies often match memory-based agents, suggesting many benchmarks do not adequately test core Dec-POMDP assumptions.
Entities (15)
Relation Signals (9)
Cooperative MARL → isframedas → Dec-POMDP
confidence 95% · Cooperative multi-agent reinforcement learning (MARL) is typically framed as a decentralised partially observable Markov decision process (Dec-POMDP)
Reactive Policies → matchperformanceof → Memory-based Agents
confidence 95% · Reactive policies match the performance of memory-based agents in over half the scenarios
Diagnostic Suite → evaluates → SMAX
confidence 90% · We apply these diagnostics to policies learned by standard baselines in 37 popular MARL scenarios, across... SMAX
Diagnostic Suite → evaluates → Overcooked
confidence 90% · We apply these diagnostics to policies learned by standard baselines in 37 popular MARL scenarios, across... Overcooked
Diagnostic Suite → evaluates → MABrax
confidence 90% · We apply these diagnostics to policies learned by standard baselines in 37 popular MARL scenarios, across... MaBrax
Diagnostic Suite → evaluates → Hanabi
confidence 90% · We apply these diagnostics to policies learned by standard baselines in 37 popular MARL scenarios, across... Hanabi
Diagnostic Suite → evaluates → MPE
confidence 90% · We apply these diagnostics to policies learned by standard baselines in 37 popular MARL scenarios, across MPE... Our analysis reveals... MPE emerges as the only suite where every scenario satisfies all four diagnostic criteria
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Cooperative multi-agent reinforcement learning (MARL) is typically framed as a decentralised partially observable Markov decision process (Dec-POMDP), a setting whose hardness stems from two key challenges: partial observability and decentralised coordination. Genuinely solving such tasks requires Dec-POMDP reasoning, where agents use history to infer hidden states and coordinate based on local information. Yet it remains unclear whether popular benchmarks actually demand this reasoning or permit success via simpler strategies. We introduce a diagnostic suite combining statistically grounded performance comparisons and information-theoretic probes to audit the behavioural complexity of baseline policies (IPPO and MAPPO) across 37 scenarios spanning MPE, SMAX, Overcooked, Hanabi, and MaBrax. Our diagnostics reveal that success on these benchmarks rarely requires genuine Dec-POMDP reasoning. Reactive policies match the performance of memory-based agents in over half the scenarios, and emergent coordination frequently relies on brittle, synchronous action coupling rather than robust temporal influence. These findings suggest that some widely used benchmarks may not adequately test core Dec-POMDP assumptions under current training paradigms, potentially leading to over-optimistic assessments of progress. We release our diagnostic tooling to support more rigorous environment design and evaluation in cooperative MARL.
Tags
Links
- Source: https://arxiv.org/abs/2602.20804v2
- Canonical: https://arxiv.org/abs/2602.20804v2
Trouble viewing inline? Open PDF directly →
Full Text
95,272 characters extracted from source content.
Expand or collapse full text
ifaamas [AAMAS ’26]Proc. of the 25th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2026)May 25 – 29, 2026 Paphos, CyprusC. Amato, L. Dennis, V. Mascardi, J. Thangarajah (eds.) 2026 2026 1235 of Edinburgh Kingdom of Edinburgh Kingdom di Milano of Edinburgh Kingdom of Edinburgh Kingdom Probing Dec-POMDP Reasoning in Cooperative MARL Kale-ab Abebe Tessera k.tessera@ed.ac.uk , Leonard Hinckeldey l.hinckeldey@ed.ac.uk , Riccardo Zamboni riccardo.zamboni@polimi.it , David Abel david.abel@ed.ac.uk and Amos Storkey a.storkey@ed.ac.uk Abstract. Cooperative multi-agent reinforcement learning (MARL) is typically framed as a decentralised partially observable Markov decision process (Dec-POMDP), a setting whose hardness stems from two key challenges: partial observability and decentralised coordination. Genuinely solving such tasks requires Dec-POMDP reasoning, where agents use history to infer hidden states and coordinate based on local information. Yet it remains unclear whether popular benchmarks actually demand this reasoning or permit success via simpler strategies. We introduce a diagnostic suite combining statistically grounded performance comparisons and information-theoretic probes to audit the behavioural complexity of baseline policies (IPPO and MAPPO) across 37 scenarios spanning MPE, SMAX, Overcooked, Hanabi, and MaBrax. Our diagnostics reveal that success on these benchmarks rarely requires genuine Dec-POMDP reasoning. Reactive policies match the performance of memory-based agents in over half the scenarios, and emergent coordination frequently relies on brittle, synchronous action coupling rather than robust temporal influence. These findings suggest that some widely used benchmarks may not adequately test core Dec-POMDP assumptions under current training paradigms, potentially leading to over-optimistic assessments of progress. We release our diagnostic tooling to support more rigorous environment design and evaluation in cooperative MARL.111The code is available at https://github.com/KaleabTessera/probing-dec-pomdps. Key words and phrases: Multi-Agent Reinforcement Learning, Cooperative Multi-Agent Reinforcement Learning, Dec-POMDPs 1. Introduction The widespread deployment of autonomous multi-agent systems is bounded by their ability to coordinate under uncertainty. In such settings, no single agent possesses a complete view of the world, yet outcomes depend on joint behaviour. This tension lies at the heart of cooperative multi-agent reinforcement learning (MARL, albrech2024multiagent). The standard formalism for these problems, decentralised partially observable Markov decision processes (Dec-POMDPs, bernstein2002complexity; oliehoek2016concise), capture this intrinsic hardness through two fundamental characteristics: partial observability, where agents cannot directly observe the full global state, and decentralised coordination, where agents must cooperate based on local and private information. The intrinsic hardness of this setting stems directly from the interaction of these two factors. In principle, to act optimally, each agent must recover a Markovian signal by maintaining a multi-agent belief over the joint state and the policies (or histories) of other agents (oliehoek2016concise). However, exact multi-agent belief computation is typically infeasible (bernstein2002complexity). Consequently, practical model-free methods approximate this reasoning using finite-memory or recurrent policies (e.g., GRUs) (hausknecht2015deep), often instantiated within the centralised training with decentralised execution (CTDE, oliehoek2008optimal; kraemer2016multi) paradigm to leverage extra information during learning. The empirical success of MARL approaches in benchmarks (among others, yu2022surprising; papoudakis2020benchmarking) is often interpreted as evidence that practical approximations (e.g., recurrent policies) effectively capture the Dec-POMDP reasoning these problems demand. We challenge this interpretation. High returns can mask a failure to learn the underlying coordination challenge, as agents may exploit reactive shortcuts permitted by the task design rather than employing genuine history-based reasoning. This distinction is critical. If valid solutions exist that ignore the theoretical challenges of partial observability and coordination, then the environment can become a weak proxy for the Dec-POMDP formalism, yielding an illusion of progress on coordination under uncertainty. We therefore use trained policies as diagnostic probes to ask: Do modern cooperative MARL environments truly test the Dec-POMDP properties that make these problems hard, or do they permit success via strategies that bypass them? To answer this, we introduce a suite of MARL diagnostics that couple statistically grounded performance comparisons with information-theoretic probes to measure history dependence, private information flow, synchronous action coupling, and directed temporal influence. Together, these reveal whether learned policies genuinely employ Dec-POMDP reasoning, or bypass it entirely. We apply these diagnostics to policies learned by standard baselines in 37 popular MARL scenarios, across MPE (lowe2017multi), SMAX222Both SMAC-V1 (samvelyan2019starcraft) and SMAC-V2 (ellis2023smacv2) maps were tested. (rutherford2023jaxmarl), Overcooked (V1 and V2) (carroll2019utility; gessler2025overcookedv), Hanabi (bard2020hanabi) and MaBrax (rutherford2023jaxmarl; peng2021facmac). Across these settings, our analysis reveals three main takeaways: (i) history dependence rarely translates to history utility—while all learned policies encode some history dependence, only 43% actually need memory to achieve high returns, indicating that current observations often suffice for strong performance; (i) hidden environment state and hidden teammate information act as separate drivers of difficulty, which our metrics successfully disentangle (e.g., empirically validating the design shift from Overcooked V1 to V2); and (i) while coordination is common, its structure is highly variable—synchronous and temporal mechanisms dissociate across benchmarks. Notably, MPE emerges as the only suite where every scenario satisfies all four diagnostic criteria, consistently requiring both meaningful history use and decentralised coordination. Ultimately, these findings suggest that, under current training paradigms, success on popular benchmarks often does not require the Dec-POMDP reasoning these tasks are intended to evaluate. Contributions. (1) Diagnostic framework. We introduce information-theoretic probes – measuring history dependence, private information flow, synchronous action coupling, and directed temporal influence – that audit whether learned policies actually exhibit Dec-POMDP reasoning, beyond what raw returns reveal. (2) Systematic benchmark audit. We evaluate 37 scenarios across seven benchmark suites, revealing that history dependence is ubiquitous but rarely performance-critical, coordination structures vary qualitatively across domains, and few environments jointly test both partial observability and coordination. (3) Open-source tooling and implications. We release diagnostic tools for researchers to audit their own environments, and discuss implications for designing tasks where partial observability and coordination are non-optional. 2. Background We introduce key concepts that will be needed throughout the paper. Interaction Protocol. As a base model for interaction, we consider a discounted Dec-POMDP (bernstein2002complexity), defined by the tuple ℳ=(,,,,μ,ii∈,ii∈,R,γ)M=(N,S,T,O,μ,\A^i\_i ,\O^i\_i ,R,γ). Here, N is the set of N∈ℕN agents and S is the set of global states. At each time step t, the system is in some state st∈s_t . Each agent i∈i selects an action ati∈ia_t^i ^i, forming a joint action t=(at1,…,atN)a_t=(a_t^1,…,a_t^N) in the joint action space =×i=1NiA=×_i=1^NA^i. This action leads to a state transition according to the probability function (st+1|st,t)T(s_t+1|s_t,a_t) and a shared reward R(st,t)R(s_t,a_t). Agents do not observe the global state sts_t, instead they receive a local observation oti∈io_t^i ^i. The joint observation to_t is drawn according to the observation function (t|st,t−1)O(o_t|s_t,a_t-1). The goal is to learn a joint policy π at which no agent has any incentive to deviate, while maximising the expected discounted return t∼,ℳ[∑t=0∞γtR(st,t)]E_a_t π,M [ _t=0^∞γ^tR(s_t,a_t) ]. These solution concepts are usually described through various notions of equilibria: we report a brief description in Appendix A. Mutual Information. To study the information embedded in agents’ policies, we propose metrics based on mutual information (MI). For two discrete random variables X and Y with joint probability mass function333For continuous variables, we use the probability density function. p(x,y)p(x,y) and marginals p(x)p(x), p(y)p(y), we can measure MI as follows: (X;Y) (X;Y) =H(X)−ℍ(X∣Y)=ℍ(Y)−ℍ(Y∣X), =H(X)-H(X Y)=H(Y)-H(Y X), (1) =∑x,yp(x,y)logp(x,y)p(x)p(y). = _x,yp(x,y) p(x,y)p(x)p(y). (2) where H is the Shannon entropy. Intuitively, (X;Y)I(X;Y) is the average amount of information that X conveys about Y, or vice versa. MI is symmetric and non-negative, and (X;Y)=0I(X;Y)=0 iff X and Y are independent. We will also use metrics based on conditional mutual information (CMI), (X;Y∣Z)I(X;Y Z). Intuitively, CMI measures the extra information that X tells us about Y, excluding what we know about Y given Z. (X;Y∣Z)=0I(X;Y Z)=0 iff X and Y are conditionally independent given Z. Ag. iiO1iO_1^iA1iA_1^iO2iO_2^iA2iA_2^i⋯·sOtiO_t^iAtiA_t^i⋯·sOTiO_T^iATiA_T^iOAR≜(Oti;Ati)OAR ( [rgb]0.14453125,0.38671875,0.921875 [named]pgfstrokecolorrgb0.14453125,0.38671875,0.921875O_t^i\,;\, [rgb]0.01953125,0.58984375,0.41015625 [named]pgfstrokecolorrgb0.01953125,0.58984375,0.41015625A_t^i)(a) Observation–Action Relevance (OAR)Ag. iiO1iO_1^iA1iA_1^iO2iO_2^iA2iA_2^i⋯·sOtiO_t^iAtiA_t^i⋯·sOTiO_T^iATiA_T^iHtiH_t^iHAR≜(Hti;Ati∣Oti)HAR ( [rgb]0.14453125,0.38671875,0.921875 [named]pgfstrokecolorrgb0.14453125,0.38671875,0.921875H_t^i\,;\, [rgb]0.01953125,0.58984375,0.41015625 [named]pgfstrokecolorrgb0.01953125,0.58984375,0.41015625A_t^i [rgb]0.8515625,0.46484375,0.0234375 [named]pgfstrokecolorrgb0.8515625,0.46484375,0.0234375O_t^i)(b) History–Action Relevance (HAR)Ag. iiO1iO_1^iA1iA_1^i⋯·sAt-1iA_t-1^iOtiO_t^iAtiA_t^i⋯·sOTiO_T^iATiA_T^iτt−1i _t-1^iAg. jjO1jO_1^jA1jA_1^j⋯·sAt-1jA_t-1^jOtjO_t^jAtjA_t^j⋯·sOTjO_T^jATjA_T^jτt−1j _t-1^jPIFi→j≜((τt−1i,Oti);Atj|(τt−1j,Otj))PIF_i→ j ( [rgb]0.14453125,0.38671875,0.921875 [named]pgfstrokecolorrgb0.14453125,0.38671875,0.921875( _t-1^i,\,O_t^i)\,;\, [rgb]0.01953125,0.58984375,0.41015625 [named]pgfstrokecolorrgb0.01953125,0.58984375,0.41015625A_t^j\, |\, [rgb]0.8515625,0.46484375,0.0234375 [named]pgfstrokecolorrgb0.8515625,0.46484375,0.0234375( _t-1^j,\,O_t^j) )(c) Private Information Flow (PIF)Ag. iiO1iO_1^iA1iA_1^iO2iO_2^iA2iA_2^i⋯·sOtiO_t^iAtiA_t^i⋯·sOTiO_T^iATiA_T^iAg. jjO1jO_1^jA1jA_1^jO2jO_2^jA2jA_2^j⋯·sOtjO_t^jAtjA_t^j⋯·sOTjO_T^jATjA_T^jA≜(Ati;Atj|Oti,Otj)A ( [rgb]0.14453125,0.38671875,0.921875 [named]pgfstrokecolorrgb0.14453125,0.38671875,0.921875A_t^i\,;\, [rgb]0.01953125,0.58984375,0.41015625 [named]pgfstrokecolorrgb0.01953125,0.58984375,0.41015625A_t^j\, |\, [rgb]0.8515625,0.46484375,0.0234375 [named]pgfstrokecolorrgb0.8515625,0.46484375,0.0234375O_t^i,\,O_t^j )(d) Action–Action Coupling (A)Ag. iiO1iO_1^iA1iA_1^i⋯·sOt-1iO_t-1^iAt-1iA_t-1^iAtiA_t^i⋯·sOTiO_T^iATiA_T^iτt−1i _t-1^iAg. jjO1jO_1^jA1jA_1^j⋯·sOt-1jO_t-1^jAt-1jA_t-1^jAtjA_t^j⋯·sOTjO_T^jATjA_T^jτt−1j _t-1^jDAIi→j≜1T∑t=1T(τt−1i;Atj|τt−1j)DAI_i→ j 1T _t=1^TI ( [rgb]0.14453125,0.38671875,0.921875 [named]pgfstrokecolorrgb0.14453125,0.38671875,0.921875 _t-1^i\,;\, [rgb]0.01953125,0.58984375,0.41015625 [named]pgfstrokecolorrgb0.01953125,0.58984375,0.41015625A_t^j\, |\, [rgb]0.8515625,0.46484375,0.0234375 [named]pgfstrokecolorrgb0.8515625,0.46484375,0.0234375 _t-1^j )(e) Directed Action Information (DAI) If high, indicates… OAR Near-reactive policies: actions largely determined by current observation. HAR Genuine memory use: agents exploit information beyond OtiO_t^i. PIF Cross-agent private information: i’s trajectory and current observation inform j’s action beyond j’s own history and observation. A Instantaneous conventions or symmetry breaking not explained by Oti,OtjO_t^i,O_t^j. DAI Temporal, directed influence: i’s past trajectory exerts sustained influence on j’s future actions. Figure 1. Summary of information-theoretic diagnostics. Colours denote source (blue), target (green), and conditioning (amber, dashed) variables. All quantities are expectations under the converged joint policy p π; OtiO_t^i is agent i’s observation, AtiA_t^i its action, HtiH_t^i the local history (RNN hidden state or finite window), τt−1iτ^i_t-1 agent i’s action–observation history up to t−1t-1, and T the episode horizon. 3. Related Work Benchmarking Partial Observability. ellis2023smacv2 found that many SMAC (samvelyan2019starcraft) maps admit open-loop solutions that ignore local observations. While they redesigned these maps to enforce ”meaningful partial observability”, they provided no metric to quantify it. In single-agent RL, tao2025benchmarking formalised memory improvability based on performance gaps between agents with access to more or less state information. Our framework provides quantitative tools for the multi-agent case, moving beyond raw performance metrics. We disentangle history dependence, private information flow, and coordination as separate dimensions of Dec-POMDP difficulty. Conventions. Co-trained agents typically develop conventions that are efficient but arbitrary and brittle when paired with unfamiliar partners (foerster2019bayesian; hu2020other). Prior work shows that grounding these conventions in observations makes coordination more robust (hu2021off). Our A and DAI diagnostics explicitly quantify these dynamics, disentangling instantaneous, ungrounded conventions from coordination that is temporally responsive to a partner’s trajectory. 4. Probing Dec-POMDPs To probe the reasoning demands specific to MARL environments, we focus on two core properties of Dec-POMDPs – partial observability and decentralised coordination. While the interaction of these factors renders the general problem class NEXP-complete444The worst case complexity of DEC-MDPs is the same as Dec-POMDPs (bernstein2002complexity), as such hardness comes from decentralisation as well, and not (only) from the presence of hidden states., theoretical worst-case hardness does not necessarily imply practical difficulty in specific benchmarks. Our goal is therefore to characterise these properties functionally, measuring them only as they matter for solving a task. Consequently, we define every diagnostic as an expectation under the trajectory distribution of a joint policy π after convergence. We do not define purely structural properties of Dec-POMDPs independent of behaviour, but rather, we quantify the specific reasoning capabilities necessitated by the task. Figure 1 presents a summary and interpretation of our proposed diagnostic measures, and we discuss the technical details in the following sections. 4.1. Partial Observability Is Partial Observability Relevant? While many environments are structurally partially observable (states are hidden), this does not guarantee that the missing information is functionally relevant to solving a task. For example, the hidden state may not affect the rewards or dynamics, or it may be redundant given the current observations. We are therefore interested in identifying when partial observability strictly affect success. If a task requires memory, it confirms that immediate observations are insufficient and that history contains decision-relevant information. Therefore, we measure this using history dependence. Definition (Relevant Partial Observability). An environment exhibits relevant partial observability if: (i) memory-based agents (πRNN _RNN) outperform reactive agents (πF _F) under matched training conditions; and (i) learned policies actively exploit history, rather than relying solely on immediate observations. This definition requires that memory is both beneficial (producing higher returns) and active (influencing decisions). We quantify this with a performance diagnostic, and with two complementary information-theoretic probes. Diagnostic 1 (Memory–Reactive Gap). We test whether memory results in a performance gain by comparing recurrent and feed-forward policies. For matched training runs (sharing seed, environment, and algorithm), let J(π)J(π) denote the mean evaluation return. We define the paired performance gap as: ΔMem≜J(πRNN)−J(πF). _Mem J( _RNN)-J( _F). We test :median(ΔMem)≤0H_0:\ median( _Mem)≤ 0 vs. :median(ΔMem)>0H_1:\ median( _Mem)>0 using a one-sided Wilcoxon signed-rank test (wilcoxon1945individual) over the paired differences. A significant result (p<0.05p<0.05) indicates a reliable performance advantage from memory under matched training. Diagnostic 2 (History–Action Relevance (HAR)). We quantify memory use beyond the current observation via conditional mutual information: HAR≜(Hti;Ati∣Oti),HARnorm≜HARℍ(Ati∣Oti)∈[0,1].HAR (H_t^i;A_t^i O_t^i), ^norm HARH(A_t^i O_t^i)∈[0,1]. (3) Here, HtiH_t^i denotes the agent’s history representation, for reactive policies, Hti=Ot−k:t−1iH_t^i=O^i_t-k:t-1 (a length-k window excluding OtiO_t^i), and for recurrent policies, HtiH_t^i is the RNN hidden state. Diagnostic 3 (Observation–Action Relevance (OAR)). We quantify reactivity by measuring how informative the current observation is about the agent’s action: OAR≜(Oti;Ati),OARnorm≜OARℍ(Ati)∈[0,1].OAR (O_t^i;A_t^i), ^norm OARH(A_t^i)∈[0,1]. (4) High OARnormOAR^norm indicates that AtiA_t^i is largely predictable from the current observation OtiO_t^i (i.e., near-reactive behaviour). Conversely, low OARnormOAR^norm combined with high HARnormHAR^norm provides evidence that history contributes information for selecting AtiA_t^i beyond what is contained in OtiO_t^i. Is Partial Observability Reliant on Private Information? The previous diagnostics measure whether agents benefit from history or memory, which acts as a behavioural proxy for functionally relevant partial observability. Crucially, history dependence alone does not imply that the hidden information is relevant for coordination. An agent may use its history only to infer latent environment state, as in single-agent POMDPs (aastrom1965optimal; kaelbling1998planning), even if this provides no additional information about coordinating with teammates. We therefore introduce a cross-agent diagnostic that quantifies whether the private information of one agent helps predict the actions of another. This metric is related to the intuition behind meaningful partial observability (ellis2023smacv2), where hidden information observed by one agent is critical for the actions of another. Such cross-agent information asymmetries are central to the hardness of Dec-POMDPs (bernstein2002complexity). Diagnostic 4 (Private Information Flow (PIF)). We measure how much additional information agent i’s history provides about agent j’s action, beyond what is already contained in j’s own history. We define this using conditional mutual information: PIFi→j≜(τt−1i,Oti;Atj∣τt−1j,Otj),PIFi→jnorm≜PIFi→jℍ(Atj∣τt−1j,Otj)∈[0,1]. splitPIF_i→ j& ( _t-1^i,O_t^i;\,A_t^j _t-1^j,O_t^j),\\ PIF^norm_i→ j& PIF_i→ jH(A_t^j _t-1^j,O_t^j)∈[0,1]. split (5) Here, τt−1 _t-1 denotes an agent’s action-observation history555In practice, we approximate τt−1 _t-1 using the RNN hidden state (for recurrent policies) or a finite window of size k, (Ot−k:t−1i,At−k:t−1i)(O^i_t-k:t-1,A^i_t-k:t-1) (for reactive policies).. We explicitly condition on the current observations Oti,OtjO_t^i,O_t^j alongside the past τt−1 _t-1 to capture information asymmetries at decision time. PIFi→jPIF_i→ j quantifies how much information about AtjA_t^j is contained in agent i’s trajectory that is not already captured by agent j. PIFi→jnormPIF^norm_i→ j rescales this as the fraction of agent j’s residual action uncertainty (given its own history τt−1j _t-1^j and observation OtjO_t^j) that is explained by agent i. 4.2. Decentralised Coordination The previous diagnostics quantify whether hidden information is relevant to decision-making, specifically, whether agents require memory of local state (HAR; Diagnostic 2) or access to a teammate’s private information (PIF; Diagnostic 4). However, they do not characterise the form of coordination that emerges in the joint behaviour induced by the converged policies (if any). We therefore introduce coordination probes that separate instantaneous action coupling from temporally extended, more directional dependence. Is Coordination Synchronous? Diagnostic 5 (Action–Action Coupling (A)). We quantify instantaneous action dependence via the coupling of actions at time t: A≜(Ati;Atj∣Oti,Otj),AAnorm≜Aℍ(Atj∣Oti,Otj).A (A_t^i;A_t^j O_t^i,O_t^j ), ^norm AH (A_t^j O_t^i,O_t^j ). (6) AA measures same-timestep dependence between agents’ actions beyond what their current observations explain, and A>0A>0 is consistent with symmetry breaking or instantaneous conventions (e.g., agents taking distinct roles such as heading to different landmarks). Is Coordination Temporally Responsive? AA alone cannot distinguish task-driven role differentiation from arbitrary, ungrounded conventions, as it detects instantaneous coupling beyond shared observations, but cannot distinguish static conventions (e.g., fixed roles) from agents adapting to evolving partner behaviours. To probe this temporally extended, directional dependence, we test whether agent i’s past provides additional predictive information about agent j’s current action, conditioned on agent j’s own history. While a lagged AA could measure this, it would rely on fixed windows that are brittle to unknown or variable delays. We instead use Directed Information (massey1990causality), which aggregates directional cross-timestep dependence over the episode, capturing dependencies regardless of the temporal lag. Diagnostic 6 (Directed Action Information (DAI)). We measure the average directional, cross-timestep dependence from agent i to agent j as follows: DAIi→j≜1T∑t=1T(τt−1i;Atj|τt−1j),DAIi→jnorm≜DAIi→j1T∑t=1Tℍ(Atj|τt−1j)∈[0,1]. splitDAI_i→ j 1T _t=1^TI (τ^i_t-1;A_t^j |τ^j_t-1 ),\\ DAI^norm_i→ j DAI_i→ j 1T _t=1^TH (A_t^j |τ^j_t-1 )∈[0,1]. split (7) Here, τt−1iτ^i_t-1 is agent i’s action-observation history up to t−1t-1, including At−1iA^i_t-1, the last act before agent j selects AtjA^j_t. Conditioning on τt−1jτ^j_t-1 controls for what is already predictable from agent j’s own past, so DAIi→j>0DAI_i→ j>0 indicates that agent i’s past carries additional predictive information about agent j’s current action. Unlike PIF, which includes current observations to capture information at decision time, DAI conditions only on the causal past (the trajectory completed before j’s action), isolating strictly temporal, directional influence. 5. Case Study: How Observation Structure shapes Behaviour (a) Simple Reference (b) Speaker Listener (c) Simple Spread Scenario Δ (RNN−-F) Simple Reference 6.50 Speaker Listener 14.84 Simple Spread 2.50 (d) ΔMem _Mem; bold: p<0.05p<0.05. Figure 2. MPE tasks and per-environment performance deltas (RNN−-F). Memory improves performance across tasks. (a) Observation–Action Relevance (OAR) (b) History–Action Relevance (HAR) (c) Private Information Flow (PIF) (d) Directed Action Information (DAI) Figure 3. Evolution of diagnostic metrics during training in MPE with recurrent MAPPO (mean and 95% CI): (a) Observation–Action Relevance, (b) History–Action Relevance, (c) Private Information Flow, and (d) Directed Action Information. For the same algorithm and training paradigm, environment modifications can have a large impact on the kinds of behaviour learned. Multi-Particle Environments (MPE) (lowe2017multi) provide a controlled testbed with differing observation and communication structures. We examine three cooperative tasks—Simple Reference, Speaker–Listener and Simple Spread—using our diagnostics (Section 4) and MAPPO. Performance. We see from Tbl. 2, recurrent policies (RNN) outperform feed-forward (F) baselines in all three tasks (p<0.05p<0.05, one-tailed Wilcoxon), confirming that memory provides a reliable advantage across MPE. What the diagnostics reveal. Viewing MPE through our diagnostics shows that learned behaviour varies sharply across tasks, not because the algorithm changes, but because the observation/communication structure does. Simple Reference (Fig. 2). In Simple Reference, two agents move and observe the other’s goal alongside a rich communication channel (dimc=10dim_c=10). Goal information is thus redundantly available at every timestep, reducing the need for history: HARnormHAR^norm is the lowest across tasks and declines over training (≈ 0.06≈\,0.06, Fig. 3(b)), and PIFPIF/DAIDAI remain low (Fig. 3(c), 3(d)). Speaker–Listener (Fig. 2). In this scenario, a stationary speaker observes a hidden goal and must guide a listener that receives no goal information except through a narrow message channel (dimc=3dim_c=3). This dependency produces the highest DAInormDAI^norm across tasks (>0.15>0.15, Fig. 3(d)), reflecting sustained directional influence from speaker to listener. HARnormHAR^norm rises over training but remains moderate (≈ 0.10≈\,0.10), suggesting that the listener’s history use, while present, is secondary to the cross-agent information channel. Simple Spread (Fig. 2). Here, agents must cover distinct landmarks without explicit communication. Consequently, HARnormHAR^norm and PIFnormPIF^norm are the highest across tasks (Fig. 3(b), Fig. 3(c)), indicating that agents condition on each other’s private trajectories to avoid overlapping landmarks. DAInormDAI^norm is also substantial (≈ 0.12≈\,0.12), confirming coordination is both temporally extended and reliant on private information. Takeaway. The form of coordination that emerges is shaped primarily by information bottlenecks in the environment. When task-relevant information is fully available at each timestep (Simple Reference), agents default to reactive behaviour despite having recurrent architectures. Conversely, when information is restricted, e.g., funnelled through a narrow channel (Speaker–Listener) or left implicit in a partner’s trajectory (Simple Spread), agents develop qualitatively different coordination structures: higher directional influence in the former, and higher private information flow in the latter. 6. Results (a) MPE (b) SMAX V1 maps (c) SMAX V2 maps (d) MaBrax (e) Hanabi (f) Overcooked V1 (g) Overcooked V2 Figure 4. Sample efficiency of IPPO and MAPPO across diverse MARL benchmarks. We show the min-max normalised interquartile mean (IQM) with 95% stratified bootstrap confidence intervals (CIs). Detailed plots in App. B. 55footnotetext: MAPPO is omitted for Overcooked V1 as full observability renders a centralised critic redundant. We apply our diagnostics (Sec. 4) to widely used cooperative MARL benchmarks, using learned policies as probes of partial observability and decentralised coordination as they arise in behaviour. Concretely, we ask a fundamental question: do these tasks genuinely elicit Dec-POMDP reasoning, where agents exploit history to infer decision-relevant hidden states and coordinate based on private information, or do they permit solutions that largely bypass these demands? Experimental Setup. We evaluate 37 scenarios across MPE (lowe2017multi), SMAX (V1 maps and V2-style maps) (rutherford2023jaxmarl; samvelyan2019starcraft), Overcooked (V1 and V2) (carroll2019utility; gessler2025overcookedv), Hanabi (bard2020hanabi) and MaBrax (rutherford2023jaxmarl; peng2021facmac). Evaluation Protocol. We train with 10 seeds, matching original training budgets, and evaluate every 5% of training (mean evaluation return over 32 episodes) (gorsane2022towards). For aggregate comparisons, we report min–max normalised interquartile mean (IQM) with 95% stratified bootstrap CIs (agarwal2021deep). Hyperparameters are tuned per scenario, full details in App. A.1. Algorithms. We use Independent PPO (IPPO, de2020independent) and Multi-Agent PPO (MAPPO, yu2022surprising) as they are widely used MARL baselines. We treat them as two training paradigms: IPPO uses independent critics, whereas MAPPO uses a centralised critic. Additionally, we compare feed-forward (F) and recurrent (RNN) policies to study the role of memory and temporal information flow in these settings. Finally, to avoid confounders from optimisation and representation choices associated with shared weights in heterogeneous tasks (christianos2021scaling; tessera2025hypermarl), we do not use parameter sharing in any baseline. 6.1. Diagnostic Probes To answer the questions from Section 4, we use a two-stage protocol. First, we compute diagnostics on converged policies. Then, we determine whether each value reflects genuine structure or finite-sample noise by comparing against a permutation null baseline. Permutation null baselines. Information-theoretic estimators (e.g., kNN/KSG (kraskov2004estimating; ross2014mutual)) can exhibit bias when working with finite samples, resulting in non-zero values even under independence. We therefore construct an empirical null by independently permuting each agent’s action sequence within each episode, which destroys temporal and cross-agent dependencies while preserving each agent’s marginal action distributions. We recompute each diagnostic on the permuted data and deem the result meaningful only if its value on the original trajectories exceeds the mean of the corresponding permutation baseline. Aggregation. We apply a two-stage aggregation to probe for the emergence of Dec-POMDP reasoning capabilities. First, within each run, we compute the maximum diagnostic value across agents, asking whether any agent exhibits the property. Second, we maximise across training configurations (IPPO/MAPPO × F/RNN) to determine if any algorithm elicits the behaviour. This returns a conservative, per-scenario verdict: a property is flagged as absent only if no agent under any tested paradigm displays it. Decision Rules. We now translate the conceptual questions from Section 4 into concrete decision rules, evaluating partial observability and coordination directly through agent behaviour. Table 1. Diagnostics of learned behaviour across cooperative MARL benchmarks. We report the share of scenarios (count/total) where trained policies satisfy our decision criteria (Sec. 6.1). Crucially, these reflect dependencies induced by the policy rather than strict environment requirements. Per-scenario metrics are detailed in App. D. MPE SMAX V1 SMAX V2 MaBrax Hanabi Overcooked V1 Overcooked V2 Do agents benefit from memory? 100% (3/3) 100% (9/9) 100% (3/3) 20% (1/5) 0% (0/1) 0% (0/5) 0% (0/11) Do agents use hidden teammate information? 100% (3/3) 67% (6/9) 67% (2/3) 100% (5/5) 0% (0/1) 20% (1/5) 82% (9/11) Does synchronous coordination emerge? 100% (3/3) 44% (4/9) 0% (0/3) 60% (3/5) 0% (0/1) 100% (5/5) 82% (9/11) Does temporal coordination emerge? 100% (3/3) 67% (6/9) 67% (2/3) 100% (5/5) 100% (1/1) 40% (2/5) 100% (11/11) Decision Rule 1 (Do agents benefit from memory?). Following Definition 4.1, agents benefit from memory iff both: (1) Significant performance gap: The memory–reactive gap ΔMem _Mem is significant (one-tailed Wilcoxon signed-rank, p<0.05p<0.05), see Diagnostic 1. (2) Meaningful history use: Under the memory-based policy, HARnormHAR^norm exceeds its permutation null baseline, see Diagnostic 2. Criterion (1) establishes a reliable performance advantage from memory, while criterion (2) confirms that this advantage reflects active use of history rather than other confounding factors, such as optimisation dynamics. Decision Rule 2 (Do agents use hidden teammate information?). Agents use hidden teammate information iff PIFnormPIF^norm exceeds its permutation null baseline, indicating that agent i’s trajectory and observation inform agent j’s action beyond agent j’s own history (Diag. 4). Decision Rule 3 (Does synchronous coordination emerge?). Instantaneous, synchronous coordination emerges iff AAnormA^norm exceeds its permutation null baseline, indicating coupling beyond shared observations (Diag. 5). Decision Rule 4 (Does temporal coordination emerge?). Temporal, directional coordination emerges iff DAInormDAI^norm exceeds its permutation null baseline, indicating genuine causal influence from past actions (Diag. 6). 6.2. The Relevance of Partial Observability How often does memory really matter? Applying Decision Rule 1, we find that memory-based policies yield a statistically significant performance advantage in 43.2% (16/37) of tested scenarios (ΔMem>0 _Mem>0; see Tbls. 1, 14, and Fig. 4). However, we observe a clear dissociation between history dependence and utility. HARnormHAR^norm exceeds its permutation null in all 37 scenarios (App. Tbl. LABEL:tab:all_norm_metrics_long), confirming that trained policies universally encode some history dependence, yet this dependence translates into a measurable performance gain in less than half of the cases. Hanabi illustrates this disconnect. Despite being a canonical partially observable task, the memory–reactive gap is not significant under our baselines (ΔMem=0.279 _Mem=0.279, Tbls. 1, 14), as IPPO/MAPPO fail to meaningfully exploit recurrent architectures to improve performance on this task (Fig. 11). This suggests that much of the observed history dependence could be redundant, i.e., policies learn to track past information that offers no functional advantage over current observations OtiO_t^i. Consequently, to genuinely test Dec-POMDP reasoning, environments should ensure decision-relevant information is exclusively available through history, rendering reactive policies insufficient. Is partial observability reliant on private information? From applying Decision Rule 2, we find that PIFnormPIF^norm exceeds its permutation null in 70.3% (26/37) of tested scenarios (Tbl. 1). Notably, many of these are not the same scenarios flagged by the HAR criterion, confirming that hidden environment state and hidden teammate information are distinct drivers of difficulty that our metrics can successfully disentangle (App. Tbl. LABEL:tab:all_norm_metrics_long). This separation is especially visible in Overcooked. Overcooked V1 is fully observable and triggers PIF in only 20% of layouts, while Overcooked V2, which introduces hidden teammate information by design (gessler2025overcookedv), rises to 82%. This serves as an external validation of our diagnostic, as PIF recovers the design intentions of the environment authors. SMAX V2 maps, following SMACv2, were similarly motivated by ”meaningful partial observability” (ellis2023smacv2), however, PIF is detected in 67% of both V1 and V2 maps. This suggests that, at least under current baselines, several V1 maps already exhibit meaningful cross-agent information flow, and the redesign may not have widened this gap as intended. 6.3. Decentralised Coordination Synchronous vs. Temporal coordination. Decision Rules 3 and 4 probe two distinct coordination mechanisms. Synchronous coordination (AA) captures instantaneous action coupling conditioned on current observations, and 64.9% (24/37) of scenarios exceed the null permutation. While high AA indicates action-action dependence, this coupling can be brittle, e.g. when it reflects rigid, ungrounded conventions that do not generalise (hu2020other). Nonetheless, it remains a signature of coordination. Directed Action Information (DAIDAI), by contrast, measures temporal influence between agents. Under this measure, 81.1% (30/37) of scenarios exceed the null permutation. Notably, 10/37 scenarios lack synchronous coupling yet exhibit significant temporal influence (App. Tbl. LABEL:tab:all_norm_metrics_long), indicating that meaningful sequential coordination can arise without simultaneous conventions. These two mechanisms dissociate systematically across benchmarks, revealing the underlying coordination structure each environment induces. SMAX V2 maps show the starkest separation—none trigger A, yet 67% elicit DAI, suggesting that SMAX V2-style combat micro-management relies on sequential positioning rather than synchronous actions. Overcooked V1 presents a contrasting profile (100% A, 40% DAI), reflecting rigid positional conventions in many scenarios. However, Overcooked V2’s introduction of hidden information strengthens temporal dependence (100% DAI) while retaining synchronous coupling (82% A). Finally, MPE stands out as the only suite where every scenario demands both coordination forms (100% A and 100% DAI). Summary. Our audit yields four main takeaways: (1) History dependence ≠ history utility. All policies exhibit detectable history dependence (HARnorm>nullHAR^norm>null in 37/37 scenarios), yet only 43.2% show a significant performance gain from memory (Fig. 4, Tbl. 14). (2) Hidden state and private information are separable. PIF flags 70.3% of scenarios, often different ones from HAR, confirming these are separate drivers of difficulty. The Overcooked V1→ 2 contrast (20% → 82%) validates PIF as an environment-agnostic audit tool. (3) Coordination is structurally diverse. A (64.9%) and DAI (81.1%) dissociate across benchmarks: SMAX V2 exhibits temporal but not synchronous coordination, Overcooked V1 the reverse, while MPE, SMAX V1, MaBrax, and Overcooked V2 trigger both. (4) Few benchmarks jointly test partial observability and coordination. MPE is the only suite in which every scenario satisfies all diagnostic criteria. Most scenarios do not require meaningful history use for strong performance despite being framed as Dec-POMDP challenges. Our diagnostics expose the divergence between what a benchmark intends to test and what it actually requires. By characterising how agents coordinate rather than just how well, these tools enable researchers to verify Dec-POMDP demands and deliberately select environments that stress-test specific capabilities. Furthermore, as demonstrated in Section 5, our metrics capture the behavioural impact of structural environment changes, potentially providing actionable guidance for designing more rigorous cooperative environments. 7. Implications (a) MPE Simple Reference (b) MPE Simple Spread Figure 5. Robustness to observational noise in MPE. We report the mean return and 95% stratified bootstrap confidence intervals for IPPO FeedForward (F) across varying noise scales. Scenarios with higher OARnormOAR^norm (Simple Spread) correlate with greater sensitivity to sensory perturbations. In many real-world cooperative systems, agents are expected to adapt to changes in their environment and to the behaviour of other agents. While our probes and metrics do not directly measure generalisation, they allow us to detect when policies exhibit weak statistical dependence between observations/histories and actions, i.e., low OARnormOAR^norm and HARnormHAR^norm. Such instances suggest that agents may be relying on learned conventions or implicit coordination strategies rather than actively conditioning on current observations. This distinction has nuanced implications. On the one hand, environments in which agents can solve the task via conventions without relying on observations may yield policies that are robust to sensory noise or partial occlusion. On the other hand, such policies may be brittle under structural changes to the environment, to the behaviour of other agents, or to even minimal variations in the task definition (zamboni2025principledunsupervised), since coordination may depend on fixed joint strategies rather than observation-driven adaptation. To examine how our diagnostics relate to behavioural robustness, we conduct controlled evaluations under noisy observations in two MPE tasks with differing OARnormOAR^norm values: Simple Spread and Simple Reference. The former exhibits substantially higher estimated mutual information between observations and actions than the latter (IPPO F, App. D). To test robustness to noise, we perturb observations x with additive Gaussian noise scaled by the feature-wise standard deviation σx _x, computed over N initial rollouts. For noise scale k∈[0,0.5]k∈[0,0.5]. For more details on how we add noise see the (App. C). Fig. 5 shows that performance in Simple Spread degrades more substantially under increasing noise than in Simple Reference. This is consistent with the higher OARnormOAR^norm observed in Simple Spread, when using IPPO F. A key take-away is that information-theoretic diagnostics can provide structured signals about how policies utilise observations and interact with other agents under the training distribution. When interpreted jointly, they can indicate whether behaviour appears observation-driven or convention-driven. However, these metrics quantify statistical dependence rather than causal relationships. As a result, high mutual information does not guarantee sensitivity to noise, and low values do not necessarily imply the absence of structured coordination. Careful behavioural evaluation alongside the use of diagnostics can however provide indications of robustness and generalisation of learned policies. 8. Challenges and Limitations Policy-dependent probes. All diagnostics are expectations under the converged joint policy p π and therefore characterise learned behaviour under IPPO/MAPPO with F/RNN architectures, not worst-case or best-case properties of the environment. This is deliberate, as we probe behaviours induced by widely used algorithms; however, stronger or weaker algorithms may yield different diagnostic profiles for the same scenario. Estimation noise. Our MI/CMI/DI estimators (kNN and KSG (kraskov2004estimating)) are biased in finite samples, especially with long histories or large action spaces. We mitigate this via permutation null baselines that account for estimator-specific bias, and report bootstrap confidence intervals throughout. Nonetheless, these probes are diagnostic tools, not hard pass/fail filters, and borderline cases should be interpreted with caution. 9. Conclusion In this work, we introduce a principled diagnostic framework to probe whether cooperative MARL agents genuinely exhibit Dec-POMDP reasoning. By coupling information-theoretic metrics with simple decision rules, our diagnostics evaluate how policies solve tasks, not just how well, moving evaluation beyond raw returns. Applied to 37 scenarios across seven environments, our analysis reveals that: (i) history dependence is ubiquitous but rarely yields a performance advantage; (i) hidden state and private teammate information are separable drivers of difficulty; and (i) synchronous and temporal coordination frequently dissociate across domains. Notably, MPE is the only environment in which every scenario satisfies all diagnostic criteria. Our case study further demonstrates that the form of emergent coordination is shaped primarily by information bottlenecks in the environment design. These findings motivate a shift toward benchmarks that strictly compel agents to exploit historical context and coordinate under private information—making partial observability and decentralised coordination non-optional for success. 10. Acknowledgements An author on this project received funding from the European Union’s Horizon Europe research and innovation programme under grant agreement No. 101120726. This work was also supported by UK Research and Innovation (UKRI) under the UK government’s Horizon Europe funding Guarantee 10085198. References Appendix Appendix A Solution Concepts As in the single-agent case, we can define value functions for any joint policy π composed of per-agent policies. The value function for agent i are defined as Vi(s):=[∑t=0∞γtRi(st,t)∣st=s]V_i π(s):=E_ π [ _t=0^∞γ^tR^i(s_t,a_t) s_t=s ] (8) We define a best-response policy for agent i as π†(π−i):i→Δiπ (π^-i):O^i→ _A_i such that Viπi,†(π−i)×π−i(s)=supπ¯iViπ¯i×π−i(s),∀s∈,V_i^π^i, (π^-i)×π^-i(s)= _ π^iV_i π^i×π^-i(s), ∀ s , (9) and we denote Vi†,π−i(s):=Viπi,†(π−i)×π−i(s),Vi†,π−i(μ):=s∼μ[Vi†,π−i(s)].V_i , _-i(s):=V_i^π^i, ( _-i)× _-i(s), V_i , _-i(μ):=E_s μ [V_i , _-i(s) ]. (10) We are now ready to define the following solution concepts for Dec-POMDPs: Definition ((Markov) Equilibria). For ϵ>0ε>0, a (Markov) policy π is a (Markov) ϵε-Approximate Coarse Correlated Equilibrium (CCE) if maxi∈[||]Vi†,π−i(μ)−Vi(μ)≤ϵ. _i∈[|N|] \V_i , _-i(μ)-V_i π(μ) \≤ε. (11) It is a (Markov) Coarse Correlated Equilibrium if ϵ=0ε=0. A product policy π satisfying the same condition is called a (ϵε-Approximate) Nash Equilibrium (NE). If the policy is non-Markov, then the equilibrium is said to be non-Markov. A.1. Hyperparameters Table 2. Default hyperparameters for MPE experiments. Hyperparameter IPPO-F IPPO-RNN MAPPO-F MAPPO-RNN Total Timesteps 1×1071× 10^7 Num. Parallel Envs 16 16 128 16 Num. Steps 128 128 128 128 FC Hidden Dim 128 128 128 128 GRU Hidden Dim – 128 – 128 Num. Minibatches 4 γ 0.99 GAE λ 0.95 Entropy Coef. 0.01 Value Function Coef. 0.5 Max Grad Norm 0.5 Activation tanh tanh relu tanh Anneal LR True Table 3. Default hyperparameters for Overcooked (V1) experiments. Hyperparameter IPPO-F IPPO-RNN Total Timesteps 1×1071× 10^7 Num. Parallel Envs 64 Num. Steps 256 FC Hidden Dim 64 GRU Hidden Dim – 64 Num. Minibatches 16 γ 0.99 GAE λ 0.95 Entropy Coef. 0.04 Value Function Coef. 0.5 Max Grad Norm 0.5 Activation relu LR Warmup 0.05 Rew. Shaping Horizon 5×1065× 10^6 Anneal LR True Table 4. Default hyperparameters for Overcooked (V2) experiments. All four algorithms share the same defaults. Hyperparameter All Algorithms Total Timesteps 1×1071× 10^7 Num. Parallel Envs 128 Num. Steps 256 FC Hidden Dim 128 GRU Hidden Dim 128 Num. Minibatches 64 γ 0.99 GAE λ 0.95 Entropy Coef. 0.01 Value Function Coef. 0.5 Max Grad Norm 0.25 Activation relu LR Warmup 0.05 Rew. Shaping Horizon 5×1065× 10^6 Anneal LR True Table 5. Default hyperparameters for SMAX experiments. Hyperparameter IPPO-F IPPO-RNN MAPPO-F MAPPO-RNN Total Timesteps 1×1071× 10^7 Num. Parallel Envs 128 Num. Steps 128 FC Hidden Dim 128 GRU Hidden Dim – 128 – 128 Num. Minibatches 4 γ 0.99 GAE λ 0.95 Entropy Coef. 0.0 Value Function Coef. 0.5 Max Grad Norm 0.5 Activation relu Anneal LR True Table 6. Default hyperparameters for Hanabi experiments. Hyperparameter IPPO-F IPPO-RNN MAPPO-F MAPPO-RNN Total Timesteps 1×10101× 10^10 Num. Parallel Envs 1024 Num. Steps 128 FC Hidden Dim 128 GRU Hidden Dim – 128 – 128 Num. Minibatches 4 Update Epochs 4 LR 5×10−45× 10^-4 γ 0.99 GAE λ 0.95 Clip ε 0.2 Entropy Coef. 0.01 Value Function Coef. 1.0 0.5 Max Grad Norm 0.5 Activation tanh relu Anneal LR True Adam ε 10−810^-8 Table 7. Default hyperparameters for MABrax experiments. Hyperparameter IPPO-F IPPO-RNN MAPPO-F MAPPO-RNN Total Timesteps 1×1081× 10^8 Num. Parallel Envs 64 Num. Steps 300 FC Hidden Dim 128 GRU Hidden Dim – 128 – 128 Num. Minibatches 4 Update Epochs 4 LR 1×10−31× 10^-3 γ 0.99 GAE λ 0.95 Clip ε 0.2 Entropy Coef. 2×10−62× 10^-6 Value Function Coef. 4.5 Max Grad Norm 0.5 Activation tanh Anneal LR True Table 8. Tuned hyperparameters for MPE environments, selected from sweep ranges shown in the sub-header. Environment Algorithm LR Clip ϵε Update Epochs 10−4,3×10−4,10−3\10^-4,3×10^-4,10^-3\ 0.1,0.2\0.1,0.2\ 2,4\2,4\ Spread IPPO-F 1×10−31× 10^-3 0.1 4 IPPO-RNN 1×10−31× 10^-3 0.1 4 MAPPO-F 1×10−31× 10^-3 0.1 4 MAPPO-RNN 1×10−31× 10^-3 0.1 2 Reference IPPO-F 1×10−41× 10^-4 0.2 4 IPPO-RNN 1×10−31× 10^-3 0.2 4 MAPPO-F 1×10−31× 10^-3 0.1 4 MAPPO-RNN 1×10−31× 10^-3 0.2 4 Speaker-Listener IPPO-F 1×10−31× 10^-3 0.2 2 IPPO-RNN 1×10−31× 10^-3 0.2 2 MAPPO-F 1×10−31× 10^-3 0.2 4 MAPPO-RNN 1×10−31× 10^-3 0.2 4 Tag IPPO-F 1×10−31× 10^-3 0.1 2 IPPO-RNN 1×10−41× 10^-4 0.2 2 MAPPO-F 1×10−31× 10^-3 0.1 2 MAPPO-RNN 1×10−41× 10^-4 0.2 4 Table 9. Tuned hyperparameters for Overcooked (V1) layouts, selected from sweep ranges shown in the sub-header. Layout Algorithm LR Clip ϵε Update Epochs Rew. Shaping 10−4,3×10−4,10−3\10^-4,3×10^-4,10^-3\ 0.1,0.2\0.1,0.2\ 2,4\2,4\ 2.5×106,1.5×107\2.5×10^6,1.5×10^7\ Cramped Room IPPO-F 5×10−45× 10^-4 0.2 4 5×1065× 10^6 IPPO-RNN 1×10−31× 10^-3 0.2 4 5×1065× 10^6 Asymm. Advantages IPPO-F 5×10−45× 10^-4 0.2 4 5×1065× 10^6 IPPO-RNN 1×10−31× 10^-3 0.2 4 5×1065× 10^6 Coord. Ring IPPO-F 1×10−31× 10^-3 0.2 4 5×1065× 10^6 IPPO-RNN 5×10−45× 10^-4 0.2 4 5×1065× 10^6 Counter Circuit IPPO-F 5×10−45× 10^-4 0.2 4 5×1065× 10^6 IPPO-RNN 1×10−31× 10^-3 0.2 4 5×1065× 10^6 Forced Coord. IPPO-F 5×10−45× 10^-4 0.2 4 5×1065× 10^6 IPPO-RNN 1×10−31× 10^-3 0.2 4 5×1065× 10^6 Table 10. Tuned hyperparameters for Overcooked (V2) existing layouts. Only LR was swept; Clip ϵ=0.2ε=0.2 and Update Epochs =4=4 were fixed. Layout Algorithm LR 10−4,4×10−4,5×10−4,10−3\10^-4,4×10^-4,5×10^-4,10^-3\ Cramped Room IPPO-F 1×10−31× 10^-3 IPPO-RNN 5×10−45× 10^-4 MAPPO-F 5×10−45× 10^-4 MAPPO-RNN 5×10−45× 10^-4 Asymm. Advantages IPPO-F 5×10−45× 10^-4 IPPO-RNN 1×10−31× 10^-3 MAPPO-F 5×10−45× 10^-4 MAPPO-RNN 1×10−31× 10^-3 Coord. Ring IPPO-F 1×10−31× 10^-3 IPPO-RNN 1×10−31× 10^-3 MAPPO-F 5×10−45× 10^-4 MAPPO-RNN 4×10−44× 10^-4 Counter Circuit IPPO-F 1×10−31× 10^-3 IPPO-RNN 1×10−31× 10^-3 MAPPO-F 4×10−44× 10^-4 MAPPO-RNN 5×10−45× 10^-4 Forced Coord. IPPO-F 1×10−31× 10^-3 IPPO-RNN 4×10−44× 10^-4 MAPPO-F 5×10−45× 10^-4 MAPPO-RNN 1×10−41× 10^-4 Table 11. Tuned hyperparameters for Overcooked (V2) new layouts. Only LR was swept; Clip ϵ=0.2ε=0.2 and Update Epochs =4=4 were fixed. Total timesteps =3×107=3× 10^7 and reward shaping horizon =1.5×107=1.5× 10^7. Layout Algorithm LR 10−4,4×10−4,5×10−4,10−3\10^-4,4×10^-4,5×10^-4,10^-3\ Demo Cook Simple IPPO-F 1×10−31× 10^-3 IPPO-RNN 1×10−31× 10^-3 MAPPO-F 5×10−45× 10^-4 MAPPO-RNN 5×10−45× 10^-4 Demo Cook Wide IPPO-F 1×10−31× 10^-3 IPPO-RNN 3×10−43× 10^-4 MAPPO-F 1×10−31× 10^-3 MAPPO-RNN 4×10−44× 10^-4 Grounded Coord. Ring IPPO-F 1×10−31× 10^-3 IPPO-RNN 1×10−41× 10^-4 MAPPO-F 1×10−31× 10^-3 MAPPO-RNN 4×10−44× 10^-4 Grounded Coord. Simple IPPO-F 1×10−31× 10^-3 IPPO-RNN 1×10−31× 10^-3 MAPPO-F 1×10−31× 10^-3 MAPPO-RNN 4×10−44× 10^-4 Test Time Simple IPPO-F 1×10−31× 10^-3 IPPO-RNN 1×10−31× 10^-3 MAPPO-F 1×10−31× 10^-3 MAPPO-RNN 1×10−31× 10^-3 Test Time Wide IPPO-F 1×10−31× 10^-3 IPPO-RNN 5×10−45× 10^-4 MAPPO-F 5×10−45× 10^-4 MAPPO-RNN 3×10−43× 10^-4 Table 12. Tuned hyperparameters for SMAX V1 maps, selected from sweep ranges shown in the sub-header. Clip ϵ=0.1ε=0.1 was selected for all configurations. UE = Update Epochs. IPPO-F IPPO-RNN MAPPO-F MAPPO-RNN Map LR UE LR UE LR UE LR UE Sweep range: LR ∈10−4,3×10−4,10−3∈\10^-4,3×10^-4,10^-3\, Clip ϵ∈0.1,0.2ε∈\0.1,0.2\, UE ∈2,4∈\2,4\ 3m 10−310^-3 4 10−310^-3 4 10−310^-3 4 10−310^-3 4 2s3z 10−310^-3 4 10−310^-3 4 10−310^-3 4 10−310^-3 4 3s5z 10−310^-3 4 10−310^-3 2 10−310^-3 4 10−310^-3 2 3s_vs_5z 10−310^-3 4 10−310^-3 2 10−310^-3 4 10−310^-3 4 3s5z_vs_3s6z 10−310^-3 4 10−310^-3 2 10−310^-3 4 10−310^-3 4 5m_vs_6m 10−410^-4 2 10−310^-3 2 10−310^-3 4 10−310^-3 4 8m 10−310^-3 4 10−310^-3 4 10−310^-3 4 10−310^-3 4 10m_vs_11m 10−410^-4 4 10−310^-3 2 10−410^-4 4 10−310^-3 4 6h_vs_8z 10−310^-3 4 10−310^-3 4 10−310^-3 4 10−310^-3 2 Table 13. Tuned hyperparameters for SMAX V2 maps, selected from sweep ranges shown in the sub-header. Clip ϵ=0.1ε=0.1 and Update Epochs =4=4 were selected for all configurations. Map Algorithm LR Clip ϵε Update Epochs 10−4,3×10−4,10−3\10^-4,3×10^-4,10^-3\ 0.1,0.2\0.1,0.2\ 2,4\2,4\ SMAX V2_5_units IPPO-F 1×10−31× 10^-3 0.1 4 IPPO-RNN 1×10−31× 10^-3 0.1 4 MAPPO-F 1×10−31× 10^-3 0.1 4 MAPPO-RNN 1×10−31× 10^-3 0.1 4 SMAX V2_10_units IPPO-F 1×10−31× 10^-3 0.1 4 IPPO-RNN 1×10−31× 10^-3 0.1 4 MAPPO-F 1×10−31× 10^-3 0.1 4 MAPPO-RNN 1×10−31× 10^-3 0.1 4 SMAX V2_20_units IPPO-F 1×10−31× 10^-3 0.1 4 IPPO-RNN 1×10−31× 10^-3 0.1 4 MAPPO-F 1×10−31× 10^-3 0.1 4 MAPPO-RNN 1×10−31× 10^-3 0.1 4 Appendix B Detailed Results B.1. Performance Results Table 14. Per-environment performance deltas, reported as the median of differences. Bold values indicate statistically significant results (p<0.05p<0.05, one-sided Wilcoxon signed rank test). We observe a statistically significant memory advantage (Δ>0 >0) in 43.2% (16/37) of the tested environments. †MAPPO comparisons are omitted for Overcooked V1 as the environment is fully observable (rendering IPPO/MAPPO identical). Environment Scenario Δ (RNN−-F) Δ (MAPPO−-IPPO) MPE MPE_simple_reference_v3 6.496 2.019 MPE_simple_speaker_listener_v4 14.840 1.596 MPE_simple_spread_v3 2.500 0.959 SMAX-V1 Maps 10m_vs_11m 0.945 -0.150 2s3z 1.241 0.009 3m 0.225 0.025 3s5z 1.169 0.280 3s5z_vs_3s6z 0.336 0.282 3s_vs_5z 0.228 0.138 5m_vs_6m 0.358 0.254 6h_vs_8z 0.168 0.047 8m 1.127 0.102 SMAX-V2 Maps SMAX V2_10_units 0.397 0.327 SMAX V2_20_units 0.205 0.151 SMAX V2_5_units 0.459 0.372 MaBrax ant_4x2 -1028.182 3487.891 halfcheetah_6x1 -184.390 1465.603 hopper_3x1 159.544 606.643 humanoid_9|8 31.723 397.284 walker2d_2x3 452.472 73.575 Hanabi Two Players 0.279 -1.313 Overcooked asymm_advantages -220.000 — coord_ring -40.000 — counter_circuit -35.625 — cramped_room 0.000 — forced_coord 0.000 — Overcooked V2 asymm_advantages -20.000 46.875 coord_ring 30.625 31.250 counter_circuit -8.750 34.062 cramped_room -25.625 8.125 demo_cook_simple -60.625 7.500 demo_cook_wide -10.938 -7.188 forced_coord -6.250 34.688 grounded_coord_ring -16.875 16.875 grounded_coord_simple -2.500 11.250 test_time_simple -2.500 10.000 test_time_wide 0.938 5.312 (a) Reference (b) Speaker–Listener (c) Spread Figure 6. Mean test returns with 95% confidence intervals in MPE. (a) 3m (b) 8m (c) 2s3z (d) 3s5z (e) 6h_vs_8z (f) 5m_vs_6m (g) 3s5z_vs_3s6z (h) 3s_vs_5z (i) 10m_vs_11m Figure 7. Mean test returns with 95% confidence intervals in SMAX V1. (a) 5 units (b) 10 units (c) 20 units Figure 8. Mean test returns with 95% confidence intervals in SMAX V2. (a) Coordination Ring (b) Asymmetric Advantages (c) Counter Circuit (d) Cramped Room (e) Forced Coordination Figure 9. Mean test returns with 95% confidence intervals in Overcooked V1. Overcooked V1 is fully observable, a centralised critic with concatenated global state is unnecessary here, so we report IPPO only. (a) Ant_4x2 (b) Halfcheetah_6x1 (c) Hopper_3x1 (d) Walker2d_2x3 (e) Humanoid_9—8 Figure 10. Mean test returns with 95% confidence intervals in MaBrax. Figure 11. Mean test returns with 95% confidence intervals in Hanabi with two players. Appendix C MPE Noise Details To add noise to our observation we define x^=x+ϵ,ϵ∼(0,(k⋅σx)2). x=x+ε, ε (0,(k· _x)^2). Under this formulation, the signal-to-noise ratio (SNR) in terms of variance is SNRpower=σx2(k⋅σx)2=1k2.SNR_power= _x^2(k· _x)^2= 1k^2. At the maximum scale k=0.5k=0.5, the SNR is 4:14:1, meaning signal variance is four times noise variance. Appendix D Detailed Diagnostic Measures Table 15. Normalised diagnostic metrics (mean with 95% stratified bootstrap CI). Values exceeding the permutation-null baseline are bolded. For F policies, HARnormHAR^norm uses an observation-history window and PIFnormPIF^norm/DAInormDAI^norm use an observation-action history window; for RNN policies these use the hidden state. AAnormA^norm is architecture-independent. †One or more metrics are undefined due to degenerate action entropy. Domain Environment Algorithm OARnormOAR^norm HARnormHAR^norm PIFnormPIF^norm AAnormA^norm DAInormDAI^norm MPE simple_reference_v3 IPPO_F 0.160.16 [0.15, 0.17][0.15,\,0.17] 0.100.10 [0.09, 0.10][0.09,\,0.10] 0.050.05 [0.05, 0.06][0.05,\,0.06] 0.070.07 [0.07, 0.07][0.07,\,0.07] 0.060.06 [0.05, 0.06][0.05,\,0.06] IPPO_RNN 0.130.13 [0.11, 0.15][0.11,\,0.15] 0.100.10 [0.10, 0.11][0.10,\,0.11] 0.040.04 [0.04, 0.04][0.04,\,0.04] 0.050.05 [0.05, 0.06][0.05,\,0.06] 0.040.04 [0.04, 0.04][0.04,\,0.04] MAPPO_F 0.160.16 [0.15, 0.17][0.15,\,0.17] 0.110.11 [0.10, 0.12][0.10,\,0.12] 0.050.05 [0.05, 0.06][0.05,\,0.06] 0.060.06 [0.06, 0.06][0.06,\,0.06] 0.060.06 [0.05, 0.07][0.05,\,0.07] MAPPO_RNN 0.190.19 [0.17, 0.21][0.17,\,0.21] 0.070.07 [0.07, 0.07][0.07,\,0.07] 0.040.04 [0.04, 0.04][0.04,\,0.04] 0.060.06 [0.06, 0.07][0.06,\,0.07] 0.030.03 [0.03, 0.04][0.03,\,0.04] simple_speaker_listener_v4 IPPO_F 0.930.93 [0.80, 1.00][0.80,\,1.00] 0.100.10 [0.10, 0.11][0.10,\,0.11] 0.100.10 [0.08, 0.11][0.08,\,0.11] 0.090.09 [0.09, 0.09][0.09,\,0.09] 0.120.12 [0.10, 0.14][0.10,\,0.14] IPPO_RNN 1.001.00 [1.00, 1.00][1.00,\,1.00] 0.110.11 [0.11, 0.12][0.11,\,0.12] 0.070.07 [0.06, 0.07][0.06,\,0.07] 0.070.07 [0.07, 0.08][0.07,\,0.08] 0.070.07 [0.06, 0.08][0.06,\,0.08] MAPPO_F 1.001.00 [1.00, 1.00][1.00,\,1.00] 0.120.12 [0.11, 0.13][0.11,\,0.13] 0.150.15 [0.13, 0.17][0.13,\,0.17] 0.110.11 [0.09, 0.13][0.09,\,0.13] 0.140.14 [0.12, 0.16][0.12,\,0.16] MAPPO_RNN 0.920.92 [0.79, 1.00][0.79,\,1.00] 0.100.10 [0.09, 0.10][0.09,\,0.10] 0.110.11 [0.07, 0.15][0.07,\,0.15] 0.060.06 [0.05, 0.06][0.05,\,0.06] 0.170.17 [0.13, 0.21][0.13,\,0.21] simple_spread_v3 IPPO_F 0.690.69 [0.54, 0.80][0.54,\,0.80] 0.070.07 [0.07, 0.07][0.07,\,0.07] 0.110.11 [0.10, 0.11][0.10,\,0.11] 0.070.07 [0.07, 0.08][0.07,\,0.08] 0.100.10 [0.10, 0.11][0.10,\,0.11] IPPO_RNN 0.250.25 [0.24, 0.25][0.24,\,0.25] 0.090.09 [0.09, 0.09][0.09,\,0.09] 0.050.05 [0.04, 0.05][0.04,\,0.05] 0.070.07 [0.07, 0.07][0.07,\,0.07] 0.040.04 [0.04, 0.04][0.04,\,0.04] MAPPO_F 0.260.26 [0.25, 0.28][0.25,\,0.28] 0.070.07 [0.07, 0.07][0.07,\,0.07] 0.110.11 [0.11, 0.11][0.11,\,0.11] 0.070.07 [0.07, 0.07][0.07,\,0.07] 0.100.10 [0.10, 0.10][0.10,\,0.10] MAPPO_RNN 0.250.25 [0.24, 0.25][0.24,\,0.25] 0.160.16 [0.16, 0.17][0.16,\,0.17] 0.180.18 [0.17, 0.19][0.17,\,0.19] 0.070.07 [0.07, 0.07][0.07,\,0.07] 0.120.12 [0.11, 0.12][0.11,\,0.12] SMAX-V1 maps 10m_vs_11m IPPO_F 0.600.60 [0.56, 0.64][0.56,\,0.64] 0.020.02 [0.02, 0.02][0.02,\,0.02] 0.010.01 [0.01, 0.01][0.01,\,0.01] 0.000.00 [0.00, 0.00][0.00,\,0.00] 0.010.01 [0.01, 0.01][0.01,\,0.01] IPPO_RNN 0.480.48 [0.47, 0.49][0.47,\,0.49] 0.060.06 [0.06, 0.07][0.06,\,0.07] 0.050.05 [0.05, 0.05][0.05,\,0.05] 0.010.01 [0.01, 0.01][0.01,\,0.01] 0.060.06 [0.05, 0.06][0.05,\,0.06] MAPPO_F 0.730.73 [0.61, 0.85][0.61,\,0.85] 0.070.07 [0.05, 0.11][0.05,\,0.11] 0.060.06 [0.04, 0.08][0.04,\,0.08] 0.010.01 [0.01, 0.01][0.01,\,0.01] 0.060.06 [0.05, 0.08][0.05,\,0.08] MAPPO_RNN 0.450.45 [0.44, 0.47][0.44,\,0.47] 0.060.06 [0.06, 0.07][0.06,\,0.07] 0.040.04 [0.04, 0.05][0.04,\,0.05] 0.010.01 [0.01, 0.01][0.01,\,0.01] 0.070.07 [0.07, 0.08][0.07,\,0.08] 2s3z IPPO_F 0.700.70 [0.67, 0.73][0.67,\,0.73] 0.080.08 [0.08, 0.09][0.08,\,0.09] 0.060.06 [0.05, 0.06][0.05,\,0.06] 0.020.02 [0.02, 0.02][0.02,\,0.02] 0.060.06 [0.06, 0.06][0.06,\,0.06] IPPO_RNN 0.560.56 [0.54, 0.58][0.54,\,0.58] 0.080.08 [0.07, 0.08][0.07,\,0.08] 0.060.06 [0.06, 0.06][0.06,\,0.06] 0.020.02 [0.02, 0.02][0.02,\,0.02] 0.060.06 [0.05, 0.06][0.05,\,0.06] MAPPO_F 0.730.73 [0.66, 0.81][0.66,\,0.81] 0.080.08 [0.08, 0.08][0.08,\,0.08] 0.060.06 [0.05, 0.06][0.05,\,0.06] 0.020.02 [0.02, 0.02][0.02,\,0.02] 0.060.06 [0.06, 0.07][0.06,\,0.07] MAPPO_RNN 0.670.67 [0.66, 0.68][0.66,\,0.68] 0.090.09 [0.08, 0.09][0.08,\,0.09] 0.060.06 [0.06, 0.06][0.06,\,0.06] 0.020.02 [0.02, 0.02][0.02,\,0.02] 0.050.05 [0.05, 0.06][0.05,\,0.06] 3m IPPO_F 0.760.76 [0.72, 0.79][0.72,\,0.79] 0.090.09 [0.08, 0.09][0.08,\,0.09] 0.030.03 [0.03, 0.04][0.03,\,0.04] 0.020.02 [0.02, 0.02][0.02,\,0.02] 0.040.04 [0.03, 0.04][0.03,\,0.04] IPPO_RNN 0.660.66 [0.65, 0.67][0.65,\,0.67] 0.130.13 [0.12, 0.13][0.12,\,0.13] 0.060.06 [0.06, 0.06][0.06,\,0.06] 0.030.03 [0.03, 0.04][0.03,\,0.04] 0.060.06 [0.06, 0.07][0.06,\,0.07] MAPPO_F 0.760.76 [0.73, 0.78][0.73,\,0.78] 0.110.11 [0.10, 0.11][0.10,\,0.11] 0.060.06 [0.05, 0.07][0.05,\,0.07] 0.030.03 [0.02, 0.03][0.02,\,0.03] 0.060.06 [0.05, 0.06][0.05,\,0.06] MAPPO_RNN 0.690.69 [0.68, 0.71][0.68,\,0.71] 0.120.12 [0.11, 0.12][0.11,\,0.12] 0.060.06 [0.05, 0.06][0.05,\,0.06] 0.030.03 [0.03, 0.03][0.03,\,0.03] 0.070.07 [0.06, 0.07][0.06,\,0.07] 3s5z IPPO_F 0.640.64 [0.62, 0.67][0.62,\,0.67] 0.070.07 [0.07, 0.07][0.07,\,0.07] 0.060.06 [0.06, 0.07][0.06,\,0.07] 0.010.01 [0.01, 0.01][0.01,\,0.01] 0.060.06 [0.06, 0.07][0.06,\,0.07] IPPO_RNN 0.550.55 [0.54, 0.56][0.54,\,0.56] 0.070.07 [0.06, 0.08][0.06,\,0.08] 0.070.07 [0.07, 0.08][0.07,\,0.08] 0.020.02 [0.02, 0.02][0.02,\,0.02] 0.070.07 [0.06, 0.07][0.06,\,0.07] MAPPO_F 0.700.70 [0.61, 0.80][0.61,\,0.80] 0.070.07 [0.07, 0.08][0.07,\,0.08] 0.060.06 [0.05, 0.06][0.05,\,0.06] 0.020.02 [0.01, 0.02][0.01,\,0.02] 0.060.06 [0.06, 0.08][0.06,\,0.08] MAPPO_RNN 0.710.71 [0.69, 0.73][0.69,\,0.73] 0.050.05 [0.05, 0.06][0.05,\,0.06] 0.080.08 [0.07, 0.08][0.07,\,0.08] 0.020.02 [0.02, 0.02][0.02,\,0.02] 0.080.08 [0.07, 0.08][0.07,\,0.08] 3s5z_vs_3s6z IPPO_F 0.620.62 [0.57, 0.67][0.57,\,0.67] 0.070.07 [0.07, 0.07][0.07,\,0.07] 0.060.06 [0.05, 0.06][0.05,\,0.06] 0.010.01 [0.01, 0.01][0.01,\,0.01] 0.060.06 [0.05, 0.06][0.05,\,0.06] IPPO_RNN 0.530.53 [0.50, 0.56][0.50,\,0.56] 0.070.07 [0.07, 0.08][0.07,\,0.08] 0.080.08 [0.07, 0.08][0.07,\,0.08] 0.020.02 [0.02, 0.02][0.02,\,0.02] 0.070.07 [0.07, 0.08][0.07,\,0.08] MAPPO_F 0.680.68 [0.59, 0.78][0.59,\,0.78] 0.070.07 [0.07, 0.07][0.07,\,0.07] 0.050.05 [0.05, 0.05][0.05,\,0.05] 0.010.01 [0.01, 0.02][0.01,\,0.02] 0.060.06 [0.05, 0.06][0.05,\,0.06] MAPPO_RNN 0.600.60 [0.59, 0.62][0.59,\,0.62] 0.060.06 [0.06, 0.06][0.06,\,0.06] 0.050.05 [0.05, 0.05][0.05,\,0.05] 0.010.01 [0.01, 0.02][0.01,\,0.02] 0.050.05 [0.05, 0.05][0.05,\,0.05] 3s_vs_5z IPPO_F 0.400.40 [0.38, 0.42][0.38,\,0.42] 0.090.09 [0.08, 0.09][0.08,\,0.09] 0.070.07 [0.06, 0.07][0.06,\,0.07] 0.030.03 [0.03, 0.03][0.03,\,0.03] 0.070.07 [0.06, 0.07][0.06,\,0.07] IPPO_RNN 0.440.44 [0.41, 0.47][0.41,\,0.47] 0.090.09 [0.09, 0.10][0.09,\,0.10] 0.070.07 [0.06, 0.07][0.06,\,0.07] 0.030.03 [0.03, 0.03][0.03,\,0.03] 0.060.06 [0.06, 0.06][0.06,\,0.06] MAPPO_F 0.630.63 [0.46, 0.81][0.46,\,0.81] 0.080.08 [0.08, 0.08][0.08,\,0.08] 0.050.05 [0.05, 0.05][0.05,\,0.05] 0.030.03 [0.02, 0.03][0.02,\,0.03] 0.060.06 [0.05, 0.06][0.05,\,0.06] MAPPO_RNN 0.410.41 [0.38, 0.44][0.38,\,0.44] 0.090.09 [0.09, 0.09][0.09,\,0.09] 0.070.07 [0.06, 0.07][0.06,\,0.07] 0.030.03 [0.03, 0.03][0.03,\,0.03] 0.060.06 [0.06, 0.06][0.06,\,0.06] 5m_vs_6m IPPO_F 0.680.68 [0.66, 0.70][0.66,\,0.70] 0.030.03 [0.03, 0.04][0.03,\,0.04] 0.010.01 [0.01, 0.01][0.01,\,0.01] 0.010.01 [0.01, 0.01][0.01,\,0.01] 0.010.01 [0.01, 0.01][0.01,\,0.01] IPPO_RNN 0.590.59 [0.58, 0.61][0.58,\,0.61] 0.080.08 [0.08, 0.09][0.08,\,0.09] 0.050.05 [0.05, 0.05][0.05,\,0.05] 0.020.02 [0.02, 0.02][0.02,\,0.02] 0.080.08 [0.08, 0.09][0.08,\,0.09] MAPPO_F 0.640.64 [0.63, 0.66][0.63,\,0.66] 0.080.08 [0.07, 0.08][0.07,\,0.08] 0.050.05 [0.05, 0.05][0.05,\,0.05] 0.020.02 [0.02, 0.02][0.02,\,0.02] 0.050.05 [0.05, 0.05][0.05,\,0.05] MAPPO_RNN 0.580.58 [0.57, 0.59][0.57,\,0.59] 0.090.09 [0.08, 0.09][0.08,\,0.09] 0.050.05 [0.04, 0.05][0.04,\,0.05] 0.020.02 [0.02, 0.02][0.02,\,0.02] 0.100.10 [0.09, 0.10][0.09,\,0.10] 6h_vs_8z IPPO_F 0.570.57 [0.53, 0.61][0.53,\,0.61] 0.090.09 [0.08, 0.09][0.08,\,0.09] 0.050.05 [0.05, 0.06][0.05,\,0.06] 0.020.02 [0.02, 0.02][0.02,\,0.02] 0.060.06 [0.05, 0.06][0.05,\,0.06] IPPO_RNN 0.370.37 [0.36, 0.38][0.36,\,0.38] 0.070.07 [0.07, 0.07][0.07,\,0.07] 0.060.06 [0.06, 0.06][0.06,\,0.06] 0.020.02 [0.02, 0.02][0.02,\,0.02] 0.060.06 [0.06, 0.06][0.06,\,0.06] MAPPO_F 0.510.51 [0.40, 0.65][0.40,\,0.65] 0.070.07 [0.07, 0.08][0.07,\,0.08] 0.050.05 [0.05, 0.06][0.05,\,0.06] 0.020.02 [0.02, 0.02][0.02,\,0.02] 0.060.06 [0.05, 0.07][0.05,\,0.07] MAPPO_RNN 0.470.47 [0.46, 0.48][0.46,\,0.48] 0.060.06 [0.06, 0.07][0.06,\,0.07] 0.060.06 [0.06, 0.06][0.06,\,0.06] 0.020.02 [0.02, 0.02][0.02,\,0.02] 0.060.06 [0.06, 0.06][0.06,\,0.06] 8m IPPO_F 0.610.61 [0.56, 0.67][0.56,\,0.67] 0.050.05 [0.05, 0.06][0.05,\,0.06] 0.030.03 [0.03, 0.03][0.03,\,0.03] 0.010.01 [0.01, 0.01][0.01,\,0.01] 0.030.03 [0.03, 0.04][0.03,\,0.04] IPPO_RNN 0.480.48 [0.46, 0.50][0.46,\,0.50] 0.070.07 [0.07, 0.08][0.07,\,0.08] 0.050.05 [0.05, 0.05][0.05,\,0.05] 0.010.01 [0.01, 0.02][0.01,\,0.02] 0.060.06 [0.06, 0.07][0.06,\,0.07] MAPPO_F 0.560.56 [0.54, 0.59][0.54,\,0.59] 0.070.07 [0.06, 0.08][0.06,\,0.08] 0.040.04 [0.04, 0.05][0.04,\,0.05] 0.010.01 [0.01, 0.01][0.01,\,0.01] 0.050.05 [0.05, 0.05][0.05,\,0.05] MAPPO_RNN 0.500.50 [0.49, 0.51][0.49,\,0.51] 0.070.07 [0.07, 0.08][0.07,\,0.08] 0.050.05 [0.05, 0.05][0.05,\,0.05] 0.010.01 [0.01, 0.01][0.01,\,0.01] 0.070.07 [0.07, 0.07][0.07,\,0.07] SMAX-V2 maps SMAX V2_5_units IPPO_F 0.290.29 [0.28, 0.29][0.28,\,0.29] 0.070.07 [0.06, 0.07][0.06,\,0.07] 0.050.05 [0.04, 0.06][0.04,\,0.06] 0.010.01 [0.01, 0.01][0.01,\,0.01] 0.060.06 [0.05, 0.07][0.05,\,0.07] IPPO_RNN 0.330.33 [0.32, 0.33][0.32,\,0.33] 0.090.09 [0.09, 0.09][0.09,\,0.09] 0.080.08 [0.08, 0.08][0.08,\,0.08] 0.010.01 [0.01, 0.01][0.01,\,0.01] 0.080.08 [0.08, 0.08][0.08,\,0.08] MAPPO_F 0.360.36 [0.35, 0.37][0.35,\,0.37] 0.110.11 [0.11, 0.11][0.11,\,0.11] 0.160.16 [0.15, 0.16][0.15,\,0.16] 0.020.02 [0.02, 0.02][0.02,\,0.02] 0.140.14 [0.14, 0.15][0.14,\,0.15] MAPPO_RNN 0.350.35 [0.34, 0.35][0.34,\,0.35] 0.120.12 [0.12, 0.13][0.12,\,0.13] 0.090.09 [0.09, 0.09][0.09,\,0.09] 0.020.02 [0.02, 0.02][0.02,\,0.02] 0.090.09 [0.09, 0.09][0.09,\,0.09] SMAX V2_10_units IPPO_F 0.120.12 [0.10, 0.14][0.10,\,0.14] 0.040.04 [0.03, 0.04][0.03,\,0.04] 0.030.03 [0.03, 0.04][0.03,\,0.04] 0.000.00 [0.00, 0.00][0.00,\,0.00] 0.040.04 [0.03, 0.04][0.03,\,0.04] IPPO_RNN 0.230.23 [0.22, 0.23][0.22,\,0.23] 0.070.07 [0.06, 0.08][0.06,\,0.08] 0.060.06 [0.06, 0.06][0.06,\,0.06] 0.010.01 [0.01, 0.01][0.01,\,0.01] 0.060.06 [0.06, 0.06][0.06,\,0.06] MAPPO_F 0.230.23 [0.22, 0.24][0.22,\,0.24] 0.110.11 [0.10, 0.13][0.10,\,0.13] 0.080.08 [0.07, 0.08][0.07,\,0.08] 0.010.01 [0.01, 0.01][0.01,\,0.01] 0.080.08 [0.07, 0.08][0.07,\,0.08] MAPPO_RNN 0.260.26 [0.25, 0.26][0.25,\,0.26] 0.090.09 [0.08, 0.09][0.08,\,0.09] 0.070.07 [0.07, 0.07][0.07,\,0.07] 0.010.01 [0.01, 0.01][0.01,\,0.01] 0.070.07 [0.07, 0.08][0.07,\,0.08] SMAX V2_20_units IPPO_F 0.000.00 [0.00, 0.00][0.00,\,0.00] 0.020.02 [0.02, 0.02][0.02,\,0.02] 0.010.01 [0.01, 0.02][0.01,\,0.02] 0.000.00 [0.00, 0.00][0.00,\,0.00] 0.020.02 [0.02, 0.02][0.02,\,0.02] IPPO_RNN 0.010.01 [0.00, 0.02][0.00,\,0.02] 0.050.05 [0.04, 0.05][0.04,\,0.05] 0.040.04 [0.04, 0.04][0.04,\,0.04] 0.000.00 [0.00, 0.00][0.00,\,0.00] 0.040.04 [0.04, 0.04][0.04,\,0.04] MAPPO_F 0.000.00 [0.00, 0.00][0.00,\,0.00] 0.060.06 [0.05, 0.07][0.05,\,0.07] 0.020.02 [0.01, 0.02][0.01,\,0.02] 0.000.00 [0.00, 0.00][0.00,\,0.00] 0.020.02 [0.02, 0.02][0.02,\,0.02] MAPPO_RNN 0.000.00 [0.00, 0.01][0.00,\,0.01] 0.060.06 [0.05, 0.07][0.05,\,0.07] 0.040.04 [0.04, 0.05][0.04,\,0.05] 0.000.00 [0.00, 0.00][0.00,\,0.00] 0.040.04 [0.04, 0.05][0.04,\,0.05] MaBrax ant_4x2 IPPO_F_NoPS 0.650.65 [0.51, 0.79][0.51,\,0.79] 0.460.46 [0.35, 0.61][0.35,\,0.61] 0.500.50 [0.39, 0.63][0.39,\,0.63] 0.000.00 [0.00, 0.01][0.00,\,0.01] 0.500.50 [0.38, 0.63][0.38,\,0.63] IPPO_RNN_NoPS 0.310.31 [0.16, 0.49][0.16,\,0.49] 0.020.02 [0.00, 0.03][0.00,\,0.03] 0.010.01 [0.00, 0.02][0.00,\,0.02] 0.010.01 [0.00, 0.02][0.00,\,0.02] 0.040.04 [0.01, 0.08][0.01,\,0.08] MAPPO_F_NoPS 0.580.58 [0.50, 0.70][0.50,\,0.70] 0.310.31 [0.21, 0.47][0.21,\,0.47] 0.380.38 [0.27, 0.54][0.27,\,0.54] 0.000.00 [0.00, 0.00][0.00,\,0.00] 0.370.37 [0.26, 0.52][0.26,\,0.52] MAPPO_RNN_NoPS 0.510.51 [0.42, 0.63][0.42,\,0.63] 0.010.01 [0.01, 0.01][0.01,\,0.01] 0.010.01 [0.00, 0.01][0.00,\,0.01] 0.000.00 [0.00, 0.00][0.00,\,0.00] 0.020.02 [0.02, 0.03][0.02,\,0.03] halfcheetah_6x1 IPPO_F_NoPS 0.130.13 [0.11, 0.15][0.11,\,0.15] 0.050.05 [0.05, 0.06][0.05,\,0.06] 0.060.06 [0.06, 0.06][0.06,\,0.06] 0.000.00 [0.00, 0.00][0.00,\,0.00] 0.060.06 [0.06, 0.06][0.06,\,0.06] IPPO_RNN_NoPS 1.001.00 [1.00, 1.00][1.00,\,1.00] 0.030.03 [0.01, 0.05][0.01,\,0.05] 0.020.02 [0.01, 0.03][0.01,\,0.03] 0.030.03 [0.01, 0.05][0.01,\,0.05] 0.050.05 [0.02, 0.09][0.02,\,0.09] MAPPO_F_NoPS —† 0.530.53 [0.30, 0.79][0.30,\,0.79] 0.620.62 [0.37, 0.87][0.37,\,0.87] —† 0.590.59 [0.36, 0.82][0.36,\,0.82] MAPPO_RNN_NoPS 1.001.00 [1.00, 1.00][1.00,\,1.00] 0.020.02 [0.01, 0.02][0.01,\,0.02] 0.020.02 [0.01, 0.04][0.01,\,0.04] 0.010.01 [0.01, 0.01][0.01,\,0.01] 0.090.09 [0.03, 0.17][0.03,\,0.17] hopper_3x1 IPPO_F_NoPS 0.290.29 [0.20, 0.40][0.20,\,0.40] 0.120.12 [0.10, 0.13][0.10,\,0.13] 0.140.14 [0.12, 0.16][0.12,\,0.16] 0.000.00 [0.00, 0.00][0.00,\,0.00] 0.120.12 [0.10, 0.13][0.10,\,0.13] IPPO_RNN_NoPS —† —† —† —† 0.100.10 † MAPPO_F_NoPS 1.001.00 [1.00, 1.00][1.00,\,1.00] 0.280.28 [0.21, 0.34][0.21,\,0.34] 0.350.35 [0.25, 0.47][0.25,\,0.47] 0.020.02 [0.00, 0.03][0.00,\,0.03] 0.360.36 [0.25, 0.47][0.25,\,0.47] MAPPO_RNN_NoPS 0.970.97 [0.93, 1.00][0.93,\,1.00] 0.010.01 [0.01, 0.02][0.01,\,0.02] 0.020.02 [0.00, 0.03][0.00,\,0.03] 0.000.00 [0.00, 0.01][0.00,\,0.01] 0.050.05 [0.02, 0.11][0.02,\,0.11] humanoid_9|8 IPPO_F_NoPS 0.000.00 [0.00, 0.00][0.00,\,0.00] 0.030.03 [0.03, 0.04][0.03,\,0.04] 0.030.03 [0.03, 0.04][0.03,\,0.04] 0.000.00 [0.00, 0.00][0.00,\,0.00] 0.030.03 [0.03, 0.04][0.03,\,0.04] IPPO_RNN_NoPS —† —† —† —† —† MAPPO_F_NoPS —† 0.070.07 [0.06, 0.09][0.06,\,0.09] 0.160.16 [0.06, 0.35][0.06,\,0.35] —† 0.070.07 [0.06, 0.09][0.06,\,0.09] MAPPO_RNN_NoPS 0.150.15 [0.06, 0.25][0.06,\,0.25] 0.010.01 [0.00, 0.02][0.00,\,0.02] 0.010.01 [0.00, 0.01][0.00,\,0.01] 0.010.01 [0.00, 0.02][0.00,\,0.02] 0.080.08 [0.01, 0.18][0.01,\,0.18] walker2d_2x3 IPPO_F_NoPS 0.560.56 [0.46, 0.69][0.46,\,0.69] 0.120.12 [0.11, 0.14][0.11,\,0.14] 0.150.15 [0.12, 0.19][0.12,\,0.19] 0.000.00 [0.00, 0.00][0.00,\,0.00] 0.140.14 [0.11, 0.16][0.11,\,0.16] IPPO_RNN_NoPS 1.001.00 [1.00, 1.00][1.00,\,1.00] 0.010.01 [0.00, 0.03][0.00,\,0.03] 0.010.01 [0.01, 0.03][0.01,\,0.03] 0.020.02 [0.00, 0.04][0.00,\,0.04] 0.040.04 [0.02, 0.10][0.02,\,0.10] MAPPO_F_NoPS 1.001.00 † 0.220.22 [0.15, 0.30][0.15,\,0.30] 0.350.35 [0.23, 0.49][0.23,\,0.49] 0.130.13 † 0.290.29 [0.21, 0.40][0.21,\,0.40] MAPPO_RNN_NoPS 1.001.00 † 0.030.03 † 0.010.01 † 0.000.00 † 0.100.10 [0.03, 0.17][0.03,\,0.17] Hanabi Two Players IPPO_F 0.110.11 [0.10, 0.12][0.10,\,0.12] 0.040.04 [0.04, 0.05][0.04,\,0.05] 0.020.02 [0.02, 0.02][0.02,\,0.02] 0.010.01 [0.01, 0.01][0.01,\,0.01] 0.020.02 [0.02, 0.02][0.02,\,0.02] IPPO_RNN 0.100.10 [0.10, 0.11][0.10,\,0.11] 0.050.05 [0.04, 0.05][0.04,\,0.05] 0.030.03 [0.03, 0.03][0.03,\,0.03] 0.010.01 [0.01, 0.01][0.01,\,0.01] 0.050.05 [0.05, 0.05][0.05,\,0.05] MAPPO_F 0.120.12 [0.10, 0.13][0.10,\,0.13] 0.040.04 [0.04, 0.05][0.04,\,0.05] 0.020.02 [0.02, 0.02][0.02,\,0.02] 0.010.01 [0.01, 0.01][0.01,\,0.01] 0.020.02 [0.02, 0.02][0.02,\,0.02] MAPPO_RNN 0.140.14 [0.13, 0.15][0.13,\,0.15] 0.050.05 [0.04, 0.05][0.04,\,0.05] 0.030.03 [0.03, 0.03][0.03,\,0.03] 0.010.01 [0.00, 0.01][0.00,\,0.01] 0.050.05 [0.05, 0.06][0.05,\,0.06] Overcooked V1 asymm_advantages IPPO_F 0.770.77 [0.77, 0.77][0.77,\,0.77] 0.100.10 [0.09, 0.10][0.09,\,0.10] 0.080.08 [0.08, 0.09][0.08,\,0.09] 0.010.01 [0.01, 0.01][0.01,\,0.01] 0.080.08 [0.08, 0.08][0.08,\,0.08] IPPO_RNN 0.730.73 [0.71, 0.75][0.71,\,0.75] 0.190.19 [0.17, 0.22][0.17,\,0.22] 0.060.06 [0.03, 0.09][0.03,\,0.09] 0.010.01 [0.01, 0.01][0.01,\,0.01] 0.070.07 [0.05, 0.10][0.05,\,0.10] coord_ring IPPO_F 0.740.74 [0.74, 0.75][0.74,\,0.75] 0.170.17 [0.16, 0.18][0.16,\,0.18] 0.020.02 [0.02, 0.03][0.02,\,0.03] 0.010.01 [0.01, 0.01][0.01,\,0.01] 0.030.03 [0.03, 0.03][0.03,\,0.03] IPPO_RNN 0.740.74 [0.71, 0.76][0.71,\,0.76] 0.090.09 [0.07, 0.10][0.07,\,0.10] 0.030.03 [0.02, 0.04][0.02,\,0.04] 0.010.01 [0.01, 0.01][0.01,\,0.01] 0.030.03 [0.02, 0.04][0.02,\,0.04] counter_circuit IPPO_F 0.750.75 [0.74, 0.75][0.74,\,0.75] 0.140.14 [0.13, 0.15][0.13,\,0.15] 0.010.01 [0.01, 0.02][0.01,\,0.02] 0.010.01 [0.01, 0.01][0.01,\,0.01] 0.020.02 [0.02, 0.03][0.02,\,0.03] IPPO_RNN 0.650.65 [0.55, 0.73][0.55,\,0.73] 0.110.11 [0.07, 0.14][0.07,\,0.14] 0.030.03 [0.02, 0.04][0.02,\,0.04] 0.010.01 [0.01, 0.01][0.01,\,0.01] 0.030.03 [0.02, 0.05][0.02,\,0.05] cramped_room IPPO_F 0.730.73 [0.72, 0.74][0.72,\,0.74] 0.160.16 [0.15, 0.17][0.15,\,0.17] 0.030.03 [0.02, 0.03][0.02,\,0.03] 0.020.02 [0.01, 0.02][0.01,\,0.02] 0.040.04 [0.03, 0.04][0.03,\,0.04] IPPO_RNN 0.720.72 [0.71, 0.72][0.71,\,0.72] 0.080.08 [0.07, 0.09][0.07,\,0.09] 0.030.03 [0.02, 0.04][0.02,\,0.04] 0.010.01 [0.01, 0.02][0.01,\,0.02] 0.040.04 [0.02, 0.06][0.02,\,0.06] forced_coord IPPO_F 0.740.74 [0.74, 0.75][0.74,\,0.75] 0.150.15 [0.14, 0.16][0.14,\,0.16] 0.050.05 [0.04, 0.05][0.04,\,0.05] 0.020.02 [0.02, 0.02][0.02,\,0.02] 0.060.06 [0.05, 0.06][0.05,\,0.06] IPPO_RNN 0.730.73 [0.72, 0.74][0.72,\,0.74] 0.120.12 [0.11, 0.13][0.11,\,0.13] 0.010.01 [0.01, 0.01][0.01,\,0.01] 0.010.01 [0.01, 0.02][0.01,\,0.02] 0.010.01 [0.01, 0.02][0.01,\,0.02] Overcooked V2 asymm_advantages IPPO_F 0.630.63 [0.47, 0.75][0.47,\,0.75] 0.240.24 [0.22, 0.26][0.22,\,0.26] 0.120.12 [0.11, 0.14][0.11,\,0.14] 0.020.02 [0.01, 0.02][0.01,\,0.02] 0.130.13 [0.11, 0.14][0.11,\,0.14] IPPO_RNN 0.630.63 [0.47, 0.75][0.47,\,0.75] 0.280.28 [0.21, 0.35][0.21,\,0.35] 0.100.10 [0.05, 0.16][0.05,\,0.16] 0.010.01 [0.01, 0.01][0.01,\,0.01] 0.110.11 [0.05, 0.17][0.05,\,0.17] MAPPO_F 0.770.77 [0.77, 0.78][0.77,\,0.78] 0.240.24 [0.22, 0.25][0.22,\,0.25] 0.130.13 [0.11, 0.15][0.11,\,0.15] 0.020.02 [0.02, 0.02][0.02,\,0.02] 0.130.13 [0.12, 0.15][0.12,\,0.15] MAPPO_RNN 0.760.76 [0.75, 0.76][0.75,\,0.76] 0.430.43 [0.41, 0.46][0.41,\,0.46] 0.060.06 [0.05, 0.07][0.05,\,0.07] 0.010.01 [0.01, 0.01][0.01,\,0.01] 0.060.06 [0.05, 0.08][0.05,\,0.08] coord_ring IPPO_F 0.730.73 [0.72, 0.74][0.72,\,0.74] 0.190.19 [0.16, 0.22][0.16,\,0.22] 0.030.03 [0.03, 0.04][0.03,\,0.04] 0.010.01 [0.00, 0.01][0.00,\,0.01] 0.040.04 [0.03, 0.04][0.03,\,0.04] IPPO_RNN 0.650.65 [0.50, 0.75][0.50,\,0.75] 0.180.18 [0.12, 0.25][0.12,\,0.25] 0.020.02 [0.01, 0.03][0.01,\,0.03] 0.000.00 [0.00, 0.00][0.00,\,0.00] 0.030.03 [0.02, 0.05][0.02,\,0.05] MAPPO_F 0.740.74 [0.73, 0.75][0.73,\,0.75] 0.210.21 [0.19, 0.24][0.19,\,0.24] 0.030.03 [0.02, 0.03][0.02,\,0.03] 0.010.01 [0.00, 0.01][0.00,\,0.01] 0.040.04 [0.03, 0.05][0.03,\,0.05] MAPPO_RNN 0.740.74 [0.72, 0.75][0.72,\,0.75] 0.220.22 [0.20, 0.24][0.20,\,0.24] 0.020.02 [0.02, 0.03][0.02,\,0.03] 0.000.00 [0.00, 0.01][0.00,\,0.01] 0.030.03 [0.02, 0.03][0.02,\,0.03] counter_circuit IPPO_F 0.720.72 [0.70, 0.73][0.70,\,0.73] 0.190.19 [0.14, 0.23][0.14,\,0.23] 0.050.05 [0.04, 0.06][0.04,\,0.06] 0.010.01 [0.01, 0.01][0.01,\,0.01] 0.060.06 [0.05, 0.08][0.05,\,0.08] IPPO_RNN 0.680.68 [0.61, 0.73][0.61,\,0.73] 0.250.25 [0.19, 0.31][0.19,\,0.31] 0.040.04 [0.03, 0.07][0.03,\,0.07] 0.000.00 [0.00, 0.01][0.00,\,0.01] 0.060.06 [0.03, 0.09][0.03,\,0.09] MAPPO_F 0.700.70 [0.67, 0.73][0.67,\,0.73] 0.120.12 [0.08, 0.16][0.08,\,0.16] 0.040.04 [0.02, 0.05][0.02,\,0.05] 0.000.00 [0.00, 0.01][0.00,\,0.01] 0.040.04 [0.03, 0.06][0.03,\,0.06] MAPPO_RNN 0.690.69 [0.65, 0.72][0.65,\,0.72] 0.270.27 [0.24, 0.29][0.24,\,0.29] 0.080.08 [0.05, 0.10][0.05,\,0.10] 0.010.01 [0.01, 0.01][0.01,\,0.01] 0.090.09 [0.06, 0.12][0.06,\,0.12] cramped_room IPPO_F 0.720.72 [0.71, 0.73][0.71,\,0.73] 0.200.20 [0.18, 0.22][0.18,\,0.22] 0.030.03 [0.03, 0.04][0.03,\,0.04] 0.010.01 [0.01, 0.02][0.01,\,0.02] 0.040.04 [0.03, 0.05][0.03,\,0.05] IPPO_RNN 0.690.69 [0.65, 0.73][0.65,\,0.73] 0.190.19 [0.13, 0.25][0.13,\,0.25] 0.020.02 [0.01, 0.03][0.01,\,0.03] 0.010.01 [0.00, 0.01][0.00,\,0.01] 0.030.03 [0.02, 0.05][0.02,\,0.05] MAPPO_F 0.720.72 [0.71, 0.74][0.71,\,0.74] 0.190.19 [0.17, 0.21][0.17,\,0.21] 0.040.04 [0.03, 0.06][0.03,\,0.06] 0.010.01 [0.01, 0.02][0.01,\,0.02] 0.050.05 [0.04, 0.06][0.04,\,0.06] MAPPO_RNN 0.640.64 [0.50, 0.72][0.50,\,0.72] 0.240.24 [0.18, 0.29][0.18,\,0.29] 0.010.01 [0.01, 0.02][0.01,\,0.02] 0.010.01 [0.01, 0.01][0.01,\,0.01] 0.020.02 [0.01, 0.02][0.01,\,0.02] forced_coord IPPO_F 0.730.73 [0.72, 0.74][0.72,\,0.74] 0.200.20 [0.19, 0.21][0.19,\,0.21] 0.070.07 [0.06, 0.08][0.06,\,0.08] 0.010.01 [0.01, 0.02][0.01,\,0.02] 0.090.09 [0.07, 0.10][0.07,\,0.10] IPPO_RNN 0.430.43 [0.24, 0.61][0.24,\,0.61] 0.180.18 [0.10, 0.26][0.10,\,0.26] 0.020.02 [0.01, 0.03][0.01,\,0.03] 0.010.01 [0.00, 0.01][0.00,\,0.01] 0.040.04 [0.02, 0.06][0.02,\,0.06] MAPPO_F 0.730.73 [0.72, 0.75][0.72,\,0.75] 0.200.20 [0.17, 0.22][0.17,\,0.22] 0.080.08 [0.07, 0.08][0.07,\,0.08] 0.020.02 [0.01, 0.02][0.01,\,0.02] 0.080.08 [0.08, 0.09][0.08,\,0.09] MAPPO_RNN 0.710.71 [0.65, 0.75][0.65,\,0.75] 0.360.36 [0.33, 0.39][0.33,\,0.39] 0.010.01 [0.00, 0.01][0.00,\,0.01] 0.010.01 [0.01, 0.02][0.01,\,0.02] 0.010.01 [0.00, 0.01][0.00,\,0.01] test_time_simple IPPO_F 0.730.73 [0.71, 0.74][0.71,\,0.74] 0.240.24 [0.19, 0.27][0.19,\,0.27] 0.090.09 [0.07, 0.11][0.07,\,0.11] 0.010.01 [0.01, 0.01][0.01,\,0.01] 0.100.10 [0.08, 0.11][0.08,\,0.11] IPPO_RNN 0.630.63 [0.57, 0.69][0.57,\,0.69] 0.240.24 [0.19, 0.28][0.19,\,0.28] 0.080.08 [0.06, 0.11][0.06,\,0.11] 0.010.01 [0.00, 0.01][0.00,\,0.01] 0.090.09 [0.07, 0.11][0.07,\,0.11] MAPPO_F 0.720.72 [0.71, 0.73][0.71,\,0.73] 0.260.26 [0.25, 0.27][0.25,\,0.27] 0.100.10 [0.09, 0.11][0.09,\,0.11] 0.010.01 [0.01, 0.01][0.01,\,0.01] 0.120.12 [0.10, 0.13][0.10,\,0.13] MAPPO_RNN 0.610.61 [0.59, 0.64][0.59,\,0.64] 0.280.28 [0.25, 0.31][0.25,\,0.31] 0.090.09 [0.05, 0.14][0.05,\,0.14] 0.010.01 [0.01, 0.01][0.01,\,0.01] 0.130.13 [0.09, 0.18][0.09,\,0.18] test_time_wide IPPO_F 0.700.70 [0.67, 0.72][0.67,\,0.72] 0.190.19 [0.18, 0.21][0.18,\,0.21] 0.100.10 [0.09, 0.10][0.09,\,0.10] 0.010.01 [0.01, 0.01][0.01,\,0.01] 0.110.11 [0.10, 0.12][0.10,\,0.12] IPPO_RNN 0.580.58 [0.51, 0.64][0.51,\,0.64] 0.210.21 [0.15, 0.28][0.15,\,0.28] 0.030.03 [0.01, 0.04][0.01,\,0.04] 0.000.00 [0.00, 0.01][0.00,\,0.01] 0.050.05 [0.03, 0.07][0.03,\,0.07] MAPPO_F 0.730.73 [0.72, 0.73][0.72,\,0.73] 0.210.21 [0.19, 0.23][0.19,\,0.23] 0.100.10 [0.09, 0.11][0.09,\,0.11] 0.010.01 [0.01, 0.01][0.01,\,0.01] 0.110.11 [0.10, 0.13][0.10,\,0.13] MAPPO_RNN 0.640.64 [0.62, 0.66][0.62,\,0.66] 0.280.28 [0.25, 0.30][0.25,\,0.30] 0.100.10 [0.08, 0.12][0.08,\,0.12] 0.010.01 [0.01, 0.01][0.01,\,0.01] 0.130.13 [0.10, 0.16][0.10,\,0.16] demo_cook_simple IPPO_F 0.710.71 [0.70, 0.72][0.70,\,0.72] 0.220.22 [0.21, 0.23][0.21,\,0.23] 0.100.10 [0.09, 0.11][0.09,\,0.11] 0.010.01 [0.01, 0.01][0.01,\,0.01] 0.110.11 [0.10, 0.11][0.10,\,0.11] IPPO_RNN 0.520.52 [0.48, 0.55][0.48,\,0.55] 0.210.21 [0.18, 0.24][0.18,\,0.24] 0.020.02 [0.02, 0.03][0.02,\,0.03] 0.000.00 [0.00, 0.00][0.00,\,0.00] 0.030.03 [0.02, 0.04][0.02,\,0.04] MAPPO_F 0.700.70 [0.67, 0.72][0.67,\,0.72] 0.210.21 [0.18, 0.24][0.18,\,0.24] 0.100.10 [0.08, 0.12][0.08,\,0.12] 0.010.01 [0.01, 0.01][0.01,\,0.01] 0.100.10 [0.08, 0.12][0.08,\,0.12] MAPPO_RNN 0.570.57 [0.56, 0.59][0.56,\,0.59] 0.230.23 [0.22, 0.25][0.22,\,0.25] 0.110.11 [0.09, 0.14][0.09,\,0.14] 0.010.01 [0.01, 0.01][0.01,\,0.01] 0.130.13 [0.11, 0.15][0.11,\,0.15] demo_cook_wide IPPO_F 0.720.72 [0.71, 0.73][0.71,\,0.73] 0.180.18 [0.16, 0.19][0.16,\,0.19] 0.090.09 [0.08, 0.10][0.08,\,0.10] 0.010.01 [0.01, 0.01][0.01,\,0.01] 0.090.09 [0.08, 0.10][0.08,\,0.10] IPPO_RNN 0.560.56 [0.51, 0.62][0.51,\,0.62] 0.180.18 [0.11, 0.24][0.11,\,0.24] 0.060.06 [0.03, 0.08][0.03,\,0.08] 0.000.00 [0.00, 0.00][0.00,\,0.00] 0.060.06 [0.04, 0.08][0.04,\,0.08] MAPPO_F 0.720.72 [0.71, 0.72][0.71,\,0.72] 0.180.18 [0.17, 0.20][0.17,\,0.20] 0.090.09 [0.08, 0.10][0.08,\,0.10] 0.010.01 [0.01, 0.01][0.01,\,0.01] 0.090.09 [0.09, 0.10][0.09,\,0.10] MAPPO_RNN 0.610.61 [0.57, 0.65][0.57,\,0.65] 0.230.23 [0.21, 0.26][0.21,\,0.26] 0.120.12 [0.10, 0.15][0.10,\,0.15] 0.010.01 [0.01, 0.01][0.01,\,0.01] 0.140.14 [0.11, 0.16][0.11,\,0.16] grounded_coord_ring IPPO_F 0.710.71 [0.70, 0.73][0.70,\,0.73] 0.230.23 [0.22, 0.25][0.22,\,0.25] 0.070.07 [0.06, 0.08][0.06,\,0.08] 0.000.00 [0.00, 0.01][0.00,\,0.01] 0.080.08 [0.08, 0.09][0.08,\,0.09] IPPO_RNN 0.470.47 [0.33, 0.59][0.33,\,0.59] 0.120.12 [0.09, 0.15][0.09,\,0.15] 0.110.11 [0.06, 0.16][0.06,\,0.16] 0.000.00 [0.00, 0.01][0.00,\,0.01] 0.130.13 [0.08, 0.19][0.08,\,0.19] MAPPO_F 0.700.70 [0.66, 0.72][0.66,\,0.72] 0.240.24 [0.23, 0.25][0.23,\,0.25] 0.080.08 [0.07, 0.08][0.07,\,0.08] 0.000.00 [0.00, 0.01][0.00,\,0.01] 0.090.09 [0.08, 0.10][0.08,\,0.10] MAPPO_RNN 0.620.62 [0.60, 0.64][0.60,\,0.64] 0.290.29 [0.27, 0.30][0.27,\,0.30] 0.110.11 [0.09, 0.13][0.09,\,0.13] 0.000.00 [0.00, 0.01][0.00,\,0.01] 0.150.15 [0.13, 0.17][0.13,\,0.17] grounded_coord_simple IPPO_F 0.720.72 [0.72, 0.73][0.72,\,0.73] 0.260.26 [0.25, 0.27][0.25,\,0.27] 0.110.11 [0.10, 0.12][0.10,\,0.12] 0.010.01 [0.01, 0.01][0.01,\,0.01] 0.120.12 [0.10, 0.13][0.10,\,0.13] IPPO_RNN 0.620.62 [0.52, 0.70][0.52,\,0.70] 0.240.24 [0.18, 0.28][0.18,\,0.28] 0.070.07 [0.05, 0.09][0.05,\,0.09] 0.010.01 [0.00, 0.01][0.00,\,0.01] 0.080.08 [0.05, 0.11][0.05,\,0.11] MAPPO_F 0.720.72 [0.71, 0.73][0.71,\,0.73] 0.250.25 [0.23, 0.27][0.23,\,0.27] 0.100.10 [0.08, 0.11][0.08,\,0.11] 0.010.01 [0.01, 0.01][0.01,\,0.01] 0.110.11 [0.09, 0.12][0.09,\,0.12] MAPPO_RNN 0.650.65 [0.62, 0.68][0.62,\,0.68] 0.290.29 [0.27, 0.31][0.27,\,0.31] 0.110.11 [0.09, 0.14][0.09,\,0.14] 0.010.01 [0.01, 0.01][0.01,\,0.01] 0.160.16 [0.13, 0.18][0.13,\,0.18]