Paper deep dive
Grounded Joint-Attention Other-Play for Zero-Shot Coordination
Giulia Benintendi, Constantin Ruhdorfer, Fabian Kögel, Andreas Bulling
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Joint attention - the human ability to share a common visual or cognitive focus with others - enables a meeting of minds that lets us coordinate even with unfamiliar partners. In this work we investigate whether equipping AI agents with a similar mechanism can enable such zero-shot coordination. We introduce Mutual Attention for zero-shot TEaming (MATE): a novel multi-agent reinforcement learning method inspired by human joint attention. MATE encourages agents to coordinate their actions by aligning their visual attention on scene-salient objects during the interaction rather than relying on arbitrary partner-dependent conventions established during training. Unlike symmetry-breaking approaches that merely prevent brittle conventions from emerging, MATE actively promotes coordination through an environment-grounded signal that is naturally shared across partners. We evaluate MATE on three benchmarks: our Card Alignment Game, designed to isolate brittle convention formation, and the more challenging Level-Based Foraging and OvercookedV2 benchmarks. Our experiments consistently show that a joint-attention-inspired signal improves coordination with unknown partners, underlining MATE's potential as a general coordination mechanism that complements and surpasses symmetry-breaking approaches.
Tags
Links
- Source: https://arxiv.org/abs/2610.06025v1
- Canonical: https://arxiv.org/abs/2610.06025v1
Trouble viewing inline? Open PDF directly →
Full Text
51,264 characters extracted from source content.
Expand or collapse full text
Grounded Joint-Attention Other-Play for Zero-Shot Coordination Giulia Benintendi †thanks: Work done while interning at the University of Stuttgart. Constantin Ruhdorfer Fabian Kögel Andreas Bulling Abstract Joint attention – the human ability to share a common visual or cognitive focus with others – enables a meeting of minds that lets us coordinate even with unfamiliar partners. In this work we investigate whether equipping AI agents with a similar mechanism can enable such zero-shot coordination. We introduce Mutual Attention for zero-shot TEaming (MATE): a novel multi-agent reinforcement learning method inspired by human joint attention. MATE encourages agents to coordinate their actions by aligning their visual attention on scene-salient objects during the interaction rather than relying on arbitrary partner-dependent conventions established during training. Unlike symmetry-breaking approaches that merely prevent brittle conventions from emerging, MATE actively promotes coordination through an environment-grounded signal that is naturally shared across partners. We evaluate MATE on three benchmarks: our Card Alignment Game, designed to isolate brittle convention formation, and the more challenging Level-Based Foraging and OvercookedV2 benchmarks. Our experiments consistently show that a joint-attention-inspired signal improves coordination with unknown partners, underlining MATE’s potential as a general coordination mechanism that complements and surpasses symmetry-breaking approaches. 1University of Zurich 2University of Stuttgart giulia.benintendi@uzh.ch 1 Introduction Figure 1: Zero-Shot Coordination using joint attention. Alice and Bob , trained independently, must pick the same card without explicit communication. Self-play agents commit to arbitrary conventions, other-play agents have no signal at all to coordinate on, while MATE agents read each other’s attention and converge on a shared choice. One of the most common approaches for training agents in multi-agent reinforcement learning (MARL) is self-play (SP) (Tesauro 1995), in which agents learn by interacting with a copy of themselves. Despite yielding strong results in competitive (Silver et al. 2016; Berner et al. 2019; Vinyals et al. 2019; Baker et al. 2020) and cooperative settings (de Witt et al. 2019; Hu and Foerster 2020; Yu et al. 2022), agents trained via SP struggle to generalise to novel partners (Carroll et al. 2019; Hu et al. 2020; Strouse et al. 2021). This motivates research on zero-shot coordination (ZSC): coordinating with unseen partners without prior coordination (Stone et al. 2010; Hu et al. 2020). ZSC performance is usually evaluated in cross-play (XP) by pairing agents from independent training runs. Performance degrades substantially (Hu et al. 2020) when agents approach the same task through incompatible arbitrary conventions that break environment symmetries during training (Foerster et al. 2019). Consider two players tasked with picking the same card out of five without explicit communication (see Figure 1). Within a training run, two agents may converge on a shared convention, such as “always pick green”, while an agent from a different run may have learned to “always pick purple”. When paired, neither agent knows which convention the other learned, and they fail to coordinate. Solving ZSC thus requires coordination signals that a novel partner can independently recover, rather than conventions tied to a training run. To solve this issue, humans use joint attention as a social signal to dynamically resolve ambiguity and establish the relevant object of joint action (Tomasello 2014; Carpenter and Call 2013). In the card example, humans would look at their intended card and track their partner’s gaze to successively converge on a joint choice. This succeeds even between strangers because human joint attention generalises across partners, providing a cue that is produced within the ongoing interaction and that can be interpreted in context. We hypothesise that joint attention could play a similar role for AI agents and ask: Can joint attention serve as a coordination signal for independently trained partners at test time? We propose Mutual Attention for zero-shot TEaming (MATE), a novel multi-agent reinforcement learning method inspired by human visual joint attention. Each agent receives its partner’s attention as a coordination signal and conditions its policy on it, making attention interactive: Agents do not merely learn to align their attention on the same object, but also to use the partner’s visual focus to guide their own attention and actions as the interaction unfolds (see Figure 1). We build MATE on top of other-play (OP) (Hu et al. 2020) that targets brittle convention formation by exploiting known environment symmetries: Rather than coordinating with an identical copy, an agent is trained with a randomly symmetry-permuted version of itself, preventing convergence on arbitrary conventions. In the card example in Figure 1, OP can prevent run-specific choices, such as always selecting purple or green, but it does not determine which card agents should choose: While OP removes one source of failure, it does not provide any within-episode signal for selecting one among several equally plausible cards. MATE supplies this missing signal through the partner’s visual attention, allowing agents to coordinate where OP alone cannot. We evaluate MATE first in our Card Alignment Game introduced above, which isolates this coordination task in its simplest form, and then in two established coordination benchmarks: Level-Based Foraging (Albrecht and Ramamoorthy 2013) and OvercookedV2 (Gessler et al. 2025). Our results suggest that successful ZSC may require not only preventing incompatible arbitrary conventions, but also enabling agents to negotiate coordination during execution by establishing a shared attentional focus. This positions MATE as a method for fostering generalisable coordination strategies that transfer to novel partners. Our contributions are: 1. We introduce MATE, a joint attention training mechanism built on top of OP, in which each agent observes its partner’s visual attention and learns to act on it. 2. We identify coordination problems in which enforcing symmetry-equivariant policies alone is insufficient, and introduce the Card Alignment Game as a minimal testbed that isolates this limitation. 3. We show that MATE outperforms competitive baselines in the Card Alignment Game as well as Level-Based Foraging and OvercookedV2, together with qualitative evidence that coordination stems from mutual attention rather than brittle conventions. 2 Related Work 2.1 Zero-Shot Coordination ZSC investigates the ability of agents to coordinate with independently trained partners at test time, without co-adaptation. This is challenging when a task admits multiple equivalent but mutually incompatible optimal joint policies. We focus on this setting rather than broader ad-hoc teamwork (AHT), where agents collaborate with previously unknown partners whose behaviours may initially be unknown (Stone et al. 2010), as AHT is commonly treated as a natural next step once ZSC performance improves (Dizdarević et al. 2025; Ruhdorfer et al. 2026). Existing ZSC methods primarily address this coordination problem by discouraging arbitrary symmetry breaking during training. OP (Hu et al. 2020) independently transforms each agent’s Dec-POMDP using sampled environment relabellings, preventing the arbitrary conventions that SP would otherwise form around environment symmetries. However, its performance is bounded by the symmetries the environment admits. Off-Belief Learning (Hu et al. 2021) on the other hand is an algorithm that prevents arbitrary conventions and improves ZSC by taking as input a reference policy π0 _0 and optimizes a new unique policy π1 _1 under the assumption that previous actions were taken by π0 _0 while future actions will be taken by π1 _1 using a belief model. More recently, Forkel et al. (2026) showed that high entropy regularisation promotes symmetry-equivariant and XP-compatible policies, with High-Entropy (HE) IPPO, a standard IPPO with an unusually large entropy coefficient, achieving state-of-the-art inter-seed XP score in Hanabi. These approaches share the premise that coordination can be recovered by suppressing arbitrary conventions. This is not always sufficient. When several possible referents remain symmetric within the current episode, removing arbitrary conventions does not by itself determine what the agents should coordinate around. We instead equip agents with a coordination signal grounded in the episode itself, namely shared attention, that remains informative even when no symmetry is left to exploit. 2.2 Joint Attention as a Coordination Signal Joint attention is the shared and mutually recognised focus of two individuals on the same object or event. It is a foundational mechanism in human social cognition (Tomasello 2014; Mundy and Jarrold 2010; Carpenter and Call 2013) and has been studied for coordination in human-robot interaction (Staudte and Crocker 2011; Anzalone et al. 2019; García-Martínez et al. 2024). In MARL, Lee et al. (2021) equip agents with spatial attention heads and an intrinsic reward that penalises divergence between their attention maps, encouraging them to attend to the same elements of a shared observation; this improves SP coordination in gridworld tasks. However, because their evaluation is limited to SP, the shared-attention incentive may act as an additional convention-forming channel: co-trained agents can simply learn to align their attention on an arbitrary, possibly task-irrelevant location to initiate a coordinated behaviour, without producing a signal that transfers to independently trained partners. To leverage communication, prior work in MARL learns protocols for exchanging task-relevant information (Foerster et al. 2016; Sukhbaatar, Szlam, and Fergus 2016; Mordatch and Abbeel 2018), but these often fail to transfer to independently trained partners (Bullard et al. 2020; Tucker et al. 2021). MATE instead uses spatial attention as an environment-grounded communication channel. Without a pre-established convention, two factors matter beyond aligned attention: agents need to be mutually aware that they share a focus (Carpenter and Call 2013) and the focus needs to be on a relevant object (Csibra and Gergely 2009). MATE addresses both limitations by enforcing attention to be not only aligned but also focused on objects, yielding an observable signal for establishing an episode-specific target with novel partners. 3 Preliminaries 3.1 Decentralised POMDPs We model the cooperative setting as a decentralised partially observable Markov decision process (Dec-POMDP) (Oliehoek and Amato 2016), a tuple (,n,ii=1n,ii=1n,,ℛ,Uii=1n,T,γ,b0) (S,n,\A^i\_i=1^n,\O^i\_i=1^n,T,R,\U^i\_i=1^n,T,γ,b_0 ), with finite state space S, n agents, local action and observation spaces iA^i and iO^i, transition kernel (st+1∣st,at)T(s_t+1 s_t,a_t), shared reward ℛ(st+1,at)R(s_t+1,a_t), observation functions oti=Ui(st)o^i_t=U^i(s_t), horizon T, reward discount factor γ, and initial state distribution b0b_0. Each agent i acts on its local action-observation history τti=(o0i,a0i,…,at−1i,oti)τ^i_t=(o^i_0,a^i_0,…,a^i_t-1,o^i_t) through a local policy πi(ati∣τti)π^i(a^i_t τ^i_t). The joint policy then factorises as π(at∣τt)=∏i=1nπi(ati∣τti).π(a_t _t)= _i=1^nπ^i(a^i_t τ^i_t). (1) The SP objective is the expected return JSP(π)=τT∼π[∑t=0T−1γtℛ(st+1,at)].J_SP(π)=E_ _T π [ _t=0^T-1γ^t\,R(s_t+1,a_t) ]. (2) 3.2 Zero-Shot Coordination A Dec-POMDP may admit multiple joint policies with high SP return that are nevertheless mutually incompatible (Hu et al. 2020). We thus measure the ZSC score through XP, pairing agents trained in different runs. Given a Dec-POMDP with n agents and n independently trained joint policies π1,…,πn _1,…, _n, we follow prior work (Hu et al. 2020; Forkel et al. 2026) and define the XP score between them as JXP(π1,…,πn):=ϕ∼Perm(n)J(πϕ(1)1,…,πϕ(n)n).J_XP( _1,…, _n):=E_φ (n)J ( _φ(1)^1,…, _φ(n)^n ). (3) where Perm(n)Perm(n) is the set of permutations of 1,…,n\1,…,n\, and πji _j^i denotes the local policy for agent slot i from joint policy πj _j. As in this work we consider two-player Dec-POMDPs, this reduces to JXP(π1,π2)=12[J(π11,π22)+J(π21,π12)]J_XP( _1, _2)= 12 [J( _1^1, _2^2)+J( _2^1, _1^2) ]. The goal of ZSC is to find training algorithms that, when run independently, produce compatible joint policies. Following prior work, we focus on the inter-seed XP setting, i.e. independent training runs vary only in their random seed. 3.3 Symmetries in Dec-POMDPs A Dec-POMDP symmetry (Hu et al. 2020; Forkel et al. 2026) is a tuple of bijections ϕ=(ϕ,(ϕi)i=1n,(ϕi)i=1n)φ= ( _S,( _A^i)_i=1^n,( _O^i)_i=1^n ) (4) over S, iA^i, and iO^i while leaving T, ℛR, and Uii=1n\U^i\_i=1^n invariant. We write Φ for the set of all such symmetries. A symmetry acts on a joint policy by relabelling its inputs and outputs, ϕ(πi)(ai∣τi):=πi(ϕ−1(ai)∣ϕ−1(τi)),φ(π^i)(a^i τ^i):=π^i (φ^-1(a^i) φ^-1(τ^i) ), (5) where ϕ−1(τi)φ^-1(τ^i) denotes the component-wise relabelling of the actions and observations in the action-observation history. A joint policy π is symmetry equivariant if ϕ(π)=πφ(π)=π for all ϕ∈Φφ∈ . 3.4 Other-play OP addresses brittle conventions by optimising SP under independently sampled Dec-POMDP symmetries (Hu et al. 2020). For each agent i, a symmetry ϕi∼Φφ^i is drawn per episode, and OP maximises JOP(π)=(ϕ1,…,ϕn)∼Φn[JSP(ϕ1(π1),…,ϕn(πn))]J_OP(π)=E_(φ^1,…,φ^n) ^n [J_SP (φ^1(π^1),…,φ^n(π^n) ) ] (6) This promotes robustness to relabellings and discourages arbitrary symmetry breaking, yet Φ must be known a priori. 3.5 Joint Attention for Social Learning Lee et al. (2021) introduced a computational analogue of joint attention that improved JSPJ_SP and sample efficiency across several gridworld tasks. For each agent k, a single h×wh× w spatial attention map AtkA_t^k is obtained by averaging the visual attention maps of the m heads. Joint attention is modelled as the Jensen-Shannon divergence (JSD) between each pair of agents’ attention maps, using an auxiliary reward term: rtJA=−∑j=1n∑i=1nJSD(Ati∥Atj).r_t^JA=- _j=1^n _i=1^nJSD\! (A_t^i\,\|\,A_t^j ). (7) Each agent’s objective combines the environment reward with the shared joint-attention intrinsic motivation: J(πi)=π[∑t=0∞γt(rt+1i+βrtJA)|s0],J(π^i)=E_π\! [\, _t=0^∞γ^t (r_t+1^i+β\,r_t^JA )\; |\;s_0 ], (8) where β controls the weight of the joint-attention objective. 4 Mutual Attention for Zero-Shot Teaming We extend OP and the spatial-attention architecture of Lee et al. (2021) by additionally grounding the attention in the environment. We call this method MATE, which equips OP-trained IPPO agents (de Witt et al. 2020; Schulman et al. 2017) with a model of joint attention. Each agent shares its focus with its partner and learns to steer its own attention toward the partner’s future focus and on task-relevant objects. 4.1 Sharing the partner’s attention At every step, agent i receives its partner’s previous attention map At−1jA^j_t-1 as a channel appended to its image observation. Because OP expresses each agent’s observation in a different permuted frame, we re-express the partner’s map in i’s frame by undoing j’s symmetry and applying i’s: A~t−1j:=(ϕAi∘(ϕAj)−1)At−1j, A_t-1^j:= ( _A^i ( _A^j)^-1 )A_t-1^j, (9) where ϕAj φ^j_A denotes the action of ϕj φ^j on spatial attention maps. 4.2 Grounding attention in task-relevant objects Parallel attention is insufficient for joint attention, as both agents must intentionally share a focus on a relevant target to support coordinated action (Carpenter and Call 2013; Csibra and Gergely 2009). Alignment on an irrelevant background patch contributes nothing to the task unless it becomes another arbitrary convention, whereas attention to task-relevant objects provides a meaningful coordination signal that novel partners can interpret within the episode. We therefore add an auxiliary loss that grounds attention and encourages convergence on task-relevant entities, aligning attention at the object level. Because each agent observes the partner’s previous attention map, this does not passively align attention maps; rather, it trains agents to intentionally condition their own focus on the partner’s. We assume a training-time object detector OBJOBJ that returns the task-relevant objects ℰt=OBJ(ot)E_t=OBJ(o_t) at step t, each object e covering a region ℛ(e)R(e) of the attention grid. Mapping agent i’s attention back to the shared frame, A¯ti:=(ϕAi)−1Ati A_t^i:=( _A^i)^-1A_t^i, and summing it over each object’s cells, leads to a distribution ptip_t^i over objects, comparable across agents. We train ptip_t^i towards the partner’s discounted future focus qtjq_t^j, which weights each object by a discount factor ηδt(e)η _t(e), where δt(e) _t(e) is the number of steps until partner j next attends object e. Agent i’s attention is matched to this target by cross-entropy, summed over both agents and the horizon, ℒMATE=−12T∑i=12∑t=0T−1∑e∈ℰtqtj(e)logpti(e).L_MATE=- 12T _i=1^2 _t=0^T-1 _e _tq_t^j(e) p_t^i(e). (10) The full objective adds this to the IPPO loss, ℒ=ℒIPPO+λMATEℒMATE.L=L_IPPO+ _MATE\,L_MATE. (11) Algorithm 1 presents pseudocode for the two-player case. Algorithm 1 MATE Require: policy πθ _θ, symmetry group Φ , detector OBJOBJ, discount η∈[0,1)η∈[0,1), weight λMATE _MATE 1: Sample OP symmetries ϕ1,ϕ2∼Φφ^1,φ^2 2: Initialise partner maps A−11,A−12A_-1^1,A_-1^2 to uniform 3: for t=0,…,T−1t=0,…,T-1 do 4: for agent i∈1,2i∈\1,2\, partner j≠ij≠ i do 5: Reframe partner map A~t−1j=(ϕAi∘(ϕAj)−1)At−1j A_t-1^j=( _A^i ( _A^j)^-1)\,A_t-1^j 6: Sample action and attention ati,Ati∼πθ(ϕi(oti),A~t−1j)a_t^i,A_t^i _θ(φ^i(o_t^i), A_t-1^j) 7: end for 8: Step environment with at=(at1,at2)a_t=(a_t^1,a_t^2). 9: end for 10: Unmap to unpermuted frame A¯ti=(ϕAi)−1Ati A_t^i=( _A^i)^-1A_t^i 11: Sum onto objects pti(e)∝∑u∈ℛ(e)A¯ti(u)p_t^i(e) _u (e) A_t^i(u) 12: for agent i, partner j≠ij≠ i do 13: Find partner’s focus mtj=argmaxeptj(e)m_t^j= _ep_t^j(e) 14: Build target qtj←ηqt+1jq_t^j←η\,q_t+1^j, qtj(mtj)←1q_t^j(m_t^j)← 1, normalise 15: end for 16: Update θ by PPO on ℒ=ℒIPPO+λMATEℒMATEL=L_IPPO+ _MATE\,L_MATE. 17: return θ 5 Experiments We first test MATE in the Card Alignment Game, a toy environment that we propose to validate the core intuition behind our method in isolation. We then evaluate MATE in two more complex domains, Level-Based Foraging (LBF) (Albrecht and Ramamoorthy 2013) and OvercookedV2 (Gessler et al. 2025), which require agents to agree on a shared target and infer coordination from their partner’s actions. 5.1 Card Alignment Game Figure 2: Sample episode in the Card Alignment Game. Alice and Bob must select the same card in the unpermuted layout (left). To prevent position or colour conventions, they observe differently permuted frames. Finally, both agents choose independently. The Card Alignment Game (see Figure 2) is a cooperative two-player game in which agents must independently pick the same card out of five. Each episode contains seven deliberation steps, during which agents point to cards in their own local frame, followed by one private decision step. At that step, the pair receives R=1R=1 if both agents pick the same card and R=0R=0 otherwise. The game is kept simple to isolate whether joint attention alone can provide a sufficiently strong signal to enable coordination. Symmetries The game admits symmetries over card colours and positions, which SP agents can exploit to form arbitrary conventions, such as always selecting “red” or “left”. To avoid this, we employ OP by transforming each agent’s observation into oi=ϕi(o)o^i=φ^i(o) where ϕi=ϕcoli∘ϕposiφ^i=φ^i_col φ^i_pos is the composition of random individual position and colour permutations. Coordination must then be recovered within the episode, rather than through colour- or position-based conventions. In the example of Figure 2, Bob sees the unpermuted layout transformed as follows: (i) the position permutation is ϕposi=(2,3,1,4,5)φ^i_pos=(2,3,1,4,5), and (i) colour relabelling ϕicol:↦,↦,↦,↦,↦. gatheredφ^i_col: -1.07639pt [height=9.04166pt,keepaspectratio]assets/card_game/cards/red_card.png\! \! -1.07639pt [height=9.04166pt,keepaspectratio]assets/card_game/cards/purple_card.png,\, -1.07639pt [height=9.04166pt,keepaspectratio]assets/card_game/cards/blue_card.png\! \! -1.07639pt [height=9.04166pt,keepaspectratio]assets/card_game/cards/blue_card.png,\, -1.07639pt [height=9.04166pt,keepaspectratio]assets/card_game/cards/green_card.png\! \! -1.07639pt [height=9.04166pt,keepaspectratio]assets/card_game/cards/yellow_card.png, -1.07639pt [height=9.04166pt,keepaspectratio]assets/card_game/cards/yellow_card.png\! \! -1.07639pt [height=9.04166pt,keepaspectratio]assets/card_game/cards/green_card.png,\, -1.07639pt [height=9.04166pt,keepaspectratio]assets/card_game/cards/purple_card.png\! \! -1.07639pt [height=9.04166pt,keepaspectratio]assets/card_game/cards/red_card.png. gathered Therefore, Bob ’s local observation is oi=ϕi(o)=(ϕcoli∘ϕposi)(o)=(,,,,).o^i=φ^i(o)=(φ^i_col φ^i_pos)(o)=( -1.07639pt [height=9.04166pt,keepaspectratio]assets/card_game/cards/blue_card.png, -1.07639pt [height=9.04166pt,keepaspectratio]assets/card_game/cards/yellow_card.png, -1.07639pt [height=9.04166pt,keepaspectratio]assets/card_game/cards/purple_card.png, -1.07639pt [height=9.04166pt,keepaspectratio]assets/card_game/cards/green_card.png, -1.07639pt [height=9.04166pt,keepaspectratio]assets/card_game/cards/red_card.png). Observation and action space Each agent observes its permuted frame and a scalar indicating whether the episode is in deliberation or at the final decision step. The action space contains five actions, each corresponding to one of the cards: during deliberation, an action points to a card in the agent’s local frame; at the final step, it privately selects that card. The cards form the object set for the attention loss, with one ℛ(e)R(e) per card. (a) Card Alignment Game (b) LBF (c) OvercookedV2 Figure 3: SP (hatched) and XP (solid) scores for the Card Alignment Game (a), LBF (b) and OvercookedV2 (c). Error bars show SEM. MATE achieves the highest XP across all three environments. 5.2 Level-Based Foraging LBF (Albrecht and Ramamoorthy 2013) is a cooperative grid-world task in which agents navigate to apples and collect them by executing a load action. We use the forced-coordination setting, where each apple can be collected only if both agents load it jointly. Experiments are run on a fully observable 12×1212× 12 grid with two agents and eight apples. We shorten the episode horizon from 100100 to 5050 steps to increase coordination pressure. Agents receive image observations to test our method in conditions that more naturally resemble real-life deployment scenarios. Under OP, each observation is independently transformed by a random symmetry of the grid, using horizontal and vertical reflections. For the object-level attention target, apples are treated as task-relevant objects, with one ℛ(e)R(e) for each apple. 5.3 OvercookedV2 OvercookedV2 (Gessler et al. 2025) introduces partial observability and asymmetric information into Overcooked, one of the most widely used ZSC benchmarks. We evaluate on Demo Cook Simple, where only one agent observes the recipe and the other must infer the target dish from its partner’s behaviour, particularly the ingredients it carries and places in the pot. The exploration of additional layouts is left to future work. This setting makes it a natural test of whether joint attention can support implicit communication through partner behaviour. We extend the JaxMARL (Rutherford et al. 2024) environment with image observations and, following Gessler et al. (2025), apply OP by independently permuting the two recipe ingredients in each agent’s observation. While Gessler et al. (2025) embed each local view in a zero-padded global grid, we centre the observation on one agent and crop it to the view radius to reduce dependence on fixed layout coordinates and improve generalisation (Ye et al. 2020). Partner attention is shared only within the receiver’s visible region, preventing leakage of private recipe information yet making OvercookedV2 particularly challenging for MATE, as attention can provide a coordination signal only intermittently. Task-relevant kitchen objects, including carried ingredients, form the object set used by the attention loss. 6 Results We evaluate whether MATE improves ZSC by comparing SP and XP return (Hu et al. 2020), with XP computed from disjoint pairs of independently trained policies following Forkel et al. (2026). Across all environments we compare IPPO, OP, HE IPPO (Forkel et al. 2026) and MATE, additionally including the joint-attention baseline of Lee et al. (2021) in the Card Alignment Game. Results are averaged over 48 seeds in the Card Alignment Game and 12 in LBF and OvercookedV2. OP and MATE are trained under the same permutation for a fair comparison; no permutations are applied during evaluation. All methods are evaluated at the checkpoint with the highest training return. For HE IPPO, we always report the SP and XP scores at the entropy coefficient achieving the highest XP return in the sweep shown in Figure 7. 6.1 Card Alignment Game In the Card Alignment Game (Figure 3(a)), IPPO solves SP but falls to chance in XP, showing that independently trained agents learn incompatible conventions. OP rules out brittle conventions, but without a coordination signal, it remains at chance in both SP and XP. HE IPPO also fails: none of the tested entropy coefficients yields XP above chance, and large coefficients collapse non-greedy SP performance (Figure 7). The Lee et al. (2021) joint-attention objective behaves similarly: varying β over three orders of magnitude leaves the outcome unchanged, with perfect SP but chance XP. MATE instead obtains high return in both settings (0.872±0.0080.872± 0.008 SP, 0.823±0.0100.823± 0.010 XP), moving substantially above chance while keeping the SP-XP gap small. Figure 4: Card Alignment Game attention maps. (a) and (b) show per-card attention in the unpermuted frame. MATE progressively aligns attention between agents and selects the same card, whereas OP maintains divergent attention and chooses different cards. (c) shows card-attention JSD over time; lower values indicate stronger alignment. Figure 4 shows the corresponding attention patterns. MATE agents progressively align attention on the same card and select it, yielding a strategy that transfers to independently trained partners. Under OP alone, attention remains agent-specific and final choices do not align. MATE Ablation Study We analyse how each component of MATE contributes by individually removing OP (w/o OP), the partner-attention input (w/o feed), and the object-level auxiliary loss (w/o aux); see Figure 3(a). MATE w/o OP reintroduces brittle conventions, resulting in perfect SP but chance-level XP, while w/o aux at chance level demonstrates that joint attention is insufficient without grounding: agents still exchange attention, but an ungrounded signal carries no meaning a novel partner can recover. MATE w/o feed performs equivalently to OP. All three components, symmetry-robust training, interaction-time attention sharing, and object-level grounding, are thus necessary. 6.2 Level-Based Foraging In LBF (Figure 3(b)), IPPO exhibits a clear SP–XP gap, dropping from 0.356±0.0570.356± 0.057 in SP to 0.229±0.0680.229± 0.068 in XP. OP reduces this gap but also lowers SP performance, while HE IPPO improves both SP and XP without fully closing the gap. MATE performs best in both settings, reaching 0.460±0.0020.460± 0.002 in SP and 0.446±0.0030.446± 0.003 in XP. (a) MATE (b) OP Figure 5: Level-Based Foraging attention maps. (a) and (b) show the combined attention of two independently trained agents in the unpermuted frame. The colour scale indicates normalised attention intensity. When several apples are equally valid targets, MATE agents converge on the same apple, whereas OP agents attend to different regions and fail to establish a common target. Attention maps in Figure 5 show the coordination mechanism. In LBF, several apples are simultaneously equally valid targets and the time budget is short: agents must converge within a few steps, or the episode is lost to wasted moves. MATE partners align their combined attention on the next apple and collect it; OP agents, instead, each attend to their own vicinity and never commit to a shared target in time. 6.3 OvercookedV2 In Demo Cook Simple (Figure 3(c)), MATE achieves the highest XP return, reaching 132.2±3.0132.2± 3.0 in XP and 143.3±1.4143.3± 1.4 in SP. IPPO drops from 167.6±8.3167.6± 8.3 in SP to 104.2±17.8104.2± 17.8 in XP. HE IPPO reaches the highest SP return in the sweep (208.6208.6), but only 120.6±51.2120.6± 51.2 in XP (see Figure 7), leaving a substantial SP–XP gap; even its best-XP configuration remains below MATE at 125.3±14.4125.3± 14.4. OP improves XP to 116.8±10.9116.8± 10.9, considerably above the 47±15.247± 15.2 originally reported for OP by Gessler et al. (2025). In our experiments, this gain comes from replacing zero-padded observations with agent-centred crops: with the original representation, agents develop additional conventions tied to fixed spatial positions, suggesting that ingredient relabelling alone may not capture all relevant symmetries. We therefore report the original result only as a reference rather than as a directly comparable baseline. The benefit MATE can provide in this layout is inherently constrained. The agents’ fields of view overlap for only 262±9262± 9 of the 400400 evaluation timesteps and, even then, the shared region covers on average 11±0.311± 0.3 cells and at most 1515 of the 5050 cells in each view, limiting how often mutual attention is effective. Figure 6 illustrates one such brief window. Once the informed agent returns to the pot carrying the required ingredient, the ingredient reveals the hidden recipe, while mutual attention directs the uninformed agent’s focus towards this cue, indicating that the prepared soup should be delivered. Nevertheless, MATE outperforms OP under the same agent-centred observations and ingredient permutations, showing that it can improve coordination even under constrained shared visibility, and extending its benefits to partial observability and asymmetric information. Figure 6: OvercookedV2 attention maps from two moments of interaction between MATE agents. Red and blue boxes delimit the agents’ respective fields of view; contour lines show their combined attention in the unpermuted frame. 7 Discussion and Limitations Our results show that MATE supports within-episode coordination across tasks requiring target agreement, as in Card Alignment and LBF, and interpretation of behaviour under partial observability, as in OvercookedV2. This suggests that some coordination problems cannot be solved by suppressing arbitrary conventions alone; for these, grounding coordination in signals recoverable within the episode offers an effective route to generalisation in cooperative multi-agent systems. A limitation is that MATE assumes access to a detector for task-relevant objects during training. We also evaluate only with AI partners; whether the learned attention signal stays interpretable and useful in human–AI coordination remains open. 8 Conclusion We introduced MATE, a multi-agent reinforcement learning method that models joint attention as a general zero-shot coordination mechanism between independently trained partners. Across three diverse environments, MATE closes the SP-XP gap where prior zero-shot coordination methods remain at chance level. Future work could extend this idea beyond shared focus, using attention-based objectives and attention-avoidance to support richer forms of coordination, including division of labour, help-seeking, warning, and complementary attention to different task-relevant objects. Appendix A Training Details Figure 7: Scores of HE IPPO for SP and XP, against the entropy coefficient. Shaded regions show SEM. A.1 Network Architecture All methods use the same network architecture, inspired by the visual-attention architecture of Lee et al. (2021). Since our observations are image-based rather than symbolic grid states, we replace their single convolutional encoder with a residual CNN. Observations are processed by a four-block residual CNN (3232 filters, 3×33×3 kernels, stride 22, ReLU (Nair and Hinton 2010)), followed by an LSTM of hidden size 128128 and two fully connected layers of size 256256 for both actor and critic. A.2 Hyperparameters Our codebase builds on JaxAHT (Wang et al. 2026), and we ran our experiments on Ubuntu servers equipped with NVIDIA Tesla V100-SXM2 GPUs with 32 GB of memory. For all methods, we use seed 42 as a base and either incremented it sequentially (OvercookedV2) or split new random seeds using JAX (LBF, Card Alignment Game). We tuned PPO hyperparameters per environment from the JaxMARL baselines (Rutherford et al. 2024), Forkel et al. (2026), and, for OvercookedV2, Gessler et al. (2025). We varied one parameter at a time and retained the best SP-performing configuration. We swept λMATE∈10−3, 3⋅10−3, 10−2, 3⋅10−2, 0.1, 0.25, 0.4, 0.5, 1, 2 _MATE∈\10^-3, \,3·10^-3, \,10^-2, \,3·10^-2, \,0.1, \,0.25, \,0.4, \,0.5, \,1, \,2\ for the Card Alignment Game and −, 5⋅10−4, 5⋅10−3, 5⋅10−2\10^-4, \,5·10^-4, \,5·10^-3, \,5·10^-2\ for LBF. For the Lee et al. (2021) baseline, we swept β∈×−, 10−3, 3×10−3, 10−2, 3×10−2, 10−1, 0.3, 1β∈\3×10^-4, \,10^-3, \,3×10^-3, \,10^-2, \,3×10^-2, \,10^-1, \,0.3, \,1\. Hyperparameter Card Align. LBF OCv2 LR 3,,8×10−4\3,5,8\\!×\!10^-4 ×10−44\!×\!10^-4 2.5,×10−4\2.5,4\\!×\!10^-4 LR annealing False True True Steps ×1065\!×\!10^6 ×1063\!×\!10^6 ×1073\!×\!10^7 Parallel envs 512,\512,1024\ 128,256\128,256\ 192,256\192,256\ Rollout 88 128128 256256 Epochs 44 Minibatches 1616 3232 ,64\16,64\ Clip ϵε 0.20.2 Entropy coef. 0.010.01 0.010.01 0.01,0.02\0.01,0.02\ Value coef. 0.50.5 GAE λ .8,.9,.95\.8,.9,.95\ .8.8 .8,.9,.95\.8,.9,.95\ Discount γ .99.99 Grad norm .5.5 MATE parameters λMATE _MATE .25.25 10−410^-4 10−3,−\10^-3,10^-4\ η .9.9 .8,.95\.8,.95\ .8,.95\.8,.95\ Table 1: Training hyperparameters for Card Alignment Game, LBF, and OvercookedV2. Braces show the values considered during tuning; bold values were selected. Empty entries inherit Card Alignment Game values. A.3 High-Entropy IPPO Baseline Following Forkel et al. (2026), we keep standard IPPO training unchanged and tune the entropy coefficient over a larger range (Figure 7). This should make independently trained policies compatible, with XP matching SP. In the Card Alignment Game, we extend the sweep to 10510^5, yet the greedy SP–XP gap remains open throughout. In LBF, greedy SP and XP become close for several coefficients, but never intersect, with SP consistently higher. In OvercookedV2, greedy XP approaches and eventually exceeds SP only after increasing entropy has substantially degraded SP performance. Acknowledgments The authors thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting C. Ruhdorfer and F. Kögel. Contribution Statement G.B. and C.R. jointly developed the method, with C.R. conceiving the initial research direction and method, while G.B. led its implementation and experimental development. G.B. conducted the experiments, analyzed the experimental results, and led the writing of the paper. C.R. provided technical leadership, contributed to the experimental analysis, and contributed to the writing of the paper. F.K. advised on key research decisions and contributed to the writing of the paper. A.B. supervised the research, acquired funding, and contributed to the writing of the paper. References Albrecht and Ramamoorthy (2013) Albrecht, S. V.; and Ramamoorthy, S. 2013. A game-theoretic model and best-response learning method for ad hoc coordination in multiagent systems. In Gini, M. L.; Shehory, O.; Ito, T.; and Jonker, C. M., eds., International conference on Autonomous Agents and Multi-Agent Systems, AAMAS ’13, Saint Paul, MN, USA, May 6-10, 2013, 1155–1156. IFAAMAS. Anzalone et al. (2019) Anzalone, S. M.; Xavier, J.; Boucenna, S.; Billeci, L.; Narzisi, A.; Muratori, F.; Cohen, D.; and Chetouani, M. 2019. Quantifying patterns of joint attention during human-robot interactions: An application for autism spectrum disorder assessment. Pattern Recognit. Lett., 118: 42–50. Baker et al. (2020) Baker, B.; Kanitscheider, I.; Markov, T. M.; Wu, Y.; Powell, G.; McGrew, B.; and Mordatch, I. 2020. Emergent Tool Use From Multi-Agent Autocurricula. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net. Berner et al. (2019) Berner, C.; Brockman, G.; Chan, B.; Cheung, V.; Debiak, P.; Dennison, C.; Farhi, D.; Fischer, Q.; Hashme, S.; Hesse, C.; Józefowicz, R.; Gray, S.; Olsson, C.; Pachocki, J.; Petrov, M.; de Oliveira Pinto, H. P.; Raiman, J.; Salimans, T.; Schlatter, J.; Schneider, J.; Sidor, S.; Sutskever, I.; Tang, J.; Wolski, F.; and Zhang, S. 2019. Dota 2 with Large Scale Deep Reinforcement Learning. CoRR, abs/1912.06680. Bullard et al. (2020) Bullard, K.; Meier, F.; Kiela, D.; Pineau, J.; and Foerster, J. N. 2020. Exploring Zero-Shot Emergent Communication in Embodied Multi-Agent Populations. CoRR, abs/2010.15896. Carpenter and Call (2013) Carpenter, M.; and Call, J. 2013. How Joint Is the Joint Attention of Apes and Human Infants? In Metcalfe, J.; and Terrace, H. S., eds., Agency and Joint Attention, 49–61. New York: Oxford University Press. ISBN 9780199988341. Carroll et al. (2019) Carroll, M.; Shah, R.; Ho, M. K.; Griffiths, T.; Seshia, S. A.; Abbeel, P.; and Dragan, A. D. 2019. On the Utility of Learning about Humans for Human-AI Coordination. In Wallach, H. M.; Larochelle, H.; Beygelzimer, A.; d’Alché-Buc, F.; Fox, E. B.; and Garnett, R., eds., Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, 5175–5186. Csibra and Gergely (2009) Csibra, G.; and Gergely, G. 2009. Natural pedagogy. Trends in Cognitive Sciences, 13(4): 148–153. de Witt et al. (2019) de Witt, C. S.; Foerster, J. N.; Farquhar, G.; Torr, P. H. S.; Boehmer, W.; and Whiteson, S. 2019. Multi-Agent Common Knowledge Reinforcement Learning. In Wallach, H. M.; Larochelle, H.; Beygelzimer, A.; d’Alché-Buc, F.; Fox, E. B.; and Garnett, R., eds., Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, 9924–9935. de Witt et al. (2020) de Witt, C. S.; Gupta, T.; Makoviichuk, D.; Makoviychuk, V.; Torr, P. H. S.; Sun, M.; and Whiteson, S. 2020. Is Independent Learning All You Need in the StarCraft Multi-Agent Challenge? CoRR, abs/2011.09533. Dizdarević et al. (2025) Dizdarević, T.; Hammond, R.; Gessler, T.; Calinescu, A.; Cook, J.; Gallici, M.; Lupu, A.; and Foerster, J. N. 2025. Ad-Hoc Human-AI Coordination Challenge. In Forty-second International Conference on Machine Learning. Foerster et al. (2016) Foerster, J. N.; Assael, Y. M.; de Freitas, N.; and Whiteson, S. 2016. Learning to Communicate with Deep Multi-Agent Reinforcement Learning. In Lee, D. D.; Sugiyama, M.; von Luxburg, U.; Guyon, I.; and Garnett, R., eds., Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, 2137–2145. Foerster et al. (2019) Foerster, J. N.; Song, H. F.; Hughes, E.; Burch, N.; Dunning, I.; Whiteson, S.; Botvinick, M. M.; and Bowling, M. 2019. Bayesian Action Decoder for Deep Multi-Agent Reinforcement Learning. In Chaudhuri, K.; and Salakhutdinov, R., eds., Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, 1942–1951. PMLR. Forkel et al. (2026) Forkel, J.; Ruhdorfer, C.; Beukman, M.; Bulling, A.; and Foerster, J. 2026. High entropy leads to symmetry-equivariant policies in Dec-POMDPs. arXiv:2511.22581. García-Martínez et al. (2024) García-Martínez, J.; Gamboa-Montero, J. J.; Castillo, J. C.; and Castro-González, Á. 2024. Analyzing the Impact of Responding to Joint Attention on the User Perception of the Robot in Human-Robot Interaction. Biomimetics, 9(12). Gessler et al. (2025) Gessler, T.; Dizdarevic, T.; Calinescu, A.; Ellis, B.; Lupu, A.; and Foerster, J. N. 2025. OvercookedV2: Rethinking Overcooked for Zero-Shot Coordination. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net. Hu and Foerster (2020) Hu, H.; and Foerster, J. N. 2020. Simplified Action Decoder for Deep Multi-Agent Reinforcement Learning. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net. Hu et al. (2021) Hu, H.; Lerer, A.; Cui, B.; Pineda, L.; Brown, N.; and Foerster, J. N. 2021. Off-Belief Learning. In Meila, M.; and Zhang, T., eds., Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, 4369–4379. PMLR. Hu et al. (2020) Hu, H.; Lerer, A.; Peysakhovich, A.; and Foerster, J. N. 2020. "Other-Play" for Zero-Shot Coordination. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, 4399–4410. PMLR. Lee et al. (2021) Lee, D.; Jaques, N.; Kew, J. C.; Eck, D.; Schuurmans, D.; and Faust, A. 2021. Joint Attention for Multi-Agent Coordination and Social Learning. CoRR, abs/2104.07750. Mordatch and Abbeel (2018) Mordatch, I.; and Abbeel, P. 2018. Emergence of Grounded Compositional Language in Multi-Agent Populations. In McIlraith, S. A.; and Weinberger, K. Q., eds., Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, 1495–1502. AAAI Press. Mundy and Jarrold (2010) Mundy, P.; and Jarrold, W. 2010. Infant joint attention, neural networks and social cognition. Neural Networks, 23(8-9): 985–997. Nair and Hinton (2010) Nair, V.; and Hinton, G. E. 2010. Rectified Linear Units Improve Restricted Boltzmann Machines. In Fürnkranz, J.; and Joachims, T., eds., Proceedings of the 27th International Conference on Machine Learning (ICML-10), June 21-24, 2010, Haifa, Israel, 807–814. Omnipress. Oliehoek and Amato (2016) Oliehoek, F. A.; and Amato, C. 2016. A Concise Introduction to Decentralized POMDPs. Springer Briefs in Intelligent Systems. Springer. ISBN 978-3-319-28927-4. Ruhdorfer et al. (2026) Ruhdorfer, C.; Bortoletto, M.; Forkel, J.; Foerster, J.; and Bulling, A. 2026. The Yokai Learning Environment: Tracking Beliefs Over Space and Time. arXiv:2508.12480. Rutherford et al. (2024) Rutherford, A.; Ellis, B.; Gallici, M.; Cook, J.; Lupu, A.; Ingvarsson, G.; Willi, T.; Hammond, R.; Khan, A.; de Witt, C. S.; Souly, A.; Bandyopadhyay, S.; Samvelyan, M.; Jiang, M.; Lange, R. T.; Whiteson, S.; Lacerda, B.; Hawes, N.; Rocktäschel, T.; Lu, C.; and Foerster, J. N. 2024. JaxMARL: Multi-Agent RL Environments and Algorithms in JAX. In Globersons, A.; Mackey, L.; Belgrave, D.; Fan, A.; Paquet, U.; Tomczak, J. M.; and Zhang, C., eds., Advances in Neural Information Processing Systems 37: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024. Schulman et al. (2017) Schulman, J.; Wolski, F.; Dhariwal, P.; Radford, A.; and Klimov, O. 2017. Proximal Policy Optimization Algorithms. CoRR, abs/1707.06347. Silver et al. (2016) Silver, D.; Huang, A.; Maddison, C. J.; Guez, A.; Sifre, L.; van den Driessche, G.; Schrittwieser, J.; Antonoglou, I.; Panneershelvam, V.; Lanctot, M.; Dieleman, S.; Grewe, D.; Nham, J.; Kalchbrenner, N.; Sutskever, I.; Lillicrap, T. P.; Leach, M.; Kavukcuoglu, K.; Graepel, T.; and Hassabis, D. 2016. Mastering the game of Go with deep neural networks and tree search. Nat., 529(7587): 484–489. Staudte and Crocker (2011) Staudte, M.; and Crocker, M. W. 2011. Investigating joint attention mechanisms through spoken human-robot interaction. Cognition, 120 2: 268–91. Stone et al. (2010) Stone, P.; Kaminka, G. A.; Kraus, S.; and Rosenschein, J. S. 2010. Ad Hoc Autonomous Agent Teams: Collaboration without Pre-Coordination. In Fox, M.; and Poole, D., eds., Proceedings of the Twenty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2010, Atlanta, Georgia, USA, July 11-15, 2010, 1504–1509. AAAI Press. Strouse et al. (2021) Strouse, D.; McKee, K. R.; Botvinick, M. M.; Hughes, E.; and Everett, R. 2021. Collaborating with Humans without Human Data. In Ranzato, M.; Beygelzimer, A.; Dauphin, Y. N.; Liang, P.; and Vaughan, J. W., eds., Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, 14502–14515. Sukhbaatar, Szlam, and Fergus (2016) Sukhbaatar, S.; Szlam, A.; and Fergus, R. 2016. Learning Multiagent Communication with Backpropagation. In Lee, D. D.; Sugiyama, M.; von Luxburg, U.; Guyon, I.; and Garnett, R., eds., Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, 2244–2252. Tesauro (1995) Tesauro, G. 1995. Temporal Difference Learning and TD-Gammon. Commun. ACM, 38(3): 58–68. Tomasello (2014) Tomasello, M. 2014. Joint attention as social cognition. In Joint attention, 103–130. Psychology Press. Tucker et al. (2021) Tucker, M.; Li, H.; Agrawal, S.; Hughes, D.; Sycara, K. P.; Lewis, M.; and Shah, J. A. 2021. Emergent Discrete Communication in Semantic Spaces. In Ranzato, M.; Beygelzimer, A.; Dauphin, Y. N.; Liang, P.; and Vaughan, J. W., eds., Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, 10574–10586. Vinyals et al. (2019) Vinyals, O.; Babuschkin, I.; Czarnecki, W. M.; Mathieu, M.; Dudzik, A.; Chung, J.; Choi, D. H.; Powell, R.; Ewalds, T.; Georgiev, P.; Oh, J.; Horgan, D.; Kroiss, M.; Danihelka, I.; Huang, A.; Sifre, L.; Cai, T.; Agapiou, J. P.; Jaderberg, M.; Vezhnevets, A. S.; Leblond, R.; Pohlen, T.; Dalibard, V.; Budden, D.; Sulsky, Y.; Molloy, J.; Paine, T. L.; Gülçehre, Ç.; Wang, Z.; Pfaff, T.; Wu, Y.; Ring, R.; Yogatama, D.; Wünsch, D.; McKinney, K.; Smith, O.; Schaul, T.; Lillicrap, T. P.; Kavukcuoglu, K.; Hassabis, D.; Apps, C.; and Silver, D. 2019. Grandmaster level in StarCraft I using multi-agent reinforcement learning. Nat., 575(7782): 350–354. Wang et al. (2026) Wang, C.; Fernandez, R.; Mustafaoglu, Z. S.; Kundan, M.; Cui, J.; Xiao, L.; Wang, Z.; Shi, D. Y.; Madhan, A.; Liu, J.; Rahman, A.; and Stone, P. 2026. JaxAHT: A JAX-Based Library for Ad Hoc Teamwork. arXiv:2609.13716. Ye et al. (2020) Ye, C.; Khalifa, A.; Bontrager, P.; and Togelius, J. 2020. Rotation, Translation, and Cropping for Zero-Shot Generalization. In IEEE Conference on Games, CoG 2020, Osaka, Japan, August 24-27, 2020, 57–64. IEEE. Yu et al. (2022) Yu, C.; Velu, A.; Vinitsky, E.; Gao, J.; Wang, Y.; Bayen, A. M.; and Wu, Y. 2022. The Surprising Effectiveness of PPO in Cooperative Multi-Agent Games. In Koyejo, S.; Mohamed, S.; Agarwal, A.; Belgrave, D.; Cho, K.; and Oh, A., eds., Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022.