Paper deep dive

Training Generalizable Collaborative Agents via Strategic Risk Aversion

Chengrui Qu, Yizhou Zhang, Nicolas Lanzetti, Eric Mazumdar

Year: 2026Venue: arXiv preprintArea: cs.LGType: PreprintEmbeddings: 145

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%

Last extracted: 7/20/2026, 12:42:12 PM

Summary

This paper introduces Strategic Risk Aversion as a principled inductive bias for multi-agent reinforcement learning (MARL) to improve generalizable collaboration. The authors propose that risk-averse agents are robust to partner deviations and exhibit less free-riding compared to standard Nash equilibrium strategies. They develop the Strategically Risk-Averse Policy Optimization (SRPO) algorithm, which integrates risk aversion into policy optimization, and demonstrate its effectiveness in achieving reliable cooperation with unseen partners across various collaborative benchmarks, including LLM-based tasks.

Entities (7)

Strategic Risk Aversion · concept · 98%SRPO · algorithm · 95%Risk-Averse Quantal Response Equilibrium (RQE) · concept · 92%Free-riding · phenomenon · 90%IPPO · algorithm · 90%Partner Generalization · problem · 90%LLM Collaboration · task · 85%

Relation Signals (6)

SRPO → implements → Strategic Risk Aversion

confidence 95% · We develop Strategically Risk-Averse Policy Optimization (SRPO), a MARL algorithm that optimizes an RQE-derived objective which integrates naturally with policy-optimization algorithms

Strategic Risk Aversion → improves → Partner Generalization

confidence 90% · This perspective inherently aligns with the goal of partner generalization, as it forces agents to be robust to deviations in their partner’s play.

Strategic Risk Aversion → induces → Collaboration

confidence 90% · We prove that in continuous quadratic aggregative games, strategic risk aversion can encourage collaboration, with higher risk aversion leading to a greater focus on the shared goal.

Risk-Averse Quantal Response Equilibrium (RQE) → isformalizedby → Strategic Risk Aversion

confidence 90% · Formalizing this principle leads to a new equilibrium concept: (Strategically) Risk-Averse Quantal Response Equilibria (RQE)

Strategic Risk Aversion → mitigates → Free-riding

confidence 90% · We prove in finite-action collaborative games with private costs that strategic risk aversion can mitigate free-riding at equilibrium.

SRPO → outperforms → IPPO

confidence 85% · We demonstrate across collaborative MARL benchmarks that SRPO consistently achieves more reliable coordination with heterogeneous and unseen partners than IPPO

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Many emerging agentic paradigms require agents to collaborate with one another (or people) to achieve shared goals. Unfortunately, existing approaches to learning policies for such collaborative problems produce brittle solutions that fail when paired with new partners. We attribute these failures to a combination of free-riding during training and a lack of strategic robustness. To address these problems, we study the concept of strategic risk aversion and interpret it as a principled inductive bias for generalizable cooperation with unseen partners. While strategically risk-averse players are robust to deviations in their partner's behavior by design, we show that, in collaborative games, they also (1) can have better equilibrium outcomes than those at classical game-theoretic concepts like Nash, and (2) exhibit less or no free-riding. Inspired by these insights, we develop a multi-agent reinforcement learning (MARL) algorithm that integrates strategic risk aversion into standard policy optimization methods. Our empirical results across collaborative benchmarks (including an LLM collaboration task) validate our theory and demonstrate that our approach consistently achieves reliable collaboration with heterogeneous and previously unseen partners across collaborative tasks.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

143,786 characters extracted from source content.

Expand or collapse full text

theorem]Definition Training Generalizable Collaborative Agents via Strategic Risk Aversion Chengrui Qu Caltech Department of Computing and Mathematical Sciences, California Institute of Technology, Pasadena, CA, USA. Yizhou Zhang111The function is strictly increasing on the domain of τ where RQE are well-defined, that is, when the risk parameters τ is “not too large”; else, risk diverges to infinity. For details, we refer the reader to the appendix and, in particular, to (14). Caltech Nicolas Lanzetti111The function is strictly increasing on the domain of τ where RQE are well-defined, that is, when the risk parameters τ is “not too large”; else, risk diverges to infinity. For details, we refer the reader to the appendix and, in particular, to (14). Caltech Eric Mazumdar111The function is strictly increasing on the domain of τ where RQE are well-defined, that is, when the risk parameters τ is “not too large”; else, risk diverges to infinity. For details, we refer the reader to the appendix and, in particular, to (14). Caltech Abstract Many emerging agentic paradigms require agents to collaborate with one another (or people) to achieve shared goals. Unfortunately, existing approaches to learning policies for such collaborative problems produce brittle solutions that fail when paired with new partners. We attribute these failures to a combination of free-riding during training and a lack of strategic robustness. To address these problems, we study the concept of strategic risk aversion and interpret it as a principled inductive bias for generalizable cooperation with unseen partners. While strategically risk-averse players are robust to deviations in their partner’s behavior by design, we show that, in collaborative games, they also (1) can have better equilibrium outcomes than those at classical game-theoretic concepts like Nash, and (2) exhibit less or no free-riding. Inspired by these insights, we develop a multi-agent reinforcement learning (MARL) algorithm that integrates strategic risk aversion into standard policy optimization methods. Our empirical results across collaborative benchmarks (including an LLM collaboration task) validate our theory and demonstrate that our approach consistently achieves reliable collaboration with heterogeneous and previously unseen partners across collaborative tasks. Keywords: Strategic risk aversion, multi-agent reinforcement learning, partner generalization 1 Introduction AI systems increasingly operate in multi-agent environments, where success depends on effective interaction with other agents. An increasingly important subset of these problems involves agents solving collaborative tasks—i.e., tasks where agents must work together towards a shared goal. Examples of such problems range from robots coordinating with people in shared physical spaces like warehouses (Kruse et al., 2013), to emerging “agentic AI” problems where multiple models collaborate to write code (Wu et al., 2024) or solve math problems (Cemri et al., 2025). A common approach to such problems is to view them as collaborative games and use concepts from game theory to design algorithms to learn collaborative policies (Dafoe et al., 2021). Viewed through this lens, a central challenge in collaborative games is partner generalization—where one seeks to maintain effective interaction across new sets of partners (Carroll et al., 2019; Hu et al., 2020; Ruhdorfer et al., 2025). Indeed, in many real-world settings, agents must navigate situations in which they interact with different partners (either algorithmic or human) who may have slightly different goals, heuristics, or levels of competence. Agents should learn strategies that work well across a broad range of partners without sacrificing performance. Unfortunately, current approaches tend to struggle with this—with learned policies often failing to generalize to new partners (Stone et al., 2010), or overfitting to other agents’ eccentricities and becoming overly reliant on specific conventions (Carroll et al., 2019; Lerer and Peysakhovich, 2019). We view the problem of partner generalization as ultimately a question of robustness of the learned policy and of alignment between agents (Leibo et al., 2017). The need for robustness emerges from the requirement that learned strategies be insensitive to changes in a partner’s strategy and degrade gracefully in performance as the partners become less collaborative. The need for alignment emerges from a desire to find agents that contribute proportionally to the task. If agents learn to under-contribute or delegate costly actions (Sunehag et al., 2018; Liu et al., 2023) to their partners—effectively free-riding on their partner’s effort—they cannot generalize to new environments and partners. While free-riding is a widely acknowledged phenomenon in game theory, it is rarely studied as a problem in collaborative multi-agent learning despite recent empirical evidence that it arises even with large AI models (Zhang et al., 2025). Prior work—which we review in Section˜2 and then in depth in Appendix A—has studied partner generalization in specific settings that fail to scale to large-scale problems like the post-training of LLM agents (Hu et al., 2020) or via largely heuristic approaches (Forkel and Foerster, 2025). In contrast, in this paper we advance strategic risk aversion as a principled and scalable approach to addressing this issue in a broad class of collaborative games. Recently proposed in multi-agent reinforcement learning (MARL) to derive human-like strategies in general games (Mazumdar et al., 2024), strategic risk aversion mirrors behaviors observed in people in experimental economics (Goeree et al., 2003). Unlike many approaches to robustness or risk aversion in MARL which primarily focus on robustness to uncertainty in the underlying environment, strategic risk aversion posits that agents should be risk-averse to uncertainty stemming from their opponents’ decisions. This perspective inherently aligns with the goal of partner generalization, as it forces agents to be robust to deviations in their partner’s play. Formalizing this principle leads to a new equilibrium concept: (Strategically) Risk-Averse Quantal Response Equilibria (RQE) (Mazumdar et al., 2024). While RQE has been shown to better capture human play in simple experiments and offer better computational tractability (Zhang and Mazumdar, 2025) than classic game theoretic concepts, its potential for collaborative games and partner generalization remains unknown. We argue that strategic risk aversion provides a suitable inductive bias for robust cooperation in collaborative games: at an RQE, an agent is trained against a structured set of plausible partner deviations, governed by the agent’s degree of risk aversion. This ensures strategies are robust without being overly conservative. Contributions: Through both a theoretical analysis of RQE in structured collaborative games and extensive experiments across a range of diverse multi-agent benchmarks, we validate that RQE are particularly suited for collaborative tasks. In particular, our contributions are as follows: • Incentivizing collaboration (Section˜4.1): We prove that in continuous quadratic aggregative games, strategic risk aversion can encourage collaboration, with higher risk aversion leading to a greater focus on the shared goal. This implies a counterintuitive result: unlike in classic robust optimization or RL, robustness does not necessarily require sacrificing performance. • Alleviating free-riding (Section˜4.2): We prove in finite-action collaborative games with private costs that strategic risk aversion can mitigate free-riding at equilibrium. Thus, risk aversion can reduce the alignment problem in collaborative MARL. • Scalable risk-averse MARL algorithms: We develop Strategically Risk-Averse Policy Optimization (SRPO), a MARL algorithm that optimizes an RQE-derived objective which integrates naturally with policy-optimization algorithms like independent proximal policy optimization (IPPO). • Empirical validation: We demonstrate across collaborative MARL benchmarks that SRPO consistently achieves more reliable coordination with heterogeneous and unseen partners than IPPO (the current scalable MARL baseline). We observe that while IPPO performs well, it consistently finds free-riding equilibria that limit its generalization. In contrast, we observe—in line with our theory—that RQE found by SRPO exhibit no free-riding and result in better partner generalization. We also provide preliminary small-scale experiments showing that these findings can even extend to the fine-tuning of collaborative agentic AI teams made up of large language models, demonstrating the scalability of our approach. 2 Related Works Our work studies strategic risk aversion in collaborative MARL. We cover the most relevant related work here and defer a more in depth discussion to Appendix A. The most related literature in collaborative MARL focuses on the partner generalization problem; i.e., how to learn collaborative policies that remain effective when paired with previously unseen partners (Barrett et al., 2014; Carroll et al., 2019; Dizdarević et al., 2025). This is sometimes referred to as zero-shot coordination (Hu et al., 2020, 2021), or ad-hoc teamwork problem (Stone et al., 2010). One large class of approaches can broadly be described as population-based methods (Vinyals et al., 2019; Zhao et al., 2023; Yu et al., 2023; Wang et al., 2024; Rahman et al., 2023; Lupu et al., 2021) in which one trains against a diverse set of learned policies—essentially performing domain randomization for partners. Though such approaches are conceptually simple, the task of generating good populations of partners is nontrivial and typically relies on heuristic, task-dependent design choices. Furthermore, such approaches quickly become computationally intractable for complex, high-dimensional settings (e.g., LLM fine-tuning) and tend not to have strong guarantees that they result in better generalization. To circumvent this limitation, Forkel and Foerster (2025) attempt to induce robustness by introducing more randomness during training (by increasing entropy regularization in policy optimization). Though this method scales well, our experiments demonstrate that it does not solve free-riding and consequently does not address the problem of generalization across collaborative tasks. In contrast, our approach based on strategic risk aversion scales as a simple modification to existing policy-optimization techniques, is grounded in a principled game theoretic formulation, and empirically consistently delivers more stable cross-play performance across collaborative benchmarks. While these forms of risk aversion have been well studied in the experimental economics literature (Gollier, 2001; Goeree and Offerman, 2002; Goeree et al., 2003), they have only recently been studied in theory (Lanzetti et al., 2025b; Mazumdar et al., 2024) and in MARL (Zhang and Mazumdar, 2025; Slumbers et al., 2023). We demonstrate its potential in collaborative games. 3 Problem Setup We consider a general-sum collaborative game with N players. Each player i∈[N]i∈[N] is endowed with actions ai∈ia_i _i, where the action spaces can be discrete or continuous, and we use the notation −i=∏j≠ijA_-i= _j≠ iA_j for the action space of all players except i. Each player aims to maximize a utility function ui:i×−i→ℝu_i:A_i×A_-i . The utility function consists of a shared reward R, which coincides for all players, and a private cost cic_i, so that ui(ai,a−i)=R(a1,…,aN)−ci(ai).u_i(a_i,a_-i)=R(a_1,…,a_N)-c_i(a_i). These games are collaborative since all players seek to maximize the reward, but pay for their own effort. They have been well studied in economics under the guise of public goods games and naturally model many real-world scenarios where players aim to collaboratively accomplish a task but are penalized for their own effort. The players select a mixed strategy xi∈Δ(i)x_i∈ (A_i), which is a probability over the available actions iA_i (here, Δ(i) (A_i) is the set of probability distributions over iA_i). In the standard game-theoretic setting, the utility of each player is then the standard risk-neutral expected utility Ui(xi,x−i)=a1∼x1…aN∼xN[ui(ai,a−i)],U_i(x_i,x_-i)=E_ subarrayca_1 x_1\\ ...\\ a_N x_N subarray[u_i(a_i,a_-i)], where x−ix_-i is the collection of the mixed strategies of all other players j≠ij≠ i. In this work, we follow Mazumdar et al. (2024) and propose to introduce two elements of human decision-making: risk aversion and bounded rationality. Risk aversion To model risk aversion, we consider players that seek to optimize a risk-adjusted utility based on the entropic risk measure with risk aversion parameter τi>0 _i>0 (e.g., see Föllmer and Schied (2002)), instead of their expected utility. Players’ resulting objectives are given by Uiτi(xi,x−i)=infp∈Δ(−i)Ui(xi,p)+1τiKL(p,x−i).U _i_i(x_i,x_-i)= _p∈ (A_-i)U_i(x_i,p)+ 1 _i *KL(p,x_-i). (1) In words, the utility results from a worst-case approach, in which a fictitious adversary tries to inflict maximum damage at the (risk-neutral) expected utility, while not deviating “too much” from the strategies of all other players. For the sake of this paper, we will measure this deviation with the KL divergence, defined as KL(p,q)=∑jpjlog⁡(pjqj) *KL(p,q)= _jp_j ( p_jq_j) (when −iA_-i is discrete) and KL(p,q)=∫ℝnρp(x)log⁡(ρp(x)ρq(x)) *KL(p,q)= _R^n _p(x) ( _p(x) _q(x)) (when −i=ℝnA_-i=R^n is continuous), where ρp _p and ρq _q are the densities of p and q (provided they exist, else KL(p,q)=+∞ *KL(p,q)=+∞). Our choice of the entropic risk measure is rooted in operations research, but also motivated by ubiquitous use of the KL divergence across machine learning and in particular in policy optimization in reinforcement learning. By focusing on the KL divergence, we derive algorithms that can be implemented with minimal changes to existing codebases. Bounded rationality To incorporate bounded rationality, we add entropy to each player’s utility, so that their utility is Uiτi,ϵi(xi,x−i) U_i _i, _i(x_i,x_-i) =Uiτi(xi,x−i)−ϵiH(xi), =U _i_i(x_i,x_-i)- _iH(x_i), (2) where ϵi>0 _i>0 and H(xi)H(x_i) is the (negative) entropy of the mixed strategy xix_i; i.e., H(p)=∑jpjlog⁡(pj)H(p)= _jp_j (p_j) (if iA_i is discrete) and H(p)=∫ℝnρp(x)log⁡(ρp(x))dxH(p)= _R^n _p(x) ( _p(x)) x (when i=ℝnA_i=R^n is continuous), where ρp _p is again the density of p (provided it exists, else H(p)=+∞H(p)=+∞). Entropy regularization of the utility leads to the celebrated quantal response model in behavioral economics (McKelvey and Palfrey, 1995) and is widely employed in machine learning for exploration purposes (e.g., see Ahmed et al. (2019)). For more general regularization schemes, we refer to Mazumdar et al. (2024). Risk-averse quantal response equilibrium (RQE). Given our new utility Uiτi,ϵi(xi,x−i)U_i _i, _i(x_i,x_-i), we now define a RQE as a set of joint mixed strategies from which no player has an incentive to unilaterally deviate: definition [Mazumdar et al. (2024), Definition 5] A tuple of mixed strategies (x1⋆,…,xN⋆)(x_1 ,…,x_N ) is a risk-averse quantal response equilibrium (RQE) with degrees of risk aversion τ1,…,τN _1,…, _N and bounded rationality ϵ1,…,ϵN _1,…, _N if for all players i we have Uiτi,ϵi(xi,x−i⋆)≤Uiτi,ϵi(xi⋆,x−i⋆)∀xi∈Δ(i).U _i, _i_i(x_i,x_-i )≤ U _i, _i_i(x_i ,x_-i )\>∀\,x_i∈ (A_i). (3) When τi,ϵi→0 _i, _i→ 0 we recover Nash equilibria, when τi→0 _i→ 0 we recover quantal response equilibria, and when ϵ→0ε→ 0 and τ→∞τ→∞ we recover security strategies. In the next section, we investigate the benefits of risk aversion, together with bounded rationality, in two classes of collaborative games: aggregative continuous quadratic games and finite symmetric collaborative games. These insights prompt us, in the subsequent section, to use RQE for the design of algorithms to train collaborative agents in MARL. 4 “Free-Lunch” Theorems for Strategic Risk Aversion in Collaborative Games Figure 1: (i). Expected utility of each player as a function of the degree of risk aversion τ at equilibrium, when ϵ=1ε=1 in Section˜4.1. Player’s utilities can first increase with risk aversion before decreasing due to over-conservatism, meaning that strategic risk aversion can yield better performing equilibria than Nash or QRE. (i). Probability that a player collaborates at a RQE as a function of the level of risk aversion, for ϵ=0.2ε=0.2 for the game in Example 4.2. Strategic risk aversion alleviates free riding entirely after a given threshold (i.e., δ→0δ→ 0) as our theory predicts. We first study strategic risk aversion in simple, structured classes of collaborative games which are more tractable to analyze than complex MARL problems we design algorithms for. We prove two “free-lunch” theorems that illustrate how strategic risk aversion can (1) increase collaboration and (2) mitigate free riding—both of which are essential to the task of partner generalization. These results serve as a principled rationale for incorporating strategic risk aversion into MARL. We empirically observe both these properties across our experiments in Section 6, validating that our insights extend beyond structured games. 4.1 Strategic Risk Aversion can Induce Collaboration Our first setting lies in continuous (=ℝnA=R^n) quadratic aggregative games, popular in economics (Corchón, 1994) and control theory (Paccagnan et al., 2018). In line with our collaborative setting, we write the utility of each player as the difference between a shared reward that depends on the aggregate action a1+…+aNa_1+…+a_N, so that R(a1,…,aN)=12⟨∑i=1Nai,H∑i=1Nai⟩+⟨h,∑i=1Nai⟩,R(a_1,…,a_N)= 12 _i=1^Na_i,H _i=1^Na_i + h, _i=1^Na_i , for some H negative definite and h of appropriate dimensions, and a private cost that only depends on the player’s own action aia_i, of the form ci(ai)=ρi2‖ai‖2c_i(a_i)= _i2 a_i^2 for ρi>0 _i>0. For simplicity of exposition, we assume here that all players have the same degree of risk aversion τi _i and bounded rationality ϵi _i, and the same ρi _i. Nevertheless, as we show in the Appendix B.1, all results extend to the case where these are player-dependent. Being a continuous game, the computation of an RQE is infinite dimensional, as it involves searching over mixed strategies over continuous action spaces. In Section˜B.1.1 in the appendix, however, we bypass this complexity by showing that there is a unique Gaussian RQE (i.e., RQE where mixed strategies are Gaussian) that can be computed efficiently. This effectively allows us to study the effect of risk aversion on the expected shared reward J(τ)≔ai∼xi⋆(τ)[R(a1,…,aN)].J(τ) _a_i x_i (τ) [R(a_1,…,a_N) ]. Remarkably, as we show in the next theorem (proven in Appendix B.1), this function is strictly increasing. That is, risk monotonically increases the shared reward and thus induces collaboration between the players. theorem [risk induces collaboration] Let xi⋆(τ)x_i (τ) be the Gaussian mixed strategy of player i at the unique Gaussian RQE of the game, as a function of the degree of risk aversion τ. Then, the expected shared reward τ↦J(τ)τ J(τ) is strictly increasing111The function is strictly increasing on the domain of τ where RQE are well-defined, that is, when the risk parameters τ is “not too large”; else, risk diverges to infinity. For details, we refer the reader to the appendix and, in particular, to (14).. That is, players contribute more to the shared reward as they become more risk-averse. This increasing contribution to the shared reward generally entails an increase in the players’ private cost. Thus, in practice, we observe a tradeoff between being risk neutral (τ→0τ→ 0) and more risk-averse (τ large) with large personal cost but large shared reward. In a one-dimensional example, we can study this tradeoff analytically. Example . Consider two robots that move an object from the origin to a target location a¯ a. Each robot exerts a force aia_i, so that the total force is a1+a2a_1+a_2. Players incur a penalty for their own actuation effort. Thus, the shared reward is −12(a1+a2−a¯)2- 12(a_1+a_2- a)^2 and the personal cost is 12ai2 12a_i^2 (with ρi=1 _i=1 for simplicity). In this case, the Gaussian RQE has mean mi⋆(τ)=a¯3−τϵm_i (τ)= a3-τε and variance Σi=ϵ2 _i= ε2. The equilibrium expected reward is concave in τ as shown in Fig.˜1, highlighting that risk aversion can yield higher utilities. Remark . We refer to Theorem 4.1 as a “free-lunch" theorem because it suggests that one can add risk aversion into a game and not incur a loss in performance. This can be observed in Fig.˜1(i) in which equilibrium utility increases from a risk-neutral baseline before decreasing again. This is in stark contrast to single-agent RL or classic robust optimization where robustness necessitates sacrificing performance. We validate our theory in broader classes of games in Section˜6 and observe that, in some games, some degree of strategic risk aversion can increase collaboration and performance over a risk-neutral baseline. In other games, however, the structure is less amenable to such phenomena, and we observe that risk aversion introduces conservatism and a corresponding decrease in performance. 4.2 Strategic Risk Aversion Alleviates Free-riding We now prove a second “free-lunch” theorem associated with strategic risk aversion in collaborative games, namely that players who are strategically risk-averse will free-ride less at equilibrium. We prove this result in the context of 2-player collaborative finite symmetric games played over the probability simplex. These are games in which player i=1,2i=1,2’s utility is given by Ui(x1,x2) U_i(x_1,x_2) =a1∼x1,a2∼x2[R(a1,a2)]−ai∼xi[c(ai)], =E_ subarrayca_1 x_1,a_2 x_2 subarray[R(a_1,a_2)]-E_a_i x_i[c(a_i)], where R is a symmetric (R(a1,a2)=R(a2,a1)R(a_1,a_2)=R(a_2,a_1)) shared reward, c is a private cost, and xix_i are strategy vectors in the probability simplex in ℝnR^n. Although the game is symmetric, equilibria can be asymmetric and thus exhibit free-riding—i.e., one player can exert significantly less effort at equilibrium while still enjoying high utility. As we observe across our experiments, free-riding equilibria are ubiquitous across many MARL benchmarks and pose a significant challenge to the generalization of agents’ strategies. Concretely, we define free riding in our game as follows. definition[free-riding] An RQE (x1,x2)∈Δ()×Δ()(x_1,x_2)∈ (A)× (A) exhibits free-riding with degree δ≥0δ≥ 0 if the difference in costs paid by each player at the RQE is δ; i.e., |a1∼x1[c(a1)]−a2∼x2[c(a2)]|=δ|E_a_1 x_1[c(a_1)]-E_a_2 x_2[c(a_2)]|=δ. If δ=0δ=0, the RQE does not exhibit free-riding. The following theorem shows that as a player’s degree of risk aversion increases, free-riding becomes less prevalent. theorem [risk removes free-riding] There exists a game-dependent constant C depending only on R, c, and the degree of bounded rationality ϵε such that if player’s degrees of risk aversion τ in a two-player collaborative game satisfy τ>Cδ2τ> Cδ^2, then the game can admit no RQE with degree of free-riding greater than δ. This theorem (proven in Appendix B.2) shows that as strategic risk aversion increases, RQE must exhibit less free-riding. The intuition for this result is as follows: suppose that player i free-rides at a RQE. If they become risk-averse, the worst-case deviation for their opponent is simply to stop putting any effort into the game. Since player i is free-riding and not putting their own effort, this deviation causes a large drop in performance. Thus, at a risk-averse equilibrium, a player must contribute some of their own effort. We validate this intuition in a simple coordination game. Example . Consider the game with =C=collaborate,D=defectA=\C=collaborate,D=defect\, R(a1,a2)=1R(a_1,a_2)=1 if a1=Ca_1=C or a2=Ca_2=C (at least one player collaborates) and R(a1,a2)=0R(a_1,a_2)=0 if a1=a2=Da_1=a_2=D (both defect), and c(ai)=0.4c(a_i)=0.4 if ai=Ca_i=C (cost of collaboration) and c(ai)=0.0c(a_i)=0.0 otherwise. We fix ϵ=0.2ε=0.2 and study the effect of the degree of risk aversion on the game in Fig.˜1. Without risk aversion, the game has one symmetric and two free-riding RQE. As the degree of risk aversion increases, players become increasingly collaborative and the degree of free-riding diminishes. Importantly, there is a threshold where the two free-riding RQEs disappear and “merge” into the symmetric one—a phenomenon known as pitchfork bifurcation. We validate this theorem through our experiments in both the Overcooked gridworld and the Tag environment. In both environments, we empirically observe that Nash agents naturally learn to free-ride at equilibrium which in turn severely degrades their ability to generalize to new agents. Our strategic risk-averse agents however tend to put more effort into the task and thus generalize better to new agents. Remark . In our ablation studies on the effects of risk aversion, we empirically observe a threshold-like phenomenon under which free riding disappears as risk aversion increases past a certain threshold. This seems to validate that the implications of Section˜4.2 hold beyond the simple classes of games covered by the theorem. 5 MARL Algorithm Design The theoretical benefits of strategic risk aversion—increased collaboration and no free-riding—prompt us to design scalable strategically risk-averse training algorithms for MARL. We do so in the general setting of policy optimization algorithms. For a more detailed mathematical background of risk aversion in MARL, we refer to Appendix C. 5.1 Meta-algorithm for Strategically Risk-averse Policy Optimization In standard policy optimization, each agent seeks a policy πθi _ _i, parametrized by parameters θi _i, that maximizes an objective function ℒi(θi,θ−i)L_i( _i, _-i), which is a function of the policies of all agents. Inspired by our strategically risk-averse utility (1), it is tempting to replace this objective function with infϕiℒi(θi,ϕi)+1τiKL(ϕi,θ−i)−ϵiH(θi), _ _iL_i( _i, _i)+ 1 _i *KL( _i, _-i)- _iH( _i), (4) where, to ease exposition, we slightly overload the notation of KL(ϕi,θ−i) *KL( _i, _-i) to be the (expected) KL-divergence between the induced policies across states (if the problem is one of MARL) . Unfortunately, the computation of this objective poses a significant computational challenge. The mere evaluation of this objective, as well as the computation of its gradient, requires solving a high-dimensional non-convex optimization problem—the infimum over ϕi _i—and is therefore computationally prohibitive. To bypass this complexity, we follow Mazumdar et al. (2024) and resort to an auxiliary game in which we augment the number of players. For each agent i, we replace the infimum in (4) by an adversary that aims to inflict maximum damage to agent i, while not deviating too much from the other agents’ policies, and is thus designed to solve the infimum in (4). As such, the objective of agent i is now a function of their parameters θi _i and those of their adversary: ℒi(θi,ϕi)+1τiKL(ϕi,θ−i)−ϵiH(θi),L_i( _i, _i)+ 1 _i *KL( _i, _-i)- _iH( _i), (5) where the KL term can now be dropped as it does not affect the agent’s parameters. The objective of the adversary agent, who controls the adversarial policy πϕi _ _i and whose goal is to minimize the objective of agent i, depends on the parameters of all agents and is therefore given by −ℒi(θi,ϕi)−1τiKL(ϕi,θ−i)+ϵiH(θi),-L_i( _i, _i)- 1 _i *KL( _i, _-i)+ _iH( _i), (6) where the entropy term can be dropped. While adversarial training has been proposed as a robustness technique, such methods are often unstable to train and result in over-conservative policies (Bukharin et al., 2023; Lauffer et al., 2025). The key insight resulting from strategic risk aversion is the idea to constrain the adversary to not deviate from the opponents policy (which is also evolving during training). This extra step stabilizes training. This framing provides us with a meta-algorithm for strategically risk-averse MARL. We now instantiate this algorithm in the case of PPO, also called independent PPO (IPPO) when deployed in multi-agent settings (Yu et al., 2022), and introduce Strategically Risk-averse Policy Optimization (SRPO), the first strategically risk-averse algorithm for MARL. 5.2 SRPO When using PPO (Schulman et al., 2017), the objective of agent i is given by ℒiIPPO(θi,θ−i)=ℒiCLIP(θi,θ−i)−ϵiH(θi)(oit), _i^IPPO( _i, _-i)=L_i^CLIP( _i, _-i)- _iH( _i)(o_i^t), (7) where the clipped surrogate objective is ℒiCLIP(θi,θ−i)= _i^CLIP( _i, _-i)= t[min(rit(θi)A^it(θi,θ−i), \>E_t [ (r_i^t( _i) A_i^t( _i, _-i), clip(rit(θi),1−δ,1+δ)A^it(θi,θ−i))], (r_i^t( _i),1-δ,1+δ) A_i^t( _i, _-i) ) ], with rit(θi)=πθi(ait|oit)πθi,old(ait|oit)r_i^t( _i)= _ _i(a_i^t|o_i^t) _ _i,old(a_i^t|o_i^t) being the importance sampling ratio, A^it(θi,θ−i) A_i^t( _i, _-i) the estimated advantage at time t (and depends on θ−i _-i as it is computed using samples of the policies of all agents), ϵiH(θi)(oit) _iH( _i)(o_i^t) a (negative) entropy regularizer, and oito_i^t the observation of agent i at time t. We can now follow the meta-algorithm above to derive strategically risk-averse proximal policy optimization. Specifically, the objective of agent i is ℒiSRPO(θi,ϕi)=ℒiCLIP(θi,ϕi)−ϵiH(θi)(oit).L_i^SRPO( _i, _i)=L_i^CLIP( _i, _i)- _iH( _i)(o_i^t). (8) Since the PPO objective already includes entropy regularization, the additional entropy term in (5) is superfluous. The objective of player i’s adversary is then ℒ¯iSRPO(ϕi,(θi,θ−i))=−ℒiCLIP(θi,ϕi)−1τiKL(ϕi,θ−i). L_i^SRPO( _i,( _i, _-i))=-L_i^CLIP( _i, _i)- 1 _i *KL( _i, _-i). (9) Overall, SRPO consists of running gradient descent on these objectives. We present pseudocode and details in Appendix C.2, and note here that the iteration structure and complexity of SRPO is very similar to that of IPPO. 6 Experiments We evaluate the proposed SRPO algorithm against IPPO on three cooperative multi-agent benchmarks and one LLM-based debate task. We use IPPO as our benchmark due to its emerging role as the dominant scalable algorithm for collaborative MARL across numerous benchmarks (Yu et al., 2022; Forkel and Foerster, 2025; de Witt et al., 2020). The experimental design closely follows the theoretical motivations developed earlier. Concretely, our experiments test the following claims: • Partner generalization. SRPO achieves higher and more stable cross-play performance than IPPO when paired with previously unseen partners. • Shared reward and free-riding. In cooperative tasks with private costs, SRPO converges to equilibria with higher shared reward and reduced free-riding behavior. • Scalability. SRPO remains practical in complex multi-agent settings when implemented with policy sharing and adversary sampling. We implement SRPO and IPPO in four representative environments: (1) a modified grid-world inspired by Overcooked AI (Carroll et al., 2019), which allows explicit modeling of both shared team rewards and private costs; (2) Tag (Lowe et al., 2017), a continuous-control coordination task; (3) 4-player Hanabi (Bard et al., 2020) with 3 colors and 3 ranks, a partially observed cooperative game requiring implicit coordination and communication used to study zero-shot coordination (Forkel and Foerster, 2025), and (4) an LLM-based multi-agent debate setting (Du et al., 2024; Park et al., 2025) on the GSM8K dataset (Cobbe et al., 2021) in which agents must collaborate to solve math problems. Training and evaluation. Across all environments, for an apples-to-apples comparison, the number of interactions with the environment for both SRPO and IPPO are kept the same. We evaluate partner generalization using cross-play performance, where a trained agent is paired with held-out partners not encountered during training. This evaluation makes the existence of free-riding-type policies extremely apparent: the policies lead to a distinct checker-board pattern, which emerges when free-riding agents are paired with one another and fail to achieve good performance. We consistently observe this phenomenon arising from IPPO-trained agents across environments. Ablation studies. Across environments, we also report results of ablation studies on both entropy and strategic risk aversion to further validate our theory, show how our results are robust to hyper-parameter selection, and highlight how entropy alone cannot guarantee generalization in general collaborative games despite positive empirical evidence in special cases (Forkel and Foerster, 2025). The code will be released upon publication, and the detailed setup is in Appendix D. Figure 2: Cross-play and ablation experiments in the overcooked environment. Each square represents the average reward across 10 episodes of length 128 for each pair of agents. Diagonal blocks represent the training performance of the agents. (i) We directly observe that IPPO (ϵ=0.1ε=0.1) learns to free-ride while SRPO (τ=10,ϵ=0.1)τ=10,ε=0.1) does not. Furthermore, mirroring Section˜4.1, we observe that SRPO yields higher utility strategies (i.e., risk improves performance). (i) Results of an ablation experiment, varying τ while holding ϵ=0.1ε=0.1. We empirically observe that free-riding completely disappears as risk aversion increases, mirroring the result in Section˜4.2. (i) Difference between Training Performance (TP) and Cross-play Performance (CP) (mean and standard deviation): the performance of IPPO drastically decreases, with lower average and larger standard deviation in cross-play, while the performance of SRPO is unaffected. 6.1 Overcooked Gridworld We construct an environment based onOvercooked Ruhdorfer et al. (2025) multi-agent benchmark to serve as a simple laboratory to verify our theory. In our Overcooked Gridworld, agents receive a shared team reward when an onion is picked up (+1+1) and placed into a pot (+10+10), while each agent incurs private costs for movement (−0.2-0.2) and collision (−2-2). This induces a canonical social dilemma: although the team objective is collaborative, each agent has an incentive to avoid costly effort and rely on the teammate to complete the task. Results are shown in Fig.˜2. SRPO reduces free-riding and exhibits higher performance. Empirically, IPPO always converges to a free-riding equilibrium where one teammate avoids movement but collects high reward due to the effort of their partner—evident in the characteristic checkerboard pattern observed in the top left block of Fig.˜2(i). In contrast, SRPO learns a policy in which both agents coordinate and contribute to the task. This behavior matches SRPO’s objective: because training explicitly optimizes performance against plausible adversarial partner deviations (Section˜5), non-contribution becomes risky. As we summarize in Fig.˜2(i), SRPO thus attains both higher performances and improved cross-play reliability, consistent with our theoretical predictions. Ablation study of the degree of strategic risk aversion τ. As shown in Fig.˜2(i), small values of τ and IPPO (τ=0τ=0) lead to free-riding and thus poor cross-play performance, reflecting reliance on fragile conventions and lack of effort on the part of agents. As τ increases, collaboration becomes more stable: policies converge to a consistent solution, free-riding disappears, and cross-play performance improves, validating that Theorem 4.2 holds more broadly. We also observe that SRPO is robust to choices of τ. Ablations show consistent gains over IPPO across orders of magnitude of τ. We make similar observations in ablation experiments in the Tag environment presented in Appendix D.6. 6.2 Tag In Tag, two chasers must coordinate to catch a runner. The chasers get a shared positive reward (+1) if a chaser collides with a runner. We first train runner policies independently via adversarial training, and then train chaser policies using either IPPO or SRPO against a fixed runner to keep the game collaborative. Cross-play performance is evaluated along two axes: (i) pairing trained chasers with an unseen chaser partner (teammate shift), and (i) evaluating chasers against a runner policy not observed during training (opponent shift). The results are shown in Fig.˜3. Figure 3: Cross-play performances of SRPO (τ=10,ϵ=0.01τ=10,ε=0.01) and IPPO (ϵ=0.01ε=0.01) agents in the Tag environment against a runner seen during training (i) and an unseen runner (i). Each square represents the average reward of two agents across 100 runs of length 100. IPPO does well in training environments (yet still clearly learns free-riding like policies), but their performance degrades drastically against an unseen runner. SRPO has slightly lower training performance but clearly learns a more generalizable policy. (i) Difference between Training Performance (TP) and Cross-play Performance (CP) (mean and standard deviation): the performance of IPPO drastically decreases, with lower average and larger standard deviation in cross-play, while the performance of SRPO is almost unaffected. SRPO learns more generalizable policies. We empirically observe that in this standard benchmark environment, IPPO-trained agents can learn high performing policies. However, we also observe that these policies often encode free-riding and can overfit to both the runner encountered during training and specific coordination conventions it develops with its training partner. While this can produce strong in-distribution performance (i.e., on the diagonal), it degrades sharply under either teammate or runner shifts, as seen in the sharp drops in performance as IPPO agents are played against each other, as we summarize in Fig.˜3(i). In contrast, SRPO has slightly worse performance in the training environment. This highlights that in some environments incorporating strategic risk aversion may sometimes require tradeoffs with performance. In the case of Tag, we can trace this back to the lack of the aggregative structure required for Section˜4.1. Nevertheless, SRPO attains consistently higher and more stable cross-play performance across both other partners (including free-riding IPPO agents) and runners, showing their potential to generalize more widely. 6.3 Hanabi Hanabi is a collaborative card game in which players must play cards in the correct sequence, with the twist that you can see everyone’s cards except your own. It is a canonical benchmark for collaboration in MARL (Bard et al., 2020). Following Lauffer et al. (2025), we consider a simplified game with 33 colors and 33 ranks. In this setting, agents may develop private communication protocols during training that fail to generalize to unseen teammates. We focus here on the 4-player variant to demonstrate the scalability of SRPO with the number of players, and present results on 2-player games in Appendix˜D. (a) (b) Figure 4: Cross-play performance of SRPO and IPPO agents in the Hanabi environment. We use policy sharing to validate the scalability of SRPO. During evaluation, we let agents 1 and 2 share a policy and agents 3 and 4 share a policy, enabling pairwise cross-play evaluation. In Fig.˜4(a), each square represents the average reward of the two agent groups across 100 runs, each of length 100. Fig.˜4(b) shows the differences between training performance (TP) and cross-play performance (CP) (mean and standard deviation) for both IPPO and SRPO. SRPO remains more robust when paired with an unseen partner. Here, we set the entropy coefficient to be ϵ=0.001ε=0.001 for both IPPO and SRPO, and τ=0.01τ=0.01 for SRPO. Scalability of SRPO. To scale SRPO to larger numbers of agents we make use of policy sharing—i.e., we share a single risk-averse policy across agents and maintain only one adversary policy, randomly assigning the adversarial role during training. The results are shown in Fig.˜4. where we observe that SRPO exhibits more stable cross-play performance than IPPO. 6.4 Multi-LLM-Agent Debate on GSM8K To conclude, we present a proof-of-concept on using SRPO in a language-based cooperative reasoning setting in which agents must collaborate to solve grade school math word problems (Cobbe et al., 2021) through structured debate. In this setup, two agents engage in a three-round iterative debate protocol. In the first round, each agent independently observes the question and produces its own reasoning and answer. In subsequent rounds, each agent observes the original question as well as both agents’ outputs from the previous round, and then refines its response. Successfully solving a problem therefore requires more than producing a correct answer in isolation: each agent must reinforce correct reasoning while remaining robust to potentially misleading or incorrect proposals from its teammate. This naturally induces a cooperative yet adversarial interaction, where robustness to the partner’s policy plays a critical role in achieving reliable joint performance. Both IPPO (the existing state-of-the-art) and SRPO agents are trained using multiple base language models, including Qwen2.5-0.5B-Instruct (Q0.5B) and Qwen2.5-3B-Instruct (Q3B) (Bai et al., 2025), as well as Qwen3-0.6B (Q0.6B) and Qwen3-4B-Instruct-2507 (Q4B) (Yang et al., 2025), using the verl training framework (Sheng et al., 2024). Here, we set the entropy coefficient to be ϵ=0ε=0 for both IPPO and SRPO, and τ=10τ=10 for SRPO. We evaluate performance along two complementary robustness dimensions. First, we measure cross-play performance between agents trained with the same method but using different model scales. Performance is quantified using joint accuracy, where a problem is counted as correct only if both agents produce the correct final answer. This metric directly reflects the ability of agents to coordinate reliably under policy heterogeneity. Second, we evaluate robustness to drastic partner shifts by pairing a trained Qwen agent with an untuned Llama 3.2-1B-Instruct model (Grattafiori et al., 2024). In this setting, we measure the accuracy of the trained agent alone, isolating its ability to maintain correct reasoning despite interacting with a potentially unreliable partner. Results are summarized in Tables˜1 and 2. Table 1: Cross-play performance (joint accuracy) across model scale combinations. Percentage improvement is computed relative to IPPO. Across all model combinations, SRPO consistently outperforms IPPO in terms of cross-play performance. Method Q0.5B&Q0.6B Q0.5B&Q3B Q0.5B&Q4B Q0.6B&Q3B Q0.6B&Q4B Q3B&Q4B SRPO 0.622 0.686 0.873 0.616 0.848 0.732 IPPO 0.605 0.651 0.846 0.573 0.711 0.709 Improvement (%) +2.81% +5.38% +3.19% +7.50% +19.27% +3.24% Table 2: Performance (the accuracy of the trained agent itself) when paired with an untuned Llama 3.2-1B-Instruct model. Percentage improvement is computed relative to IPPO. Across all model sizes, SRPO agents consistently outperform IPPO, indicating that SRPO endows the trained the power to be robust to the teammate. Method Q0.5B Q0.6B Q3B Q4B SRPO 0.405 0.671 0.632 0.917 IPPO 0.378 0.587 0.552 0.901 Improvement (%) +7.14% +14.31% +14.49% +1.78% SRPO extends to LLM-based multi-agent systems. As shown in Table˜1, SRPO consistently improves joint accuracy over IPPO across all cross-play combinations, with gains of up to 19.27%19.27\%. This demonstrates that SRPO enhances coordination robustness across heterogeneous policies, enabling agents to reliably reach correct joint outcomes. Furthermore, as shown in Table˜2, SRPO substantially improves individual accuracy when paired with an untuned partner, achieving gains of up to 14.49%14.49\%. This indicates that SRPO-trained agents are significantly more robust to severe partner mismatch, maintaining correct reasoning even when interacting with unreliable teammates. Together, these results highlight SRPO’s effectiveness in promoting robust cooperative reasoning in multi-agent language environments. In Section˜D.5, we provide evidence that SRPO primarily improves collaboration, rather than the individual reasoning ability of each agent. 7 Conclusion In this work, we introduce strategic risk aversion as a principled inductive bias for learning collaborative policies that generalize to unseen partners. Using the risk-averse quantal response equilibrium (RQE) framework, we show that strategic risk aversion can both increase contributions to shared rewards and eliminate free-riding at equilibrium, demonstrating that robustness need not be purely conservative. Guided by these insights, we propose Strategically Risk-Averse Policy Optimization (SRPO), a scalable modification of standard policy optimization. Across cooperative benchmarks (Overcooked, Tag, Hanabi) and an LLM-based debate task on GSM8K, SRPO yields improved robustness to heterogeneous or unreliable teammates, including across model scales. Future work includes extending strategic risk aversion to broader agentic AI settings such as human–AI collaboration and multi-agent foundation model systems. References Z. Ahmed, N. Le Roux, M. Norouzi, and D. Schuurmans (2019) Understanding the impact of entropy on policy optimization. In International Conference on Machine Learning, p. 151–160. Cited by: §3. S. Bai, K. Chen, X. Liu, J. Wang, W. Ge, S. Song, K. Dang, P. Wang, S. Wang, J. Tang, et al. (2025) Qwen2.5-VL technical report. arXiv preprint arXiv:2502.13923. Cited by: §D.5, §6.4. N. Bard, J. N. Foerster, S. Chandar, N. Burch, M. Lanctot, H. F. Song, E. Parisotto, V. Dumoulin, S. Moitra, E. Hughes, I. Dunning, S. Mourad, H. Larochelle, M. G. Bellemare, and M. Bowling (2020) The Hanabi challenge: a new frontier for AI research. Artificial Intelligence 280 (C). Cited by: §6.3, §6. S. Barrett, N. Agmon, N. Hazon, S. Kraus, and P. Stone (2014) Communicating with unknown teammates. In International Conference on Autonomous Agents and Multi-Agent Systems, p. 1433–1434. Cited by: §2. A. Bukharin, Y. Li, Y. Yu, Q. Zhang, Z. Chen, S. Zuo, C. Zhang, S. Zhang, and T. Zhao (2023) Robust multi-agent reinforcement learning via adversarial regularization: theoretical foundation and stable algorithms. In Advances in Neural Information Processing Systems, Cited by: §5.1. Y. Cao, W. Yu, W. Ren, and G. Chen (2013) An overview of recent progress in the study of distributed multi-agent coordination. IEEE Transactions on Industrial Informatics 9 (1), p. 427–438. Cited by: Appendix A. M. Carroll, R. Shah, M. K. Ho, T. Griffiths, S. Seshia, P. Abbeel, and A. Dragan (2019) On the utility of learning about humans for human-ai coordination. In Advances in Neural Information Processing Systems, H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (Eds.), Vol. 32, p. . External Links: Link Cited by: Appendix A, §1, §2, §6. M. Cemri, M. Z. Pan, S. Yang, L. A. Agrawal, B. Chopra, R. Tiwari, K. Keutzer, A. Parameswaran, D. Klein, K. Ramchandran, M. Zaharia, J. E. Gonzalez, and I. Stoica (2025) Why do multi-agent LLM systems fail?. arXiv preprint arXiv:2503.13657. External Links: 2503.13657 Cited by: §1. K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, et al. (2021) Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168. Cited by: §6.4, §6. L. C. Corchón (1994) Comparative statics for aggregative games the strong concavity case. Mathematical Social Sciences 28 (3), p. 151–165. Cited by: §4.1. A. Dafoe, Y. Bachrach, G. Hadfield, E. Horvitz, K. Larson, and T. Graepel (2021) Cooperative AI: machines must learn to find common ground. Nature 593 (7857), p. 33–36. Cited by: §1. C. S. de Witt, T. Gupta, D. Makoviichuk, V. Makoviychuk, P. H. S. Torr, M. Sun, and S. Whiteson (2020) Is independent learning all you need in the starcraft multi-agent challenge?. arXiv preprint arXiv:2011.09533. External Links: 2011.09533 Cited by: Appendix A, §6. T. Dizdarević, R. Hammond, T. Gessler, A. Calinescu, J. Cook, M. Gallici, A. Lupu, and J. N. Foerster (2025) Ad-hoc human-AI coordination challenge. In International Conference on Machine Learning, Vol. 267, p. 13900–13937. Cited by: §2. Y. Du, S. Li, A. Torralba, J. B. Tenenbaum, and I. Mordatch (2024) Improving factuality and reasoning in language models through multiagent debate. In International Conference on Machine Learning, Cited by: §6. H. Eriksson, D. Basu, M. Alibeigi, and C. Dimitrakakis (2022) Risk-sensitive bayesian games for multi-agent reinforcement learning under policy uncertainty. arXiv preprint arXiv:2203.10045. Cited by: Appendix A. F. Facchinei and J. Pang (2003) Finite-dimensional variational inequalities and complementarity problems. Springer. Cited by: §B.1.1. H. Föllmer and A. Schied (2002) Convex measures of risk and trading constraints. Finance and stochastics 6, p. 429–447. Cited by: §B.1.1, §3. J. Forkel and J. Foerster (2025) Entropy is all you need for inter-seed cross-play in hanabi. arXiv preprint arXiv:2511.22581. External Links: 2511.22581 Cited by: Appendix A, §D.6, §1, §2, §6, §6, §6. S. Ganesh, N. Vadori, M. Xu, H. Zheng, P. Reddy, and M. Veloso (2019) Reinforcement learning for market making in a multi-agent dealer market. arXiv preprint arXiv:1911.05892. Cited by: Appendix A. J. K. Goeree, C. A. Holt, and T. R. Palfrey (2003) Risk averse behavior in generalized matching pennies games. Games and Economic Behavior 45 (1), p. 97–113. Cited by: §1, §2. J. K. Goeree and T. Offerman (2002) Efficiency in auctions with private and common values: an experimental study. American Economic Review 92 (3), p. 625–643. Cited by: §2. C. Gollier (2001) The economics of risk and time. MIT press. Cited by: §2. A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan, et al. (2024) The Llama 3 herd of models. arXiv preprint arXiv:2407.21783. Cited by: §6.4. H. Hu, A. Lerer, B. Cui, L. Pineda, N. Brown, and J. Foerster (2021) Off-belief learning. In International Conference on Machine Learning, p. 4369–4379. Cited by: §2. H. Hu, A. Lerer, A. Peysakhovich, and J. Foerster (2020) “Other-play” for zero-shot coordination. In International Conference on Machine Learning, p. 4399–4410. Cited by: Appendix A, §1, §1, §2. T. Kruse, A. K. Pandey, R. Alami, and A. Kirsch (2013) Human-aware robot navigation: a survey. Robot. Auton. Syst. 61 (12), p. 1726–1743. Cited by: §1. G. Lan (2023) Policy mirror descent for reinforcement learning: linear convergence, new sampling complexity, and generalized problem classes. Mathematical programming 198 (1), p. 1059–1106. Cited by: §C.3. N. Lanzetti, S. Bolognani, and F. Dörfler (2025a) First-order conditions for optimization in the Wasserstein space. SIAM Journal on Mathematics of Data Science 7 (1), p. 274–300. Cited by: §B.1.1, §B.1.1, §B.1.1. N. Lanzetti, S. Fricker, S. Bolognani, F. Dörfler, and D. Paccagnan (2025b) Strategically robust game theory via optimal transport. arXiv preprint arXiv:2507.15325. Cited by: Appendix A, §2. N. Lanzetti, A. Terpin, and F. Dörfler (2024) Variational analysis in the Wasserstein space. arXiv preprint arXiv:2406.10676. Cited by: §B.1.1. N. Lauffer, A. Shah, M. Carroll, S. A. Seshia, S. Russell, and M. D. Dennis (2025) Robust and diverse multi-agent learning via rational policy gradient. In Advances in Neural Information Processing Systems, Cited by: §D.4, §5.1, §6.3. J. Z. Leibo, V. Zambaldi, M. Lanctot, J. Marecki, and T. Graepel (2017) Multi-agent reinforcement learning in sequential social dilemmas. In Conference on Autonomous Agents and MultiAgent Systems, Richland, SC, p. 464–473. Cited by: §1. A. Lerer and A. Peysakhovich (2019) Learning existing social conventions via observationally augmented self-play. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society, AIES ’19, New York, NY, USA, p. 107–114. Cited by: §1. Y. Liang, D. Chen, A. Gupta, S. S. Du, and N. Jaques (2024) Learning to cooperate with humans using generative agents. In Advances in Neural Information Processing Systems, Cited by: Appendix A. B. Liu, Z. Pu, Y. Pan, J. Yi, Y. Liang, and D. Zhang (2023) Lazy agents: a new perspective on solving sparse reward problem in multi-agent reinforcement learning. In International Conference on Machine Learning, p. 21937–21950. Cited by: §1. R. Lowe, Y. Wu, A. Tamar, J. Harb, P. Abbeel, and I. Mordatch (2017) Multi-agent actor-critic for mixed cooperative-competitive environments. In Advances in Neural Information Processing Systems, NIPS’17, Red Hook, NY, USA, p. 6382–6393. Cited by: §6. A. Lupu, B. Cui, H. Hu, and J. Foerster (2021) Trajectory diversity for zero-shot coordination. In International Conference on Machine Learning, p. 7204–7213. Cited by: §2. E. Mazumdar, K. Panaganti, and L. Shi (2024) Tractable equilibrium computation in Markov games through risk aversion. In International Conference on Learning Representations, Cited by: Appendix A, §1, §1, §2, §3, §3, §3, §5.1. R. D. McKelvey and T. R. Palfrey (1995) Quantal response equilibria for normal form games. Games and economic behavior 10 (1), p. 6–38. Cited by: §3. Meta Fundamental AI Research Diplomacy Team (FAIR)†, A. Bakhtin, N. Brown, E. Dinan, G. Farina, C. Flaherty, D. Fried, A. Goff, J. Gray, H. Hu, et al. (2022) Human-level play in the game of diplomacy by combining language models with strategic reasoning. Science 378 (6624), p. 1067–1074. Cited by: Appendix A. D. Muglich, C. Schroeder de Witt, E. van der Pol, S. Whiteson, and J. Foerster (2022) Equivariant networks for zero-shot coordination. In Advances in Neural Information Processing Systems, Vol. 35, p. 6410–6423. Cited by: Appendix A. D. Paccagnan, B. Gentile, F. Parise, M. Kamgarpour, and J. Lygeros (2018) Nash and Wardrop equilibria in aggregative games with coupling constraints. IEEE Transactions on Automatic Control 64 (4), p. 1373–1388. Cited by: §4.1. C. Park, S. Han, X. Guo, A. E. Ozdaglar, K. Zhang, and J. Kim (2025) MAPoRL: multi-agent post-co-training for collaborative large language models with reinforcement learning. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 30215–30248. Cited by: §6. W. Qiu, X. Wang, R. Yu, R. Wang, X. He, B. An, S. Obraztsova, and Z. Rabinovich (2021) RMIX: learning risk-sensitive policies for cooperative reinforcement learning agents. In Advances in Neural Information Processing Systems, Vol. 34, p. 23049–23062. Cited by: Appendix A. A. Rahman, E. Fosong, I. Carlucho, and S. V. Albrecht (2023) Generating teammates for training robust ad hoc teamwork agents via best-response diversity. Transactions on Machine Learning Research. Note: Cited by: §2. C. Ruhdorfer, M. Bortoletto, A. Penzkofer, and A. Bulling (2025) The overcooked generalisation challenge: evaluating cooperation with novel partners in unknown environments using unsupervised environment design. Transactions on Machine Learning Research. Cited by: §1, §6.1. J. Schulman, S. Levine, P. Abbeel, M. Jordan, and P. Moritz (2015) Trust region policy optimization. In International Conference on Machine Learning, p. 1889–1897. Cited by: §C.5, §C.5. J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov (2017) Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. External Links: 1707.06347 Cited by: Appendix A, §C.5, §5.2. S. Shen, C. Ma, C. Li, W. Liu, Y. Fu, S. Mei, X. Liu, and C. Wang (2023) RiskQ: risk-sensitive multi-agent reinforcement learning value factorization. In Advances in Neural Information Processing Systems, Vol. 36, p. 34791–34825. Cited by: Appendix A. G. Sheng, C. Zhang, Z. Ye, X. Wu, W. Zhang, R. Zhang, Y. Peng, H. Lin, and C. Wu (2024) HybridFlow: a flexible and efficient rlhf framework. arXiv preprint arXiv: 2409.19256. Cited by: §D.5, §6.4. L. Shi, E. Mazumdar, Y. Chi, and A. Wierman (2024) Sample-efficient robust multi-agent reinforcement learning in the face of environmental uncertainty. In International Conference on Machine Learning, Cited by: Appendix A. O. Slumbers, D. H. Mguni, S. B. Blumberg, S. M. Mcaleer, Y. Yang, and J. Wang (2023) A game-theoretic framework for managing risk in multi-agent systems. In International Conference on Machine Learning, p. 32059–32087. Cited by: Appendix A, §2. P. Stone, G. Kaminka, S. Kraus, and J. Rosenschein (2010) Ad hoc autonomous agent teams: collaboration without pre-coordination. In Proceedings of the AAAI Conference On Artificial Intelligence, Vol. 24, p. 1504–1509. Cited by: §1, §2. P. Sunehag, G. Lever, A. Gruslys, W. M. Czarnecki, V. Zambaldi, M. Jaderberg, M. Lanctot, N. Sonnerat, J. Z. Leibo, K. Tuyls, and T. Graepel (2018) Value-decomposition networks for cooperative multi-agent learning based on team reward. In Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, p. 2085–2087. Cited by: §1. J. K. Terry, B. Black, N. Grammel, M. Jayakumar, A. Hari, R. Sullivan, L. Santos, R. Perez, C. Horsch, C. Dieffendahl, N. L. Williams, Y. Lokesh, and P. Ravi (2021) PettingZoo: gym for multi-agent reinforcement learning. arXiv preprint arXiv:2009.14471. External Links: 2009.14471 Cited by: §D.3. K. Tran, D. Dao, M. Nguyen, Q. Pham, B. O’Sullivan, and H. D. Nguyen (2025) Multi-agent collaboration mechanisms: a survey of LLMs. arXiv preprint arXiv:2501.06322. Cited by: Appendix A. J. Treutlein, M. Dennis, C. Oesterheld, and J. Foerster (2021) A new formalism, method and open issues for zero-shot coordination. In International Conference on Machine Learning, p. 10413–10423. Cited by: Appendix A. J.N. Tsitsiklis (1984) Problems in decentralized decision making and computation. Massuchusetts Institute of Technology. Cited by: Appendix A. O. Vinyals, I. Babuschkin, W. M. Czarnecki, M. Mathieu, A. Dudzik, J. Chung, D. H. Choi, R. Powell, T. Ewalds, P. Georgiev, J. Oh, D. Horgan, M. Kroiss, I. Danihelka, A. Huang, L. Sifre, T. Cai, J. P. Agapiou, M. Jaderberg, A. S. Vezhnevets, R. Leblond, T. Pohlen, V. Dalibard, D. Budden, Y. Sulsky, J. Molloy, T. L. Paine, C. Gulcehre, Z. Wang, T. Pfaff, Y. Wu, R. Ring, D. Yogatama, D. Wünsch, K. McKinney, O. Smith, T. Schaul, T. Lillicrap, K. Kavukcuoglu, D. Hassabis, C. Apps, and D. Silver (2019) Grandmaster level in StarCraft I using multi-agent reinforcement learning. Nature 575 (7782), p. 350–354. Cited by: §2. X. Wang, S. Zhang, W. Zhang, W. Dong, J. Chen, Y. Wen, and W. Zhang (2024) ZSC-eval: an evaluation toolkit and benchmark for multi-agent zero-shot coordination. In Advances in Neural Information Processing Systems Datasets and Benchmarks Track, Cited by: §2. Q. Wu, G. Bansal, J. Zhang, Y. Wu, B. Li, E. Zhu, L. Jiang, X. Zhang, S. Zhang, J. Liu, A. H. Awadallah, R. W. White, D. Burger, and C. Wang (2024) AutoGen: enabling next-gen LLM applications via multi-agent conversations. In First Conference on Language Modeling, Cited by: §1. A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, et al. (2025) Qwen3 technical report. arXiv preprint arXiv:2505.09388. Cited by: §D.5, §6.4. A. Yekkehkhany, T. Murray, and R. Nagi (2020) Risk-averse equilibrium for games. arXiv preprint arXiv:2002.08414. Cited by: Appendix A. C. Yu, J. Gao, W. Liu, B. Xu, H. Tang, J. Yang, Y. Wang, and Y. Wu (2023) Learning zero-shot cooperation with humans, assuming humans are biased. In International Conference on Learning Representations, Cited by: §2. C. Yu, A. Velu, E. Vinitsky, J. Gao, Y. Wang, A. Bayen, and Y. Wu (2022) The surprising effectiveness of PPO in cooperative multi-agent games. In Advances in Neural Information Processing Systems, Vol. 35, p. 24611–24624. Cited by: Appendix A, §5.1, §6. K. Zhang, T. SUN, Y. Tao, S. Genc, S. Mallya, and T. Basar (2020) Robust multi-agent reinforcement learning with model uncertainty. In Advances in Neural Information Processing Systems, Vol. 33, p. 10571–10583. Cited by: Appendix A. Y. Zhang and E. Mazumdar (2025) Convergent Q-learning for infinite-horizon general-sum Markov games through behavioral economics. In 2025 IEEE 64th Conference on Decision and Control (CDC), p. 5899–5904. Cited by: §1, §2. Z. Zhang, X. Li, Y. Lin, H. Liu, R. Chandradevan, L. Wu, M. Lin, F. Wang, X. Tang, Q. He, and S. Wang (2025) Unlocking the power of multi-agent LLM for reasoning: from lazy agents to deliberation. arXiv preprint arXiv:2511.02303. External Links: 2511.02303 Cited by: §1. R. Zhao, J. Song, Y. Yuan, H. Hu, Y. Gao, Y. Wu, Z. Sun, and W. Yang (2023) Maximum entropy population-based training for zero-shot human-AI coordination. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 37, p. 6145–6153. Cited by: §2. Appendix A Further Discussion on Related Work In this section, we discuss a broader set of related work to better position our work in the landscape of collaborative multi-agent learning and risk-averse/robust MARL. Collaborative MARL and Partner Generalization Research into solving collaborative decision-making problems using MARL has a long history (Cao et al., 2013), originating from a desire for solving decentralized decision-making problems (Tsitsiklis, 1984). Far from fading, these issues remain at the fore of research interest with the advent of powerful AI systems and a desire to build large teams of AI agents (Tran et al., 2025) for solving collaborative tasks. Emerging from this literature, is a general consensus that classic single-agent policy optimization algorithms like PPO (Schulman et al., 2017), used in a decentralized way, can yield state-of-the-art or near state-of-the-art performance without needing to be overly specialized to the multi-agent nature of the problem (Yu et al., 2022; de Witt et al., 2020; Forkel and Foerster, 2025). Despite such results, learned policies resulting from such approaches can fail to generalize to unseen agents (Forkel and Foerster, 2025) and new environments. We validate this through our experiments and further observe that these state-of-the-art algorithms are consistently prone to learning free-riding strategies. Beyond the lines of work mentioned in Section 2, other important lines of work on addressing the partner generalization problem seek to avoid brittle coordination conventions by learning symmetry-invariant or partner-agnostic policies (Hu et al., 2020; Treutlein et al., 2021; Muglich et al., 2022). These often require strong structural assumptions and may not scale to complex collaborative environments. Another set of approaches attempt to achieve robustness by training against partner policies derived from human data (Carroll et al., 2019; Meta Fundamental AI Research Diplomacy Team (FAIR)† et al., 2022; Liang et al., 2024), but their effectiveness is constrained by data availability and coverage. Neither of these approaches are fully agnostic to the structure of the underlying problem and are thus not general principles that can be broadly applied across different tasks. Consequently, it remains unclear how to systematically achieve partner generalization without relying on human data or ad-hoc constructions. In this paper we showed that strategic risk aversion gives us one potential approach. Furthermore, we show that it can readily be combined with policy optimization algorithms like PPO, resulting in scalable and performant algorithms for learning generalizable policies. Robustness and Risk Aversion in MARL A second related line of work is the emerging literature on robust and risk-averse MARL (Zhang et al., 2020; Shi et al., 2024; Yekkehkhany et al., 2020; Slumbers et al., 2023; Qiu et al., 2021). While most of the papers study robustness or risk aversion to changes in the underlying environment (Eriksson et al., 2022; Ganesh et al., 2019; Qiu et al., 2021; Shen et al., 2023; Shi et al., 2024; Zhang et al., 2020) or to tail risks in large populations of agents (Yekkehkhany et al., 2020), in this work we study strategic risk aversion. Strategic risk aversion and robustness yield solutions that are more amenable to computation than classic game-theoretic solution concepts (Mazumdar et al., 2024), ensure robustness to unseen behaviors of the other agents, and sometimes even induce a coordination effect between the agents (Lanzetti et al., 2025b)—an empirical observation that we make rigorous in Section 4. Despite the prior work in this area, the broader benefits of the concept in collaborative games are unknown. This is the problem that we study in this paper. Appendix B Proofs for Section˜4 In this section, we provide the proofs of our two main theorems, Section˜4.1 and Section˜4.2, as well as the derivation of Section˜4.1 used in Fig.˜1. B.1 Proof of Section˜4.1 We begin with the proof of Section˜4.1. We first show that there exists a RQE in the space of Gaussian strategies in the aggregative games studied in this section. We then provide the proof of a more general version of Section˜4.1. B.1.1 Preliminaries Before proving Section˜B.1.1, we study RQE for quadratic games. We start with the risk-averse quantal best response. We conduct our analysis in slightly more general settings where the players’ utilities are ui(ai,a−i)=12⟨[aia−i],[Hi,iHi,−iH−i,iH−i,−i][aia−i]⟩+⟨[hih−i],[aia−i]⟩,u_i(a_i,a_-i)= 12 bmatrixa_i\\ a_-i bmatrix, bmatrixH_i,i&H_i,-i\\ H_-i,i&H_-i,-i bmatrix bmatrixa_i\\ a_-i bmatrix + bmatrixh_i\\ h_-i bmatrix, bmatrixa_i\\ a_-i bmatrix , (10) where the matrix [Hi,iHi,−iH−i,iH−i,−i] bmatrixH_i,i&H_i,-i\\ H_-i,i&H_-i,-i bmatrix is assumed to be symmetric negative semidefinite, and Hi,iH_i,i is assumed to be symmetric negative definite for all i. Throughout this section, with slight abuse of notation, we denote by x−ix_-i the product measure of the mixed strategies of all players except i. Lemma (risk-averse quantal best response). Let x−ix_-i be Gaussian with mean m−im_-i and covariance matrix Σ−i≻0 _-i 0. Then, the risk-averse quantal best response, defined as the maximizer of Uiτi,ϵi(xi,x−i)U_i _i, _i(x_i,x_-i), is uniquely given by a Gaussian mixed strategy with the following covariance matrix and mean: Σi _i =−εiHi,i−1 =- _iH_i,i^-1 mi m_i =(−Hi,i+Hi,−iP−i−1Hi,−i⊤)−1(hi+Hi,−iP−i−1(1τiΣ−i−1m−i−h−i)), = (-H_i,i+H_i,-iP_-i^-1H_i,-i )^-1 (h_i+H_i,-iP_-i^-1 ( 1 _i _-i^-1m_-i-h_-i ) ), where P−i=1τiΣ−i−1+H−i,−iP_-i= 1 _i _-i^-1+H_-i,-i, provided that P−iP_-i is positive definite. If instead P−iP_-i is not positive definite, then risk is infinite. Proof. To start, recall that the duality representation of the entropic risk measure (e.g., see Föllmer and Schied (2002)) gives supp∈Δ(ℝn(N−1))ai∼xia−i∼x−i[−ui(ai,a−i)]−1τiKL(p,x−i)=1τilog⁡ai∼xia−i∼x−i[exp⁡(−τiui(ai,a−i))]. _p∈ (R^n(N-1))E_ subarrayca_i x_i\\ a_-i x_-i subarray[-u_i(a_i,a_-i)]- 1 _i *KL(p,x_-i)= 1 _i _ subarrayca_i x_i\\ a_-i x_-i subarray[ (- _iu_i(a_i,a_-i) )]. Thus, we can reformulate the optimization problem for best responses as follows: supxi∈Δ(ℝn) _x_i∈ (R^n) Uiτi,ϵi(xi,x−i) U_i _i, _i(x_i,x_-i) =−infxi∈Δ(ℝn)−Uiτi,ϵi(xi,x−i) =- _x_i∈ (R^n)-U_i _i, _i(x_i,x_-i) =−infxi∈Δ(ℝn)supp∈Δ(ℝn(N−1))ai∼xia−i∼x−i[−ui(ai,a−i)]−1τiKL(p,x−i)+ϵiH(xi) =- _x_i∈ (R^n) _p∈ (R^n(N-1))E_ subarrayca_i x_i\\ a_-i x_-i subarray[-u_i(a_i,a_-i)]- 1 _i *KL(p,x_-i)+ _iH(x_i) =−infxi∈Δ(ℝn)1τilog⁡a−i∼x−i[exp⁡(−τi(12⟨a−i,H−i,−ia−i⟩+ai∼xi[⟨ai,Hi,−ia−i⟩]+⟨h−i,a−i⟩))] =- _x_i∈ (R^n) 1 _i _a_-i x_-i [ (- _i( 12 a_-i,H_-i,-ia_-i +E_a_i x_i [ a_i,H_i,-ia_-i ]+ h_-i,a_-i ) ) ] −ai∼xi[12⟨ai,Hi,iai⟩+⟨hi,ai⟩]+ϵiH(xi) -E_a_i x_i [ 12 a_i,H_i,ia_i + h_i,a_i ]+ _iH(x_i) =supxi∈Δ(ℝn)−1τilog⁡a−i∼x−i[exp⁡(−τi(12⟨a−i,H−i,−ia−i⟩+ai∼xi[⟨ai,Hi,−ia−i⟩]+⟨h−i,a−i⟩))] = _x_i∈ (R^n)- 1 _i _a_-i x_-i [ (- _i( 12 a_-i,H_-i,-ia_-i +E_a_i x_i [ a_i,H_i,-ia_-i ]+ h_-i,a_-i ) ) ] +ai∼xi[12⟨ai,Hi,iai⟩+⟨hi,ai⟩]−ϵiH(xi) +E_a_i x_i [ 12 a_i,H_i,ia_i + h_i,a_i ]- _iH(x_i) =supxi∈Δ(ℝn)−1τilog⁡a−i∼x−i[exp⁡(−τi(12⟨a−i,H−i,−ia−i⟩+⟨mi,Hi,−ia−i⟩+⟨h−i,a−i⟩))] = _x_i∈ (R^n)- 1 _i _a_-i x_-i [ (- _i ( 12 a_-i,H_-i,-ia_-i + m_i,H_i,-ia_-i + h_-i,a_-i ) ) ] +ai∼xi[12⟨ai,Hi,iai⟩+⟨hi,ai⟩]−ϵiH(xi). +E_a_i x_i [ 12 a_i,H_i,ia_i + h_i,a_i ]- _iH(x_i). When x−ix_-i is Gaussian with mean m−im_-i and variance Σ−i _-i, the expression simplifies to a−i∼x−i _a_-i x_-i [exp⁡(−τi(12⟨a−i,H−i,−ia−i⟩+⟨ai,Hi,−ia−i⟩+⟨h−i,a−i⟩))] [ (- _i ( 12 a_-i,H_-i,-ia_-i + a_i,H_i,-ia_-i + h_-i,a_-i ) ) ] =1(2π)ndet⁡(Σ−i)∫ℝnexp(−12⟨a−i−m¯,Σ¯−i−1(a−i−m¯)⟩ = 1 (2π)^n ( _-i) _R^n (- 12 a_-i- m, _-i^-1(a_-i- m) . +12⟨m¯−i,Σ¯−i−1m¯−i⟩−12⟨m−i,Σ−i−1m−i⟩)da−i .+ 12 m_-i, _-i^-1 m_-i - 12 m_-i, _-i^-1m_-i )da_-i =(2π)ndet⁡(Σ¯−i)(2π)ndet⁡(Σ−i)exp⁡(12⟨m¯−i,Σ¯−im¯−i⟩−12⟨m−i,Σ−im−i⟩) = (2π)^n ( _-i)(2π)^n ( _-i) ( 12 m_-i, _-i m_-i - 12 m_-i, _-im_-i ) =1det⁡(Σ−iΣ¯−i−1)exp⁡(12⟨m¯−i,Σ¯−i−1m¯−i⟩−12⟨m−i,Σ−i−1m−i⟩), = 1 ( _-i _-i^-1) ( 12 m_-i, _-i^-1 m_-i - 12 m_-i, _-i^-1m_-i ), where Σ¯−i−1=Σ−i−1+τiH−i,−i=τiP−i _-i^-1= _-i^-1+ _iH_-i,-i= _iP_-i and m¯−i=Σ¯(Σ−i−1m−i−τi(Hi,−i⊤mi+h−i)) m_-i= ( _-i^-1m_-i- _i(H_i,-i m_i+h_-i)). Moreover, det⁡(Σ−iΣ¯−i−1)=det⁡(I+τiΣ−iH−i,−i). ( _-i _-i^-1)= (I+ _i _-iH_-i,-i). Overall, we therefore have log⁡a−i∼x−i[exp⁡(−τi(12⟨a−i,H−i,−ia−i⟩+⟨ai,Hi,−ia−i⟩+⟨h−i,a−i⟩))]=−12log⁡det⁡(I+τiΣ−iH−i,−i)+12⟨m¯−i,Σ¯−i−1m¯−i⟩−12⟨m−i,Σ−i−1m−i⟩. _a_-i x_-i [ (- _i ( 12 a_-i,H_-i,-ia_-i + a_i,H_i,-ia_-i + h_-i,a_-i ) ) ]\\ =- 12 (I+ _i _-iH_-i,-i)+ 12 m_-i, _-i^-1 m_-i - 12 m_-i, _-i^-1m_-i . We can now plug this expression into the optimization problem for best responses and get supxi∈Δ(ℝn)Uiτi,ϵi(xi,x−i)=supxi∈Δ(ℝn) _x_i∈ (R^n)U_i _i, _i(x_i,x_-i)= _x_i∈ (R^n) 12τilog⁡det⁡(I+τiΣ−iH−i,−i)−12τi⟨m¯−i,Σ¯−i−1m¯−i⟩+12τi⟨m−i,Σ−i−1m−i⟩ 12 _i (I+ _i _-iH_-i,-i)- 12 _i m_-i, _-i^-1 m_-i + 12 _i m_-i, _-i^-1m_-i +ai∼xi[12⟨ai,Hi,iai⟩+⟨hi,ai⟩]−ϵiH(xi). +E_a_i x_i [ 12 a_i,H_i,ia_i + h_i,a_i ]- _iH(x_i). This is an unconstrained optimization problem in the space of probability measures. We will use the first-order necessary conditions in the Wasserstein space in Lanzetti et al. (2024, 2025a) to construct an optimal solution and then use the sufficient conditions to establish optimality. Using Theorem 3.2 in Lanzetti et al. (2025a), we can set the Wasserstein gradient to 0 to obtain that, at optimality, the following first-order necessary condition must hold: 1τi(−τiHi,−iΣ¯−i)Σ¯−i−1m¯−Hi,iai−hi+ϵi∇ρ(ai)ρ(ai)=0 for xi-almost all ai∈ℝn, 1 _i(- _iH_i,-i _-i) _-i^-1 m-H_i,ia_i-h_i+ _i ∇ρ(a_i)ρ(a_i)=0 for $x_i$-almost all $a_i ^n$, (11) where ρ is the density of the optimal xix_i (if it exists). The first term follows from a generalization of the chain rule (Proposition 2.16 in Lanzetti et al. (2025a)), while the other terms follow from the well-known Wasserstein gradients of expected values (Proposition 2.20 in Lanzetti et al. (2025a)) and of entropy (Example 2.27 in Lanzetti et al. (2025a)). We now make the “ansatz” that xix_i is Gaussian with mean mim_i and covariance Σi _i. Since xix_i is Gaussian, we have ∇ρ(ai)ρ(ai)=∇(−12⟨ai−mi,Σi−1(ai−mi)⟩)=−Σi−1(ai−mi). ∇ρ(a_i)ρ(a_i)=∇ (- 12 a_i-m_i, _i^-1(a_i-m_i) )=- _i^-1(a_i-m_i). Thus, together with the expression for m¯−i m_-i, (11) reduces −Hi,−iΣ¯−i(Σ−i−1m−i−τi(Hi,−i⊤mi+h−i))−Hi,iai−hi−ϵiΣi−1(ai−mi)=0-H_i,-i _-i( _-i^-1m_-i- _i(H_i,-i m_i+h_-i))-H_i,ia_i-h_i- _i _i^-1(a_i-m_i)=0 (12) We study mean and covariance separately. • Mean: We can then integrate (12) with respect to xix_i to obtain ∫ℝn−Hi,−iΣ¯−i(Σ−i−1m−i−τi(Hi,−i⊤mi+h−i))−Hi,iai−hi−ϵiΣi−1(ai−mi)dxi(ai)=0. _R^n-H_i,-i _-i( _-i^-1m_-i- _i(H_i,-i m_i+h_-i))-H_i,ia_i-h_i- _i _i^-1(a_i-m_i) x_i(a_i)=0. This yields −Hi,−iΣ¯−i(Σ−i−1m−i−τi(Hi,−i⊤mi+h−i))−(Hi,i+Qi)mi−hi=0,-H_i,-i _-i( _-i^-1m_-i- _i(H_i,-i m_i+h_-i))-(H_i,i+Q_i)m_i-h_i=0, and so (−Hi,i+τiHi,−iΣ¯−iHi,−i⊤)mi−Hi,−iΣ¯−iΣ−i−1m−i+τiHi,−iΣ¯−ih−i−hi=0. (-H_i,i+ _iH_i,-i _-iH_i,-i )m_i-H_i,-i _-i _-i^-1m_-i+ _iH_i,-i _-ih_-i-h_i=0. The unique solution to this linear equation is mi=(−Hi,i+τiHi,−iΣ¯−iHi,−i⊤)−1(hi+Hi,−iΣ¯−i(Σ−i−1m−i−τih−i).m_i= (-H_i,i+ _iH_i,-i _-iH_i,-i )^-1 (h_i+H_i,-i _-i( _-i^-1m_-i- _ih_-i ). With Σ¯−i=1τiP−i−1 _-i= 1 _iP_-i^-1 we obtain the desired expression. • Covariance: We right multiply (12) with (ai−mi)⊤(a_i-m_i) and integrate with respect to xix_i to get −∫ℝnHi,iai(ai−mi)⊤dxi(ai)−ϵiI=0- _R^nH_i,ia_i(a_i-m_i) dx_i(a_i)- _iI=0 and so −Hi,i(Σi+mimi⊤−mimi⊤)−ϵiI=0.-H_i,i( _i+m_im_i -m_im_i )- _iI=0. Overall, we therefore have Σi=−ϵiHi,i−1. _i=- _iH_i,i^-1. Finally, we notice that the objective function is strongly geodesically convex in the distribution. Thus, the candidate solution is the unique minimizer and, therefore, the unique best response to x−ix_-i (see Theorem 3.3 in Lanzetti et al. (2025a)). ∎ With this lemma, we obtain a way to compute Gaussian RQE in continuous games: Lemma . Consider a surrogate “game of the means”, where each player selects a vector mi∈ℝnm_i ^n to maximize the surrogate concave quadratic utility U¯i(mi,m−i)=12⟨mi,(Hi,i−Hi,−iP−i−1Hi,−i⊤)mi⟩+1τi⟨mi,Hi,−iP−i−1Σ−i−1m−i⟩+⟨mi,hi−Hi,−iP−i−1h−i⟩. U_i(m_i,m_-i)= 12 m_i,(H_i,i-H_i,-iP_-i^-1H_i,-i )m_i + 1 _i m_i,H_i,-iP_-i^-1 _-i^-1m_-i + m_i,h_i-H_i,-iP_-i^-1h_-i . (13) Then, (m1⋆,…,mN⋆)(m_1 ,…,m_N ) is a Nash equilibrium of the game with utility (13) if and only if (x1⋆,…,xN⋆)(x _1,…,x_N ), where xi⋆x_i is a Gaussian distribution with mean mi⋆m_i and variance −ϵiHi,i−1- _iH_i,i^-1, is an RQE of the game with utilities (10). Moreover, (m1⋆,…,mN⋆)(m_1 ,…,m_N ) is the unique Nash equilibrium of this surrogate game if and only if (x1⋆,…,xN⋆)(x _1,…,x_N ) is the unique RQE in which one or more mixed strategies are Gaussian. Proof. The proof follows directly from Section˜B.1.1. Uniqueness in Gaussian mixed strategies follows from uniqueness of the best response. ∎ At this point, we can study RQE for games in which R(a1,…,aN)=−12‖∑i=1Nai−a¯‖H2=−12⟨∑i=1Nai−a¯,H(∑i=1Nai−a¯)⟩ci(ai)=ρi2‖ai‖2,R(a_1,…,a_N)=- 12 \| _i=1^Na_i- a \|_H^2=- 12 _i=1^Na_i- a,H ( _i=1^Na_i- a ) c_i(a_i)= _i2 a_i^2, where we assume that the matrix H, used in the weighted norm, is symmetric positive definite and ρi>0 _i>0. As we will show below, considering this class is sufficient. Proposition (computation of RQE). Let ϵi>0 _i>0 and τi>0 _i>0 so that 1τiϵj(ρj+H)−H≻0∀j≠i. 1 _i _j( _j+H)-H 0 ∀ j≠ i. (14) Let (m1⋆,…,mN⋆)∈ℝn×…×ℝn(m_1 ,…,m_N ) ^n×…×R^n be a Nash equilibrium of the surrogate game where the utility of each player is U¯i(mi,m−i)=12⟨mi,(−ρiI−H−∑j≠iHPij−1H)mi⟩−1τi⟨mi,H(∑j≠iPij−1Σj−1mj)⟩+⟨mi,Ha¯+H∑j≠iPij−1Ha¯⟩, U_i(m_i,m_-i)= 12 m_i, (- _iI-H- _j≠ iHP_ij^-1H )m_i \\ - 1 _i m_i,H ( _j≠ iP_ij^-1 _j^-1m_j ) + m_i,H a+H _j≠ iP_ij^-1H a , where Σi=ϵi(ρiI+H)−1≻0 _i= _i( _iI+H)^-1 0 and Pij=1τiϵj(ρjI+H)−H≻0P_ij= 1 _i _j( _jI+H)-H 0. Then, the Gaussian mixed strategies (x1⋆,…,xN⋆)(x_1 ,…,x_N ) form an RQE of the game and, at equilibrium, each player has finite costs. Moreover, if the equilibrium (m1⋆,…,mN⋆)(m_1 ,…,m_N ) is unique, then (x1⋆,…,xN⋆)(x_1 ,…,x_N ) is the unique Gaussian RQE. In particular, this is the case ρi _i, the degrees of risk aversion, and the degrees of bounded rationality are player-independent (i.e., ρi=ρ _i=ρ, τi=τ _i=τ, and ϵ=ϵiε= _i). Proof. The proof follows directly from Section˜B.1.1, where we have Hi,i=−H−ρiIH_i,i=-H- _iI, H−i,−i=IN−1⊗(−H)H_-i,-i=I_N-1 (-H), Hi,−i=[−H…−H]H_i,-i= bmatrix-H&…&-H bmatrix, hi=Ha¯h_i=H a, and h−i=[(Ha¯)⊤⋯(Ha¯)⊤]⊤h_-i= bmatrix(H a) &·s&(H a) bmatrix . We now show that if ρi=ρ _i=ρ, τi=τ _i=τ, and ϵi=ϵ _i=ε we have uniqueness. We show this in the case of two players; the general case is then a straightforward extension. For uniqueness, it suffices to check that the surrogate game is strongly monotone and therefore will have a unique Nash equilibrium (Facchinei and Pang, 2003). Since ρi=ρ _i=ρ, τi=τ _i=τ, and ϵi=ϵ _i=ε, we also have Σ2=Σ2=Σ _2= _2= and P12=P21=P_12=P_21=P. In this case, the game map reads ℱ(m1,m2)=[∂m1U¯1(m1,m2)∂m2U¯2(m2,m1)]=F[m1m2]+f,F(m_1,m_2)= bmatrix _m_1 U_1(m_1,m_2)\\ _m_2 U_2(m_2,m_1) bmatrix=F bmatrixm_1\\ m_2 bmatrix+f, where F≔[−ρI−H−HP−1H⊤−1τHP−1Σ−1−1τHP−1Σ−1−ρI−H−HP−1H⊤]f=[Ha¯+HP−1Ha¯Ha¯+HP−1Ha¯].F bmatrix-ρ I-H-HP^-1H &- 1τHP^-1 ^-1\\ - 1τHP^-1 ^-1&-ρ I-H-HP^-1H bmatrix f= bmatrixH a+HP^-1H a\\ H a+HP^-1H a bmatrix. With −1τHP−1Σ−1=−1τH(1τΣ−1−H)−1Σ−1=−H(I−τΣH)−1=−H+τHΣH(I−τΣH)−1=−H−H(1τΣ−1−H)−1H=−H−HP−1H- 1τHP^-1 ^-1=- 1τH ( 1τ ^-1-H )^-1 ^-1=-H(I-τ H)^-1\\ =-H+τ H H(I-τ H)^-1=-H-H ( 1τ ^-1-H )^-1H=-H-HP^-1H we have F=[−ρI00−ρI]+[−H−H−H−H]+[−HP−1H−HP−1H−HP−1H−HP−1H].F= bmatrix-ρ I&0\\ 0&-ρ I bmatrix+ bmatrix-H&-H\\ -H&-H bmatrix+ bmatrix-HP^-1H&-HP^-1H\\ -HP^-1H&-HP^-1H bmatrix. Clearly, F is negative definite, as it results from the sum of a negative definite matrix and two positive semidefinite matrices. Thus, the game map is strongly monotone, and we have uniqueness. ∎ Discussion A few observations on this result. First, the surrogate game can be solved by setting the gradient of each player’s utility to zero, which yields a linear system of equations which can be shown to possess a unique solution whenever ρi>0 _i>0. Second, if the assumption (14) is violated, then the utility of player i will generally diverge −∞-∞ and, thus, the player has “infinite” risk. To rule out this triviality, we therefore assume (14). Third, uniqueness is in the space of Gaussian mixed strategies. Thus, there might exist another equilibrium in which all mixed strategies are non-Gaussian. B.1.2 Proof of Section˜4.1 We prove a more general version of Section˜4.1. theorem [risk induces collaboration] Consider the game with utilities in (4.1). On the domain of τi _i where (14) and where the Gaussian RQE in unique22footnotemark: 2, let xi⋆(τ1,…,τN)x_i ( _1,…, _N) be the mixed strategy of player i at the unique Gaussian RQE of the game, as a function of the degrees of risk aversion τ1,…,τN _1,…, _N. Then, the expected shared reward (τ1,…,τN)↦J(τ1,…,τN)( _1,…, _N) J( _1,…, _N), where J(τ1,…,τN)≔ai∼xi⋆(τ1,…,τN)[R(a1,…,aN)],J( _1,…, _N) _a_i x_i ( _1,…, _N) [R(a_1,…,a_N) ], is strictly increasing333Uniqueness holds whenever the game is symmetric or τi _i is sufficiently small. In general, uniqueness of Gaussian RQE can be checked studying the uniqueness of Nash equilibria for the surrogate game Section˜B.1.1, and, amounts to checking positive (or negative) positiveness of a matrix.That is, players contribute more to the joint reward as they become more risk-averse. 33footnotetext: Here, strictly increasing means that if τi′≥τi _i ≥ _i for all i with one inequality strict, then J(τ1′,…,τN′)>J(τ1,…,τN)J( _1 ,…, _N )>J( _1,…, _N). Proof of Section˜B.1.2. For simplicity of notation, we conduct the proof in the case of two players, with the extension to more players being straightforward. To start, we show that, without loss of generality, the problem can be simplified, via the following three steps. First, we notice R(a1,a2)=12⟨a1+a2,H(a1+a2)⟩+⟨h,a1+a2⟩=12⟨a1+a2+H−1h,H(a1+a2+H−1h)⟩−⟨h,H−1h⟩.R(a_1,a_2)= 12 a_1+a_2,H(a_1+a_2) + h,a_1+a_2 = 12 a_1+a_2+H^-1h,H(a_1+a_2+H^-1h) - h,H^-1h . Since the term ⟨h,H−1h⟩ h,H^-1h is constant, we can therefore without loss of generality focus on the reward R(a1,a2)=12⟨a1+a2+H−1h,H(a1+a2+H−1h)⟩.R(a_1,a_2)= 12 a_1+a_2+H^-1h,H(a_1+a_2+H^-1h) . Additionally, if we let H¯=−H H=-H, which is positive definite, we have R(a1,a2)=−12⟨a1+a2−H¯−1h,H¯(a1+a2−H¯−1h)⟩=−12‖a1+a2−a¯‖H¯2R(a_1,a_2)=- 12 a_1+a_2- H^-1h, H(a_1+a_2- H^-1h) =- 12\|a_1+a_2- a\|_ H^2 where a¯≔H¯−1h=−H−1h a H^-1h=-H^-1h. Second, up to replacing Qi≔ρiIQ_i _iI with ρiH¯ _i H and rescaling a¯ a, we can assume, without loss of generality, that H¯=I H=I (i.e., H=−IH=-I). Third, we note that, since the variance of each mixed strategy xi⋆(τ1,τ2)x_i ( _1, _2) does not depend on risk parameters τ1,τ2 _1, _2, it suffices to study the monotonicity of the function J¯(τ1,τ2)=−12‖m1⋆(τ1,τ2)+m2⋆(τ1,τ2)−a¯‖2, J( _1, _2)=- 12 m_1 ( _1, _2)+m_2 ( _1, _2)- a^2, where mi⋆(τ1,τ2)m_i ( _1, _2) is the mean of xi⋆(τ1,τ2)x_i ( _1, _2) at the unique Gaussian RQE. Thus, we now study the monotonicity of J¯(τ1,τ2) J( _1, _2). From basic sensitivity analysis, J¯(τ1,τ2) J( _1, _2) is smooth and, thus, all its directional derivatives can be expressed via its gradient. To evaluate the gradient of J¯(τ1,τ2) J( _1, _2), we compute its partial derivatives via the product rule as follows: ∂τiJ¯(τ1,τ2)=−⟨m1⋆(τ1,τ2)+m2⋆(τ1,τ2)−a¯,∂τim1⋆(τ1,τ2)+∂τim2⋆(τ1,τ2)⟩. _ _i J( _1, _2)=- m_1 ( _1, _2)+m_2 ( _1, _2)- a, _ _im_1 ( _1, _2)+ _ _im_2 ( _1, _2) . (15) We evaluate this derivative for i=1i=1, with the case i=2i=2 being analogous. As a preliminary, we note that [m1⋆(τ1,τ2)m2⋆(τ1,τ2)] bmatrixm_1 ( _1, _2)\\ m_2 ( _1, _2) bmatrix =[−Q1−I−P12−1−1τ1P12−1Σ2−1−1τ2P21−1Σ1−1−Q2−I−P21−1]⏟S(τ1,τ2)[−a¯−P12−1a¯−a¯−P12−1a¯] = bmatrix-Q_1-I-P_12^-1&- 1 _1P_12^-1 _2^-1\\ - 1 _2P_21^-1 _1^-1&-Q_2-I-P_21^-1 bmatrix_S( _1, _2) bmatrix- a-P_12^-1 a\\ - a-P_12^-1 a\\ bmatrix =[Q1+I+P12−11τ1P12−1Σ2−11τ2P21−1Σ1−1Q2+I+P21−1]⏟S(τ1,τ2)[a¯+P12−1a¯a¯+P12−1a¯]⏟s(τ1,τ2), = bmatrixQ_1+I+P_12^-1& 1 _1P_12^-1 _2^-1\\ 1 _2P_21^-1 _1^-1&Q_2+I+P_21^-1 bmatrix_S( _1, _2) bmatrix a+P_12^-1 a\\ a+P_12^-1 a\\ bmatrix_s( _1, _2), where S(τ1,τ2)S( _1, _2) is a matrix whose symmetric part is positive definite. Thus, by the product and chain rule, we conclude that [∂τ1m1⋆(τ1,τ2)∂τ1m2⋆(τ1,τ2)]=−S(τ1,τ2)−1(∂τ1S(τ1,τ2)[m1⋆(τ1,τ2)m2⋆(τ1,τ2)]−∂τ1s(τ1,τ2)). bmatrix _ _1m_1 ( _1, _2)\\ _ _1m_2 ( _1, _2) bmatrix=-S( _1, _2)^-1 ( _ _1S( _1, _2) bmatrixm_1 ( _1, _2)\\ m_2 ( _1, _2) bmatrix- _ _1s( _1, _2) ). Using ∂τ1P12−1 _ _1P_12^-1 =(Σ2−1−τ1I)−1Σ2−1(Σ2−1−τ1I)−1=(Σ2−1−τ1I)−2Σ2−1=∂τ1(1τ1P12−1Σ2−1), =( ^-1_2- _1I)^-1 _2^-1( ^-1_2- _1I)^-1=( ^-1_2- _1I)^-2 _2^-1= _ _1 ( 1 _1P_12^-1 _2^-1 ), we now plug in all expressions to obtain [∂τ1m1⋆(τ1,τ2)∂τ1m2⋆(τ1,τ2)] bmatrix _ _1m_1 ( _1, _2)\\ _ _1m_2 ( _1, _2) bmatrix =−S(τ1,τ2)−1([∂τ1P12−1∂τ11τiP12−1Σ2−100][m1⋆(τ1,τ2)m2⋆(τ1,τ2)]−[∂τ1P12−1a¯0]) =-S( _1, _2)^-1 ( bmatrix _ _1P_12^-1& _ _1 1 _iP_12^-1 _2^-1\\ 0&0 bmatrix bmatrixm_1 ( _1, _2)\\ m_2 ( _1, _2) bmatrix- bmatrix _ _1P_12^-1 a\\ 0 bmatrix ) =−S(τ1,τ2)−1([(Σ2−1−τ1I)−1Σ2−1(Σ2−1−τ1I)−1(Σ2−1−τ1I)−2Σ2−100][m1⋆(τ1,τ2)m2⋆(τ1,τ2)] =-S( _1, _2)^-1 ( bmatrix( ^-1_2- _1I)^-1 _2^-1( ^-1_2- _1I)^-1&( ^-1_2- _1I)^-2 _2^-1\\ 0&0 bmatrix bmatrixm_1 ( _1, _2)\\ m_2 ( _1, _2) bmatrix −[(Σ2−1−τ1I)−1Σ2−1(Σ2−1−τ1I)−1a¯0]) 56.9055pt- bmatrix( _2^-1- _1I)^-1 _2^-1( _2^-1- _1I)^-1 a\\ 0 bmatrix ) =−S(τ1,τ2)−1[(Σ2−1−τ1I)−2Σ2−10](m1⋆(τ1,τ2)+m2⋆(τ1,τ2)−a¯) =-S( _1, _2)^-1 bmatrix( _2^-1- _1I)^-2 _2^-1\\ 0 bmatrix(m_1 ( _1, _2)+m_2 ( _1, _2)- a) =−1τ12S(τ1,τ2)−1[P2−2Σ2−10](m1⋆(τ1,τ2)+m2⋆(τ1,τ2)−a¯). =- 1 _1^2S( _1, _2)^-1 bmatrixP_2^-2 _2^-1\\ 0 bmatrix(m_1 ( _1, _2)+m_2 ( _1, _2)- a). Thus, we have ∂τ1m1⋆(τ1,τ2) _ _1m_1 ( _1, _2) +∂τ1m2⋆(τ1,τ2) + _ _1m_2 ( _1, _2) =[I][∂τ1m1⋆(τ1,τ2)∂τ1m2⋆(τ1,τ2)] = bmatrixI&I bmatrix bmatrix _ _1m_1 ( _1, _2)\\ _ _1m_2 ( _1, _2) bmatrix =−1τ12[I]S(τ1,τ2)−1[P2−2Σ2−10]⏟N(m1⋆(τ1,τ2)+m2⋆(τ1,τ2)−a¯), =- 1 _1^2 bmatrixI&I bmatrixS( _1, _2)^-1 bmatrixP_2^-2 _2^-1\\ 0 bmatrix_N(m_1 ( _1, _2)+m_2 ( _1, _2)- a), so that (15) becomes ∂τ1J¯(τ1,τ2)=1τ12⟨m1⋆(τ1,τ2)+m2⋆(τ1,τ2)−a¯,N(m1⋆(τ1,τ2)+m2⋆(τ1,τ2)−a¯)⟩. _ _1 J( _1, _2)= 1 _1^2 m_1 ( _1, _2)+m_2 ( _1, _2)- a,N(m_1 ( _1, _2)+m_2 ( _1, _2)- a) . To conclude the proof, it suffices to show that N is positive definite for all τi≥0 _i≥ 0. Indeed, in this case, ∂τiJ¯(τ1,τ2) _ _i J( _1, _2) is always positive (unless m1⋆(τ1,τ2)+m2⋆(τ1,τ2)=a¯m_1 ( _1, _2)+m_2 ( _1, _2)= a, which however cannot happen if ρ1,ρ2>0 _1, _2>0) and, thus, the shared reward J¯(τ1,τ2) J( _1, _2) is strictly increasing. Thus, in the rest of the proof, we will show that N is positive definite. Using the formula for the inverse of block matrices, we conclude that N=(I+(Q2+I+P12−1)−11τ2P21−1Σ1−1)⏟N1(Q1+I+P12−1+1τ1P12−1Σ2−1(Q2+I+P21−1)−11τ2P21−1Σ1−1)−1⏟N2−1P12−2Σ2−1⏟N3−1.N= (I+(Q_2+I+P_12^-1)^-1 1 _2P_21^-1 _1^-1 )_N_1\\ (Q_1+I+P_12^-1+ 1 _1P_12^-1 _2^-1(Q_2+I+P_21^-1)^-1 1 _2P^-1_21 _1^-1 )^-1_N_2^-1 P_12^-2 _2^-1_N_3^-1. Since Q1=ρ1HQ_1= _1H and Q2=ρ2HQ_2= _2H commute, we have that Σ1,Σ2,P12,P21 _1, _2,P_12,P_21, and thus N1,N2,N3N_1,N_2,N_3 belong to the subalgebra induced by Q1,Q2\Q_1,Q_2\ and, thus, commute. Therefore, each NiN_i results from the sum and product of symmetric and positive definite matrices (and their inverses) that commute and, as such, is symmetric and positive definite. As a consequence, the matrix N is also symmetric and positive definite. ∎ The proof of Section˜4.1 is now a direct consequence: Proof of Section˜4.1. The proof follows from Section˜B.1.2 and Section˜B.1.1, using ρi=ρ _i=ρ, τi=τ _i=τ, and ϵi=ϵ _i=ε for all agents. ∎ To conclude, we provide more details on Section˜4.1: Example (Details on Section˜4.1). For the game in Section˜4.1, we have H=1H=1, ρi=1 _i=1, and ci=0c_i=0, so that Σi=ϵi2 _i= _i2 and Pij=2τiϵj−1=2−τiϵjτiϵjP_ij= 2 _i _j-1= 2- _i _j _i _j so that U¯i(mi,m−i)=−12(2+τiϵj2−τiϵj)mi2−1τiτiϵj2−τiϵj2ϵjmim−i+mi(1+τiϵj2−τiϵj)a¯. U_i(m_i,m_-i)=- 12 (2+ _i _j2- _i _j )m_i^2- 1 _i _i _j2- _i _j 2 _jm_im_-i+m_i (1+ _i _j2- _i _j ) a. With τ1=τ2=τ _1= _2=τ and ϵ1=ϵ2=ϵ _1= _2=ε, we obtain the expected reward of each player from the shared reward J(τ)J(τ) minus the personal reward and it can be shown to be −122−τϵ+(τϵ2)2(3−τϵ2)2+C,- 12 2-τε+( τε2)^2(3- τε2)^2+C, where C is a constant that results from the variance of the mixed strategies and is therefore not dependent on τ but only on ϵε. B.2 Proof of Section˜4.2 In this section, we provide the proof of Section˜4.2. To do so, we first derive several helper-lemmas which guarantee us that (i) players’ strategies in an RQE are on the interior of the simplex, (i) the KL-divergence between any strategy and an RQE strategy is uniformly bounded, and (i) that we can relate the degree of free-riding to a lower bound on the distance between equilibrium strategies. To begin, we define some useful quantities that capture measures of the spread of players’ costs and reward functions cmax≔maxa⁡c(a)c_max _ac(a), cmin≔mina⁡c(a)c_min _ac(a), vmin≔mina1,a2⁡R(a1,a2)−cmaxv_min _a_1,a_2R(a_1,a_2)-c_max, vmax≔maxa1,a2⁡R(a1,a2)−cminv_max _a_1,a_2R(a_1,a_2)-c_min, and v¯≔vmax−vmin v v_max-v_min. Throughout the proof, we let n≔||n |A| be the number of actions and Δn=Δ() _n= (A) be the probability simplex. For a mixed strategy x∈Δnx∈ _n, we denote by x(a)∈[0,1]x(a)∈[0,1] the probability mass assigned to the action a. Lemma . Suppose that players have degree of bounded rationality ϵε. At a RQE, a player’s strategy xix_i has support on all actions, with probability at least m=exp⁡(−v¯/ϵ)nm= (- v/ε)n. Proof. In a RQE, a player’s strategy xi∗x_i^* must satisfy xi∗=argmaxxi∈Δn⟨xi,Rp∗⟩−⟨c,xi⟩−ϵH(xi),x_i^*= *arg\,max_x_i∈ _n\ \ x_i,Rp^* - c,x_i -ε H(x_i), where p∗=arg⁡minp∈Δn⁡⟨xi,Rp∗⟩+1τKL(p,x−i∗)p^*= _p∈ _n x_i,Rp^* + 1τ *KL(p,x_-i^*). Letting v≔Rp∗−cv Rp^*-c, the classic results show that xix_i is a Boltzmann distribution of the form: xi(a)=exp⁡(v(a)/ϵ)∑a′=1nexp⁡(v(a′)/ϵ),x_i(a)= (v(a)/ε) _a =1^n (v(a )/ε), where xi(a)x_i(a) is the probability mass assigned to action aia_i. By minimizing the numerator and maximizing the denominator of the right hand side we find that: xi(a)≥exp⁡(−v¯/ϵ)n,x_i(a)≥ (- v/ε)n, which completes the proof. ∎ Lemma . Suppose x2∈σ(Δn)x_2∈σ( _n) where σ(Δn)≔x∈Δ:x(a)≥m for all a∈σ( _n) \x∈ :x(a)≥ m for all a \ where m∈(0,1n)m∈(0, 1n). Then the KL divergence between any distribution p∈Δnp∈ _n and x2x_2 is bounded by supq∈ΔnKL(q,x2)≤log⁡(1m). _q∈ _n *KL(q,x_2)≤ ( 1m). Proof. We directly have KL(q,x2)=∑a∈q(a)log⁡(q(a)x2(a))≤∑aq(a)log⁡(1m)=log⁡(1m) *KL(q,x_2)= _a q(a) ( q(a)x_2(a))≤ _aq(a) ( 1m)= ( 1m) since x2(a)≥mx_2(a)≥ m. ∎ Lemma . Let x1,x2∈Δnx_1,x_2∈ _n. If |⟨c,x1−x2⟩|≥δ| c,x_1-x_2 |≥δ, then ‖x1−x2‖1≥2δcmax−cmin.\|x_1-x_2\|_1≥ 2δc_ -c_ . Proof. Let d≔x1−x2d x_1-x_2. Since x1,x2∈Δnx_1,x_2∈ _n, ∑ada=0 _ad_a=0. We use the fact that, for any d∈ℝnd ^n with ∑ad(a)=0 _ad(a)=0, |⟨c,d⟩|≤cmax−cmin2‖d‖1| c,d |≤ c_ -c_ 2\|d\|_1. This directly yields δ≤|⟨c,d⟩|≤cmax−cmin2‖x1−x2‖1.δ≤| c,d |≤ c_ -c_ 2\|x_1-x_2\|_1. Rearranging yields the lower bound. ∎ We can now prove Theorem 4.2, restated below with all constants made explicit. theorem Let δ>0δ>0. Suppose that players have a degree of bounded rationality ϵ>0ε>0 and degrees of risk aversion τ>0τ>0 that satisfies τ>2(ϵlog⁡(n)+v¯)(cmax−cmin)2ϵδ2,τ> 2(ε n+ v)(c_max-c_min)^2εδ^2, then the game cannot admit a RQE with degree of free-riding greater than δ. Proof. We prove this by contradiction. We assume that τ>2(ϵlog⁡(n)+v¯)(cmax−cmin)2ϵδ2τ> 2(ε n+ v)(c_max-c_min)^2εδ^2 and that x1x_1 and x2x_2 constitute a resulting RQE with a level of free riding larger than δ; i.e., |⟨c,x1−x2⟩|≥δ| c,x_1-x_2 |≥δ . Define the worst-case utility W(x)W(x) and the robust-regularized objective Φ(x) (x): U(x,q) U(x,q) ≔⟨x,Rq⟩−⟨c,x⟩ x,Rq - c,x W(x) W(x) ≔infq∈ΔnU(x,q) _q∈ _nU(x,q) Φ(x) (x) ≔W(x)−ϵH(x) W(x)-ε H(x) Since W is concave (as the infimum of affine functions) and H is strongly convex on Δn _n with respect to the ℓ1 _1 norm, Φ is ϵε-strongly concave on Δn _n with respect to the ℓ1 _1 norm. Let us consider player 11’s risk-adjusted utility. Without loss of generality, we develop the following for x1x_1, the results hold for x2x_2 by symmetry. We first show that we can lower bound the risk-adjusted objective by our worst case objective for any x1,x2∈Δnx_1,x_2∈ _n: Uτ(x1,x2)=minq∈Δn⁡U(x1,q)+1τKL(q,x2)≥minq∈Δn⁡U(x1,q)=W(x1).U^τ(x_1,x_2)= _q∈ _nU(x_1,q)+ 1τ *KL(q,x_2)≥ _q∈ _nU(x_1,q)=W(x_1). Now consider an RQE made up of strategies (x1,x2)(x_1,x_2), we upper bound the risk-adjusted objective by using the fact that x1x_1 and x2x_2 are on the interior of the simplex since they are entropy-regularized best responses (Section˜B.2), and thus the KL *KL term is bounded by Section˜B.2. Letting C≔log⁡(n)+v¯ϵC n+ vε be the resulting upper bound (which depends solely on R,c,ϵR,c,ε and the dimension n) we conclude that Uτ(x1,x2)=minq∈Δn⁡U(x1,q)+1τKL(q,x2)≤W(x1)+Cτ.U^τ(x_1,x_2)= _q∈ _nU(x_1,q)+ 1τ *KL(q,x_2)≤ W(x_1)+ Cτ. Putting these together, we have Φ(x1)=W(x1)−ϵH(x1)≤Uτ(x1,x2)−ϵH(x1)≤W(x1)+Cτ−ϵH(x1)=Φ(x1)+Cτ. (x_1)=W(x_1)-ε H(x_1)≤ U^τ(x_1,x_2)-ε H(x_1)≤ W(x_1)+ Cτ-ε H(x_1)= (x_1)+ Cτ. Using the definition of an RQE, we can further develop this to find that, for all x∈Δnx∈ _n: Φ(x1)+Cτ≥Uτ(x1,x2)−ϵH(x1)≥Uτ(x,x2)−ϵH(x)≥Φ(x) (x_1)+ Cτ≥ U^τ(x_1,x_2)-ε H(x_1)≥ U^τ(x,x_2)-ε H(x)≥ (x) Re-arranging, and using the fact that Φ is strongly concave with respect to the ℓ1 _1 norm on the simplex, we have Φ(x)≤Φ(x⋆)−ϵ2‖x1−x⋆‖12 (x)≤ (x )- ε2\|x_1-x \|_1^2 where x⋆=arg⁡minx∈Δn⁡Φ(x)x = _x∈ _n (x) is the unique minimizer of Φ over the simplex. Thus, ϵ2‖x1−x⋆‖12≤Φ(x∗)−Φ(x1)≤Cτ⟹‖x1−x⋆‖1≤2Cτ. ε2\|x_1-x \|_1^2≤ (x^*)- (x_1)≤ Cτ \|x_1-x \|_1≤ 2Cτ. By symmetry the same bound holds for x2x_2, and thus via the triangle inequality we obtain ‖x1−x2‖1≤‖x1−x⋆‖1+‖x2−x⋆‖1≤22Cτ.\|x_1-x_2\|_1≤\|x_1-x \|_1+\|x_2-x \|_1≤ 2 2Cτ. Via Section˜B.2, however we have that 2δcmax−cmin≤‖x1−x2‖1≤‖x1−x⋆‖1+‖x2−x⋆‖1≤22Cτ. 2δc_max-c_min≤\|x_1-x_2\|_1≤\|x_1-x \|_1+\|x_2-x \|_1≤ 2 2Cτ. Re-arranging, we find that τ≤2C(cmax−cmin)2δ2.τ≤ 2C(c_max-c_min)^2δ^2. Plugging in our value for C=log⁡(n)+v¯ϵC= n+ vε, this implies that τ≤2(ϵlog⁡(n)+v¯)(cmax−cmin)2ϵδ2.τ≤ 2(ε n+ v)(c_max-c_min)^2εδ^2. However, we assumed that the RQE resulted from the use of a τ>2(ϵlog⁡(n)+v¯)(cmax−cmin)2ϵδ2τ> 2(ε n+ v)(c_max-c_min)^2εδ^2 which is a contradiction. ∎ Appendix C Mathematical Background of SRPO In this section, we provide a formal framework for risk aversion and bounded rationality in MARL, and present the motivation for our algorithm SRPO. We first introduce the mathematical framework of discounted general-sum Markov games, and then discuss how to extend risk aversion and bounded rationality from normal-form games to Markov games. Next, we present the pseudocode of SRPO and a short discussion. To better motivate SRPO, we provide policy gradient theorems for each original player and adversary as well as their performance difference lemmas (PDLs), and introduce how to (mathematically) obtain the SRPO loss from the PDLs. C.1 Discounted General-sum Markov Games We consider a discounted N-player general-sum Markov game is specified by a tuple: ℳ=,ii=1N,rii=1N,γ,P,ρ0,MG=\S,\A_i\_i=1^N,\r_i\_i=1^N,γ,P, _0\, where S is the state space of the underlying MDP, iA_i is the action space of player i∈[N]i∈[N], and we use the notation =∏i=1NiA= _i=1^NA_i to denote the product action space of both players. We assume that |||S| and |i||A_i| are all finite. Here, ri:×→[0,1]r_i:S×A→[0,1] is the reward function of player i, which we assume to be deterministic. We use r to denote the joint reward function ≔(ri)i=1Nr (r_i)_i=1^N. Moreover, γ∈[0,1)γ∈[0,1) is the discount factor and P:×→Δ()P:S×A→ (S) is the transition kernel, where Δ() (S) is the probability simplex of S and P(s′|s,)P(s |s,a) is the probability of the next state being s′s given the current state s and the current actions =(ai)i=1Na=(a_i)_i=1^N of the players. We use ρ0∈Δ() _0∈ (S) to denote the initial state distribution. We focus on Markov policies, the class of policies where the action selection probability only depends on the current state instead of the entire gameplay trajectory, i.e., π=(πi)i=1Nπ=( _i)_i=1^N where πi:→Δ(i),i∈[N] _i:S→ (A_i),i∈[N]. Given a product Markov policy π, without considering risk aversion and bounded rationality, player i has an expected discounted cumulative reward given by π[∑t=0∞γtri(st,at)]E_π[ _t=0^∞γ^tr_i(s_t,a_t)]. To incorporate risk aversion and bounded rationality in discounted infinite-horizon Markov games, we slightly overload the notations in normal-form games and consider the following risk-adjusted objective of player i that minimizes fi(πi,π−i)=maxpi:→Δ(−i)⁡Ji(πi,π−i,pi)f_i( _i, _-i)= _p_i:S→ (A_-i)J_i( _i, _-i,p_i) where JiJ_i is defined as Ji(πi,π−i,pi)=π,p,s0∼ρ0[∑t=0∞γt(ri(st,t)+1τiDi(pi,π−i;st)−ϵiνi(πi;st))] J_i( _i, _-i,p_i)=E_π,p,s_0 _0 [ _t=0^∞γ^t (r_i(s_t,a_t)+ 1 _iD_i (p_i, _-i;s_t )- _i _i( _i;s_t) ) ] (16) where the joint actions ta_t are sampled through ai,t∼πi(⋅|st)a_i,t _i(·|s_t) and −i,t∼pi(⋅|st)a_-i,t p_i(·|s_t) and the next state st+1s_t+1 is sampled from st+1∼P(⋅|st,t)s_t+1 P(·|s_t,a_t), and the notations of Di(pi,π−i;s)D_i(p_i, _-i;s) and νi(πi;s) _i( _i;s) are abbreviations of Di(pi(⋅|s),π−i(⋅|s))D_i(p_i(·|s), _-i(·|s)) and νi(πi(⋅|s)) _i( _i(·|s)) respectively. Here, we allow for more generality of the regularizers, and, if we want to be consistent with the main body, we can choose DiD_i to be KL *KL, and νi _i to be negative entropy H. Given a set of original player policies πi _i and adversarial policies pip_i, we define the value function for each state s∈s as: Viπ,p(s)=π,p,s0=s[∑t=0∞γt(ri(st,t)+1τiDi(pi,π−i;st)−ϵiνi(πi;st))] V_i^π,p(s)=E_π,p,s_0=s [ _t=0^∞γ^t (r_i(s_t,a_t)+ 1 _iD_i (p_i, _-i;s_t )- _i _i( _i;s_t) ) ] (17) so that Ji(π,p)=s∼ρ0[Viπ,p(s)]J_i(π,p)=E_s _0[V_i^π,p(s)], and the Q function as: Qiπ,p(s,)=ri(s,)+γs′∼P(⋅|s,)Viπ,p(s′).Q_i^π,p(s,a)=r_i(s,a)+ _s P(·|s,a)V_i^π,p(s ). (18) It is easy to verify that Viπ,p(s)= V_i^π,p(s)= πi(⋅|s)TQiπ,p(s,⋅)pi(⋅|s)+1τiDi(pi,π−i;s)−ϵiνi(πi;s). _i(·|s)^TQ_i^π,p(s,·)p_i(·|s)+ 1 _iD_i(p_i, _-i;s)- _i _i( _i;s). (19) Finally, we define given policies πi,pi _i,p_i, we define the discounted state visitation probability as ds0πi,pi(s)≔(1−γ)∑t=0∞γtPrπi,pi⁡(st=s|s0)=(1−γ)∑t=0∞γtes0T(Pπi,pi)tes, d_s_0 _i,p_i(s) (1-γ) _t=0^∞γ^t _ _i,p_i(s_t=s|s_0)=(1-γ) _t=0^∞γ^te_s_0^T(P _i,p_i)^te_s, (20) where ese_s denotes the one-hot vector corresponding to state s. To learn the RQE of the Markov game, we have to optimize the policies πi _i to minimize (and pip_i to maximize) the risk-adjusted objective (16). Here, we provide some useful tools for optimizing these policies. C.2 Details of SRPO We present the pseudocode for SRPO in Algorithm˜1. Notice that although the agents and adversaries are maximizing two losses ℒiSRPOL_i^SRPO and ℒ¯iSRPO L_i^SRPO respectively, we unify these two losses by considering the joint loss: ℒi,jointSRPO(θi,θ−i,β)=ℒiCLIP(θi,β)+1τiKL(β,θ−i)−ϵiH(θi)(oit).L_i,joint^SRPO( _i, _-i,β)=L_i^CLIP( _i,β)+ 1 _i *KL(β, _-i)- _iH( _i)(o_i^t). (21) The update of agent i can be seen as maximizing ℒi,jointSRPOL_i,joint^SRPO as a function of θi _i, and the update of adversary i can be seen as minimizing ℒi,jointSRPOL_i,joint^SRPO as a function of β. Algorithm 1 Strategically Risk-averse Policy Optimization (SRPO) for 2n2n-agents 1: Initialize: Policies πθii=1n\ _ _i\_i=1^n, adversaries πϕii=1n\ _ _i\_i=1^n, and critics Vψii=1n\V_ _i\_i=1^n. 2: Parameters: Risk aversion τii=1n\ _i\_i=1^n, bounded rationality ϵii=1n\ _i\_i=1^n, learning rates ηθ,ηϕ,ηψ _θ, _φ, _ψ, clipping δ. 3: for iteration k=1,2,…k=1,2,… do 4: for each agent i∈1,…,ni∈\1,…,n\ do 5: // Data Collection 6: Run joint policy θi,ϕi=(πθi,πϕi) π_ _i, _i=( _ _i, _ _i) of agent i and adversary i in the environment for T steps. 7: Store trajectories i=(oit,ait,rit)i=1,t=1n,TD_i=\(o_i^t,a_i^t,r_i^t)\_i=1,t=1^n,T. 8: Compute advantage estimates A^it A_i^t using local critic VψiV_ _i. 9: // Maximin-Optimization 10: for step in steps do 11: # Update Agent by maximizing ℒiSRPOL_i^SRPO 12: θi←θi+ηθ∇θiℒiSRPO _i← _i+ _θ _ _iL_i^SRPO 13: # Update Adversary by maximizing ℒ¯iSRPO L_i^SRPO 14: ϕi←ϕi+ηϕ∇ϕiℒ¯iSRPO _i← _i+ _φ _ _i L_i^SRPO 15: # Update Local Critic 16: ψi←ψi−ηψ∇ψiℒiVF(ψi) _i← _i- _ψ _ _iL_i^VF( _i) 17: end for 18: end for 19: Update old policy parameters: θi,old←θi _i,old← _i for all i. 20: end for Discussion The iteration structure of SRPO is very similar to that of IPPO. In the data collection phase, unlike IPPO, agent i computes the advantage estimate based on rollouts played against adversary i instead of other real agents. In the optimization phase, both agent i and their adversary update their policies to maximize the SRPO losses (8) and (9). Here, we have introduced an additional critic with parameter ψi _i with the MSE loss function ℒiVF(θi,θ−i)L_i^VF( _i, _-i), as a standard practice for variance reduction in IPPO, which is only used when computing the advantage estimates. C.3 Policy Gradient Theorems Policy gradient theorems are the most fundamental theoretical tool for nearly all policy-based algorithms. As the foundation of our framework, we provide the expression of policy gradients for the original agents and the adversaries under the risk-adjusted objectives, respectively. Although our results are the first to present these, similar results for single-agent version can be found in (Lan, 2023). For the original agents, the policy gradient theorem has the following form: Lemma (Policy Gradient for πi _i). The gradient ∇πiViπ,p(s) _ _iV_i^π,p(s) can be written as ∇πi(⋅|x)Viπ,p(s)=11−γdsπi,pi(x)[−i∼pi(⋅|x)[Qiπ,p(x,⋅,−i)]−ϵi∇πiνi(πi;x)]. _ _i(·|x)V_i^π,p(s)= 11-γd_s _i,p_i(x) [E_a_-i p_i(·|x)[Q_i^π,p(x,·,a_-i)]- _i _ _i _i( _i;x) ]. If πi _i is parameterized by θi _i, we have ∇θiViπ,p(s)=11−γx∼dsπi,pi,∼(πi⊗pi)(⋅|x)[∇θilog⁡(πi(ai|x))(Qiπ,p(x,)−ϵi∂νi(πi;x)∂πi(ai|x))], _ _iV_i^π,p(s)= 11-γE_x d_s _i,p_i,a ( _i p_i)(·|x) [ _ _i _i(a_i|x) (Q_i^π,p(x,a)- _i ∂ _i( _i;x)∂ _i(a_i|x) ) ], where =(ai,−i)a=(a_i,a_-i). Proof. We have ∇πi(⋅|x)Viπ,p(s)= _ _i(·|x)V_i^π,p(s)= ⟨∂πi(⋅|x)Qiπ,p(s,⋅),(πi⊗pi)(⋅|s)⟩ ∂ _i(·|x)Q_i^π,p(s,·),( _i p_i)(·|s) +(a−i∼pi(⋅|s)[Qiπ,p(s,⋅,a−i)]−ϵi∇πiνi(πi;s))[s=x] + (E_a_-i p_i(·|s)[Q_i^π,p(s,·,a_-i)]- _i _ _i _i( _i;s) )I[s=x] = = γs′∼Pπi,pi(s)[∇πiViπ,p(s′)]+(a−i∼pi(⋅|s)[Qiπ,p(s,⋅,a−i)]−ϵi∇πiνi(πi;s))[s=x] _s P _i,p_i(s)[ _ _iV_i^π,p(s )]+ (E_a_-i p_i(·|s)[Q_i^π,p(s,·,a_-i)]- _i _ _i _i( _i;s) )I[s=x] = = … … = = 11−γdsπi,pi(x)[a−i∼pi(⋅|x)[Qiπ,p(x,⋅,a−i)]−ϵi∇πiνi(πi;x)]. 11-γd_s _i,p_i(x) [E_a_-i p_i(·|x)[Q_i^π,p(x,·,a_-i)]- _i _ _i _i( _i;x) ]. Similarly, for the parameterized case, we have ∂πi(ai|x) ∂ _i(a_i|x) Viπ,p(s) V_i^π,p(s) = = ⟨∂πi(ai|x)Qiπ,p(s,⋅),(πi⊗pi)(⋅|s)⟩ ∂ _i(a_i|x)Q_i^π,p(s,·),( _i p_i)(·|s) +(a−i∼pi(⋅|s)[Qiπ,p(s,ai,a−i)]−ϵi∂πi(ai|x)νi(πi;s))[s=x] + (E_a_-i p_i(·|s)[Q_i^π,p(s,a_i,a_-i)]- _i ∂ _i(a_i|x) _i( _i;s) )I[s=x] = = γs′∼Pπi,pi(s)[∂πi(ai|x)Viπ,p(s′)]+(a−i∼pi(⋅|s)[Qiπ,p(s,ai,a−i)]−ϵi∂πi(ai|x)νi(πi;s))[s=x] _s P _i,p_i(s)[ ∂ _i(a_i|x)V_i^π,p(s )]+ (E_a_-i p_i(·|s)[Q_i^π,p(s,a_i,a_-i)]- _i ∂ _i(a_i|x) _i( _i;s) )I[s=x] = = … … = = 11−γdsπi,pi(x)[a−i∼pi(⋅|x)[Qiπ,p(x,ai,a−i)]−ϵi∂πi(ai|x)νi(πi;x)], 11-γd_s _i,p_i(x) [E_a_-i p_i(·|x)[Q_i^π,p(x,a_i,a_-i)]- _i ∂ _i(a_i|x) _i( _i;x) ], therefore, ∇θiViπ,p(s)= _ _iV_i^π,p(s)= ∑x,ai∂πi(ai|x)Viπ,p(s)∇θiπi(ai|x) _x,a_i ∂ _i(a_i|x)V_i^π,p(s) _ _i _i(a_i|x) = = 11−γ∑x,aidsπi,pi(x)[a−i∼pi(⋅|x)[Qiπ,p(x,ai,a−i)]−ϵi∂πi(ai|x)νi(πi;x)]∇θiπi(ai|x) 11-γ _x,a_id_s _i,p_i(x) [E_a_-i p_i(·|x)[Q_i^π,p(x,a_i,a_-i)]- _i ∂ _i(a_i|x) _i( _i;x) ] _ _i _i(a_i|x) = = 11−γx∼dsπi,pia∼(πi⊗pi)(⋅|x)[∇θilog⁡(πi(ai|x))(Qiπ,p(x,a)−ϵi∂πi(ai|x)νi(πi;x))]. 11-γE_x d_s _i,p_iE_a ( _i p_i)(·|x) [ _ _i _i(a_i|x) (Q_i^π,p(x,a)- _i ∂ _i(a_i|x) _i( _i;x) ) ]. This concludes the proof. ∎ Similarly, we have the policy gradient for pip_i as follows: Lemma (Policy gradient for pip_i). The gradient ∇piViπ,p(s) _p_iV_i^π,p(s) can be written as ∇pi(⋅|x)Viπ,p(s)=11−γdsπi,pi(x)[ai∼πi(⋅|x)[Qiπ,p(x,ai,⋅)]+1τi∇piDi(pi,π−i;x)] _p_i(·|x)V_i^π,p(s)= 11-γd_s _i,p_i(x) [E_a_i _i(·|x)[Q_i^π,p(x,a_i,·)]+ 1 _i _p_iD_i(p_i, _-i;x) ] and if pip_i is parameterized by θ¯i θ_i, we have ∇θ¯iViπ,p(s)=11−γx∼dsπi,pi,∼(πi⊗pi)(⋅|x)[∇θ¯ilog⁡pi(−i|x)(Qiπ,p(x,)+1τi∂Di(pi,π−i;x)∂pi(−i|x))], _ θ_iV_i^π,p(s)= 11-γE_x d_s _i,p_i,a ( _i p_i)(·|x) [ _ θ_i p_i(a_-i|x) (Q_i^π,p(x,a)+ 1 _i ∂ D_i(p_i, _-i;x)∂ p_i(a_-i|x) ) ], where =(ai,−i)a=(a_i,a_-i). Proof. We have ∇pi(⋅|x)Viπ,p(s)= _p_i(·|x)V_i^π,p(s)= ∇pi(⋅|x)(⟨Qiπ,p(s,⋅),(πi⊗pi)(⋅|s)⟩)+1τi∇piDi(pi,π−i;s)[s=x] _p_i(·|x)( Q_i^π,p(s,·),( _i p_i)(·|s) )+ 1 _i _p_iD_i(p_i, _-i;s)I[s=x] = = ⟨∂pi(⋅|x)Qiπ,p(s,⋅),(πi⊗pi)(⋅|s)⟩ ∂ p_i(·|x)Q_i^π,p(s,·),( _i p_i)(·|s) +(ai∼πi(⋅|s)[Qiπ,p(s,ai,⋅)]+1τi∇piDi(pi,π−i;s))[s=x] + (E_a_i _i(·|s)[Q_i^π,p(s,a_i,·)]+ 1 _i _p_iD_i(p_i, _-i;s) )I[s=x] = = γs′∼Pπi,pi(s)[∇pi(⋅|x)Viπ,p(s′)]+(ai∼πi(⋅|s)[Qiπ,p(s,ai,⋅)]+1τi∇piDi(pi,π−i;s))[s=x] _s P _i,p_i(s)[ _p_i(·|x)V_i^π,p(s )]+ (E_a_i _i(·|s)[Q_i^π,p(s,a_i,·)]+ 1 _i _p_iD_i(p_i, _-i;s) )I[s=x] = = … … = = 11−γdsπi,pi(x)[ai∼πi(⋅|x)[Qiπ,p(x,ai,⋅)]+1τi∇piDi(pi,π−i;x)], 11-γd_s _i,p_i(x) [E_a_i _i(·|x)[Q_i^π,p(x,a_i,·)]+ 1 _i _p_iD_i(p_i, _-i;x) ], and similarly for the parameterized case, ∇θ¯iViπ,p(s) _ θ_iV_i^π,p(s) = = 11−γx∼dsπi,pia∼(πi⊗pi)(⋅|x)[∇θ¯ilog⁡pi(a−i|x)(Qiπ,p(x,a)+1τi∂pi(a−i|x)Di(pi,π−i;x))]. 11-γE_x d_s _i,p_iE_a ( _i p_i)(·|x) [ _ θ_i p_i(a_-i|x) (Q_i^π,p(x,a)+ 1 _i ∂ p_i(a_-i|x)D_i(p_i, _-i;x) ) ]. This concludes the proof. ∎ C.4 Performance Difference Lemmas Another crucial tool for quantifying the difference in value functions (hence expected returns) between different policies through the advantage function is the performance difference lemma (PDL). For our 4-player risk-adjusted game, we state the PDLs for original agents and adversaries below: Lemma (Performance Difference Lemma for πi _i). For two policies πi _i and πi′ _i of player i, we have Viπi,z(s)−Viπi′,z(s)=11−γs′∼dsπi,pi,∼(πi⊗pi)(⋅|s′)[Aiπi′,z(s′,)−ϵiνi(πi;s′)] V_i _i,z(s)-V_i _i ,z(s)= 11-γE_s d_s _i,p_i,a ( _i p_i)(·|s ) [A_i _i ,z(s ,a)- _i _i( _i;s ) ] (22) where the advantage function AiA_i is defined as: Aiπi,z(s,)=Qiπi,z(s,)−Viπi,z(s)+1τiDi(z;s).A_i _i,z(s,a)=Q_i _i,z(s,a)-V_i _i,z(s)+ 1 _iD_i(z;s). Here, z is the shorthand notation of z=(π−i,pi)z=( _-i,p_i). Proof. We can write the difference in value function as Viπi′,z(s)−Viπi,z(s) V_i _i ,z(s)-V_i _i,z(s) = = πi′,z,s0=s[∑t=0∞γt(ri(st,t)+1τiDi(z;st)−ϵiνi(πi′;st))]−Viπi,z(s) _ _i ,z,s_0=s [ _t=0^∞γ^t (r_i(s_t,a_t)+ 1 _iD_i (z;s_t )- _i _i( _i ;s_t) ) ]-V_i _i,z(s) = = Viπi′,z(s)−(ai,0,−i,0)∼(πi⊗pi)(⋅|s)[ri(s,0)+1τiDi(z;s)−ϵiνi(πi;s)+γs′∼P(⋅|s,0)Viπi′,z(s′)] V_i _i ,z(s)-E_(a_i,0,a_-i,0) ( _i p_i)(·|s) [r_i(s,a_0)+ 1 _iD_i(z;s)- _i _i( _i;s)+ _s P(·|s,a_0)V_i _i ,z(s ) ] +(ai,0,−i,0)∼(πi⊗pi)(⋅|s)[ri(s,0)+1τiDi(z;s)−ϵiνi(πi;s)+γs′∼P(⋅|s,0)Viπi′,z(s′)]−Viπi,z(s) +E_(a_i,0,a_-i,0) ( _i p_i)(·|s) [r_i(s,a_0)+ 1 _iD_i(z;s)- _i _i( _i;s)+ _s P(·|s,a_0)V_i _i ,z(s ) ]-V_i _i,z(s) = = (ai,0,−i,0)∼(πi⊗pi)(⋅|s)[Viπi′,z(s)−Qiπi′,z(s,0)−1τiDi(z;s)+ϵiνi(πi;s)] _(a_i,0,a_-i,0) ( _i p_i)(·|s) [V_i _i ,z(s)-Q_i _i ,z(s,a_0)- 1 _iD_i(z;s)+ _i _i( _i;s) ] +γ(ai,0,−i,0)∼(πi⊗pi)(⋅|s)s′∼P(s,0)[Viπi′,z(s′)−Viπi,z(s′)] + _(a_i,0,a_-i,0) ( _i p_i)(·|s)E_s P(s,a_0) [V_i _i ,z(s )-V_i _i,z(s ) ] = = ⋯ ·s = = 11−γs′∼dsπi,pi∼(πi⊗pi)(⋅|s′)[Viπi′,z(s′)−Qiπi′,z(s′,)−1τiDi(z;s′)+ϵiνi(πi;s′)] 11-γE_s d_s _i,p_iE_a ( _i p_i)(·|s ) [V_i _i ,z(s )-Q_i _i ,z(s ,a)- 1 _iD_i(z;s )+ _i _i( _i;s ) ] This concludes the proof. ∎ Lemma (Performance Difference Lemma for pip_i). For two policies pip_i and pi′p_i of adversary i, we have: Vipi,π(s)−Vipi′,π(s)=11−γs′∼dsπi,pi,∼(πi⊗pi)(⋅|s′)[Aipi′,π(s′,)+1τiDi(pi,π−i;s′)] V_i^p_i,π(s)-V_i^p_i ,π(s)= 11-γE_s d_s _i,p_i,a ( _i p_i)(·|s ) [A_i^p_i ,π(s ,a)+ 1 _iD_i(p_i, _-i;s ) ] where Aipi,π(s,)=Qipi,π(s,)−Vipi,π(s)−ϵiνi(πi;s).A_i^p_i,π(s,a)=Q_i^p_i,π(s,a)-V_i^p_i,π(s)- _i _i( _i;s). Proof. We can write the value function difference as Vipi,π(s)−Vipi′,π(s) V_i^p_i,π(s)-V_i^p_i ,π(s) = = Vipi,π(s)−(ai,0,−i,0)∼(πi⊗pi)(⋅|s)[ri(s,0)+1τiDi(pi,π−i;s)−ϵiνi(πi;s)+γs′∼P(⋅|s,0)Vipi′,π(s′)] V_i^p_i,π(s)-E_(a_i,0,a_-i,0) ( _i p_i)(·|s) [r_i(s,a_0)+ 1 _iD_i(p_i, _-i;s)- _i _i( _i;s)+ _s P(·|s,a_0)V_i^p_i ,π(s ) ] +(ai,0,−i,0)∼(πi⊗pi)(⋅|s)[ri(s,0)+1τiDi(pi,π−i;s)−ϵiνi(πi;s)+γs′∼P(⋅|s,0)Vipi′,π(s′)]−Vipi′,π(s) +E_(a_i,0,a_-i,0) ( _i p_i)(·|s) [r_i(s,a_0)+ 1 _iD_i(p_i, _-i;s)- _i _i( _i;s)+ _s P(·|s,a_0)V_i^p_i ,π(s ) ]-V_i^p_i ,π(s) = = γ(ai,0,−i,0)∼(πi⊗pi)(⋅|s)s′∼P(⋅|s,0)[Vipi,π(s′)−Vipi′,π(s′)] _(a_i,0,a_-i,0) ( _i p_i)(·|s)E_s P(·|s,a_0)[V_i^p_i,π(s )-V_i^p_i ,π(s )] +(ai,0,−i,0)∼(πi⊗pi)(⋅|s)[Qipi′,π(s,0)+1τiDi(pi,π−i;s)−ϵiνi(πi;s)−Vipi′,π(s)] +E_(a_i,0,a_-i,0) ( _i p_i)(·|s) [Q_i^p_i ,π(s,a_0)+ 1 _iD_i(p_i, _-i;s)- _i _i( _i;s)-V_i^p_i ,π(s) ] = = … … = = 11−γs′∼dsπi,pi∼(πi⊗pi)(⋅|s′)[Qipi′,π(s′,0)+1τiDi(pi,π−i;s′)−ϵiνi(πi;s′)−Vipi′,π(s′)]. 11-γE_s d_s _i,p_iE_a ( _i p_i)(·|s ) [Q_i^p_i ,π(s ,a_0)+ 1 _iD_i(p_i, _-i;s )- _i _i( _i;s )-V_i^p_i ,π(s ) ]. This concludes the proof. ∎ C.5 Motivating the SRPO Objective Having stated the policy gradient theorems and PDLs, we now motivate our SRPO objective following the same rationale as that in TRPO (Schulman et al., 2015). Let z=(π0,p0)z=(π^0,p^0) be the joint policy at the current timestep, by Section˜C.4, consider policy update for πi _i, we can write the value function for any policy πi _i as Jiπi,π−i0,pi0=Jiπ0,p0+11−γs∼dπi,pi0,∼(πi⊗pi0)(⋅|s)[Aiπi0,π−i0,pi0(s,)−ϵiνi(πi;s)], J_i _i, _-i^0,p_i^0=J_i^π^0,p^0+ 11-γE_s d _i,p_i^0,a ( _i p_i^0)(·|s) [A_i _i^0, _-i^0,p_i^0(s,a)- _i _i( _i;s) ], where dπi,pi0d _i,p_i^0 is the state distribution for the initial state distribution ρ0 _0. When πi _i is constrained to be close to πi0 _i^0, we use the following surrogate objective first proposed in TRPO (Schulman et al., 2015) to approximate the expectation term by replacing the sample distribution from dπi,pi0d _i,p_i^0 to dπi0,pi0d _i^0,p_i^0: s∼dπi0,pi0,∼(πi0⊗pi0)(⋅|s)[πi(ai|s)πi0(ai|s)Aiπ0,pi0(s,)−ϵiνi(πi;s)], _s d _i^0,p_i^0,a ( _i^0 p_i^0)(·|s) [ _i(a_i|s) _i^0(a_i|s)A_i^π^0,p_i^0(s,a)- _i _i( _i;s) ], (23) where the approximation dπi0,pi0≈dπi,pi0d _i^0,p_i^0≈ d _i,p_i^0 holds when π0π^0 stays close to π, and importance sampling through the ratio πi(ai|s)πi0(ai|s) _i(a_i|s) _i^0(a_i|s) on aia_i. Similarly by Section˜C.4, we have the following surrogate objective for pip_i as follows: s∼dπi0,pi0,∼(πi0⊗pi0)(⋅|s)[ _s d _i^0,p_i^0,a ( _i^0 p_i^0)(·|s) [ pi(−i|s)pi0(−i|s)Aipi0,π0(s,)+1τiDi(pi,π−i;s)]. p_i(a_-i|s)p_i^0(a_-i|s)A_i^p_i^0,π^0(s,a)+ 1 _iD_i(p_i, _-i;s) ]. (24) The original TRPO objective suggests optimizing (23) and (24) subject to the constraints dπi0,pi0[KL(πi(⋅|s)∥πi0(⋅|s))]≤δ,E_d _i^0,p_i^0[ *KL( _i(·|s)\| _i^0(·|s))]≤δ, and dπi0,pi0[KL(pi(⋅|s)∥pi0(⋅|s))]≤δ,E_d _i^0,p_i^0[ *KL(p_i(·|s)\|p_i^0(·|s))]≤δ, respectively. A later adaptation PPO (Schulman et al., 2017) replaces these hard constraints with the clipped surrogate objective. Following this adaptation and replacing the policy to condition on observations instead of the global states, we obtain our SRPO losses (8) and (9) respectively. Appendix D Experimental Details D.1 Overall Setup Across all experiments, we compare SRPO against IPPO under a unified training and evaluation protocol. Each method is trained from scratch using multiple random seeds, and performance is reported as episodic return averaged over rollouts. Training. Agents are trained for a fixed interaction budget per environment, with periodic evaluation during training. Unless otherwise specified, SRPO and IPPO agents share the same architecture, optimizer settings, amount of interaction with the environment, and entropy regularization, differing only in the inclusion of the strategic risk-averse objective. Given the same total amount of interactions, SRPO and IPPO take roughly the same time to train in all environments we have. Cross-play evaluation. To assess unseen partner generalization, we adopt a cross-play protocol. For each environment, we collect a population of independently trained agents and evaluate all pairwise combinations without further learning. Cross-play performance is reported as a matrix, where each entry corresponds to the average episodic return of a fixed agent pair evaluated over multiple episodes. This protocol measures zero-shot coordination with previously unseen partners. D.2 Overcooked Gridworld Environments. We implemented a revised version of Overcooked AI, shown in Fig.˜6. The task is defined on a 5×55× 5 grid with two agents acting simultaneously. Agents start from one of two symmetric initial configurations in the top left and must cooperatively pick up onions from two fixed sources and deliver them to a central pot. Each agent selects from five discrete actions (up, down, left, right, stay); non-stay actions incur a private movement cost of 0.2. Agent updates are applied sequentially in random order each step, with collisions blocked and penalized by 2.0. An agent can carry at most one onion; stepping onto an available onion grants a shared reward of +1 and removes the onion, while delivering an onion by attempting to move into the (solid) pot yields a shared reward of +10. Each onion independently respawns with probability 0.2 per step. Observations are fully shared and consist of both agents’ positions, carry status, and onion availability. In this setup, rewards combine private movement penalties with shared team rewards, inducing cooperative behavior under individual costs. Figure 5: The Overcooked Gridworld environment. Figure 6: The Tag environment. Training and evaluation. Each agent is trained for approximately 2×1062× 10^6 environment interactions. Training statistics are recorded every 10240 interactions, where each evaluation point reports the average return over 5 rollouts of length 128. We perform 30 independent runs for each method. IPPO agents use an entropy coefficient of ϵ=0.1ε=0.1, while SRPO agents use τ=10τ=10 with the same entropy coefficient. For the main experiments, we construct a 60×6060× 60 cross-play matrix consisting of 30 IPPO agents and 30 SRPO agents. Each entry is averaged over 100 evaluation episodes of length 100. We report cross-play results for both the training environment and the held-out test environment. The results are shown in Fig.˜2. D.3 Tag Environment. We evaluate on the PettingZoo Multi-Agent Particle Environment (MPE) simple_tag_v3 (Terry et al., 2021), configured with 1 runner (prey), 2 chasers, and 2 static obstacles, using discrete actions and a horizon of 100 cycles per episode. The environment is shown in Fig.˜6. Rewards are taken directly from the environment and scaled by 0.1, and an episode terminates when any agent is terminated or truncated. Training and evaluation. Each agent is trained for approximately 3×1073× 10^7 environment interactions. Training performance is recorded every 4096040960 interactions, where each evaluation point reports the average episodic return over 64 rollouts of length 100. We perform 30 independent runs for each method. IPPO agents use an entropy coefficient of ϵ=0.01ε=0.01, while SRPO agents use the risk aversion parameter τ=10τ=10 with the same entropy coefficient. To evaluate partner generalization, we construct a 60×6060× 60 cross-play matrix consisting of 30 IPPO agents and 30 SRPO agents. Each matrix entry corresponds to the average return obtained by a fixed agent pair, evaluated over 100 episodes of length 100 without further learning. In addition to cross-play on the training environment, we evaluate all agent pairs in a held-out test environment with a different runner configuration and report these results separately. The results are shown in Fig.˜3. D.4 Hanabi Environments. Hanabi is a cooperative card game where players act as distracted pyrotechnicians who must work together to launch a spectacular fireworks display by playing cards in the correct sequence, with the twist that you can see everyone’s cards except your own and is a canonical benchmark for collaboration. At each turn, the active player chooses from moves such as playing a card to the shared firework piles, discarding a card to regain an information token, or hinting another player about the color or rank of cards in their hand; hints consume a finite pool of information tokens, and illegal plays consume life tokens. Following Lauffer et al. (2025), we consider a version with 33 colors and 33 ranks. We conduct our experiments in both the 22 player and the 44 player settings. Training and evaluation. Each run uses a single GPU with 1 training thread and 1000 rollout threads, collecting trajectories of episode length 100100. We train for 3×1073× 10^7 environment steps with PPO updates using 15 epochs and 1 minibatch per update. We set the learning rates to 7×10−47× 10^-4 (actor) and 1×10−31× 10^-3 (critic), and set the initialization gain to 0.010.01. We set the entropy regularization to 0.0010.001 and the risk aversion parameter τ=0.01τ=0.01. In the 22-player setting, policies use a 22-layer MLP with hidden size 128. In the 44-player setting, policies use a 22-layer MLP with hidden size 256, and each agent is randomly set to be adversary agent during each rollout for SRPO. For each setting, we construct a 20×2020× 20 cross-play matrix consisting of 1010 IPPO agents and 1010 SRPO agents, trained independently with different seeds. Each entry is averaged over 5050 evaluation episodes of length 100100. The results are shown in Figs.˜7 and 8. (a) Hanabi 22-player. (b) Hanabi 44-player. Figure 7: Cross-play performances of SRPO (ϵ=0.001,τ=0.01ε=0.001,τ=0.01) and IPPO (ϵ=0.001ε=0.001) in both 22-player and 44-player hanabi games. Each square represents the average reward of two agents across 50 runs of length 100. (a) Hanabi 22-player. (b) Hanabi 44-player. Figure 8: Performance change (mean and standard deviation) between Training Performance (TP) and Cross-play Performance (CP): the performance of IPPO drastically decreases, with lower average and larger standard deviation in cross-play, while the performance of SRPO drops less severely. D.5 Multi-agent debate on GSM8k Environment. In this setup, two agents engage in a three-round iterative debate protocol. In the first round, each agent independently observes the question and produces its own reasoning and answer. In subsequent rounds, each agent observes the original question as well as both agents’ outputs from the previous round, and then refines its response. Successfully solving a problem therefore requires more than producing a correct answer in isolation: each agent must reinforce correct reasoning while remaining robust to potentially misleading or incorrect proposals from its teammate. This naturally induces a cooperative yet adversarial interaction, where robustness to the partner’s policy plays a critical role in achieving reliable joint performance. Both IPPO (the existing state-of-the-art) and SRPO agents are trained using multiple base language models, including Qwen2.5-0.5B-Instruct (Q0.5B) and Qwen2.5-3B-Instruct (Q3B) (Bai et al., 2025), as well as Qwen3-0.6B (Q0.6B) and Qwen3-4B-Instruct-2507 (Q4B) (Yang et al., 2025), using the verl training framework (Sheng et al., 2024). Here, we set the entropy coefficient to be ϵ=0ε=0 for both IPPO and SRPO, and τ=10τ=10 for SRPO. Qwen2.5-0.5B-Instruct is trained with 200200 epochs, and Qwen3-0.6B is trained with 300300 steps. Every 1010 epochs, we evaluate the performances on a held-out validation set using the same debate setup. The results are shown in Figs.˜10, 9, 11 and 12. Specifically, the initial accuracy before debate reflects the agent’s reasoning abilities, while the final accuracy after debate reflects the agent’s coordination abilities. The results show that, compared to IPPO, SRPO only changes the agent’s coordination abilities, instead of the reasoning abilities. The final debate accuracy achieved by SRPO during training is slightly lower than that of IPPO, as SRPO is trained in the presence of an adversarial partner, whereas IPPO is not. (a) Initial accuracy before debate. (b) Final accuracy after debate. Figure 9: The training curve for SRPO and IPPO with Qwen2.5-0.5B-Instruct. We point out that SRPO is trained against an adversary—meaning lower training reward (post debate) is expected as the adversary learns to mislead the agent to minimize reward. (a) Initial accuracy before debate. (b) Final accuracy after debate. Figure 10: The training curve for SRPO and IPPO with Qwen3-0.6B. We point out that SRPO is trained against an adversary—meaning lower training reward (post debate) is expected as the adversary learns to mislead the agent to minimize reward. (a) Initial accuracy before debate. (b) Final accuracy after debate. Figure 11: The training curve for SRPO and IPPO with Qwen2.5-3B-Instruct. We point out that SRPO is trained against an adversary—meaning lower training reward (post debate) is expected as the adversary learns to mislead the agent to minimize reward. (a) Initial accuracy before debate. (b) Final accuracy after debate. Figure 12: The training curve for SRPO and IPPO with Qwen3-4B-Instruct-2507. We point out that SRPO is trained against an adversary—meaning lower training reward (post debate) is expected as the adversary learns to mislead the agent to minimize reward. D.6 Ablation study We conduct ablation studies on both the risk aversion parameter τ and the entropy coefficient ϵε in the Overcooked and Tag environments. The results are shown in Figs.˜13 and 14. While prior work (Forkel and Foerster, 2025) demonstrates that tuning the entropy coefficient ϵε can improve cross-play performance of IPPO in the Hanabi environment, we find that this strategy does not generalize to Overcooked or Tag. In these environments, entropy tuning alone fails to yield robust partner generalization, suggesting that entropy regularization is not a principled mechanism for addressing coordination under partner shifts. (a) Overcooked. (b) Tag. Figure 13: Ablation study on the entropy parameter ϵε. (a) Overcooked. (b) Tag. Figure 14: Ablation study on the degree of risk aversion parameter τ; τ=0τ=0 refers to the risk-neutral case, i.e., IPPO.

Instant research discovery