Paper deep dive
Phi-Actor-Critic: Steering General-Sum Games to Pareto-Efficient Correlated Equilibria
Wongyu Lee, Francesco Lelli, Omran Ayoub, Massimo Tornatore
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 7/9/2026, 4:39:58 AM
Summary
The paper introduces Phi-Actor-Critic (Phi-AC), a deep multi-agent reinforcement learning framework that leverages swap regret minimization and a centralized attention critic to steer general-sum games toward Pareto-efficient correlated equilibria, overcoming the tendency of standard MARL methods to converge to suboptimal Nash equilibria.
Entities (9)
Relation Signals (9)
Phi-Actor-Critic → addresses → General-Sum Games
confidence 95% · To address this limitation, we propose Phi-Actor-Critic (Phi-AC), a framework that leverages swap regret minimization to steer learning toward high-welfare correlated equilibria (CE).
Phi-Actor-Critic → employs → Swap Regret Minimization
confidence 94% · leverages swap regret minimization to steer learning toward high-welfare correlated equilibria (CE).
Phi-Actor-Critic → steerslearningtoward → Correlated Equilibria
confidence 93% · steer learning toward high-welfare correlated equilibria (CE).
Phi-Actor-Critic → uses → Centralized Attention Critic
confidence 92% · Phi-AC employs a centralized attention critic that predicts vector-valued regrets in a single forward pass
Phi-Actor-Critic → introduces → RB-SWO
confidence 91% · We therefore introduce the Regret-Balancing Social Welfare Objective (RB-SWO), which biases learning toward Pareto-efficient CE
Phi-Actor-Critic → evaluatedon → MPE
confidence 90% · Experiments on matrix games, Multi-Agent Particle Environments (MPE), and the Melting Pot Harvest scenario demonstrate
RB-SWO → uses → Lagrangian Dual Optimization
confidence 90% · We solve this via the primal-dual method. Introducing the dual variables... we minimize the following objective for actor parameters
Phi-Actor-Critic → outperforms → QMIX
confidence 88% · outperforming strong MARL baselines in both sustainability and fairness.
Phi-Actor-Critic → outperforms → MAPPO
confidence 88% · outperforming strong MARL baselines in both sustainability and fairness.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Real-world multi-agent systems, from traffic coordination to resource allocation, are often modeled as general-sum games where individual incentives conflict with collective welfare. In these settings, the central challenge is not merely finding an equilibrium, but selecting socially desirable outcomes among many suboptimal Nash equilibria. Standard deep multi-agent reinforcement learning (MARL) methods struggle with this problem, as value-decomposition approaches are constrained by monotonicity assumptions and policy-gradient methods often converge to stable but socially inefficient equilibria. To address this limitation, we propose $\Phi$-Actor-Critic ($\Phi$-AC), a framework that leverages swap regret minimization to steer learning toward high-welfare correlated equilibria (CE). To make counterfactual regret estimation tractable in deep MARL, $\Phi$-AC employs a centralized attention critic that predicts vector-valued regrets in a single forward pass, avoiding computationally expensive counterfactual simulations. We further introduce a Lagrangian-based equilibrium selection mechanism that optimizes social welfare while enforcing stability through regret constraints. Experiments on matrix games, Multi-Agent Particle Environments (MPE), and the Melting Pot Harvest scenario demonstrate that $\Phi$-AC learns efficient and stable coordination strategies across diverse mixed-motive settings while maintaining high collective return and competitive fairness.
Tags
Links
- Source: https://arxiv.org/abs/2606.11284v1
- Canonical: https://arxiv.org/abs/2606.11284v1
Trouble viewing inline? Open PDF directly →
Full Text
57,491 characters extracted from source content.
Expand or collapse full text
Phi-Actor-Critic: Steering General-Sum Games to Pareto-Efficient Correlated Equilibria Wongyu Lee1 Francesco Lelli2 Omran Ayoub3&Massimo Tornatore1 1Politecnico di Milano 2Tilburg University 3University of Applied Sciences and Arts of Southern Switzerland wongyu.lee@polimi.it, f.lelli@tilburguniversity.edu, omran.ayoub@supsi.ch, massimo.tornatore@polimi.it Abstract Real-world multi-agent systems, from traffic coordination to resource allocation, are often modeled as general-sum games where individual incentives conflict with collective welfare. In these settings, the central challenge is not merely finding an equilibrium, but selecting socially desirable outcomes among many suboptimal Nash equilibria. Standard deep multi-agent reinforcement learning (MARL) methods struggle with this problem, as value-decomposition approaches are constrained by monotonicity assumptions and policy-gradient methods often converge to stable but socially inefficient equilibria. To address this limitation, we propose Φ -Actor-Critic (Φ -AC), a framework that leverages swap regret minimization to steer learning toward high-welfare correlated equilibria (CE). To make counterfactual regret estimation tractable in deep MARL, Φ -AC employs a centralized attention critic that predicts vector-valued regrets in a single forward pass, avoiding computationally expensive counterfactual simulations. We further introduce a Lagrangian-based equilibrium selection mechanism that optimizes social welfare while enforcing stability through regret constraints. Experiments on matrix games, Multi-Agent Particle Environments (MPE), and the Melting Pot Harvest scenario demonstrate that Φ -AC learns efficient and stable coordination strategies across diverse mixed-motive settings while maintaining high collective return and competitive fairness. 1 Introduction Achieving socially efficient coordination, where agents collectively obtain higher welfare, remains challenging in multi-agent systems with partially conflicting incentives. Even when higher-welfare collective strategies exist, independent learning dynamics in mixed-motive environments often converge to individually rational but socially inefficient behaviors, such as risk-averse coordination or persistent asymmetric roles Shoham and Leyton-Brown (2008). Existing deep multi-agent reinforcement learning (MARL) methods often struggle to avoid such outcomes. Value-decomposition approaches such as QMIX Rashid et al. (2020) rely on restrictive monotonicity assumptions, while policy-gradient methods (PGMs) such as MAPPO Yu et al. (2022) lack mechanisms that explicitly select socially desirable equilibria. As a result, these methods may converge to stable but low-welfare Nash equilibria (NE). To address this limitation, we shift the focus from standard reward maximization to regret-based equilibrium selection rooted in algorithmic game theory (AGT) Hart and Mas-Colell (2000); Blum and Mansour (2007); Cesa-Bianchi and Lugosi (2006). Rather than targeting only independent NE, regret-based learning enables correlated solution concepts that support richer coordination in general-sum games. In particular, while external regret minimization is associated with broad coarse correlated equilibria (CCE), swap-regret-based learning refines the target to the tighter class of correlated equilibria (CE) by evaluating whether an agent would benefit from conditionally replacing one action with another Hart and Mas-Colell (2000); Blum and Mansour (2007). Based on this insight, we propose Φ -Actor-Critic (Φ -AC), a regret-aware framework designed to operationalize swap regret for equilibrium selection in deep MARL. We address two practical challenges in applying swap regret to deep MARL: scalable regret estimation and equilibrium selection. First, since calculating swap regret typically requires computationally expensive counterfactual evaluation, we introduce a centralized attention critic that efficiently predicts vector-valued regrets in a single forward pass (O(1)O(1)). Second, since low regret only encourages behavior consistent with approximate CE conditions, it does not specify which equilibrium within the CE set should be selected. We therefore introduce the Regret-Balancing Social Welfare Objective (RB-SWO), which biases learning toward Pareto-efficient CE by optimizing social welfare under regret constraints. By formulating the learning problem as a constrained Markov game optimized via a Lagrangian objective, Φ -AC encourages socially efficient behavior while maintaining low swap regret. Our contributions are threefold: 1. Efficient Regret Estimation. We introduce a regret-conditioned critic that predicts swap regret in a single forward pass, avoiding expensive counterfactual simulations in high-dimensional environments. 2. Principled Equilibrium Selection. We introduce a Lagrangian-based selection mechanism that steers learning toward efficient and fair equilibria under regret constraints. 3. Scalable Coordination in Social Dilemmas. We demonstrate that Φ -AC learns high-welfare coordination strategies in matrix games and scales to complex Sequential Social Dilemmas (SSDs), outperforming strong MARL baselines in both sustainability and fairness. 2 Related Work Scalable Deep MARL. Centralized-training decentralized-execution (CTDE) has become a standard paradigm for scalable MARL, allowing methods such as MADDPG Lowe et al. (2017) and COMA Foerster et al. (2018) to use centralized critics during training while preserving decentralized execution. Within this paradigm, COMA improves credit assignment through counterfactual baselines, while value-decomposition methods such as QMIX Rashid et al. (2020) further improve scalability by factorizing the joint action-value function through a monotonic mixing assumption. Recent PGM such as MAPPO Yu et al. (2022) also demonstrate strong empirical performance across cooperative MARL benchmarks. Despite these advances, these methods remain primarily optimized for expected return. QMIX is further constrained by its monotonic mixing assumption, which can be restrictive in mixed-motive games, while PGMs do not explicitly impose regret-based equilibrium selection. Thus, they may achieve stable learning or high returns without controlling whether the selected outcome is socially desirable. Regret-Based Learning. Regret-minimization methods provide a principled route to equilibrium computation. Counterfactual Regret Minimization (CFR) and its neural variants, including Deep CFR and DREAM, minimize counterfactual regret and provide convergence guarantees primarily in two-player zero-sum imperfect-information games Zinkevich et al. (2007); Brown et al. (2019); Steinberger et al. (2020). Similarly, NeuRD connects policy-gradient learning to Hedge Freund and Schapire (1997) and replicator dynamics in settings where the equilibrium objective is commonly formulated as approximate Nash convergence Hennes et al. (2020). In contrast, Φ -AC targets general-sum MARL, where swap-regret-inspired learning naturally relates to correlated-equilibrium conditions and where the main challenge is selecting efficient CE rather than computing a zero-sum Nash solution. Positioning of Φ -AC. Φ -AC bridges these lines of work by combining scalable centralized-critic MARL with swap-regret-based equilibrium selection. Rather than only improving return maximization or Nash convergence, Φ -AC uses learned swap-regret signals to steer policies toward Pareto-efficient CE while remaining applicable to high-dimensional mixed-motive environments. 3 Preliminaries We formulate the environment as a general-sum Markov game and study equilibrium selection through the lens of CE. 3.1 General-Sum Markov Games We model the multi-agent environment as a General-Sum Markov Game (MG), formally defined by the tuple =⟨,,ii∈,,rii∈,γ,Ω,⟩.G= ,S,\A_i\_i ,P,\r_i\_i ,γ, ,O . Here, =1,…,NN=\1,…,N\ is the set of agents, S is the global state space, :×→Δ()P:S×A→ (S) denotes the state transition function, ri:×→ℝr_i:S×A is the reward function for agent i, and γ∈[0,1)γ∈[0,1) is the discount factor. We assume a partially observable setting where each agent i receives a local observation oi∈Ωio_i∈ _i through the observation function :×→ΩiO:S×N→ _i. The joint action is denoted by =(ai,−i)∈≡∏ia=(a_i,a_-i) ≡ _iA_i. 3.2 Correlated Equilibrium and OSDP While NE assumes independent policies, CE Aumann (1974) allows coordination via a correlation device, often enabling higher-welfare outcomes. In Markov games, however, equilibrium conditions involve deviations over sequential decision processes rather than a single normal-form interaction. The One-Shot Deviation Principle (OSDP) states that, under standard dynamic-game conditions, it is sufficient to check whether any agent can profit by deviating at a single decision point while following the original policy thereafter Hendon et al. (1996). We use this principle as a local approximation of sequential deviation incentives. In our setting, this amounts to measuring whether an agent can improve its return by unilaterally changing its current action, using the stationary Q-function Qi(s,)Q_i π(s,a) as a local estimate of deviation incentives. We utilize the framework of Φ -Equilibria Greenwald and Jafari (2003). Let Φi _i be a set of deviation functions ϕ:i→iφ:A_i _i for agent i, where each deviation maps an action aia_i to an alternative action ϕ(ai)φ(a_i). Definition 1 (Φ -Equilibrium Greenwald and Jafari (2003)). A joint policy π constitutes a Φ -Equilibrium at state s if no agent i can improve their utility by applying any deviation ϕ∈Φiφ∈ _i: ∑∈(|s)Qi(s,)≥∑∈(ϕ∘i)(|s)Qi(s,),∀i∈,∀ϕ∈Φi. _a π(a|s)Q_i π(s,a)≥ _a (φ _i π)(a|s)Q_i π(s,a),\\ ∀ i ,∀φ∈ _i. (1) Here, (ϕ∘i)(φ _i π) denotes the joint distribution where agent i follows the deviation ϕφ while others follow π. When Φi _i includes all possible swap deviations, Eq. (1) characterizes CE. 3.3 Convergence via No-Swap-Regret Dynamics A fundamental result in AGT links regret minimization to equilibrium convergence Blum and Mansour (2007); Cesa-Bianchi and Lugosi (2006). We define the instantaneous swap regret vector following standard swap-regret formulations Hart and Mas-Colell (2000); Blum and Mansour (2007). iΦ(s,)∈ℝ|i| R_i (s,a) ^|A_i| where each element corresponds to the counterfactual gain of swapping the chosen action aia_i to a specific alternative ai′∈ia _i _i: iΦ(s,)[a′]=[Qi(s,(a′,−i))−Qi(s,)]+. R_i (s,a)[a ]= [Q_i π(s,(a ,a_-i))-Q_i π(s,a) ]^+. (2) where [⋅]+[·]^+ denotes the ReLU operator. This vector formulation allows the critic to estimate regret for all possible deviations simultaneously, which is crucial for scalable optimization. Proposition 1 (No-Φ -Regret to Approximate Φ -Equilibrium Hart and Mas-Colell (2000); Greenwald and Jafari (2003); Blum and Mansour (2007)). Under standard finite repeated-game dynamics, let xt∈Δ()x_t∈ (A) be the joint action distribution induced at round t, and let ui(xt)u_i(x_t) denote agent i’s expected utility under xtx_t. Define RTΦ,i=maxϕ∈Φi∑t=1T[ui(ϕ∘ixt)−ui(xt)].R_T ,i= _φ∈ _i _t=1^T [u_i(φ _ix_t)-u_i(x_t) ]. If every agent satisfies RTΦ,i/T≤ϵR_T ,i/T≤ε, then the empirical distribution x¯T=1T∑t=1Txt x_T= 1T _t=1^Tx_t is an ϵε-Φ -equilibrium. When Φi _i contains all swap deviations, x¯T x_T is an ϵε-CE. The Selection Gap. While Proposition 1 establishes that minimizing swap regret leads to approximate CE behavior, it does not determine which equilibrium is selected. In general-sum games, some equilibria may still be socially inefficient or unfair, necessitating an explicit selection mechanism. In Section 4, we address this challenge via a constrained Markov game formulation using a Lagrangian objective that favors socially efficient equilibria. 4 Methodology: Φ -Actor-Critic We propose Φ -AC, a deep MARL framework designed to encourage efficient coordination through swap-regret-based learning. The framework consists of three components: scalable regret estimation (Sec. 4.1), constrained equilibrium selection (Sec. 4.2), and exploration dynamics (Sec. 4.3). The overall centralized training optimization orchestrating these components is summarized in Algorithm 1. 4.1 Scalability: The Regret-Conditioned Critic From Simulation to Prediction. The theoretical convergence to CE relies on calculating Instantaneous Swap Regret (Eq. 2). In tabular settings with a simulator, one can simply reset the environment to query “what if” for every unchosen action (O(N||)O(N|A|) complexity). However, in real-time deployment where resetting is impossible, explicitly computing these counterfactual rewards becomes prohibitively expensive. Φ -AC solves this bottleneck by training a centralized attention mechanism inspired by MAAC Iqbal and Sha (2019) to explicitly predict these counterfactual values. This provides dense regret estimates through a single forward pass, improving scalability in high-dimensional environments. As depicted in Figure 1, we design a dual-head critic to jointly model value estimation and swap-regret prediction. To account for non-stationary learning dynamics induced by concurrently adapting agents, we condition the critic on running cumulative regret cum R^cum via a Feature-wise Linear Modulation (FiLM) layer, which dynamically scales and shifts critic features Perez et al. (2018). The critic consists of two heads: • Q-Head: Estimates action-values Qiψ(s,,cum)Q^ψ_i(s,a, R^cum). • Regret-Head: Predicts the regret vector ^iψ∈ℝ|i| R^ψ_i ^|A_i|, where each element approximates the counterfactual gain of switching from aia_i to ai′a _i. To stabilize regret estimation, we compute the regression target itarget R^target_i using a Target Network ψ′ψ with parameters lagging ψ: itarget(a′)=[Qiψ′(s,(a′,−i))−Qiψ′(s,)]+. R^target_i(a )= [Q^ψ _i(s,(a ,a_-i))-Q^ψ _i(s,a) ]^+. (3) The critic minimizes the joint loss, including ℒR=1N∑i‖itarget−^iψ‖22L_R= 1N _i\| R^target_i- R^ψ_i\|_2^2. Algorithm 1 Centralized Training Optimization in Φ -AC 1:Input: Minibatch ℬ=(,,,′)B=(o,a,r,o ) from D, Current Cumulative Regret cum R^cum. 2:Hyperparameters: δregret _regret (Tolerance), δ (EMA Decay rate), ηα _α (Dual LR). 3:// 1. Regret-Conditioned Critic Update (Sec. 4.1) 4:Calculate TD targets yiy_i and regret targets itarget R^target_i using target networks ψ′,θ′ψ ,θ . 5:Get outputs from dual-head Critic conditioned on cum R^cum via FiLM: 6: Qiψ,^iψ←Critic(,,cum)\Q^ψ_i, R^ψ_i\ (o,a, R^cum) 7:Update Critic ψ by minimizing the joint loss: 8: ℒCritic=1N∑i(ℒQ(Qiψ,yi)+ℒR(^iψ,itarget))L_Critic= 1N _i (L_Q(Q^ψ_i,y_i)+L_R( R^ψ_i, R^target_i) ) 9:// 2. Actor & Dual Update via RB-SWO (Sec. 4.2) Sample differentiable actions ~∼θ() a π_θ(o) via Gumbel-Softmax using regret-biased logits (Eq. (10)) 10:Get Critic predictions Qiψ(~),^iψ(~)\Q^ψ_i( a), R^ψ_i( a)\ evaluated at the sampled actions. 11:Calculate Regret Magnitudes ℳi←‖[^iψ(~)]+‖2M_i←\|[ R^ψ_i( a)]^+\|_2. 12:[Primal] Update Actor θ by minimizing: 13: ℒtotal←ℒRB-SWO(θ,)L_total _RB-SWO(θ, α) (Eq. (8)) 14:[Dual] Update multipliers αifair,αient _i^fair, _i^ent via gradient ascent: 15: αifair←[αifair+ηα(ℳi−δregret)]+ _i^fair← [ _i^fair+ _α(M_i- _regret) ]^+ 16: αient←[αient+ηα(ℋitarget−ℋ(πθi))]+ _i^ent← [ _i^ent+ _α(H^target_i-H( _ _i)) ]^+ 17:// 3. Cumulative Regret & Target Networks Update (Sec. 4.3) 18:Update Cumulative Regret using Exponential Moving Average (EMA): 19: icum←δicum+(1−δ)^iψ() R^cum_i←δ R^cum_i+(1-δ) R^ψ_i(a) 20:Soft update target networks ψ′ψ and θ′θ . 4.2 Selection: Lagrangian Dual Optimization Minimizing swap regret encourages stability, but it does not determine which equilibrium is selected when multiple low-regret equilibria exist. To address this selection gap, we introduce the Regret-Balancing Social Welfare Objective (RB-SWO), which maximizes collective welfare while constraining each agent’s regret magnitude. This formulation favors high-welfare coordination without allowing the policy to move far from the low-regret region. Formulation as Constrained Optimization. We formalize RB-SWO as a constrained optimization problem over expected discounted returns. We explicitly define the Regret Magnitude ℳi()M_i( π) as the expected L2L_2-norm of the positive swap regret vector: ℳi()=[‖[^iψ(s,)]+‖2]M_i( π)=E [ \| [ R^ψ_i(s,a) ]^+ \|_2 ] (4) We also define the policy entropy as ℋ(πi)=ai∼πi[−logπi(ai|oi)]H( _i)=E_a_i _i[- _i(a_i|o_i)], with ℋitarget(t)H^target_i(t) denoting a time-varying minimum entropy target. The optimization problem is: max _ π J()=s∼d,∼[∑i∈Qi(s,)] J( π)=E_s d π,a π [ _i Q_i π(s,a) ] (5) s.t. ℳi()≤δregret,∀i∈ _i( π)≤ _regret, ∀ i (6) ℋ(πi)≥ℋitarget(t),∀i∈. ( _i) ^target_i(t), ∀ i . (7) Figure 1: The architecture of Φ -AC. (Left) Decentralized actors select actions using local observations and cumulative regret bias. (Right) The centralized attention critic processes global states and cumulative regret to output both Q-values and explicit regret predictions. (Center) The Regret Coordinator modulates the critic via FiLM based on the cumulative regret feedback loop. where δregret _regret controls the allowed regret magnitude, and ℋitarget(t)H^target_i(t) is annealed over training. We employ the Gumbel-Softmax reparameterization trick Jang et al. (2016) to enable differentiable optimization of the regret constraint. Lagrangian Dual Optimization. We solve this via the primal-dual method. Introducing the dual variables =αifair,αienti=1N, α=\ _i^fair, _i^ent\_i=1^N, where αifair≥0 _i^fair≥ 0 controls regret-based regularization and αient≥0 _i^ent≥ 0 encourages exploration, we minimize the following objective for actor parameters θ: ℒtotal(θ,)=[−∑iQi⏟Max Welfare+∑iαifair(ℳi−δregret)+∑iαient(ℋitarget−ℋ(πi))]. splitL_total(θ, α)=E [& - _iQ π_i_Max Welfare+ _i _i^fair(M_i- _regret)\\ &+ _i _i^ent(H^target_i-H( _i)) ]. split (8) The optimization proceeds as an alternating game: • Primal Update (Actor): Update θ to minimize ℒtotalL_total. • Dual Update (Multipliers): Update multipliers via gradient ascent with step size ηα>0 _α>0: αifair←[αifair+ηα(ℳi−δregret)]+. _i^fair← [ _i^fair+ _α(M_i- _regret) ]^+. (9) When regret increases, the corresponding dual variable also increases, placing greater emphasis on stability during optimization. 4.3 Dynamics: Entropy Annealing & Warm-up A practical challenge in regret-based learning is that agents cannot regret actions they have never explored. To resolve this, we implement an explicit schedule: Discovery (Warm-up). For the first TwarmT_warm steps (a predefined hyperparameter), we suppress the fairness penalty (αfair→0α^fair→ 0) and enforce a high entropy target. This forces agents to explore the state space purely via Maximum Entropy RL (MaxEnt RL) Haarnoja et al. (2018), accumulating the reward experiences necessary to define well-defined regret. Convergence (Annealing). Post warm-up, we activate the regret constraints and linearly decay the entropy, encouraging diverse exploration before convergence to stable coordinated behaviors. 5 Theoretical Analysis In this section, we provide a theoretical interpretation of Φ -AC by connecting its policy update to no-swap-regret learning and by explaining how the proposed Lagrangian objective biases equilibrium selection. Rather than relying on exact tabular regret computation, Φ -AC uses a learned critic to approximate regret signals in a differentiable deep MARL setting. 5.1 Approximating Regret Matching via Deep Learning We next interpret the actor update in Φ -AC through the lens of regret matching. Actor as Smooth Regret Matching. Classically, convergence to CE can be achieved via Regret Matching (RM) Hart and Mas-Colell (2000), where agents sample actions proportionally to their positive regret values, i.e., π(ai)∝[iΦ(ai)]+π(a_i) [ R _i(a_i)]^+. However, the non-differentiability of the standard RM operator makes it unsuitable for gradient-based optimization in deep networks. To bridge this gap, Φ -AC operationalizes RM via a Regret-Biased Softmax Policy. By injecting the predicted cumulative regret ^cum R^cum into the logits, the policy becomes: πθ(ai|oi)=exp((Li(oi,ai)+β[^icum(ai)]+)/τ)∑a′exp((Li(oi,a′)+β[^icum(a′)]+)/τ). _θ(a_i|o_i)= ((L_i(o_i,a_i)+β[ R^cum_i(a_i)]^+)/τ ) _a ((L_i(o_i,a )+β[ R^cum_i(a )]^+)/τ ). (10) Here, Li(oi,ai)L_i(o_i,a_i) denotes the raw logits output by the actor network prior to softmax normalization, τ is the temperature, and β controls the strength of the regret bias. Lower temperatures make the policy more sensitive to positive regret, while larger β places greater weight on actions with accumulated regret. Together, these terms yield a smooth and differentiable approximation to regret-driven action selection. This formulation can be interpreted as a smooth approximation of RM. While standard RM selects actions in proportion to positive regret, our softmax-based formulation is conceptually related to exponential weights and Hedge algorithms Freund and Schapire (1997); Cesa-Bianchi and Lugosi (2006), which are also known to induce no-regret behavior under standard online learning assumptions. Approximate Regret under Function Approximation. In deep MARL, regret estimates are inherently approximate due to function approximation. Following standard analyses of perturbed no-regret dynamics Cesa-Bianchi and Lugosi (2006), we consider the setting where the centralized critic provides uniformly bounded regret estimation error: maxs,‖true−^ψ‖∞≤ϵc. _s,a\| R^true- R^ψ\|_∞≤ _c. Under policy updates that are sufficiently smooth with respect to the regret estimates, the induced perturbation scales with ϵc _c, implying that the resulting average swap regret remains bounded up to an (ϵc)O( _c) approximation term. Consequently, the learned dynamics may approach an approximate CE despite imperfect regret estimation. 5.2 Geometry of Equilibrium Selection While regret minimization encourages behavior consistent with approximate CE conditions, it does not by itself distinguish high-welfare equilibria from inefficient low-regret outcomes. RB-SWO addresses this selection gap by adding welfare maximization under individual regret constraints, yielding a Lagrangian objective that favors high-welfare coordination. Dual Forces in Optimization. We formulate equilibrium selection as maximizing social welfare (W=∑QiW=Σ Q_i) subject to stability constraints (ℳi≤δregretM_i≤ _regret). The Lagrangian objective ℒ(θ,α)=W−∑αi(ℳi−δregret)L(θ,α)=W-Σ _i(M_i- _regret) induces two distinct gradient forces: Efficiency Force (∇θW _θW): This term drives the joint policy along the “flat” directions of the CE polytope (where regret is zero) toward higher-welfare regions of the CE set. Restoring Force (−αi∇θℳi- _i _θM_i): If an agent deviates from equilibrium (increasing ℳiM_i), the dual variable αi _i increases (Dual optimization). This increases the optimization pressure toward lower-regret regions. Fairness via Individual Constraints. Individual regret constraints also play an important role in equilibrium selection. Geometrically, asymmetric equilibria (where one agent yields and another benefits) imply that while the sum of regrets might be low, the individual regret for the yielding agent is high. By enforcing ℳi≤δM_i≤δ for each agent rather than only controlling the average regret, RB-SWO discourages solutions in which one agent absorbs most of the deviation incentive. This encourages more balanced outcomes and reduces winner-takes-all behavior. 6 Experimental Evaluation While the previous sections introduced the regret-aware architecture and optimization framework of Φ -AC, this section empirically evaluates its ability to learn socially efficient coordination. We evaluate Φ -AC across a hierarchy of complexity: diagnostic matrix games, scalable MPE domains, and Melting Pot Harvest. These settings respectively test equilibrium selection in transparent games, scalability in higher-dimensional interactions, and sustainable coordination under shared resource depletion. We compare Φ -AC against representative MARL baselines that capture different algorithmic design choices. MADDPG represents deterministic policy-gradient learning, MAPPO and COMA represent stochastic policy-gradient methods with centralized training, and QMIX represents value decomposition under monotonic factorization. Experimental Setup. Training configurations were tailored to the complexity of each domain: Iterated Matrix Games (IMG) were trained for 1k episodes with a horizon of 25 steps, MPE environments Lowe et al. (2017) for 30k episodes (horizon 25), and Melting Pot (Harvest) Leibo et al. (2021) for 10k episodes (horizon 500). To ensure statistical robustness, all results for MPE and Melting Pot are reported as mean ± standard deviations over 5 independent runs. Figure 2: Policy Trajectories. (Top-L) Prisoner’s Dilemma, (Top-R) Chicken, (Bot) Stag Hunt. Φ -AC (Red) consistently reaches the intended high-welfare equilibria. 6.1 Diagnostic Analysis: Iterated Matrix Games We first employ Iterated Matrix Games to empirically verify the equilibrium convergence of each model. We consider a two-player setting where agents interact repeatedly, allowing them to adapt strategies based on history. We select three canonical games, each representing a distinct coordination failure mode. Fig. 2 visualizes the learning trajectories. Prisoner’s Dilemma. This game tests whether agents can overcome the temptation to defect and coordinate on mutual cooperation. As shown in Fig. 2 (Top-L), MADDPG and MAPPO tend to converge toward the inefficient NE (D,D), while QMIX and Φ -AC reach the social optimum (C,C). Unlike QMIX, which benefits from the monotonic payoff structure in this setting, Φ -AC reaches mutual cooperation through regret-aware coordination. Chicken. This game tests coordination under two asymmetric equilibria, where either agent can benefit if the other yields. As shown in Fig. 2 (Top-R), baselines often drift toward one-sided or unstable outcomes, whereas Φ -AC promotes balanced coordination by reducing regret disparities between agents. Stag Hunt. This game tests whether agents can coordinate on a high-payoff action that is beneficial only when both agents choose it, rather than falling back to the safer but lower-reward option. As shown in Fig. 2 (Bottom), baselines tend to favor the conservative risk-dominant outcome, while Φ -AC consistently reaches the payoff-dominant cooperative equilibrium. MADDPG MAPPO QMIX COMA Φ -AC (a) Coop: Reward (b) Coop: Regret (c) Zero-Sum: Reward (d) Zero-Sum: Regret (e) Mixed: Reward (f) Mixed: Regret Figure 3: Training Dynamics. Rows represent environments: Cooperative, Zero-Sum, and Mixed. Left panels show evaluation rewards, and right panels show cumulative regret. 6.2 Selection under Scale: MPE We extend our evaluation to the MPE to verify whether the selection capability scales to high-dimensional state-action spaces via our centralized attention critic. We report Mean Reward (efficiency), Cumulative Regret (Convergence), and Regret Gap (|ℛicum−ℛjcum||R^cum_i-R^cum_j|, fairness). Table 1 summarizes the quantitative results, and Figure 3 visualizes the training dynamics. Cooperative: Avoiding Inefficient Equilibria. In simple_spread, agents must cover landmarks while avoiding collisions. COMA and QMIX achieve low regret but converge to lower-welfare solutions, suggesting that stability alone does not imply efficient equilibrium selection. In contrast, Φ -AC achieves the highest reward (-42.24). By balancing welfare maximization with regret constraints, it avoids premature stabilization and reaches a higher-welfare configuration. Zero-Sum: Stability vs. Exploitation. High individual rewards in simple_adversary often signal exploitation rather than equilibrium. MAPPO achieves a high reward (91.17) but suffers from large regret gap (5926.82), indicating a non-stationary cycle where one agent continually exploits a weak opponent without convergence. Φ -AC converges to a reward near zero (-3.27) with significantly lower regret, suggesting a more stable low-regret regime in which unilateral exploitation incentives are reduced. Mixed-Motive: Scalable Fairness. simple_tag involves conflicting goals within a general-sum framework. Baselines exhibit large regret gaps (>2000>2000), indicating highly imbalanced deviation incentives consistent with winner-takes-all behavior. Φ -AC reduces this gap by nearly two orders of magnitude (53.79). This suggests that the RB-SWO mechanism can scale to high-dimensional domains, enforcing equity by ensuring that the dissatisfaction (regret) is distributed equitably among agents. Model Reward Cum. Regret Regret Gap Cooperative (Spread) MADDPG −53.55±0.55-53.55± 0.55 2177.97±46.982177.97± 46.98 165.32±61.98165.32± 61.98 MAPPO −55.68±3.11-55.68± 3.11 988.00±82.29988.00± 82.29 527.92±272.01527.92± 272.01 COMA −91.24±13.08-91.24± 13.08 43.72 ± 11.54 39.93 ± 25.37 QMIX −88.49±11.54-88.49± 11.54 390.13±41.66390.13± 41.66 248.44±60.42248.44± 60.42 Φ -AC -42.24 ± 0.21 460.46±34.69460.46± 34.69 260.64±119.98260.64± 119.98 Zero-Sum (Adversary) MADDPG −4.91±3.65-4.91± 3.65 1316.86±105.131316.86± 105.13 187.11±122.92187.11± 122.92 MAPPO 91.17±2.9691.17± 2.96 5470.50±1845.025470.50± 1845.02 5926.82±6044.885926.82± 6044.88 Φ -AC -3.27 ± 1.63 281.30 ± 11.10 43.32 ± 18.28 Mixed-Motive (Tag) MADDPG 6.83±0.936.83± 0.93 1409.98±50.541409.98± 50.54 2321.88±140.222321.88± 140.22 MAPPO 227.69±13.24227.69± 13.24 1084.68±202.971084.68± 202.97 2192.30±654.542192.30± 654.54 Φ -AC 12.55 ± 7.32 481.56 ± 20.51 53.79 ± 12.09 Table 1: Scalability results on MPE. Φ -AC achieves the best balance of efficiency and stability. Note: In zero-sum settings, a reward near 0 with low regret implies a stable NE, whereas high rewards indicate exploitation. 6.3 Ablation on Learning Objectives We dissect the contribution of our proposed learning objectives in simple_tag. The results, summarized in Table 2, reveal that each component is critical for preventing unstable learning outcomes. 1. w/o Regret Matching (RM) Bonus: Removing the regret-bias term (β=0β=0) leads to unstable learning dynamics. The total regret increases substantially to 1933.04 with an extreme standard deviation (± 2007.34). This indicates that without the RM bonus, the learning dynamics fail to converge, oscillating severely or diverging completely. 2. w/o Fairness Objective (RB-SWO): Removing the fairness constraint (αfair→0α^fair→ 0) degrades both stability (481.56→510.11481.56→ 510.11 in Regret) and efficiency. This suggests that the fairness objective acts as a dual-purpose regularizer: it not only prevents inefficient outcomes (“Coordination Collapse”) but also accelerates convergence by constraining the search space to the stable region. Model Variant Final Reward Total Regret Full Φ -AC (Proposed) 12.55 ± 7.32 481.56 ± 20.51 w/o RM Bonus 12.53 ± 7.95 1933.04 ± 2007.34 w/o Fairness Objective 10.94 ± 7.19 510.11 ± 28.46 Table 2: Ablation results on simple_tag. Removing the RM Bonus leads to severe instability (large regret variance), while removing Fairness degrades both convergence and reward. Φ -AC achieves the most stable and efficient equilibrium. 6.4 Robustness in Dilemmas: Melting Pot Finally, we evaluate Φ -AC on Harvest, an SSD that serves as a challenging benchmark for sustainable coordination under shared resource constraints. In this environment, apples regenerate based on the density of nearby unharvested apples, creating a “Tragedy of the Commons” where individually greedy behavior can undermine long-term collective productivity. We compare Φ -AC against MAPPO (centralized policy-gradient baseline) and QMIX (value decomposition baseline) using the Sustainability Index (SI), defined as SI=(1−Gini)×∑iReturni,SI=(1-Gini)× _iReturn_i, which captures the trade-off between productivity and fairness. (a) Collective Reward (b) Gini Coefficient (c) Empirical CE Gap (d) Reward-Regret Dynamics Figure 4: Dynamics in Harvest. (a,b) Φ -AC achieves high collective reward while maintaining competitive fairness. (c) The empirical CE-gap proxy remains low during evaluation. (d) Training reward-regret dynamics show decreasing regret alongside sustained collective reward. Escaping Resource Collapse (Fig. 4 (a,b)). As shown in Table 3, Φ -AC achieves the highest collective reward (134.33) while maintaining competitive fairness relative to MAPPO. In contrast, QMIX suffers from severe inequality and substantially lower collective productivity, resulting in the weakest Sustainability Index. Although MAPPO achieves the lowest Gini coefficient, its lower collective reward leads to a smaller overall sustainability score than Φ -AC. These results suggest that regret-based equilibrium selection helps maintain productive coordination without collapsing into resource-depleting behavior. Reward–Regret Dynamics (Fig. 4 (c,d)). To further analyze the learned coordination dynamics, we additionally measure the empirical CE gap, defined as the maximum positive swap regret estimated by the critic during evaluation. As shown in Fig. 4(c), the empirical CE gap of Φ -AC remains consistently low throughout training, suggesting reduced unilateral deviation incentives among agents. This interpretation is further supported by the reward-regret dynamics in Fig. 4(d). During early training, swap regret temporarily increases as agents explore alternative coordination strategies. As learning progresses, however, the regret signal decreases while collective reward remains high. This indicates that agents gradually settle into a stable high-welfare coordination pattern rather than maximizing reward through unstable exploitative behavior. Model Coll. Reward Gini Coeff Sust. Index Φ -AC (Ours) 134.33 ± 3.46 0.26 ± 0.03 99.41 ± 2.67 MAPPO 116.55 ± 7.96 0.18 ± 0.03 95.48 ± 7.60 QMIX 94.49 ± 25.08 0.40 ± 0.05 59.27 ± 18.27 Table 3: Performance summary in harvest (Evaluation, Last 10%). Φ -AC achieves the highest Sustainability Index while maintaining strong collective productivity. 7 Conclusion In this work, we introduced Φ -AC, a principled framework designed to integrate game-theoretic swap regret minimization with deep MARL. By shifting the paradigm from reward-only optimization to equilibrium selection, Φ -AC addresses the limitations of conventional MARL objectives in distinguishing efficient coordination from suboptimal stability. Our empirical evaluation supports Φ -AC across a hierarchy of complexity. At the diagnostic level (Matrix Games), we visualized robust steering toward social optima. At the scalable level (MPE), we demonstrated effective fairness-aware coordination. Most significantly, in the complex SSD (Harvest), we provided empirical evidence of high-welfare coordination behavior consistent with correlated-equilibrium principles. The observed decrease in smoothed swap regret alongside increasing collective reward suggests that Φ -AC promotes sustained coordination rather than short-term reward maximization. A current limitation is its reliance on discrete action spaces. Future work will extend regret estimation to continuous control, broadening game-theoretic equilibrium selection to physical decision-making domains. References R. J. Aumann (1974) Subjectivity and correlation in randomized strategies. Journal of mathematical Economics 1 (1), p. 67–96. Cited by: §3.2. A. Blum and Y. Mansour (2007) From external to internal regret.. Journal of Machine Learning Research 8 (6). Cited by: §A.1, §1, §3.3, Proposition 1. N. Brown, A. Lerer, S. Gross, and T. Sandholm (2019) Deep counterfactual regret minimization. In International conference on machine learning, p. 793–802. Cited by: §2. N. Cesa-Bianchi and G. Lugosi (2006) Prediction, learning, and games. Cambridge university press. Cited by: §1, §3.3, §5.1, §5.1. J. Foerster, G. Farquhar, T. Afouras, N. Nardelli, and S. Whiteson (2018) Counterfactual multi-agent policy gradients. In Proceedings of the AAAI conference on artificial intelligence, Vol. 32. Cited by: §2. Y. Freund and R. E. Schapire (1997) A decision-theoretic generalization of on-line learning and an application to boosting. Journal of computer and system sciences 55 (1), p. 119–139. Cited by: §2, §5.1. A. Greenwald and A. Jafari (2003) A general class of no-regret learning algorithms and game-theoretic equilibria. In Learning Theory and Kernel Machines: 16th Annual Conference on Learning Theory and 7th Kernel Workshop, COLT/Kernel 2003, Washington, DC, USA, August 24-27, 2003. Proceedings, p. 2–12. Cited by: §3.2, Definition 1, Proposition 1. T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine (2018) Soft actor-critic: off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, p. 1861–1870. Cited by: §4.3. S. Hart and A. Mas-Colell (2000) A simple adaptive procedure leading to correlated equilibrium. Econometrica 68 (5), p. 1127–1150. Cited by: §A.1, §A.1, §1, §3.3, §5.1, Proposition 1. E. Hendon, H. J. Jacobsen, and B. Sloth (1996) The one-shot-deviation principle for sequential rationality. Games and Economic Behavior 12 (2), p. 274–282. Cited by: §3.2. D. Hennes, D. Morrill, S. Omidshafiei, R. Munos, J. Perolat, M. Lanctot, A. Gruslys, J. Lespiau, P. Parmas, E. Duéñez-Guzmán, et al. (2020) Neural replicator dynamics: multiagent learning via hedging policy gradients. In Proceedings of the 19th international conference on autonomous agents and multiagent systems, p. 492–501. Cited by: §2. S. Iqbal and F. Sha (2019) Actor-attention-critic for multi-agent reinforcement learning. In International conference on machine learning, p. 2961–2970. Cited by: §4.1. E. Jang, S. Gu, and B. Poole (2016) Categorical reparameterization with gumbel-softmax. arXiv preprint arXiv:1611.01144. Cited by: §4.2. J. Z. Leibo, E. A. Dueñez-Guzman, A. Vezhnevets, J. P. Agapiou, P. Sunehag, R. Koster, J. Matyas, C. Beattie, I. Mordatch, and T. Graepel (2021) Scalable evaluation of multi-agent reinforcement learning with melting pot. In International conference on machine learning, p. 6187–6199. Cited by: §6. R. Lowe, Y. I. Wu, A. Tamar, J. Harb, O. Pieter Abbeel, and I. Mordatch (2017) Multi-agent actor-critic for mixed cooperative-competitive environments. Advances in neural information processing systems 30. Cited by: §2, §6. E. Perez, F. Strub, H. De Vries, V. Dumoulin, and A. Courville (2018) Film: visual reasoning with a general conditioning layer. In Proceedings of the AAAI conference on artificial intelligence, Vol. 32. Cited by: §4.1. T. Rashid, M. Samvelyan, C. S. De Witt, G. Farquhar, J. Foerster, and S. Whiteson (2020) Monotonic value function factorisation for deep multi-agent reinforcement learning. Journal of Machine Learning Research 21 (178), p. 1–51. Cited by: §1, §2. Y. Shoham and K. Leyton-Brown (2008) Multiagent systems: algorithmic, game-theoretic, and logical foundations. Cambridge University Press. Cited by: §1. E. Steinberger, A. Lerer, and N. Brown (2020) Dream: deep regret minimization with advantage baselines and model-free learning. arXiv preprint arXiv:2006.10410. Cited by: §2. C. Yu, A. Velu, E. Vinitsky, J. Gao, Y. Wang, A. Bayen, and Y. Wu (2022) The surprising effectiveness of ppo in cooperative multi-agent games. Advances in neural information processing systems 35, p. 24611–24624. Cited by: §1, §2. M. Zinkevich, M. Johanson, M. Bowling, and C. Piccione (2007) Regret minimization in games with incomplete information. Advances in neural information processing systems 20. Cited by: §2. Appendix A Theoretical Proofs and Background In this section, we provide the theoretical foundations connecting swap regret minimization to CE and outline the proof for the convergence proposition presented in the main text. A.1 Connection between Swap Regret and CE As established in foundational algorithmic game theory Hart and Mas-Colell [2000]; Blum and Mansour [2007], there is a fundamental equivalence between no-swap-regret dynamics and the set of CE. Recall the definition of the swap-regret vector i,TΦ∈ℝ|Φi| R_i,T ^| _i|, whose components are indexed by deviation mappings ϕ∈Φiφ∈ _i. Let t∈1,…,Tt∈\1,…,T\ denote the discrete time step, ai,t∈ia_i,t _i be the action taken by agent i, and −i,ta_-i,t be the joint action of all other agents. For each deviation ϕφ, the average swap regret over T steps is evaluated using the reward function rir_i: ℛi,TΦ(ϕ)=1T∑t=1T[ri(ϕ(ai,t),−i,t)−ri(ai,t,−i,t)].R_i,T (φ)= 1T _t=1^T [r_i(φ(a_i,t),a_-i,t)-r_i(a_i,t,a_-i,t) ]. (11) A distribution z∈Δ()z∈ (A) over the joint action space A constitutes a CE if no agent can improve their expected utility by unilaterally applying any swap ϕφ. Driving the maximum positive regret, denoted as maxϕ∈Φi[ℛi,TΦ(ϕ)]+ φ∈ _i[Ri,T (φ)]+ (where [⋅]+[·]_+ indicates the positive part operator), to zero for every agent implies that the empirical distribution of play approaches the CE set Hart and Mas-Colell [2000]. Intuition of Regret Matching. To provide deeper intuition for Regret Matching (RM), it is instructive to consider the rationale behind increasing the probability of actions with high positive regret. Intuitively, a positive swap regret ℛ(a→a′)>0R(a→ a )>0 represents a ”missed opportunity”: the additional utility the agent would have gained had they played a′a instead of a. By strictly following RM dynamics (or its smooth approximation via Softmax), the agent probabilistically corrects these past errors, steering the joint policy toward a state where no such missed opportunities exist (i.e., equilibrium). From Normal-Form to Sequential Games. While classical swap regret is defined for normal-form games, extending it to Markov games requires evaluating deviations over entire trajectories. Motivated by the one-shot deviation principle, we approximate trajectory-level deviation incentives through local Markovian deviations evaluated by the stationary action-value function Qiπ(s,)Q_i^π(s,a). This approximation is most naturally interpreted under stationary Markov policies and Markovian deviation classes. Furthermore, while exact theoretical regret relies on the arithmetic time-average, in our implementation, we use an Exponential Moving Average (EMA) as a practical approximation to stabilize regret estimates in non-stationary deep MARL. Unlike the arithmetic average used in the classical convergence proof, EMA emphasizes recent deviation incentives and should be interpreted as an implementation-level stabilization mechanism rather than a separate convergence guarantee. A.2 Proof Sketch of Proposition 1 Proposition 1. Let the critic’s regret prediction error be bounded by ‖^ψ−true‖∞≤ϵc\| R^ψ- R^true\|_∞≤ _c. If agents follow smooth regret matching dynamics using ^ψ R^ψ, the joint empirical distribution converges to the set of ϵε-CE, where ϵ=(ϵc+δregret+τlog|i|)ε=O( _c+ _regret+τ |A_i|). Proof. Let ttrue R^true_t be the true regret vector at time t, and ^tψ R^ψ_t be the critic’s estimate such that ‖^tψ−ttrue‖∞≤ϵc\| R^ψ_t- R^true_t\|_∞≤ _c. Standard no-regret algorithms guarantee that the time-averaged regret vanishes at a rate of (1/T)O(1/ T) using the true regret values. When using the approximate regret ^ψ R^ψ, the policy update rule induces a perturbed dynamic. Specifically, for any agent i and deviation ϕ∈Φiφ∈ _i, the uniform approximation bound implies over finite time T: 1T∑t=1Tℛi,ttrue(ϕ)≤1T∑t=1Tℛ^i,tψ(ϕ)+ϵc. 1T _t=1^TR^true_i,t(φ)≤ 1T _t=1^T R^ψ_i,t(φ)+ _c. (12) Taking the maximum over ϕφ and considering only the positive deviation incentives, the average true regret is bounded by: [maxϕ∈Φi1T∑t=1Tℛi,ttrue(ϕ)]+≤[maxϕ∈Φi1T∑t=1Tℛ^i,tψ(ϕ)]++ϵc. [ _φ∈ _i 1T _t=1^TR^true_i,t(φ) ]_+≤ [ _φ∈ _i 1T _t=1^T R^ψ_i,t(φ) ]_++ _c. (13) Recall that our Lagrangian objective explicitly constrains the L2L_2 norm of the predicted positive regret vector, ℳi≤δregretM_i≤ _regret. Because bounding the L2L_2 norm strictly bounds the maximum element (‖x‖∞≤‖x‖2\|x\|_∞≤\|x\|_2), the learning dynamics successfully minimize the first term on the right-hand side up to the slack δregret _regret. Furthermore, because our actor utilizes a Softmax approximation with temperature τ (as defined in the main text) rather than a strict step-function Regret Matching operator (πt+1∝[^tψ]+ _t+1 [ R^ψ_t]^+), an additional entropy-induced approximation error proportional to (τlog|i|)O(τ |A_i|) is introduced. Taking the asymptotic limit T→∞T→∞, the (1/T)O(1/ T) convergence term vanishes. Thus, the empirical distribution converges to the set of ϵε-CE with ϵ=(ϵc+δregret+τlog|i|)ε=O( _c+ _regret+τ |A_i|). When τ is annealed to zero, this last term vanishes. ∎ Appendix B Detailed Architecture and Implementation B.1 Centralized Attention Critic To avoid explicit enumeration of counterfactual rollouts, we implement a centralized attention critic that predicts vector-valued regret estimates in a single forward pass. Regret-Head Justification & Gradient Flow. While the raw counterfactual advantage can be computed as Qi(s,(ai′,−i))−Qi(s,)Q_i(s,(a_i ,a_-i))-Q_i(s,a), the regret constraint depends on its positive part: [Qi(s,(ai′,−i))−Qi(s,)]+ [Q_i(s,(a_i ,a_-i))-Q_i(s,a) ]_+. Accordingly, the explicit regret head predicts a smoothed non-negative approximation of this positive-part regret. Crucially, as mentioned in the main text, we ensure end-to-end differentiability: 1. Softplus Activation: The Regret-Head uses a Softplus activation to strictly enforce non-negativity (ℛ^≥0 R≥ 0) while maintaining smooth gradients, unlike ReLU which has zero gradients in the negative regime. 2. Gumbel-Softmax: The actor’s policy outputs are sampled via the Gumbel-Softmax trick (Temperature τ=1.0τ=1.0, annealed to 0.10.1). This allows the gradient from the stability constraint ℳi=‖^i‖2M_i=\| R_i\|_2 to backpropagate through the critic to the actor. Network Initialization. To prevent value explosion in the early stages of training, we apply a specific initialization to the Q-Head’s output layer: weights are scaled by 0.010.01 and biases are initialized to 0. This ensures that the initial Q-values are near zero. For the regret head, we initialize the final bias to a small negative value so that the Softplus output starts close to zero, allowing the agents to begin with unbiased exploration. FiLM Modulation. To handle non-stationarity, the Regret Coordinator module uses Feature-wise Linear Modulation (FiLM). The cumulative regret cum R^cum is processed by an MLP to generate scale (γFiLM _FiLM) and shift (βFiLM _FiLM) parameters. Let F denote the intermediate feature representation of the global state and joint action. The modulated feature F′F is computed as: F′=γFiLM(cum)⊙F+βFiLM(cum)F = _FiLM( R^cum) F+ _FiLM( R^cum) (14) This allows the critic to dynamically adapt its attention focus based on the current distance to equilibrium. B.2 Actor Architecture • Vector Inputs (MPE): MLP with 2 hidden layers (128 units), ReLU activation. • Pixel Inputs (Melting Pot): Standard Nature-CNN architecture. – Input: 68×6868× 68 Grayscale (Stacked 4 frames) – Conv1: 32 filters, 8×88× 8, stride 4 – Conv2: 64 filters, 4×44× 4, stride 2 – Conv3: 64 filters, 3×33× 3, stride 1 – FC: 1024 → 128 → Action Dim B.3 Lagrangian Auto-Tuning and Exploration We automate the selection of penalty coefficients α via Dual Gradient Descent: • Fairness: αfairα^fair increases only when Regret Norm >δregret> _regret (0.15). • Entropy Annealing: In our implementation, the entropy target decays from a ratio of 0.8 to 0.05 over the first 10% of episodes to facilitate the transition from Discovery to Convergence. • Regret Exploration Bonus: As listed in the hyperparameters, we utilize an exploration bonus (λregret _regret) to scale intrinsic exploration signals during the initial discovery phase. Appendix C Environment Settings C.1 Matrix Games Payoff For reproducibility, we explicitly detail the payoff matrices used in Section 5.1. The values represent the joint reward (r1,r2)(r_1,r_2) for the row player (Agent 1) and column player (Agent 2). Table 4: Payoff Matrices for Iterated Matrix Games. Actions are denoted as C (Cooperate) and D (Defect). Game Ag 1 2 Agent 2 Action C D Prisoner’s Dilemma C (3, 3) (0, 5) D (5, 0) (1, 1) Chicken C (3, 3) (1, 4) D (4, 1) (0, 0) Stag Hunt C (Stag) (4, 4) (0, 3) D (Hare) (3, 0) (3, 3) C.2 MPE Zero-Sum Modification In simple_adversary, to ensure a rigorous zero-sum evaluation, we modified the reward structure such that the sum of rewards is strictly zero (+10+10 for the adversary, −10-10 for the agents). Since the modified reward is strictly zero-sum, the total collective reward across both sides is zero by construction and cannot be used as a convergence metric. Instead, we evaluate convergence using side-specific returns, regret, and the reduction of exploitability-like performance gaps. In this setting, Φ -AC successfully stabilizes these metrics without oscillating into a regime where one side continually exploits the other, unlike the baselines. C.3 Melting Pot (Harvest) We use the commons_harvest_open substrate with the following preprocessing: • Resolution: Resized to 68×6868× 68. • Frame Stack: Last 4 frames are stacked to capture temporal dynamics. • Action Repeat: Actions are repeated for 4 steps. • Reward: +1+1 for collecting an apple. Zapping removes a target from the game for 25 steps. Appendix D Hyperparameters We provide detailed hyperparameters for both experimental domains. Note that the simpler MPE environment utilizes larger batch sizes and learning rates compared to the pixel-based Melting Pot environment. D.1 MPE and Matrix Games Table 5 lists the settings used for MPE environments (simple_spread, simple_tag, simple_adversary) and matrix games. Table 5: Hyperparameters for MPE & Matrix Games Parameter Value Optimization Total Episodes 30,000 Episode Length 25 Actor Learning Rate 3×10−43× 10^-4 Critic Learning Rate 3×10−43× 10^-4 Batch Size 1024 Buffer Capacity 100,000 Discount Factor (γ) 0.99 Soft Update (τ) 0.01 Φ -AC Specific Regret Exploration Bonus (λregret _regret) 1.0 Regret Bias Scale (β) 1.0 Fairness Threshold (δregret _regret) 0.15 FiLM Scale 1.0 Entropy Decay (Ratio) 1.0→0.051.0→ 0.05 Regret EMA Decay 0.995 D.2 Melting Pot (Harvest) Table 6 lists the settings for the sequential social dilemma experiments. Table 6: Hyperparameters for Melting Pot (Harvest) Parameter Value Optimization Actor Learning Rate 1×10−41× 10^-4 Critic Learning Rate 1×10−41× 10^-4 Batch Size 512 Buffer Capacity 30,000 Discount Factor (γ) 0.99 Soft Update (τ) 0.01 Φ -AC Specific Regret Exploration Bonus (λregret _regret) 1.0 Regret Bias Scale (β) 0.75 Fairness Threshold (δregret _regret) 0.15 FiLM Scale 1.5 Entropy Decay (Ratio) 0.8→0.050.8→ 0.05 Regret EMA Decay 0.90