Paper deep dive
Conservative Equilibrium Discovery in Offline Game-Theoretic Multiagent Reinforcement Learning
Austin A. Nguyen, Michael P. Wellman
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 7/20/2026, 7:50:12 PM
Summary
The paper introduces COffeE-PSRO, a novel framework for offline game-theoretic multiagent reinforcement learning that extends Policy Space Response Oracles (PSRO) with conservatism principles. By training an ensemble dynamics model to quantify uncertainty and modifying the RL objective to penalize high-uncertainty transitions, the method skews strategy exploration toward solutions with low regret in the true game. A new meta-strategy solver, Robust Replicator Dynamics (R2D), is proposed to guide this exploration, demonstrating superior performance in extracting low-regret equilibria compared to state-of-the-art offline approaches.
Entities (8)
Relation Signals (6)
COffeE-PSRO → extends → Policy Space Response Oracles
confidence 95% · Specifically, we extend Policy Space Response Oracles (PSRO), an online game-solving approach, by quantifying game dynamics uncertainty...
COffeE-PSRO → incorporates → Conservatism
confidence 95% · Our incorporation of Conservatism principles from Offline reinforcement learning approaches for strategy Exploration gives our approach its name: COffeE-PSRO.
Robust Replicator Dynamics → isa → meta-strategy solver
confidence 92% · We further propose a novel meta-strategy solver... Our approach accounts for robust offline evaluation, giving our MSS its name: Robust Replicator Dynamics (R2D).
Ensemble Dynamics Model → quantifies → Uncertainty
confidence 90% · Given a dataset of trajectories... we first train an ensemble dynamics model... and use prediction differences within the ensemble to quantify uncertainty...
COffeE-PSRO → uses → Robust Replicator Dynamics
confidence 90% · Our experiments suggest that R2D yields the best results.
COffeE-PSRO → outperforms → state-of-the-art offline approaches
confidence 85% · Experiments demonstrate COffeE-PSRO's ability to extract lower-regret solutions than state-of-the-art offline approaches...
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Offline learning of strategies takes data efficiency to its extreme by restricting algorithms to a fixed dataset of state-action trajectories. We consider the problem in a mixed-motive multiagent setting, where the goal is to solve a game under the offline learning constraint. We first frame this problem in terms of selecting among candidate equilibria. Since datasets may inform only a small fraction of game dynamics, it is generally infeasible in offline game-solving to even verify a proposed solution is a true equilibrium. Therefore, we consider the relative probability of low regret (i.e., closeness to equilibrium) across candidates based on the information available. Specifically, we extend Policy Space Response Oracles (PSRO), an online game-solving approach, by quantifying game dynamics uncertainty and modifying the RL objective to skew towards solutions more likely to have low regret in the true game. We further propose a novel meta-strategy solver, tailored for the offline setting, to guide strategy exploration in PSRO. Our incorporation of Conservatism principles from Offline reinforcement learning approaches for strategy Exploration gives our approach its name: COffeE-PSRO. Experiments demonstrate COffeE-PSRO's ability to extract lower-regret solutions than state-of-the-art offline approaches and reveal relationships between algorithmic components empirical game fidelity, and overall performance.
Tags
Links
- Source: https://arxiv.org/abs/2603.00374v2
- Canonical: https://arxiv.org/abs/2603.00374v2
Trouble viewing inline? Open PDF directly →
Full Text
56,996 characters extracted from source content.
Expand or collapse full text
ifaamas [AAMAS ’26]Proc. of the 25th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2026)May 25 – 29, 2026 Paphos, CyprusC. Amato, L. Dennis, V. Mascardi, J. Thangarajah (eds.) 2026 2026 25 of Michigan Arbor States of Michigan Arbor States Conservative Equilibrium Discovery in Offline Game-Theoretic Multiagent Reinforcement Learning AAAI Track Austin A. Nguyen ngaustin@umich.edu and Michael P. Wellman wellman@umich.edu Abstract. Offline learning of strategies takes data efficiency to its extreme by restricting algorithms to a fixed dataset of state-action trajectories. We consider the problem in a mixed-motive multiagent setting, where the goal is to solve a game under the offline learning constraint. We first frame this problem in terms of selecting among candidate equilibria. Since datasets may inform only a small fraction of game dynamics, it is generally infeasible in offline game-solving to even verify a proposed solution is a true equilibrium. Therefore, we consider the relative probability of low regret (i.e., closeness to equilibrium) across candidates based on the information available. Specifically, we extend Policy Space Response Oracles (PSRO), an online game-solving approach, by quantifying game dynamics uncertainty and modifying the RL objective to skew towards solutions more likely to have low regret in the true game. We further propose a novel meta-strategy solver, tailored for the offline setting, to guide strategy exploration in PSRO. Our incorporation of Conservatism principles from Offline reinforcement learning approaches for strategy Exploration gives our approach its name: . Experiments demonstrate ’s ability to extract lower-regret solutions than state-of-the-art offline approaches and reveal relationships between algorithmic components, empirical game fidelity, and overall performance. Key words and phrases: Empirical game-theoretic analysis; game-solving; multiagent reinforcement learning; offline reinforcement learning doi: XEYY6214 1. Introduction Game-theoretic analysis is a powerful tool when evaluating multiagent systems because it focuses attention on configurations where agents behave rationally relative to each other. In complex games with large strategy spaces, identifying such equilibrium profiles through multiagent reinforcement learning (MARL) requires substantial computation and data. Gathering ample data for game-theoretic analysis can thus be difficult and costly in real-world applications. A capability to perform such analysis offline from pre-generated, fixed-size datasets addresses the extreme case of limits on real-time data collection. Strategy generation using offline reinforcement learning is well-studied in single-agent and cooperative multiagent domains. Many offline learning algorithms adopt the conservatism principle: strategies should stay in dataset-informed regions and assume unknown regions yield low returns. This can be interpreted loosely as policy selection: among policies that seem approximately optimal, favor ones for which the dataset provides strong evidence and thus better relative confidence of high actual (true) return. Analogously, we frame offline game-solving as equilibrium selection, where we choose among profiles apparently in approximate equilibrium, aiming to minimize actual (true-game) regret. Our contribution is in the application and extension of conservatism to offline game-solving. This extension poses a special challenge: consideration of a candidate strategy profile’s potential strategic deviations. Our approach achieves this by combining offline RL algorithmic concepts and Policy Space Response Oracles (PSRO) (lanctot2017unified_psro), an online game-solving framework that iteratively extends a strategy set using deep RL best-responses to mixed strategies in an empirically estimated game model. Given a dataset of trajectories (sequences of true-game multiagent interactions), we first train an ensemble dynamics model to replace an online simulator and use prediction differences within the ensemble to quantify uncertainty under the dataset. We apply conservatism by introducing a response objective that balances reward, uncertainty given the current response target, and uncertainty of potential strategic deviations. Further, we propose a novel meta-strategy solver that minimizes a heuristic-based, pessimistic estimate of regret. This extension of Conservatism from Offline RL approaches to aid strategy Exploration in PSRO gives our approach its name: . In experiments on a sequential bargaining game, we find that to output lower-regret solutions than tested baselines. Our results highlight the role of conservatism in improving model fidelity, robustness to dataset changes, and solution regret. 2. Related Work Conservatism is a common theme in offline RL approaches (levine2020offline_tutorial). Model-free methods have incorporated conservatism through policy regularization (kumar2019_bear; wu2019_brac), value penalties (kumar2020_cql), and implicit behavioral constraints (nair2020awac; peng2019advantage_awr), while model-based methods have used uncertainty quantification and reward penalties (yu2020_mopo; kidambi2020_morel). Conservative methods in offline MARL have addressed centralized coordination (wang2024offline_global_implicit) and nonstationarity in decentralized optimization (jiang2023offline_transition_normalization). yang2020overview_game_theoretic_marl provide an overview of game-theoretic approaches in MARL. The framework of Policy Space Response Oracles (PSRO) (Bighashdel24; lanctot2017unified_psro) employs an empirical game modeling approach (Wellman25tg), iteratively extending the strategy set using deep RL to derive policies maximizing return against an other-player response target defined by a meta-strategy solver (MSS). Substantial work has investigated strategy exploration (Jordan10sw) in PSRO, in particular through MSS selection of response targets (balduzzi2019open_rectified_nash; marris2021multi_joint_psro; muller2020generalized_alpha_psro; smith2021iterative_qmix; wang2023regularization_rrd) or through definition of response objectives (Perez-Nieves21; wang2024generalized_gro; ex2psronguyenexplicit). In offline game-theoretic MARL, theoretical work has investigated the necessary conditions for equilibrium extraction (cui2022offline_solvable; zhong2022pessimistic), defining unilateral concentration as the property where a dataset covers an equilibrium and all its unilateral strategic deviations. Other approaches focus on offline evaluation of given strategies to determine which are most likely equilibria (zhang2023_offline_markov_games_general_function; zhong2022pessimistic). shao2024copsro employ conservative critics and low-rank matrix completions to select equilibria given a fixed dataset of payoff data for select policy profiles. Most relevant here is work on strategy generation for offline equilibrium extraction (li2022offline_equilibrium_finding), which uses a trained dynamics model in place of a simulator and mixes the final solution with a behavior cloning policy. chen2024offline_fictitious_self_play adapts fictitious self-play using single-agent offline RL and importance sampling. 3. Notation We model a multiagent environment as a stochastic game defined by ⟨,,,ℛ,Z,O,n,γ⟩ ,A,T,R,Z,O,n,γ . The environment follows trajectories of states s∈s , influenced by actions a∈a taken alternately by each of n agents. Each action prompts a transition according to the state transition function (s′∣s,a):×→[0,1]T(s s,a):S×A×S→[0,1]. Agent rewards are given by ℛ(s,a):×→ℝnR(s,a):S×A ^n, and γ∈[0,1)γ∈[0,1) is a discount factor. The environment is partially observable, meaning agents draw observations z∈Zz∈ Z from an observation function O(s):→ZO(s):S→ Z. We denote an action-observation history by hth_t, with ℋt≡(Z×)tH_t≡(Z×A)^t the space of conceivable t-length histories. A policy or pure strategy πi _i for player i is defined as a mapping from hth_t to a probability distribution over legal actions πi:ℋ→Δ() _i:H→ (A). We omit subscript i to refer to a joint policy (or strategy profile) π=(π1,…,πn)π=( _1,…c, _n). A trajectory τ describes a rollout of game play and is represented by a sequence of tuples τ=((st,oi,t,ai,t,ri,t,i))t=1|τ|τ= ((s_t,o_i,t,a_i,t,r_i,t,i) )_t=1 |τ |, where t indexes the timestep. A dataset D is a collection of trajectories τjj=1||\τ^j\_j=1 |D |, indexed here by j. We may subscript a trajectory τπj _π^j to indicate the joint policy π generating the rollout. The joint probability of being in state s and executing action a under joint policy π is denoted by μπ(s,a)μ^π(s,a). Multiagent interactions may also be expressed in normal form. A normal form symmetric game Γ is represented by a tuple (Π,U,n)( ,U,n), where n is the number of players, Π=Π1×⋯×Πn = _1×…m× _n is a set of joint strategies (π∈Ππ∈ ), and payoff function U:Π→ℝnU: ^n maps strategy profiles π to vectors of payoffs (i.e., returns). A probability distribution σi∈Δ(Πi) _i∈ ( _i) over pure strategies is called a mixed strategy. We also use σi,s _i,s to indicate player i’s strategy applied at step s. A subscript −i-i indicates all players except i. We write player i’s payoff for playing πi _i when the others play π−i _-i as ui(πi,π−i)u_i( _i, _-i), where u is a length-n utility vector. For mixed strategies, the payoffs are given by expectation: ui(πi,σ−i) u_i( _i, _-i) =π−i∼σ−iui(πi,π−i), =E_ _-i _-iu_i( _i, _-i), ui(σ) u_i(σ) =πi∼σiui(πi,σ−i). =E_ _i _iu_i( _i, _-i). Player i’s best response to all others playing σ is given by i(σ)=argmaxσi∗∈Δ(Πi)ui(σi∗,σ−i)BR_i(σ)= _ _i^*∈ ( _i)u_i( _i^*, _-i). Mixed strategy σ∗σ^* is a Nash equilibrium (NE) iff ∀i.σi∗∈i(σ∗)∀ i.\ _i^* _i(σ^*). The potential gain to deviating from σ is called regret: i(σ)=ui(πi∗,σ−i)−ui(σ)Regret_i(σ)=u_i( _i^*, _-i)-u_i(σ), where πi∗∈i(σ) _i^* _i(σ). By definition, if σ∗σ^* is a NE, its regret is zero. An empirical game model Γ^=(Π^,U^,n) =( , U,n) is a normal-form game over a finite joint-strategy subspace Π^=Π^1×⋯×Π^n = _1×…m× _n, Π^i⊂Πi _i⊂ _i, with payoff function U U estimated through simulation. 4. Conservatism in Offline Game-Solving Empirical game updates and best-response training in PSRO require explicit access to a game simulator. The offline setting, in contrast, works with fixed datasets of unspecified quality. Assessing candidate solutions requires reliable information on the included strategies and relevant deviations, which is contingent on the dataset’s behavior coverage, or lack thereof. Since we cannot draw assumptions on the extent of this coverage, offline game-solving must account for uncertainty in regret assessments. A joint strategy π is considered covered if the dataset contains all state-action pairs visitable under π: ∀s,a.μπ(s,a)>0⟹(s,a)∈∀ s,a.\ μ^π(s,a)>0 (s,a) . In large games, full coverage of any policy π and its unilateral deviations is infeasible. Therefore, we consider the extent to which a policy π is covered. Our approach frames offline game-solving as an equilibrium selection problem: among solutions that seem to have low regret offline, we prefer those more certain to have low regret in the true game. By skewing offline strategy exploration toward high-confidence policy spaces, we seek to produce lower-regret solutions than alternative exploration methods with some consistency. 4.1. Training the Ensemble Dynamics Model Our model-based approach adapts analogous single-agent offline RL approaches. We train an ensemble of K multi-layer perceptrons as dynamics models, each estimating the transition function T, reward function ℛR, observation mapping O, and state-dependent action space A, from which dataset D was generated. Henceforth, our references to these MDP components denote predictions of a learned dynamics model. Ensemble predictions are the average prediction across all K networks: ℛ¯i(st,at)=1K∑j=1Kℛij(st,at) R_i(s_t,a_t)= 1K _j=1^KR_i^j(s_t,a_t), where ℛij(st,at)R^j_i(s_t,a_t) refers to model j’s prediction of player i’s reward given state-action pair (st,ats_t,a_t), and so forth for other MDP components. Additional details, including terminal handling, information state generation, and training parameters are provided in Supp. 4.1 . Prediction differences between intra-ensemble models can serve as proxy for uncertainty quantification (kidambi2020_morel). We follow yu2020_mopo and define ρ based on the range of model predictions:111The original approach of yu2020_mopo used transition function differences. We instead employ reward differences, which achieved stronger results in our experiments. ρ(st,at,ℛ)=maxj,k∑i=1n|ℛij(st,at)−ℛik(st,at)|.ρ(s_t,a_t,R)= _j,k _i=1^n |R_i^j(s_t,a_t)-R_i^k(s_t,a_t) |. (1) Equation 1 reflects an assumption that models tend to provide different reward predictions on information-deprived transitions, and similar predictions for sufficiently informed transitions (whether informed explicitly or implicitly by dataset D). The maximum prediction difference is a conservative indicator (yu2020_mopo; kidambi2020_morel). 4.2. A Conservative Response Objective At each iteration of online PSRO, strategies are trained by optimizing a single agent’s expected return against a fixed, stationary response target, σ−i _-i. πi _i =argmaxπθJi(θ,σ−i), = _ _θJ_i(θ, _-i), Ji(θ,σ−i) J_i(θ, _-i) =at∼(πθ,σ−i)∑t=1Tℛ¯i(st,at). =E_a_t ( _θ, _-i) _t=1^T R_i(s_t,a_t). (2) Under a fixed dataset, we cannot assume availability of covered best-responses to σ−i _-i. PSRO’s response objective by optimizing an augmented reward function, Ji′(θ,σ−i)J_i (θ, _-i) =at∼(πθ,σ−i)[∑t=1Tℛ¯i(st,at)−λρ(st,at,ℛ)],=E_a_t ( _θ, _-i) [ _t=1^T R_i(s_t,a_t)-λρ(s_t,a_t,R) ], (3) where λ is a hyperparameter. Equation 3 incorporates model-based uncertainty given σ−i _-i. Although this objective is sufficient in single-agent RL problems, offline game-solving requires information on strategy deviations to additionally verify equilibrium properties (cui2022offline_solvable). We hypothesize that skewing strategy exploration toward equilibria with low ρ on unilateral deviations will also improve performance. Suppose player i is creating a strategy exclusively optimizing for unilateral concentration. This entails creating a strategy πi _i that provides other players the least uncertainty ρ on all potential responses to πi _i. Fig. 1 demonstrates we can achieve this by sampling a different other-player policy at each training episode and minimizing ρ. A roughly equivalent formulation222Equivalence requires that the uniform distribution over strategies be behaviorally equivalent to the uniform-random strategy, which does not generally hold. In N-player games, only one player can use uniform-random action at any given episode. is to respond to a uniform-random action policy exclusively optimizing ρ: Ji′(θ,σ−i)=(1−α)Ji′(θ,σ−i)−αat∼(πθ,())∑t=1Tλρ(st,at,ℛ),J_i (θ, _-i)=(1-α)J_i (θ, _-i)\\ - _a_t ( _θ,U(A)) _t=1^Tλρ(s_t,a_t,R), (4) where α is a hyperparameter denoting the probability of optimizing strategic deviation coverage in each training episode. In practice, we linearly anneal α to zero over PSRO iterations. In the context of strategy generation, this objective balances model-predicted reward, uncertainty given the response target σ−i _-i, and uncertainty of potential other-player deviations. Under the lens of equilibrium selection, Equation 4 can be viewed as skewing strategy exploration toward higher-certainty policy regions (wang2024generalized_gro). 4.3. A Conservative Meta-Strategy Solver We propose a novel MSS that uses ensemble model predictions to derive a conservative response target. This MSS takes concepts from the Bellman-consistent equilibrium learning (BCEL) method of zhang2023_offline_markov_games_general_function, incorporated in a game-solver based on replicator dynamics (RD) (RD). BCEL employs lower and upper utility bounds for given joint strategies and identifies the solution minimizing worst-case regret. We design an RD update that computes a stable point in heuristically estimated, worst-case regret. We define the utility predicted by model j for player i under joint policy π as uij(π)=1N∑k=1N∑(st,at)∈τπkℛij(st,at)u_i^j(π)= 1N _k=1^N _(s_t,a_t)∈τ^k_πR_i^j(s_t,a_t), where returns are averaged over N trajectories τπkτ^k_π. Lower and upper utility bounds are replaced by the minimum and maximum utility estimates over the ensemble ℛR, respectively: u¯i(π)=maxj∈1…Kuij(π) u_i(π)= _j∈\1…c K\u_i^j(π). Pessimistic utilities u¯(π) u(π) are minimized over K models. We modify the RD update by myopically optimizing for the current pessimistic regret estimate at each step: ≐u¯i(πik,σ−i)−u¯i(σ) u_i( _i^k, _-i)- u_i(σ) ≐maxπij∈(Π^i\πik)u¯(πij,σ−i)−u¯i(πik,σ−i) _π^j_i∈( _i _i^k) u(π^j_i, _-i)- u_i( _i^k, _-i) Δσik _i^k =σik[−] = _i^k[UBDP-UBDR] where σikσ^k_i is the probability player i plays strategy k and Δ indicates the amount of change applied to a variable. The first term captures an upper-bound deviation payoff (UBDP) for playing πikπ^k_i against σ−i _-i. The second represents an upper-bound deviating regret (UBDR) if the player deviates to πikπ^k_i. Our approach accounts for robust offline evaluation, giving our MSS its name: Robust Replicator Dynamics (R2D). Figure 1. An iteration of . During strategy generation, a rollout optimizes ρ against a uniformly sampled policy with probability α, or the conservative response to the target (Equation 3) otherwise. The two cases are indicated by red and blue arrows. The newly trained policy is added to the strategy set, updating the empirical game using pessimistic and optimistic utility estimates. R2D determines the next iteration’s response target. Coffee-PSRO algorithm schematic. Neither RD nor R2D offer theoretical convergence guarantees. Nevertheless, in practice, RD reliably reaches exact or approximate equilibria in empirical games (Wellman25tg) with demonstrated success within the PSRO framework. Analogously, we demonstrate R2D’s empirical benefits within . A complementary RD variant, Regularized RD (RRD) (wang2023regularization_rrd), employs an early-stop condition based on a regret threshold to avoid overfitting to the current empirical game solution. R2D can analogously adopt a stopping threshold measured against worst-case regret, resulting in another MSS called Robust Regularized Replicator Dynamics (R3D). Algorithm 1 Conservative Offline Exploration PSRO ( ) Input: Dataset D, Penalty weight λ, Coverage weight α _init, Steps anneal coverage weight SαS_α, meta-strategy solver ℳM, PSRO iterations S, Number of simulations N Output: Player strategy sets Π , Player profiles σS _S 1: Initialize empty empirical game Γ^:(Π^,U^,n) :( , U,n) 2: Initialize strategy sets Π^i←πi0,i∈1…n _i←\π^0_i\,i∈\1…c n\ 3: Train ensemble dynamics model of size K on D 4: Update Γ using dynamics model with N simulations 5: for each PSRO iteration s from 1 to S do 6: for each player i from 1 to n do 7: πi←argmaxπθJi′(θ,σ−i) _i← _ _θJ_i (θ, _-i) (4), using λ and α 8: Update strategy sets Π^i←Π^i∪πi _i← _i∪\ _i\ 9: end for 10: Update Γ using dynamics model with N simulations 11: Update joint profile σs←ℳ(Γ^) _s ( ) 12: Anneal α←max(0,α(Sα−i)Sα)α← (0, _init(S_α-i)S_α) 13: end for 14: return Π , σS _S 4.4. Conservative Offline Exploration in PSRO (COffeE-PSRO) Algorithm 1 summarizes , depicted schematically in Fig. 1. the basic structure of PSRO, using a trained dynamics model in place of the online simulator, and using Equation 4 for strategy generation. Any meta-strategy solver ℳM can be used with ,333For conventional MSSs, empirical game payoffs would be estimated using average reward predictions rather than bounds. but our experiments suggest that R2D yields the best results. 5. Experiments 5.1. Bargaining We test our approach by adapting a 2-player, turn-based, imperfect information, symmetric bargaining game (bargaining). The game (which we refer to as Bargaining) features a pool of items that players must divide between them. Each player has a randomly drawn, private valuation that gives their per-unit utility value for each item. Players alternate making offers of division until one accepts or the game reaches a maximum number of iterations. If a player accepts, players receive utility according to the accepted allocations and respective value functions, discounted by γtγ^t where 0<γ<10<γ<1 is a hyperparameter and t is the number of bargaining rounds so far. We set γ to .99.99 If no agreement is reached by the turn limit (T=10T=10), both players receive zero utility. Supp. B provides further details. Table 1. Regret of final solutions, averaged over 10 trials with corresponding standard deviation and P-values, each consisting of S=40S=40 iterations. Bolded algorithmic components indicate our contributions. Datasets contain a varied number of trajectories N∈500,1000,2000N∈\500,1000,2000\ and are generated by two different behavior policies: uniform random π^U, and a mixture over online-generated equilibria σ^eq. We distinguish between the MSS ℳM_expl used during training versus ℳM_eval during evaluation to isolate effects on strategy exploration and immediate regret (wang2022evaluating). All P-values are calculated with Welch’s t-test to determine whether +R2D yields significant regret differences relative to the corresponding approach. True-game regret evaluations are discussed in Supp. E. Approach (500, σ^eq) (500, π^U) (1000, σ^eq) (1000, π^U) (2000, σ^eq) (2000, π^U) Alg, ℳM_expl ℳM_eval: RD , R2D 2.02±0.40∣2.02 0.40 N/A 4.23±0.77∣4.23± 0.77 N/A 1.10±0.32∣1.10 0.32 N/A 2.40±0.55∣2.40 0.55 N/A 0.88±0.35∣0.88 0.35 N/A 1.66±0.57∣1.66 0.57 N/A , RD 2.58±0.40∣2.58± 0.40 .01 4.20±0.81∣4.20± 0.81 .53 1.47±0.74∣1.47± 0.74 .09 2.64±0.55∣2.64± 0.55 .17 1.64±0.94∣1.64± 0.94 .02 3.15±2.09∣3.15± 2.09 .03 OEF, R2D 2.71±0.85∣2.71± 0.85 .02 5.01±1.07∣5.01± 1.07 .04 1.42±0.52∣1.42± 0.52 .05 3.20±0.37∣3.20± 0.37 .01 1.03±0.67∣1.03± 0.67 .27 2.24±0.84∣2.24± 0.84 .01 OEF, RD 2.38±0.55∣2.38± 0.55 .06 5.09±1.19∣5.09± 1.19 .04 1.19±0.29∣1.19± 0.29 .29 3.74±2.10∣3.74± 2.10 .04 1.40±0.79∣1.40± 0.79 .04 3.28±2.16∣3.28± 2.16 .02 OEF-BC, R2D 2.91±0.61∣2.91± 0.61 .01 4.00±0.65∣4.00± 0.65 .76 2.00±0.22∣2.00± 0.22 .01 2.62±0.22∣2.62± 0.22 .13 1.85±0.29∣1.85± 0.29 .01 2.15±0.23∣2.15± 0.23 .01 OEF-BC, RD 2.82±0.38∣2.82± 0.38 .01 3.85±0.54∣3.85 0.54 .89 1.92±0.19∣1.92± 0.19 .01 2.96±1.07∣2.96± 1.07 .08 1.92±0.49∣1.92± 0.49 .01 2.83±1.12∣2.83± 1.12 .01 Alg, ℳM_expl ℳM_eval: R2D , R2D 1.96±0.45∣1.96 0.45 N/A 3.98±0.60∣3.98± 0.60 N/A 1.22±0.41∣1.22± 0.41 N/A 2.12±0.27∣2.12 0.27 N/A 0.64±0.12∣0.64 0.12 N/A 1.40±0.36∣1.40 0.36 N/A , RD 2.15±0.43∣2.15± 0.43 .17 3.76±0.75∣3.76 0.75 .76 1.34±0.42∣1.34± 0.42 .26 2.85±0.91∣2.85± 0.91 .02 1.30±0.59∣1.30± 0.59 .01 2.42±1.43∣2.42± 1.43 .05 OEF, R2D 2.47±0.82∣2.47± 0.82 .05 4.90±1.11∣4.90± 1.11 .02 1.08±0.37∣1.08± 0.37 .78 3.25±1.00∣3.25± 1.00 .35 0.73±0.52∣0.73± 0.52 .28 1.93±0.49∣1.93± 0.49 .01 OEF, RD 2.11±0.39∣2.11± 0.39 .22 4.88±1.03∣4.88± 1.03 .02 1.01±0.19∣1.01 0.19 .92 3.68±1.63∣3.68± 1.63 .01 0.96±0.59∣0.96± 0.59 .06 2.79±1.98∣2.79± 1.98 .03 OEF-BC, R2D 2.77±0.33∣2.77± 0.33 .01 3.97±0.65∣3.97± 0.65 .51 1.89±0.16∣1.89± 0.16 .01 2.70±0.39∣2.70± 0.39 .01 1.68±0.22∣1.68± 0.22 .01 2.79±1.98∣2.79± 1.98 .03 OEF-BC, RD 2.65±0.20∣2.65± 0.20 .01 3.79±0.46∣3.79± 0.46 .78 1.98±0.19∣1.98± 0.19 .01 2.92±0.75∣2.92± 0.75 .01 1.82±0.18∣1.82± 0.18 .01 2.65±0.94∣2.65± 0.94 .01 5.2. Implementation and Reproducibility Our experiments build on an existing codebase, Deepmind’s OpenSpiel (openSpiel). We refer to datasets using tuples (N,πβ)(N,π^β) where N is the number of trajectories and πβπ^β is the joint behavior policy. Our experiments vary N∈500,1000,2000N∈\500,1000,2000\ and πβ∈π,σπ^β∈\π^U,σ^eq\, where π^U is a uniform-random action policy, and σ^eq is a uniform mixture over 5 equilibrium policies generated by online PSRO (Supp. C). Under these settings, coverage of Bargaining’s full strategy space is highly unlikely, providing benchmarks that measure ability to extract approximate equilibria under limited information. All trials use Double Deep Q-Networks (DDQN) (van2016deep) for best-response calculations, yielding deterministic, pure strategies πi _i. Supp. A discusses hyperparameters for our best-response and dynamics model. We compare our approach to two benchmarks, both variants of Offline Equilibrium Finding PSRO (li2022offline_equilibrium_finding). These approaches train a dynamics model, querying it in place of an online simulator (equivalent to λ=α=0λ=α=0), and either use the output solution, or mix that solution with a behavior-cloned policy πiπ^bc_i on D by weight α _bc. We refer to the former as OEF and the latter as OEF-BC. All approaches use symmetric implementations, where players share a strategy set Π^i _i and profile σi _i, Γ is a symmetric game, and a single policy πi _i is generated at each iteration. Analogous to how offline RL uses online simulations for return evaluation, we use online best responses (detailed in Supp. E) to evaluate true-game regret. All trials consist of S=40S=40 iterations, each iteration corresponding to one strategy generation. We tune OEF-BC using =(1000,π)D=(1000,π^U).444OEF has no additional parameters. tuned across (λ, α) for both ℳ∈RD,R2DM∈\RD,R2D\, running 5 trials for each, and an additional 5 for the initially lowest regret configuration. OEF-BC is tuned exhaustively across α _bc, where all algorithmic variants immediately run 10 trials and then select the lowest regret parameter setting. We fix tuned parameters across datasets. Note that this tuning scheme is more exhaustive than ’s. We chose to use R2D as opposed to R3D in order to isolate the effects of optimizing for conservative regret estimates over expected regret.555This additionally removes the need to tune additional parameters λ _rrd and λr3d _r3d for fair comparison. Hyperparameters are detailed in Supp. A. 5.3. Regret Analysis Table 1 compares tuned versions of , OEF, and OEF-BC. We isolate and measure the improvement from an MSS used in strategy exploration ℳM_expl by using a consistent evaluation MSS ℳM_eval across different approaches (wang2022evaluating). Results suggest that separately applying R2D yields variable improvement. However, the combined use of R2D tends to outperform tested baselines, suggesting a strong, beneficial interplay between conservatism in strategy generation and MSS. (a) ρ, δ(Π^)δ( ), and regret for trials. (b) ρ, δ(Π^)δ( ), and (λ, α) for trials. (c) regret and δ(Π^)δ( ) for trials using =(1000,π)D=(1000,π^U). Figure 2. Relationships among model fidelity, performance, and conservatism. Each datapoint is a single trial (ℳ=R2DM_eval=R2D). Graphs displaying the relationship between regret, conservatism, and model fidelity. Comparing with OEF: Benefits of Conservatism - Except for (1000,σ)(1000,σ^eq), + R2D tends to outperform OEF, indicating that intentional conservatism and uncertainty quantification improve performance. Particularly, under a uniform policy π^U, D provides relatively little information on valuable policy spaces, widening the performance margin between + R2D and OEF. Under the same argument of conservatism, OEF-BC also outperforms OEF when using π^U datasets. Comparing with OEF-BC: Strategic vs. Uninformed Conservatism - consistently stronger performance than OEF-BC under all datasets except (500,π)(500,π^U). These two approaches differ conceptually in how they apply conservatism. Through strategy exploration, by prioritizing search in dataset-informed spaces while OEF-BC enforces similarity to dataset behavior. These results support the notion that low-data settings provide little information on how to be conservative in a calculated manner during exploration. Thus, ’s strategic conservatism is outperformed by OEF-BC’s simple method of conservatism in low data, weak behavior policy regimes. Behavior Cloning in General-Sum Games - We highlight that OEF-BC is substantially outperformed by all approaches under σ^eq datasets, underscoring a core property of general-sum games. Although σ^eq provides information on several ”strong” strategies, the mixture of equilibria in general-sum games is not necessarily an equilibrium itself, making mixing with a BC-trained policy arbitrarily effective. This emphasizes a vital consideration distinguishing conservative approaches in single and multiagent environments: imitating datasets that contain strong behavior does not imply the generation of strong strategies in the latter. Immediate Regret Benefits of R2D - We now discuss ℳM_eval’s effect on immediately evaluated regret independent of strategy exploration. For a given approach and dataset, R2D tends to produce lower regret solutions than RD, suggesting the optimization of conservative regret estimates benefits offline equilibrium extraction. These findings are consistent with prior work (zhang2023_offline_markov_games_general_function). 5.4. Analyzing Model Fidelity While online strategy exploration focuses on creating a set of strategies that sufficiently capture game structure, a challenge of offline game-solving is the added uncertainty of utility estimations due to limited access to data. The extent to which our algorithm should trust its offline game model for equilibrium extraction should, intuitively, be a function of its utility approximation accuracy. We refer to a game model’s utility estimation accuracy as model fidelity and hypothesize that it is a key component for robust offline equilibrium discovery. More precisely, since the optimization of Equation 4 can be interpreted as skewing strategy exploration to create a higher fidelity game model, we investigate the relationship between ρ, parameter strength, model fidelity, and solution quality. For each experimental trial depicted in Fig. 2, we use offline-generated strategies Π to reconstruct an empirical game Γ with true game simulations, calculating the average uncertainty ρ across all simulations and average utility error across all entries in Γ : δ(Π^)=1Π^∑π∈Π^∑i=1n|ui(π)−u^i(π)|δ( )= 1 _π∈ _i=1^n |u_i(π)- u_i(π) |, where ui(π)u_i(π) and u^i(π) u_i(π) are player i’s true game and dynamics model estimated utilities, respectively. Figs. 2(a) and 2(b) present scatter plots relating ρ and δ(Π^)δ( ), color-coded proportionally to the magnitude of regret and parameters (λ,α)(λ,α) applied, respectively. Plots show a clear positive relationship between ρ and δ(Π^)δ( ), indicating ρ is an effective proxy for uncertainty quantification. Parameters λ and α strongly correlate with improved model fidelity but not necessarily regret. Instead, moderate hyperparameter choices, and therefore modest model fidelity, yield the lowest regret, shown in Fig. 2(b), suggesting that excessive conservatism prevents the discovery of valuable strategies even if the subgame’s utility estimates are relatively accurate. These experiments demonstrate how effective offline game-solving relies on policy generation balancing conservatism and potential strategic value (model fidelity vs. regret). This tradeoff between model fidelity and regret is reflected in Fig. 2(c), where the use of R2D when =(1000,π)D=(1000,π^U) tends to decrease regret across all approaches despite compromising model fidelity. 5.5. Ablations We provide experiments varying λ and α in Fig. 3, showing the subset of plots that ablate from the strongest parameter settings: (λ=4λ=4, α=.2α=.2) and (λ=4λ=4, α=.3α=.3) for R2D and RD, respectively. All remaining ablations of provided in Supp. F. Moderate levels of λ tend to produce the strongest results, both in final regret and convergence speed. However, when using ℳ=M= R2D, a more conservative MSS, smaller values of both λ and α are favored. This indicates a threshold at which excessive conservatism from all sources becomes detrimental. Generally, higher values of α correlate to slower initial convergence, indicated by regret spikes in early iterations. These spikes are commonly followed by steep dips, sometimes finding lower regret solutions than configurations with initially stronger performance. This suggests that α, by skewing strategy exploration toward regions with higher certainty on strategic deviations, benefits lower-regret, higher-certainty equilibrium selection in later iterations. Figure 3. Regret over iterations plots under various parameter settings. Regret entries for (ℳ=RD,λ=4,α=.3)(M=RD,λ=4,α=.3) and (ℳ=R2D,λ=4,α=.2)(M=R2D,λ=4,α=.2) are averaged over 10 trials; all others are over 5. Graphs displaying ablations over Coffee-PSRO hyperparameters. ℳM λ↓λ α=.1α=.1 α=.2α=.2 α=.3α=.3 RD 2 4.15±1.794.15± 1.79 3.50±0.783.50± 0.78 3.39±0.983.39 0.98 4 3.37±0.80¯ 3.37± 0.80 3.33±1.18¯ 3.33± 1.18 2.58±0.56¯ 2.58 0.56 6 4.91±0.614.91± 0.61 4.03±1.354.03 1.35 5.37±2.135.37± 2.13 R2D 2 2.89±1.512.89± 1.51 2.26±0.472.26 0.47 2.50±0.93¯ 2.50± 0.93 4 2.49±0.52¯ 2.49± 0.52 2.12±0.27¯ 2.12 0.27 2.87±0.612.87± 0.61 6 3.43±0.553.43 0.55 3.75±0.813.75± 0.81 3.64±1.013.64± 1.01 Figure 4. Final regret under various parameter settings. Regret entries for (ℳ=RD,λ=4,α=.3)(M=RD,λ=4,α=.3) and (ℳ=R2D,λ=4,α=.2)(M=R2D,λ=4,α=.2) are averaged over 10 trials; all others are over 5. Bold and underlined numbers indicate the lowest regret for fixed λ and α, respectively. During tuning, we use the same MSS ℳM for exploration and evaluation. 6. Limitations An inherent limitation of PSRO, and by extension of , is the computational cost of training a sequence of deep RL strategies in sequence, which can be exorbitantly expensive for complex games. Several approaches have focused on reducing this computational cost (Bighashdel24; smith2021iterative_qmix). Our experiments find that over baselines if the dataset D provides enough information to quantify or extrapolate uncertainty in uncovered strategy spaces (strategic conservatism). Otherwise, no more benefit than uninformed conservatism. Dataset coverage is especially difficult for multiagent environments, which tend to have larger search spaces than single-agent. Lastly, experiments demonstrate the efficacy of one hyperparameter setting (λ, α) across different datasets. However, it is unclear whether these same parameters generalize to other games with different reward structures and scales. 7. Conclusion We investigated a novel approach to offline game-solving, which addresses the discovery of equilibria from fixed datasets of multiagent trajectories. These datasets may provide incomplete information on relevant strategy spaces, making game-solving strictly more difficult in the offline setting. Under this added uncertainty, we framed this task as an equilibrium selection problem: of solutions apparently in approximate equilibrium, prefer those likely to minimize actual, true-game regret. We address equilibrium selection under the lens of strategy exploration, which refers to the process of deciding what strategies to include in an iteratively extended game model, by incorporating conservative algorithmic techniques from single-agent, offline RL algorithms. Our approach, , extends PSRO by training a dynamics model to use in place of an online simulator. Then, we design a modified response objective that optimizes model-quantified uncertainty given the current response target and potential strategy deviations. Lastly, we propose a novel MSS, R2D, that optimizes conservative regret estimates. We test our algorithm in a general-sum, sequential Bargaining game against several SOTA baselines. Our results demonstrate that + R2D tends to produce lower regret solutions with higher consistency than SOTA approaches. We also investigated the unique relationship between algorithmic components, behavior policy, and performance in general-sum games, highlighting ’s strategic method of applying conservatism. As a final solver (ℳM_eval), R2D tends to identify lower regret solutions than the RD baseline, underscoring the benefits of optimizing for pessimistic regret estimates in offline settings. Lastly, experiments analyzing model fidelity and its relationship to regret demonstrate the importance of balancing conservatism and potential strategic value in strategy exploration. References Appendix A Hyperparameters Table 2. All hyperparameters Parameter Description Chosen Parameter DDQN Best-Response Parameters Network Width 200 Network Depth 2 Replay Buffer Size 5e4 Batch Size 64 Learning Rate 1e-4 Update Target Every 1000 Learn Every 2 Discount Factor .99 Minimum Buffer Size to Learn 5e4 Epsilon Start 1.0 Epsilon End .02 Epsilon Linear Decay Duration 2e5 Training Steps 2e5 Dynamics Model Parameters Model Width 250 Model Depth 2 Ensemble Size 4 Model Training Batch Size 64 Model Training Learning Rate 3e-4 Model Training Steps 1e4 Game-Solving Parameters Iterations 40 Simulations per Entry 1000 Steps anneal α 10 OEF, OEF-BC, and used the same dynamics model parameters and architecture for fairness. Each model within an ensemble consisted of a state-transition network and a reward network, two disparate networks using the same architecture as described above. All inputs and prediction targets were normalized with the dataset’s mean and standard deviation. The state-transition network was trained to predict the change in state Δ=st+1−st∣at =s_t+1-s_t a_t. An observation-legal-action network was trained to share across all models within the ensemble, mapping true state to player-specific observations and legal-action masks. Notably, since Bargaining only consists of terminal rewards, the reward network was trained exclusively on the rewards of terminal transitions for all players to prevent network outputs from collapsing to 0. During training, players received a reward of 0 for all non-terminal transitions and network-predicted rewards otherwise. Furthermore, in , the penalties from prediction differences ρ were only applied during strategy training, not empirical game updates (where we exclusively used the ensemble’s mean predictions). We train each dynamics model using mean-squared-error loss and the parameters listed in Table 2. Observation action histories are converted to the game’s defined information state through an internal module. An episode is considered finished if either a rollout reaches a maximum length or the resulting state prediction is close (mean absolute difference ¡ .5) to a predefined terminal state. Bargaining-specific details for dynamics model training are provided in Sec B. Best-response parameters for the DDQN are provided in Table 2. These parameters were not tuned but fixed at sensible values. Appendix B Bargaining Game Adaptation We designed a generalized version of the Bargaining (bargaining) provided by OpenSpiel (openSpiel). Our game is defined by several parameters, summarized in Table 3. Table 3. Parameters in the bottom half are sampled at the beginning of each episode while those in the top half are held constant or define distributions that bottom-half parameters sample from. Symbol Chosen Description NitemsN_items 3 Number of item types [vmin,vmax][v_min,v_max] [5, 10] Summed valuation bounds for both players i∈1,2i∈\1,2\. [cmin,cmax][c_min,c_max] [5, 7] Summed item count bounds for any sampled pool T 10 Max game length γ .99 Discount factor of valuation cjj=1Nitems\c^j\_j=1^N_items N/A The number of items of each item type in the pool vijj=1Nitems\v^j_i\_j=1^N_items N/A Valuations for players i∈1,2i∈\1,2\ At the beginning of each episode, an item pool is sampled uniformly from all possible pools that satisfy two conditions: cj≥1∀j∈1…Nitems c^j≥ 1\;∀ j∈\1…c N_items\ ∑j=1Nitemscj∈[cmin,cmax] _j=1^N_itemsc^j∈[c_min,c_max] where cjc^j is a positive integer. An item pool is represented by a vector of item counts C≐cjj=1NitemsC \c^j\_j=1^N_items. Next, player valuations are sampled uniformly from all possible valuations that satisfy two conditions: vij≥1∀j∈1…Nitems,i∈1,2 v_i^j≥ 1\;∀ j∈\1…c N_items\,i∈\1,2\ ∑i=jNitemsvij∈[vmin,vmax]∀i∈1,2 _i=j^N_itemsv_i^j∈[v_min,v_max]\;∀ i∈\1,2\ where vijv_i^j is a real number greater than or equal to 1. Player i’s valuation is represented by the vector Vi≐vijj=1NitemsV_i \v_i^j\_j=1^N_items. For symmetry, the first player is decided with a coin flip and valuation sample distributions are identical for both players. The game terminates if a player accepts an offer or the number of turns exceeds T. Our chosen parameters are listed in Table 3 The state, observation, information-state (history), and action vectors are represented by st=[at−1=]+t+C+V1+V2+at−1+icurr s_t=\\I[a_t-1\ =ACCEPT]\+\t\+C+V_1+V_2+\a_t-1\+\i_curr\ sterminal=−1∣s∈∣ s_terminal=\-1\ s at=njj=1Nitems∣nj∈0,…cj a_t=\n^j\_j=1^N_items n^j∈\0,…c c^j\ oi,t=[at−1=]+t+C+Vi+at−1 o_i,t=\I[a_t-1=ACCEPT]\+\t\+C+V_i+\a_t-1\ hi,t=[at−1=]+t+C+Vi+∪t′=0t′−1at′ h_i,t=\I[a_t-1=ACCEPT]\+\t\+C+V_i+ _t =0^t -1a_t where ”++” denotes concatenation. Note that the output of DDQN assumes a discrete action space represented by an index, not an integer vector ata_t. Therefore, the network output size equals the number of all possible offers under all possible pools, computable given the parameters in Table 3. Furthermore, since the DDQN is not able to output vectorized actions as shown above, we create an internal module that converts action indices atindexa^index_t to offer vectors ata_t of length NitemsN_items. We also create a module that converts an explicit history oi,t′,a1,t′,a2,t′t′=1t\o_i,t ,a_1,t ,a_2,t \_t =1^t into a condensed information state hi,th_i,t. Appendix C Dataset Generation Experiments used three different dataset sizes N∈500,1000,2000N∈\500,1000,2000\ and two different behavior policies π∈π,σπ∈\π^U,σ^eq\, referring to each dataset by the tuple (N,π)(N,π). π^U denotes a joint policy that selects legal actions at uniform random: π:()π^U:U(A). σ^eq denotes a uniform strategy mixture over 5 profiles generated trials of PSRO, each consisting of 20 iterations: σ:(σii=15)σ^eq:U(\σ^PSRO_i\_i=1^5), where uniform sampling occurs at the beginning of each episode. For independence, all offline experiments generated independent datasets D at the beginning of every trial using the listed predefined joint policies. Appendix D Other Algorithmic Details D.1. Policy Space Response Oracles The PSRO algorithm (lanctot2017unified_psro) iteratively extends an empirical game model Γ^=(Π^,U^,n) =( , U,n) using RL. We initialize Π^i _i, for example, with a singleton random policy for each player i. PSRO extends Γ at each iteration through these steps: (1) update U U by estimating payoffs for all strategy profiles (i.e., joint strategies Π ) through simulation, (2) extract a target profile σ by applying a meta-strategy solver (MSS) to Γ , and (3) derive a best-response policy πi⋆∈Πiπ _i∈ _i for each player i by applying deep RL in an environment where other players are fixed to play σ−i _-i and add πi⋆π _i to Π^i _i. PSRO is summarized in Algorithm 2. Algorithm 2 Policy Space Response Oracles (PSRO) Input: Meta-strategy solver ℳM, PSRO iterations S, Number of Simulations N Output: Player strategy sets Π , Player Profiles σS _S 1: Initialize empty empirical game Γ^:(Π^,U^,n) :( , U,n) 2: Initialize strategy set w/ random policy Π^i←πi0,i∈1,…,n _i←\ _i^0\,i∈\1,…c,n\ 3: Update empirical game Γ with N simulations 4: for each PSRO iteration s to S do 5: for each player i to n do 6: Define Ji(θ)=π−i∼σ−i[∑t=1Tri,t]J_i(θ)=E_ _-i _-i[ _t=1^Tr_i,t] 7: πi⋆≈argmaxπJi(θ)π _i≈ _πJ_i(θ) 8: Update strategy sets Π^i←Π^i∪πi⋆ _i← _i∪π _i 9: end for 10: Update empirical game Γ with N simulations 11: Update joint profile σs←ℳ(Γ^) _s ( ) 12: end for 13: return Π , σS _S D.2. Offline Equilibrium Finding Algorithm 3 Offline Equilibrium Finding - Behavior Cloning (OEF, OEF-BC) Input: Dataset D of trajectories, meta-strategy solver ℳM, PSRO iterations S, Number of Simulations N, Behavior cloning mixing weight α _bc Output: Player bc-mixed strategy sets Π ^bc, Player profiles σS _S 1: Initialize empty empirical game Γ^:(Π^,U^,n) :( , U,n) 2: Initialize strategy set w/ random policy Π^←πi0 ←\ _i^0\ 3: Train ensemble dynamics model of size K on D 4: Train πi _i^bc using behavior cloning on D for i∈1,…ni∈\1,…c n\ 5: Update empirical game Γ using dynamics model with N simulations. 6: for each PSRO iteration s to S do 7: for each player i to n do 8: Define Ji(θ)=at∼(πθ,σ−i)[∑t=1Tℛ¯i(st,at)]J_i(θ)=E_a_t ( _θ, _-i)[ _t=1^T R_i(s_t,a_t)] 9: πi≈argmaxπθJ(θ) _i≈ _ _θJ(θ) 10: Update strategy sets Π^i←Π^i∪πi _i← _i∪ _i 11: end for 12: Update empirical game Γ using dynamics model with N simulations 13: Update joint profile σS←ℳ(Γ^) _S ( ) 14: end for 15: for each player i to n do 16: Mix policies Πi=∪s=1Sπis, ^bc_i= _s=1^S _i^s,mix where πis,(a∣s)=πis(a∣s)(1−α)+απi(a∣s) _i^s,mix(a s)= _i^s(a s)(1- _bc)+ _bcπ^bc_i(a s) 17: end for 18: return Πi _i^bc, σSσ^S Offline equilibrium finding (li2022offline_equilibrium_finding), as adapted to PSRO, is summarized in Alg. 3. The approach trains a dynamics model on dataset D and uses it in place of an online simulator. In our approach, we use the ensemble architecture and Double Deep Q-Learning as described in Sections 4.1 and 5.2, respectively, for a fair comparison with . OEF also trains a behavior cloned policy π^bc on D, which is used at the end of training to mix with the output policy π∈Ππ∈ as indicated by Line 16. Note that mixing implies sampling action ata_t from πiπ^bc_i or πisπ^s_i at each timestep. li2022offline_equilibrium_finding chooses α _bc at test time using a true game simulator to determine which setting has the lowest exploitability. While this can be seen as a violation of the offline learning assumption, we evaluate OEF-BC(α _bc) with the strongest test-time results. The OEF baseline is identical to OEF-BC but sets α _bc to 0. Appendix E True Game Regret Offline-generated profiles are evaluated in the true game. We train an online best response to profiles outputted at the last S_eval iterations: Πionline=i(σi,s)∣s∈T−S,…T _i^online=\BR_i( _i,s) s∈\T-S_eval,…c T\\, where s indicates the s-th profile generated and T denotes the number of PSRO iterations. We construct a combined empirical game that includes offline and online trained strategies Πi=Πi∪Πionline ^eval_i= _i∪ _i^online. Then we construct a normal-form game using Π _eval, recomputing all entries using N=1000N=1000 true game simulations. Player i’s regret given iteration s’s joint profile is approximated by the following: i(σs)≈maxπi∈Πiui(πi,σ−i,s)−ui(σs)Regret_i( _s)≈ _ _i^eval∈ _i^evalu_i( _i^eval, _-i,s)-u_i( _s) All results report the sum of regret across players ∑i=1ni(σs) _i=1^nRegret_i( _s). Ablations use S=T=40S_eval=T=40 while all other experiments use S=T2=20S_eval= T2=20. Appendix F Additional Results F.1. Regret Results During Tuning Table 4. True game regret values of final profiles outputted by baseline runs. Each trial consists of 40 iterations, where an iteration is one new strategy, and reported values are averaged over 10 trials. Note that when α=0.0 _bc=0.0, we refer to the baseline named OEF. α _bc OEF-BC + RD OEF-BC + R2D 0.0 3.74±2.103.74± 2.10 3.25±1.003.25± 1.00 0.2 2.96±1.072.96± 1.07 2.70±0.392.70± 0.39 0.4 3.49±0.253.49± 0.25 3.50±0.223.50± 0.22 0.6 4.79±0.254.79± 0.25 4.87±0.194.87± 0.19 0.8 6.56±0.286.56± 0.28 6.70±0.196.70± 0.19 We provide final true game regret values for tested baselines in Table 4 and all ablations over (λ, α, ℳM) in Table 5 and Fig. 5. Table 5. True game regret values of final profiles outputted by tuning. Underlined values are averaged over 10 trials, 5 otherwise. ℳM λ↓λ α=.1α=.1 α=.2α=.2 α=.3α=.3 RD 2 4.15±1.794.15± 1.79 3.50±0.773.50± 0.77 3.39±0.983.39± 0.98 4 3.37±0.803.37± 0.80 3.33±1.183.33± 1.18 2.64±0.55¯ 2.64± 0.55 6 4.91±0.614.91± 0.61 4.03±1.354.03± 1.35 5.37±2.135.37± 2.13 R2D 2 2.89±1.512.89± 1.51 2.26±0.472.26± 0.47 2.50±0.932.50± 0.93 4 2.49±0.522.49± 0.52 2.12±0.27¯ 2.12± 0.27 2.87±0.612.87± 0.61 6 3.43±0.553.43± 0.55 3.75±0.813.75± 0.81 3.64±1.013.64± 1.01 Figure 5. Ablation over . The top two rows ablate over α and λ, fixing ℳM to RD, while the next two rows fix ℳM to R2D. Graphs displaying additional ablations over Coffee-PSRO hyperaparameters.