Paper deep dive
DiffNash: Finding Nash Equilibrium in High-Dimensional Continuous Action Spaces with Diffusion Policies
Haichuan Wang, Anagha Satish, Jiale Zheng, Saisneha Ghatti, Lingkai Kong, Kai Wang, Milind Tambe
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Computing equilibria in high-dimensional continuous games requires learning and representing rich, multi-modal strategy distributions, something conventional neural policies struggle to achieve. Diffusion models offer a natural solution, but standard diffusion training requires direct samples from a fixed target distribution, which are generally unavailable under iterative game-solving updates. We propose a diffusion-based framework for equilibrium computation that enables diffusion-based variants of Online Mirror Descent (OMD) and Fictitious Play (FP). Our approach derives a reweighted score-matching objective that trains diffusion policies without requiring direct samples from each updated policy. Our main method, DiffNash, combines the fast convergence of OMD with the sample efficiency of FP by reusing past actions through a replay buffer. To correct the resulting distribution shift from sample reuse, we derive an unbiased estimator of the difference between current and past policies' Evidence Lower Bounds, whose exponential provides a tractable proxy for the corresponding importance weight without evaluating diffusion likelihoods. We evaluate our approach on forest protection games and continuous Colonel Blotto games, reducing the Nash gap over the strongest baseline by 35.6% and 24.7%, respectively, with minimal hyperparameter tuning.
Tags
Links
- Source: https://arxiv.org/abs/2610.15932v1
- Canonical: https://arxiv.org/abs/2610.15932v1
Trouble viewing inline? Open PDF directly →
Full Text
126,263 characters extracted from source content.
Expand or collapse full text
DiffNash: Finding Nash Equilibrium in High-Dimensional Continuous Action Spaces with Diffusion Policies Haichuan Wang Affiliation: Harvard University Email: haichuanwang@g.harvard.edu Anagha Satish Affiliation: Harvard University Jiale Zheng Affiliation: Harvard University Saisneha Ghatti Affiliation: Massachusetts Institute of Technology Lingkai Kong Affiliation: Harvard University Kai Wang Affiliation: Georgia Institute of Technology Milind Tambe Affiliation: Harvard University Abstract Computing equilibria in high-dimensional continuous games requires learning and representing rich, multi-modal strategy distributions, something conventional neural policies struggle to achieve. Diffusion models offer a natural solution, but standard diffusion training requires direct samples from a fixed target distribution, which are generally unavailable under iterative game-solving updates. We propose a diffusion-based framework for equilibrium computation that enables diffusion-based variants of Online Mirror Descent (OMD) and Fictitious Play (FP). Our approach derives a reweighted score-matching objective that trains diffusion policies without requiring direct samples from each updated policy. Our main method, DiffNash, combines the fast convergence of OMD with the sample efficiency of FP by reusing past actions through a replay buffer. To correct the resulting distribution shift from sample reuse, we derive an unbiased estimator of the difference between current and past policies’ Evidence Lower Bounds, whose exponential provides a tractable proxy for the corresponding importance weight without evaluating diffusion likelihoods. We evaluate our approach on forest protection games and continuous Colonel Blotto games, reducing the Nash gap over the strongest baseline by 35.6% and 24.7%, respectively, with minimal hyperparameter tuning. $ $$ $footnotetext: Equal Contribution. 1 Introduction In real-world decision making, many choices are strategic: a party’s outcome depends not only on its own decisions, but also on those of adversaries with competing objectives. Game theory provides a framework for modeling these interactions. In many applications, however, strategic decisions are not naturally enumerable and instead lie in continuous action spaces (Stein et al., 2008; Ganzfried, 2021; Kroer and Sandholm, 2015). In spatial security games, for example, a defender allocates resources while an adversary chooses where to attack (Kamra et al., 2018; Gan et al., 2017; Alpern et al., 2022); these decisions are naturally represented by locations or trajectories rather than discrete actions. Continuous decisions also arise in economic competition, where firms choose production quantities (Shi and Zhang, 2020), and in resource-allocation games, where players divide continuous budgets across competing objectives (Roberson, 2006). Such settings are difficult to capture with finite strategy spaces and require reasoning over distributions on large continuous action spaces. A Nash equilibrium is a strategy profile in which neither player can improve its outcome by unilaterally changing its strategy. A mixed strategy is a probability distribution over a player’s possible actions; in continuous games, this distribution may be defined over an infinite action space (Glicksberg, 1952). As the dimensionality and complexity of the action space grow, representing the mixed strategy itself becomes challenging, since equilibrium play may require complex, multimodal distributions over high-dimensional continuous actions (Perkins et al., 2015). Continuous-action games, even in the more well-understood two player zero-sum (2p0s) setting, present unique computation challenges. While efficient algorithms (Dantzig, 1951) and practical solvers (Jain et al., 2011; Cerny et al., 2024) exists for 2p0s games with finite action set, they cannot directly handle infinitely many actions in continuous-action games. Therefore, prior works (Kamra et al., 2019; Martin and Sandholm, 2023) parameterize player strategies with neural networks and optimizes them through iterative algorithms such as Fictitious Play (FP) or Online Mirror Descent (OMD) to approximate Nash equilibria. However, these neural policies can suffer from mode collapse in high-dimensional action spaces, failing to capture the multimodal distributions that naturally arise in continuous-action games. Since diffusion model is capable of modeling multimodality in continuous vectors (Ho et al., 2020; Wang et al., 2023; Li et al., 2024), we ask Can we parameterize player strategy with diffusion policies in iterative game-solving algorithms? In this work we propose DiffNash, a method for introducing diffusion models into equilibrium computation for games with continuous action. We parameterize each player’s mixed strategy with a diffusion model and use Online Mirror Descent (OMD) Blair (1985) to iteratively update these strategies towards equilibrium. At each iteration, we sample candidate actions from the player’s current diffusion policy and evaluate how useful those actions would be against the opponent’s current policy. Standard diffusion training assumes access to samples from the target distribution, while the OMD update defines this distribution only implicitly. Hence, we adapt the reweighting technique of Ma et al. (2025) to modify the diffusion training loss, provably recovering the correct target update. To reduce the cost of repeatedly sampling news actions from diffusion and stabilize training along OMD iterations, we reuse samples from past policies through a replay buffer. We correct the resulting distribution shift with importance weighting and approximate the importance weights using Evidence Lower Bounds (ELBOs). Our main contributions are: (1) We develop a diffusion-policy framework for equilibrium computation in continuous action games using utility-weighted score matching to represent complex, potentially multimodal mixed strategies (Section 3.1) (2) We incorporate a diffusion update into OMD, with an empirical comparison showing substantially faster convergence than a corresponding diffusion-FP approach (Section 3.2) (3) We construct an ELBO-difference based importance weighting that allows past actions to be reused for opponent evaluation (Section 3.3) and stabilizing training (Section 3.4) (4) We evaluate our approach on two continuous-action games, reducing the Nash gap over the strongest baseline by 35.6% and 24.7%, respectively, with minimal hyperparameter tuning (Section 5). 2 Background 2.1 Problem Formulation We consider a two-player normal-form game =(1,2,u1,u2)G=(A^1,A^2,u^1,u^2), where i⊆ℝdiA^i ^d^i is a non-empty compact action space of player i and ui:1×2→ℝu^i:A^1×A^2 be its bounded and measurable utility function 11 1 We use superscripts for game-theoretic notation, such as player and equilibrium-solver iteration indices, and subscripts for diffusion model notation.. We consider high-dimensional continuous action set, which may also subject to global feasibility constraints, such as a fixed resource budget or a bounded spatial domain. A mixed strategy πi∈Δ(i)π^i∈ (A^i) is a probability distribution over iA^i. Given a strategy profile π=(π1,π2)π=(π^1,π^2), the expected payoff to player 1 is Ui(π1,π2)=1∼π12∼π2[ui(1,2)]U^i(π^1,π^2)=E_ subarrayca^1 π^1\\ a^2 π^2 subarray [u^i(a^1,a^2) ]. Our goal is to find the Nash equilibrium of the game, where neither player has the incentive to deviate from their chosen strategy. Formally, Definition 1 (Nash Equilibrium). π⋆=(π1,⋆,π2,⋆)π =(π^1, ,π^2, ) is a Nash equilibrium if, for i∈1,2i∈\1,2\, Ui(πi,⋆,π−i,⋆)≥Ui(πi,π−i,⋆)for all πi∈Δ(i).U^i(π^i, ,π^-i, )≥ U^i(π^i,π^-i, ) all π^i∈ (A^i). We focus primarily on two-player zero-sum (2p0s) games, where u1(1,2)=−u2(1,2)u^1(a^1,a^2)=-u^2(a^1,a^2). 2p0s games with finite action can be solved by linear programming (Dantzig, 1951). For games with large combinatorial action spaces, such as network security games, practical methods exploit problem structure and propose practical solvers (Jain et al., 2011; Cerny et al., 2024). Continuous-action games require representing and optimizing distributions over an infinite action space. Recent approaches therefore use neural parameterizations of mixed strategies within iterative game-solving algorithms (Kamra et al., 2019; Martin and Sandholm, 2023). Our work follows this latter setting, using diffusion models as expressive representations of continuous mixed strategies. We primarily study the 2p0s setting, where equilibrium objective and associated learning dynamics are well understood. Methods with strong convergence guarantees in 2p0s games are also commonly studied and applied in general-sum settings, although convergence to Nash equilibrium is no longer guaranteed (Anagnostides et al., 2022). We also include general-sum games, where player’s objectives are not necessarily opposed and equilibrium computation is generally more challenging. We measure solution quality using the Nash gap, defined as the total improvement available to players through unilateral best responses (Lanctot et al., 2017). It is non-negative and equals zero at Nash equilibrium, measuring how exploitable the strategy profile is. Definition 2 (Nash Gap). Gap(π1,π2)=maxπ1′∈Δ(1)U(π1′,π2)−minπ2′∈Δ(2)U(π1,π2′).Gap( _1, _2)= _ _1 ∈ (A_1)U( _1 , _2)- _ _2 ∈ (A_2)U( _1, _2 ). A low Nash gap limits gains from opponent’s unilateral deviations. We seek such strategies in high-dimensional action spaces, where conventional neural policies struggle (Kamra et al., 2019). 2.2 Diffusion Models Denoising diffusion probabilistic models (DDPMs) (Sohl-Dickstein et al., 2015; Ho et al., 2020) is a generative model consisting of two stochastic processes: one forward process which progressively adds noise to the data, and a reverse (denoising) process which learns to gradually remove noise. Formally, given a clean data from the target distribution 0∼p0z_0 p_0, the forward process gradually perturbs the data by q(t∣t−1)=(t∣1−βtt−1,βt)q(z_t _t-1)=N\! (z_t 1- _tz_t-1,\, _tI ), where tz_t is the perturbed data at time t and βt _t is the noise variance at time step t=1,⋯,Tt=1,·s,T. The corresponding joint distribution of forward trajectory is q0:T(0:T)=p0(0)∏t=1Tqt|t−1(t|t−1)q_0:T(z_0:T)=p_0(z_0) _t=1^Tq_t|t-1(z_t|z_t-1). Since each step adds independent Gaussian noise, we can sample tz_t by t∼pt(t)=∫p0(0)qt|0(t∣0)d0z_t p_t(z_t)= p_0(z_0)\,q_t 0(z_t _0)\,dz_0, where qt|0(t∣0):=(t,α¯t0,(1−α¯t)),α¯t:=∏ℓ=1tαℓ,αt:=1−βtq_t 0(z_t _0):=N\! (z_t; α_t\,z_0,\,(1- α_t)I ),\ α_t:= _ =1^t _ ,\ _t:=1- _t. 22 2 Both p and q are density function. We use q to denote tractable distributions like Gaussian corruption. When there is no ambiguity, we use p and p0p_0 interchangably. The backward process recovers the data distribution from a noise distribution pTp_T with a iterative process. When βt _t is small, one can approximate the denoising process by the score function s(t):=∇tlogp(t)s(z_t):= _z_t p(z_t). Specifically, q(t−1|t)≈N(t−1,1αt(xt+βts(xt)),σt2)q(z_t-1|z_t)≈ N(z_t-1; 1 _t(x_t+ _ts(x_t)), _t^2I), where σt=1−α¯t−11−αt¯βt _t= 1- α_t-11- _t _t. Therefore, one can follow the following iterative procedure to draw samples from p0p_0: t−1=1αt(t+βts(t))+1−α¯t−11−α¯tβtϵt _t-1= 1 _t(z_t+ _ts(z_t))+ 1- α_t-11- α_t _t _t (1) for t from T to 11 and ϵt∼(0,1) _t (0,1). However, we do not know s(t)s(z_t) in closed form, so we train a score network sθ(t,t)s_θ(z_t,t) to approximate it, with the following tractable loss function (Ho et al., 2020): L(θ)=t∈[T](1−α¯t)0∼p0,t∼qt|0[||sθ(t;t)−∇tlogqt|0(t|0)||2] L(θ)=E_t∈[T](1- α_t)E_z_0 p_0,z_t q_t|0[||s_θ(z_t;t)- _z_t q_t|0(z_t|z_0)||^2] (2) Ma et al. (2025) shows that fix a time t, Equation 3 has the same global minimizer as Equation 2: L(θ;t)=t∼pt[||sθ(t;t)−∇tlogpt(t)] L(θ;t)=E_z_t p_t[||s_θ(z_t;t)- _z_t p_t(z_t)] (3) Though Equation 3 is intractable because we do not know ∇tlogpt(t) _z_t p_t(z_t), it’s important for the reweighting technique used in Section 3.1. 2.3 Equilibrium Solvers: Online Mirror Descent and Fictitious Play Let ui(i,π−i):=−i∼π−i[ui(i,−i)]u^i(a^i;π^-i):=E_a^-i π^-i[u^i(a^i,a^-i)] denote player i’s expected utility for action ia^i against opponent strategy π−iπ^-i. We introduce two major iterative algorithms used to compute Nash equilibrium for continuous action in ML literature: Online Mirror Descent (OMD) and Fictitious Play (FP). Online mirror descent. In OMD (Kivinen and Warmuth, 1997; Beck and Teboulle, 2003), each player computes a KL-regularized best response to the opponent’s strategy from the previous iteration. At iteration k, player i computes πi,k+1(i)=πi,k(i)exp(1ηui(i,π−i,k))/Zi,k, π^i,k+1(a^i)=π^i,k(a^i) ( 1η u_i( ;π^-i,k))/Z_OMD^i,k\ , (4) where 1η 1η is called temperature and Zi,k=∫iπi,k(i)exp(1ηui(i,π−i,k))daiZ_OMD^i,k= _A^iπ^i,k(a^i) \! ( 1η\,u_i(a^i;π^-i,k) )\,da^i is an intractable unknown constant. We define empirical average of OMD responses as Πi,k=1k∑h=1kπi,h ^i,k= 1k _h=1^k _OMD^i,h. Under standard regularity conditions and an appropriate learning rate schedule, OMD is a no-regret algorithm on continuous action space (Krichene et al., 2015). When both players play such algorithms, their empirical strategies converge asymptotically to the Nash equilibrium set (Balandat et al., 2016). Fictitious play. While in OMD player computes a KL-regularized best response to the opponent’s current strategy, in FP (Brown, 1951) player best responds to the empirical distribution of the opponent’s past play (Brown, 1951). FP is known to converge to Nash equilibrium with continuous action space and continuous utility function (Danskin, 1954). We consider a special variant called Generalised Weakened Fictitious Play (GWFP) (Leslie and Collins, 2006). Let πi,k _GWFP^i,k denote player i’s best response at iteration k, and let Πi,k:=1k∑h=1kπi,h ^i,k:= 1k _h=1^k _GWFP^i,h denote the empirical average of its past responses. Under GWFP, player i computes πi,k+1=exp(1ηkui(i,Π−i,k))/Zi,k, _GWFP^i,k+1= ( 1η^ku^i(a^i, ^-i,k) )/Z^i,k\ , (5) where ηkη^k is called temperature and Zi,k=∫iexp(ui(ai,Π−i,k)ηk)diZ^i,k= _A^i ( u^i(a^i, ^-i,k) _k )da^i is an intractable normalization constant. Under vanishing temperature condition, (Π1,k,Π2,k)( ^1,k, ^2,k) converges to the set of Nash equilibria asymptotically under GWFP (Leslie and Collins, 2006). To represent multimodal mixed strategies, we represent player i’s latent action with a diffusion model pθip_θ^i. A game-specific map f^G will convert the latent sample to a feasible player action, i.e., i=f(),∼pθia^i=f^G(z),z p_θ^i. The resulting mixed strategy is then πθi=f#pθi _θ^i=f_\#^Gp_θ^i. We call πθi _θ^i player i’s diffusion policy following Ma et al. (2025)’s notation, abstracting away latent space details. Understanding Convergence Dynamics. Computing the Nash gap requires solving an intractable best response problem, and approximating it at every iteration is prohibitively expensive. To study convergence dynamics, we use a multi-market Cournot game (Kamra et al., 2019) whose unique Nash equilibrium is an analytically derivable singleton, which we denote as (1,∗,2,∗)(a^1,*,a^2,*). In Cournot game, two firms compete by choosing production level across d independent markets to maximizes total profits. We defer Cournot game’s details to Appendix D.2. We define the distance of a strategy profile Π to the equilibrium by 12∑i=12ai∼πi‖ai−ai,⋆‖2. 12 _i=1^2E_a^i π^i||a^i-a^i, ||_2. Because this distance is easy to calculate, we compute it at each iteration in the Cournot game to study convergence dynamics. 3 Proposed Method Figure 1: Overview of DiffNash. Each player samples actions from a diffusion policy into a replay buffer. Both lanes pass through a shared replay weight, obtained from a single-call ELBO difference (Proposition 2), and scored against the opponent’s buffered actions by importance sampling. The OMD weight then reweights the denoising loss. Player −i-i is updated symmetrically. 3.1 Enabling Diffusion-based GWFP and OMD via Reweighted Score Matching The game-solving updates in GWFP Equation 5 and OMD Equation 4 both involve unknown normalization constants. Such unnormalized distributions can be viewed as energy-based models, which are generally difficult to train and sample from (Song and Kingma, 2021). Moreover, standard diffusion training requires samples from the target distribution, which are not directly available for the GWFP and OMD updates. Ma et al. (2025) show that, instead of integrating the squared score loss in Equation 3 with respect to ptp_t, the objective can be reweighted by any strictly positive function g(t)g(a_t) without changing its global minimizer: L(θ;t)=∫g(t)||sθ(t;t)−∇tlogpt(t)||2dt L(θ;t)= g(a_t)||s_θ(a_t;t)- _a_t p_t(a_t)||^2da_t (6) By choosing appropriate g(t)g(a_t), they derive tractable loss to learn the score function of diffusion policy in online RL. We use πt,solveri,k+1(t) _t,solver^i,k+1(a_t) to denote noise-corrupted target update at time t, i.e., πt,solveri,k+1(t):=∫πsolveri,k+1(0)qt|0(t∣0)d0 _t,solver^i,k+1(a_t):= _solver^i,k+1(a_0)\,q_t 0(a_t _0)\,da_0, where solver∈GWFP,OMDsolver∈\GWFP,OMD\. We adapt the reweighting technique to our setting by setting pt=πt,solveri,k+1p_t= _t,solver^i,k+1 and choosing proper g(t)g(a_t), so that the resulting objectives can recover score functions of noise-corrupted updated in OMD and GWFP. Proposition 1 (OMD and GWFP Loss). By choosing an appropriate gOMD(t)g_OMD(a_t) and substitute it into Equation 6, we can equivalently write their loss functions as ℒOMD(θ;t)=0∼πi,k[t∼qt|0(⋅∣0)[exp(1ηui(0;π−i,k))‖sθ(t,t)−∇tlogqt|0(t∣0)‖22]].L_OMD(θ;t)=E_a_0 π^i,k [E_a_t q_t|0(· _0) [ ( 1ηu^i(a_0;π^-i,k) ) \|s_θ(a_t,t)- _a_t q_t|0(a_t _0) \|_2^2 ] ]. (7) For every t∈(0,T]t∈(0,T], the minimizer of Equation 7 over s is the score function of πt,OMDi,k+1(t)π^i,k+1_t,OMD(a_t). In Appendix E.1, we show with a proper gGWFP(t)g_GWFP(a_t), we can recover the score function of πt,GWFPi,k+1(t)π^i,k+1_t,GWFP(a_t). We defer Proposition 1’s proof to Appendix E.1. The intuition of choosing g(t)g(a_t) is to ensure we can sample from all distributions appear in the resulting loss. For example, at iteration k+1k+1, we have sampling access to πi,kπ^i,k and qt|0q_t|0, while ui(~0,π−i,k)u^i( a_0;π^-i,k) can be unbiasedly estimated with samples from opponent’s current strategy π−i,kπ^-i,k. Thus, the objectives in Proposition 1 are tractable, and optimizing them recover the corresponding score function ∇tlogπt,solveri,k+1(t) _a_t _t,solver^i,k+1(a_t). Recall we only need the score function to perform the iterative diffusion sampling process specified in Equation 1. Hence, these objectives suffice to implement diffusion-based OMD and GWFP updates. 3.2 Comparison Between FP and OMD Figure 2: We run diffusion-based GWFP and OMD on the Cournot game for 600600 iterations across three random seeds and report their mean distances to the Nash equilibrium. In this section, we examine the complementary strengths of FP and OMD to guide the design of an algorithm that combines their advantages. (Perolat et al., 2021; Lauriere et al., 2022) report faster convergence for OMD, partly because best responding to stale historical strategies can slow FP’s convergence. We confirm this finding in Figure 2: we run diffusion-based GWFP and OMD for 600600 iterations on the Cournot game. OMD converges substantially faster in both dimensionalities. With action dimension d=30d=30, GWFP’s distance to equilibrium decreases only marginally, highlighting the difficulty of convergence in high-dimensional action spaces. While prior works are often based on FP Kamra et al. (2019); Karthikeyan and Pant (2025), this result motivates using OMD as our equilibrium solver’s backbone. Evaluating ℒOMD(θ,t)L_OMD(θ;t) in Equation 7 requires estimating the expected utility against the current opponent policy. Using L freshly generated opponent actions, this expectation can be estimated as u^i(0,π−i,k)=1L∑ℓ=1Lui(0,(ℓ)−i,k),(ℓ)−i,k∼π−i,k. u^i(a_0;π^-i,k)= 1L _ =1^Lu^i\! (a_0,a_( )^-i,k ),a_( )^-i,k π^-i,k. However, since diffusion generation involves an iterative denoising process, repeatedly sampling fresh actions from the current opponent policy can be computationally expensive. FP largely avoids this cost because each player best-responds to the opponent’s historical mixture Π−i,k ^-i,k. Following Kamra et al. (2019), samples from this mixture can be maintained in a replay buffer, making them cheap to reuse. To improve the sample efficiency of our OMD solver, we similarly maintain a replay buffer for each player and use importance sampling (IS) to correct the distribution shift between the policy that generates a sample and π−i,kπ^-i,k. Formally, let k(ℓ)k^( ) denote the iteration at which the ℓ -th sampled action was generated, so that (ℓ)−i,k(ℓ)∼π−i,k(ℓ)a_( )^-i,k^( ) π^-i,k^( ). We then have u^i(0,π−i,k)=1L∑ℓ=1Lr−i,k,k(ℓ)ui(0,(ℓ)−i,k(ℓ)),where r−i,k,k(ℓ)=πθ−i,k((ℓ)−i,k(ℓ))πθ−i,kℓ((ℓ)−i,k(ℓ)), u^i(a_0;π^-i,k)= 1L _ =1^Lr^-i,k,k^( )u^i\! (a_0,a_( )^-i,k^( ) ),\ where r^-i,k,k^( )= _θ^-i,k\! (a_( )^-i,k^( ) ) _θ^-i,k_ \! (a_( )^-i,k^( ) ), (8) r−i,k,k(ℓ)r^-i,k,k^( ) is sample l’s importance weight33 3 In practice, we use minr−i,k,k(ℓ),C \r^-i,k,k^( ),C \ as the weight in Equation 8, where C caps the weight to prevent excessively large values (Schulman et al., 2017).. However, computing πθ(⋅) _θ(·) in r−i,k,k(ℓ)r^-i,k,k^( ) exactly remains expensive for diffusion models (Song et al., 2020), motivating us to develop a principled proxy. 3.3 Enable Importance Sampling In this section, we fix a player and suppress the player index. Consider a diffusion sample 0a_0. Lemma 1. OPENrk,k(ℓ)=exp(logπθk(0))−logπθk(ℓ)(0))r^k,k^( )= ( _θ^k\! (a_0) )- _θ^k^( )\! (a_0 ) ). Lemma 1 is proved by direct calculation. By Lemma 1, we only need to approximate logπθ(0) _θ(a_0) to enable efficient importance sampling. We achieve this by using evidence lower bounds (ELBO). Definition 3. The ELBO of a sample 0a_0 is ELBOθ(0)≔q(1:T∣0)[logpθ(0:T)q(1:T∣0)]ELBO_θ(a_0) _q(a_1:T _0) [ p_θ(a_0:T)q(a_1:T _0) ]. ELBO lower bounds the log-likelihood, i.e., ELBOθ(0)≤logπθ(0)ELBO_θ(a_0)≤ _θ(a_0) (Ho et al., 2020). Since logπθ(0) _θ(a_0) is intractable for diffusion, the ELBO is commonly used as a tractable surrogate for training diffusion models. McAllister et al. (2026) also uses ELBO to approximate IS weight, though their method is for flow matching policy. We develop new method for diffusion policy. Substituting log likelihood with ELBO in Lemma 1, it remains to compute ELBOθk(0)−ELBOθk(ℓ)(0)ELBO_θ^k(a_0)-ELBO_θ^k^( )(a_0) to obtain IS weights. Although evaluating each ELBO is still expensive (Ho et al., 2020), Proposition 2 shows that their difference admits an unbiased estimator requiring only one score network evaluation. Proposition 2. Suppose πk(ℓ)π^k^( ) and πkπ^k share the same forward-noising schedule βtt=1T\ _t\_t=1^T. Denote D(θ,0,t,ϵ)=Ct‖sθ(t,t)+ϵ1−α¯t‖22D(θ;a_0,t,ε)=C_t||s_θ(a_t,t)+ ε 1- α_t||_2^2, where CtC_t a constant and t=α¯t0+1−αt¯ϵa_t= α_ta_0+ 1- _tε. If t∼Unif1,…,Tt \1,…,T\ and ϵ∼(0,I)ε (0,I), then Δ^k,k(ℓ):=T(D(θk(ℓ),0,t,ϵ)−D(θk,0,t,ϵ)) ^k,k^( ):=T (D(θ^k^( );a_0,t,ε)-D(θ^k;a_0,t,ε) ) is ELBO difference’s unbiased estimator: t,ϵ[Δ^k,k(ℓ)]=ELBOk(0)−ELBOk(ℓ)(0).E_t,ε[ ^k,k^( )]=ELBO^k(a_0)-ELBO^k^( )(a_0). The proof is deferred to Section E.2. In iteration k(ℓ)k^( ), when we sample 0∼πk(ℓ)a_0 π^k^( ), we record [0,t,ϵ,D(θk(ℓ),0,t,ϵ)] [a_0,t,ε,D(θ^k^( );a_0,t,ε) ] in a buffer, so that we only need to compute D(θk,0,t,ϵ)D(θ^k;a_0,t,ε) in future iteration k to construct Δ^k,k(ℓ) ^k,k^( ). Drawing sample from πθk _θ^k requires T iterative score network calls by Equation 1, while computing D(θk,0,t,ϵ)D(θ^k;a_0,t,ε) only requires one. Hence, our method reduces much sampling cost. We plug exp(Δ^k,k(ℓ)) ( ^k, k^( )) back to Equation 8 to enable reusing opponent buffers Π−i,k ^-i,k. 3.4 Self-Action Importance Sampling Stabilizes Training Figure 3: We use PCA to visualize the self-action buffer at iteration 101101 for Cournot d=30d=30. Recall evaluating the OMD loss Equation 7 requires samples from a player’s self action πi,kπ^i,k. Since we are maintaining replay buffers for both players, we can also use replay samples for a player’s own action, not just opponent action. Denote a player’s self action batch BselfB_self. Instead of always getting fresh samples from πi,kπ^i,k, we use ⌈ζ|Bself|⌉ ζ|B_self| IS samples with the rest from πi,kπ^i,k. Similar to opponent buffer reuse, we use the IS weight to correct distribution shift induced by replay samples. We defer the full algorithm details to Appendix C. We find that adding importance-weighted replay stabilizes training and prevents premature plateaus. The policy starts concentrate on a narrow set of high-utility actions as it approaches equilibrium, limiting the coverage of fresh samples. Replaying diverse past actions broadens the training distribution and prevents the policy from overfitting to a narrow region. We visualize the impact of replay mechanism in Figure 3, where self action IS significantly improves coverage. We introduce an additional hyperparameter called trigger κ, and we initialize the self-action IS once standard deviation of BselfB_self falls below a fraction κ of its initial value. We analyze the impact of the self action IS mechanism in Section 5.3. 4 Related Work Equilibrium Computation. We provide a comprehensive review in Appendix B and focus here on learning-based approaches for continuous action spaces. Prior work learns generative approximate best responses (Kamra et al., 2019), trains randomized policy networks with zeroth-order equilibrium dynamics (Martin and Sandholm, 2023), or minimizes exploitability using learned best responses (Martin and Sandholm, 2025). In contrast, we parameterize each player’s evolving mixed strategy with a diffusion model to capture more expressive distributions. Diffusion Models And Its Application in Online RL Diffusion models learn complex, multimodal distributions through iterative denoising and score matching (Ho et al., 2020; Song et al., 2021). Their ability to model complex distributions motivated their use as policy parameterizations in RL (Wang et al., 2024; Psenka et al., 2024; Ren et al., 2025). Most closely related to ours, Ma et al. (2025) show that exponential reweighting of samples from the current diffusion policy is sufficient to implement a KL-regularized policy-improvement step. We adapt the reweighting technique and address it in the equilibrium computation setting. We give a comprehensive review of the field in Appendix B. Diffusion Applications in Strategic Environment To our knowledge, the only direct application of diffusion to our setting is Karthikeyan and Pant (2025), which we include as a baseline and discuss in Section 5. We discuss diffusion model applications in other related strategic settings in Appendix B. 5 Experiments Beyond the Cournot game, we evaluate DiffNash on a forest security game and a continuous variant of Colonel Blotto, two widely studied games that also admit intuitive visualizations. Metrics and Baselines: We use Nash Gap (Definition 2) as our main evaluation metric. The best response optimization required in Nash Gap cannot be computed exactly. Following Kamra et al. (2019), we use an evolutionary search that repeatedly retains high-utility candidates and perturbs them with Gaussian noise to approximate the Nash Gap. We defer computation details to Section D.1. We compare against three baselines. DeepFP (Kamra et al., 2019) represents each player’s strategy in FP with a neural generator θ()T_θ(z), where ∼(0,)z (0,I). Neural OMD adapts the mirror descent policy optimization framework of Tomar et al. (2022) to equilibrium computation, using a Gaussian neural policy for each player. DiffFP (Karthikeyan and Pant, 2025) parameterizes player strategies with diffusion models within FP. Since FP updates do not directly provide samples from the target strategy, DiffFP instead refines sampled actions ia^i toward higher-utility regions using ∇iui(i,Π−i,k) _a^iu^i(a^i; ^-i,k) and trains the diffusion policy on the refined actions. These baselines help isolate the benefits of diffusion policy and principled policy updates. For fair comparison, we train each player’s policy for 40K40K gradient updates across experiments. We use default hyperparameters for baselines. 5.1 Forest Security Game Forest security game (Johnson et al., 2012) originated from forest economics (Albers, 2010). We consider the continuous version studied by Kamra et al. (2018) and DeepFP (Kamra et al., 2019). A defender selects M locations to patrol, while an attacker selects N locations to target in an endangered circular forest. Each location is parameterized by its radial distance and radian angle. Hence, defender and attacker’s action dimensionality are 2M2M and 2N2N, respectively. As M and N increase, the resulting strategy space becomes high-dimensional, a setting where Kamra et al. (2019) reports degraded performance. An attacker can collect trees within RAR_A of its location, and a defender can stop any attacker within RDR_D of them. The attacker’s utility is the value of the harvested trees but receives no value and incurs a penalty if caught by any defender. The defender’s utility is the negative of the attacker’s utility. Across experiments, we keep the interaction and payoff structure fixed and vary only the spatial distribution of trees along with the number of defenders and attackers. Changing the geometry creates qualitatively different optimal strategies by altering which attack locations are valuable, which regions can be protected together, and how many spatially separated patches the strategy must cover. In this game, we aim to find a robust patrol strategy for defender, so besides Nash gap, we also compute ExploitDExploit_D, measured as the attacker’s best-response reward against the learned average defender. We seek strategies profiles with low ExploitDExploit_D and Low Nash Gap GAPGAP. Experimental results. Table 1 shows that DiffNash achieves the best mean performance across all six forests, reducing Nash gap by an average of 35.6% relative to the strongest baseline. The gap is most pronounced in heterogeneous, multi-resource settings, where equilibrium strategies must distribute probability mass across several strategically distinct regions. DiffNash also achieves broader effective coverage. In F3 of Figure 4, for example, DiffNash’s guards (blue) concentrate around all four tree clusters (green), whereas DeepFP and DiffFP learn more diffuse patrol strategies and Neural OMD places substantial mass in low-tree-density regions. Importantly, diffuse coverage is not necessarily effective coverage: allocating guard probability to regions with few or no trees provides little protection value. We analyze the failure modes of all baselines: • DeepFP/NeuralOMD: Both exhibit mode collapse. In F2 of Figure 4, Neural OMD concentrates on a single forest patch. For DeepFP, Figure 4 and additional Appendix F.1 visualizations show both players’ strategies forming narrow circular bands. Since actions are represented in polar coordinates, this pattern indicates collapse along the radial dimension. • DiffFP: In large forests, small changes in guard locations are unlikely to change whether attackers are captured. Consequently, the defender utility gradient, i.e., ∇iui(i,Π−i,k) _a^iu^i(a^i; ^-i,k), is nearly zero for most locations, causing DiffFP’s defender strategy to get stuck, as shown by Figure 4’s F2. In contrast, DiffNash does not rely on local utility gradients. We update the entire defender policy in a principled way, yielding substantially better performance. Table 1: Forest security game results. We report the attacker best-response reward ExploitDExploit_D against the learned average defender and the two-sided Nash gap G as mean ± 95% Student-t confidence intervals over 55 random seeds; lower is better for both. F1 and F2 are taken from Kamra et al. (2019). DeepFP Neural OMD DiffFP DiffNash (Ours) Forest m=nm=n ExploitDExploit_D Gap ExploitDExploit_D Gap ExploitDExploit_D Gap ExploitDExploit_D Gap F1 1 244.5±164.2244.5± 164.2 245.8±160.6245.8± 160.6 212.6±160.4212.6± 160.4 216.6±160.2216.6± 160.2 490.2±3.4490.2± 3.4 493.0±3.6493.0± 3.6 96.7±21.296.7± 21.2 79.5±28.879.5± 28.8 F1 3 370.5±309.2370.5± 309.2 373.3±297.8373.3± 297.8 307.8±287.6307.8± 287.6 314.5±290.0314.5± 290.0 735.8±40.8735.8± 40.8 741.3±41.0741.3± 41.0 130.4±9.3130.4± 9.3 122.1±12.5122.1± 12.5 F2 1 338.6±6.5338.6± 6.5 330.3±5.8330.3± 5.8 309.7±28.3309.7± 28.3 313.7±28.4313.7± 28.4 354.9±0.9354.9± 0.9 357.8±0.7357.8± 0.7 300.8±5.2300.8± 5.2 264.4±95.5264.4± 95.5 F2 3 702.1±84.3702.1± 84.3 687.4±78.5687.4± 78.5 501.0±179.6501.0± 179.6 500.1±177.2500.1± 177.2 817.3±101.5817.3± 101.5 821.6±101.3821.6± 101.3 322.5±110.1322.5± 110.1 300.7±126.4300.7± 126.4 F3 5 578.8±109.4578.8± 109.4 565.7±119.7565.7± 119.7 767.1±278.4767.1± 278.4 761.1±271.1761.1± 271.1 818.5±123.7818.5± 123.7 800.9±121.3800.9± 121.3 559.7±76.4559.7± 76.4 528.2±142.7528.2± 142.7 F4 5 804.4±118.5804.4± 118.5 776.6±117.4776.6± 117.4 1145.9±191.61145.9± 191.6 1140.9±187.31140.9± 187.3 836.4±114.7836.4± 114.7 822.7±112.8822.7± 112.8 660.4±39.8660.4± 39.8 622.6±52.3622.6± 52.3 Figure 4: Empirical strategies in the forest security game Top: F2 (m=n=3)(m=n=3). Bottom: F3 (m=n=5)(m=n=5). Green points denote trees. Each blue/red points denote a sample from defender/attacker policy, respectively. For each method and environment, the displayed strategy is taken from 55 random seeds whose Nash gap is the median. Quantitative results are in Table 1. 5.2 Continuous Colonel Blotto We evaluate DiffNash on the continuous Colonel Blotto game (Adam et al., 2021) where two players simultaneously allocate a fixed budget across n weighted battlefields. Player’s utility depends on how much they and opponent invest in each battlefield. Because allocations are coupled by the budget constraint, equilibrium play generally requires nontrivial randomization over a continuous action space. We consider games with n=6n=6 and n=10n=10. Adam et al. (2021) provides a discrete algorithm which approximates the target equilibrium well 44 4 Such oracle-based methods do not directly extend to general continuous-action games without a tractable best-response oracle; see Appendix D.4.2., visualized in the first column of Figure 5. We vary player utilty function to get distinct target equilibrium, one with symmetric and another with radial shape. Full game details are provided in Appendix D.4. Table 2: Continuous Colonel Blotto results. Nash gap G on symmetric and radial-band Blotto with n∈6,10n∈\6,10\. Values are mean ± 95% Student-t confidence intervals over five seeds; lower is better. n Equilibrium DeepFP Neural OMD DiffFP DiffNash (Ours) 6 Symmetric 6.854±1.3666.854± 1.366 6.807±1.8326.807± 1.832 0.919±0.0650.919± 0.065 0.942±0.1680.942± 0.168 6 Radial band 0.688±0.0010.688± 0.001 0.836±0.2360.836± 0.236 1.038±0.0561.038± 0.056 0.501±0.1390.501± 0.139 10 Symmetric 9.525±1.7079.525± 1.707 7.233±1.3047.233± 1.304 1.506±0.0561.506± 0.056 1.724±0.3461.724± 0.346 10 Radial band 0.696±0.0020.696± 0.002 0.793±0.3180.793± 0.318 1.446±0.0371.446± 0.037 0.531±0.1160.531± 0.116 Experimental results. Overall DiffNash reduces the Nash Gap by 24.7% relative to the strongest baseline. In the symmetric setting of Table 2, diffusion-based policies achieve substantially lower Nash gaps than the neural baselines, highlighting their ability to represent multimodal strategies. In the radial-band setting, DiffNash consistently outperforms DiffFP. As in the forest game, when high-utility actions lie within a narrow region (the radial band), DiffFP can receive little local gradient signal from low-utility initializations. In Figure 5, this causes its policy to remain concentrated near the center, outside the support of the target equilibrium, whereas DiffNash recovers the correct equilibrium geometry. We visualize all settings in Appendix Figure 8. Figure 5: Continuous Colonel Blotto (radial-band variant,n=6n=6). Green denotes samples from the target equilibrium, while purple denotes samples from the learned strategy. Each point represents an allocation, where proximity to []j[a]_j indicates a larger allocation to the j-th battlefield. The point cloud visualizes the distribution of allocations induced by each strategy. Figure 6: Performance under different IS configurations. Solid lines denote mean, and shades denote the min to max range across runs. Robustness Across experiments in Section 5.1 and Section 5.2 we perform minimal hyperparameter tuning: we fix self action ratio ζ=1%ζ=1\% and only choose IS trigger κ=0,10%κ=\0,10\%\. DiffNash shows strong robustness to hyperparameters. 5.3 Ablation and Hyperparameter Study Ablation We ablate the two importance-sampling components of DiffNash on Cournot (d=30d=30). In Figure 6, the green and blue curves almost overlap, indicating that opponent IS has little effect on convergence. Its primary benefit is computational: in this experiment, opponent IS reduces the number of score-network calls to roughly 66%66\% of that required without opponent IS. Moreover, DiffNash (orange) achieves the lowest Nash Gap, showing that self action IS prevents the early plateau observed without self action IS and reduces variance across runs. We observe a similar pattern in the forest ablation in Appendix F.4. These results support using IS for both opponent and self-action replay. We additionally show our method’s performance in a general sum version of the Forest game in Section F.3. Hyperparameter Study We provide hyperparameter study for self action IS ratio ζ, trigger κ, self action buffer size and opponent action buffer size in Appendix F.5. 6 Conclusion We develop DiffNash, a framework that parameterizes player strategies with diffusion models within OMD to compute Nash equilibria in high-dimensional continuous action spaces. We derive a reweighted score-matching objective that trains the diffusion model to realize OMD updates. To improve sample efficiency and training stability, we reuse past actions through a replay buffer and correct the resulting distribution shift with an ELBO-based importance-sampling strategy. On forest security and continuous Colonel Blotto games, DiffNash better recovers equilibrium distributions and reduces Nash gap by 35.6% and 24.7% compared to strongest baselines, respectively. AI Usage Disclosure In this work, we used generative AI tools (GPT-5.6 Sol, GPT-6 Sol, and Claude Opus 5) for polishing and drafting sections of the manuscript, flowcharts, and assists with coding and proofs. Generating synthetic data is not applicable to this work. Additionally, we used generative AI tools for brainstorming research ideas and searching related work. At least one coauthor has reviewed AI-assisted work. We manually check the generated code and proof. The main technical contributions, including reweighted score matching objective and importance sampling strategy, are the authors’ original ideas. We take responsibility for the final content of this work, including text, claims, and artifacts produced with the aid of generative AI. Ethics Statement This work studies algorithms for computing Nash equilibria in simulated continuous-action games and does not involve human subjects, personal or sensitive data, or real-world interventions. While equilibrium-computation methods may be applicable to domains such as security and resource allocation, all experiments in this work use synthetic benchmark environments and do not involve deployed decision-making systems. We are not aware of additional ethical concerns arising from this work. Reproducibility Statement We provide complete pseudocode for DiffNash and describe the training procedure, replay mechanism, game definitions, environment parameters, reference solutions, hyperparameters, best-response and Nash-gap evaluation procedures, ablations, and computing infrastructure in the appendix. These details are intended to make our method and reported experiments reproducible from the paper. Some baseline components and environment code, including the original Forest Security Game and DeepFP implementations, were not publicly available and were obtained directly from the original authors. Reported results are averaged over multiple random seeds with confidence intervals, and proofs of all theoretical results are included. References Adam et al. (2021) L. Adam, R. Horčík, T. Kasl, and T. Kroupa Double oracle algorithm for computing equilibria in continuous games. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 35, p. 5070–5077. Cited by: Appendix B, §D.1, §D.4.1, §D.4.2, §D.4.2, §5.2. Albers (2010) H. J. Albers Spatial modeling of extraction and enforcement in developing country protected areas. Resource and Energy Economics 32 (2), p. 165–179. Cited by: §5.1. Alpern et al. (2022) S. Alpern, T. Bui, T. Lidbetter, and K. Papadaki Continuous patrolling games. Operations Research 70 (6), p. 3076–3089. Cited by: §1. Anagnostides et al. (2022) I. Anagnostides, G. Farina, I. Panageas, and T. Sandholm Optimistic mirror descent either converges to nash or to strong coarse correlated equilibria in bimatrix games. Advances in Neural Information Processing Systems 35, p. 16439–16454. Cited by: §2.1. Balandat et al. (2016) M. Balandat, W. Krichene, C. Tomlin, and A. Bayen Minimizing regret on reflexive banach spaces and nash equilibria in continuous zero-sum games. Advances in Neural Information Processing Systems 29. Cited by: §2.3. Beck and Teboulle (2003) A. Beck and M. Teboulle Mirror descent and nonlinear projected subgradient methods for convex optimization. Operations Research Letters 31 (3), p. 167–175. Cited by: §2.3. Blair (1985) C. Blair Problem complexity and method efficiency in optimization (as nemirovsky and db yudin). Siam Review 27 (2), p. 264. Cited by: §1. Brown (1951) G. W. Brown Iterative solution of games by fictitious play. Act. Anal. Prod Allocation 13 (1), p. 374. Cited by: §2.3. Cerny et al. (2024) J. Cerny, C. K. Ling, C. Kroer, and G. Iyengar Layered graph security games. In Proceedings of the Thirty-ThirdInternational Joint Conference on Artificial Intelligence, p. 2695–2703. External Links: Document, Link Cited by: §1, §2.1. Chen et al. (2009) X. Chen, X. Deng, and S. Teng Settling the complexity of computing two-player nash equilibria. Journal of the ACM (JACM) 56 (3), p. 1–57. Cited by: Appendix B. Danskin (1954) J. Danskin Fictitious play for continuous games. Naval Research Logistics Quarterly 1 (4), p. 313–320. Cited by: §2.3. Dantzig (1951) G. B. Dantzig A proof of the equivalence of the programming problem and the game problem, in “activity analysis of production and allocation”(ed. tc koopmans), cowles commission monograph, no. 13. John Wiley and Sons. Cited by: Appendix B, §1, §2.1. Gan et al. (2017) J. Gan, B. An, Y. Vorobeychik, and B. Gauch Security games on a plane. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 31. Cited by: §1. Ganzfried (2021) S. Ganzfried Algorithm for computing approximate nash equilibrium in continuous games with application to continuous blotto. Games 12 (2), p. 47. Cited by: §1. Glicksberg (1952) I. L. Glicksberg A further generalization of the kakutani fixed theorem, with application to nash equilibrium points. Proceedings of the American Mathematical Society 3 (1), p. 170–174. Cited by: §1. Ho et al. (2020) J. Ho, A. Jain, and P. Abbeel Denoising diffusion probabilistic models. Advances in neural information processing systems 33, p. 6840–6851. Cited by: §E.2, §E.2, §1, §2.2, §2.2, §3.3, §4. Jain et al. (2011) M. Jain, D. Korzhyk, O. Vaněk, V. Conitzer, M. Pěchouček, and M. Tambe A double oracle algorithm for zero-sum security games on graphs. In The 10th International Conference on Autonomous Agents and Multiagent Systems-Volume 1, p. 327–334. Cited by: §1, §2.1. Johnson et al. (2012) M. Johnson, F. Fang, and M. Tambe Patrol strategies to maximize pristine forest area. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 26, p. 295–301. Cited by: §5.1. Kamra et al. (2018) N. Kamra, U. Gupta, F. Fang, Y. Liu, and M. Tambe Policy learning for continuous space security games using neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 32. Cited by: §D.3.1, §1, §5.1. Kamra et al. (2019) N. Kamra, U. Gupta, K. Wang, F. Fang, Y. Liu, and M. Tambe Deep fictitious play for games with continuous action spaces. In Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems, p. 2042–2044. Cited by: Appendix B, §D.1, §D.3.1, §D.3.1, §D.3.2, §F.3, §1, §2.1, §2.1, §2.3, §3.2, §3.2, §4, §5.1, Table 1, Table 1, §5, §5. Karthikeyan and Pant (2025) A. Karthikeyan and Y. V. Pant DiffFP: learning behaviors from scratch via diffusion-based fictitious play. arXiv preprint arXiv:2511.13186. Cited by: §D.3.2, §3.2, §4, §5. Kingma and Ba (2014) D. P. Kingma and J. Ba Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980. Cited by: §D.3.2, §D.4.2. Kivinen and Warmuth (1997) J. Kivinen and M. K. Warmuth Exponentiated gradient versus gradient descent for linear predictors. Information and computation 132 (1), p. 1–63. Cited by: §2.3. Kong et al. (2026a) L. Kong, A. Satish, H. Jiang, A. Kangaslahti, A. Ma, W. Chen, M. Song, L. Xu, and M. Tambe Latent spherical flow policy for reinforcement learning with combinatorial actions. In Forty-third International Conference on Machine Learning, External Links: Link Cited by: Appendix B. Kong et al. (2026b) L. Kong, H. Wang, C. A. Emogor, V. Böersch-Supan, L. Xu, and M. Tambe Generative ai against poaching: latent composite flow matching for poaching prediction. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 40, p. 40279–40286. Cited by: Appendix B. Kong et al. (2025) L. Kong, H. Wang, Y. Pan, C. W. Kim, M. Song, A. Nguyen, T. Wang, H. Xu, and M. Tambe Robust optimization with diffusion models for green security. In The 41st Conference on Uncertainty in Artificial Intelligence, External Links: Link Cited by: Appendix B. Kong et al. (2026c) L. Kong, H. Wang, T. Wang, G. Xiong, and M. Tambe Composite flow matching for reinforcement learning with shifted-dynamics data. Advances in Neural Information Processing Systems 38, p. 92500–92541. Cited by: Appendix B. Krichene et al. (2015) W. Krichene, M. Balandat, C. Tomlin, and A. Bayen The hedge algorithm on a continuum. In International Conference on Machine Learning, p. 824–832. Cited by: §2.3. Kroer and Sandholm (2015) C. Kroer and T. Sandholm Discretization of continuous action spaces in extensive-form games. In Proceedings of the 2015 international conference on autonomous agents and multiagent systems, p. 47–56. Cited by: §1. Kroupa and Votroubek (2022) T. Kroupa and T. Votroubek Multiple oracle algorithm to solve continuous games. In International Conference on Decision and Game Theory for Security, p. 149–167. Cited by: Appendix B. Lanctot et al. (2017) M. Lanctot, V. Zambaldi, A. Gruslys, A. Lazaridou, K. Tuyls, J. Pérolat, D. Silver, and T. Graepel A unified game-theoretic approach to multiagent reinforcement learning. Advances in neural information processing systems 30. Cited by: §2.1. Lauriere et al. (2022) M. Lauriere, S. Perrin, S. Girgin, P. Muller, A. Jain, T. Cabannes, G. Piliouras, J. Pérolat, R. Elie, O. Pietquin, et al. Scalable deep reinforcement learning algorithms for mean field games. In International conference on machine learning, p. 12078–12095. Cited by: §3.2. Leslie and Collins (2006) D. S. Leslie and E. J. Collins Generalised weakened fictitious play. Games and Economic Behavior 56 (2), p. 285–298. Cited by: §2.3, §2.3. Li et al. (2024) S. Li, R. Krohn, T. Chen, A. Ajay, P. Agrawal, and G. Chalvatzaki Learning multimodal behaviors from scratch with diffusion policy gradient. Advances in Neural Information Processing Systems 37, p. 38456–38479. Cited by: §1. Li et al. (2026) Z. Li, H. Zhong, X. Wang, Q. Xia, L. Zhang, and L. Huang Diffusing to coordinate: efficient online multi-agent diffusion policies. In Forty-third International Conference on Machine Learning, External Links: Link Cited by: Appendix B. Lipman et al. (2023) Y. Lipman, R. T. Q. Chen, H. Ben-Hamu, M. Nickel, and M. Le Flow matching for generative modeling. In The Eleventh International Conference on Learning Representations, External Links: Link Cited by: Appendix B. Ma et al. (2025) H. Ma, T. Chen, K. Wang, N. Li, and B. Dai Efficient online reinforcement learning for diffusion policy. In Forty-second International Conference on Machine Learning, External Links: Link Cited by: Appendix B, §1, §2.2, §2.3, §3.1, §4. Martin and Sandholm (2023) C. Martin and T. Sandholm Finding mixed-strategy equilibria of continuous-action games without gradients using randomized policy networks. In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, IJCAI-23, E. Elkind (Ed.), p. 2844–2852. Note: Main Track External Links: Document, Link Cited by: Appendix B, §1, §2.1, §4. Martin and Sandholm (2025) C. Martin and T. Sandholm ApproxED: approximate exploitability descent via learned best responses. In International Joint Conference on Autonomous Agents and Multiagent Systems, p. 1454–1463. External Links: Document, Link Cited by: Appendix B, §4. McAllister et al. (2026) D. McAllister, S. Ge, B. Yi, C. M. Kim, E. Weber, H. Choi, H. Feng, and A. Kanazawa Flow matching policy gradients. In The Fourteenth International Conference on Learning Representations, External Links: Link Cited by: Appendix B, §3.3. Park et al. (2025) S. Park, Q. Li, and S. Levine Flow q-learning. In Forty-second International Conference on Machine Learning, External Links: Link Cited by: Appendix B. Perkins and Leslie (2014) S. Perkins and D. S. Leslie Stochastic fictitious play with continuous action sets. Journal of Economic Theory 152, p. 179–213. Cited by: Appendix B. Perkins et al. (2015) S. Perkins, P. Mertikopoulos, and D. S. Leslie Mixed-strategy learning with continuous action sets. IEEE Transactions on Automatic Control 62 (1), p. 379–384. Cited by: §1. Perolat et al. (2021) J. Perolat, S. Perrin, R. Elie, M. Laurière, G. Piliouras, M. Geist, K. Tuyls, and O. Pietquin Scaling up mean field games with online mirror descent. arXiv preprint arXiv:2103.00623. Cited by: §3.2. Psenka et al. (2024) M. Psenka, A. Escontrela, P. Abbeel, and Y. Ma Learning a diffusion model policy from rewards via q-score matching. In Forty-first International Conference on Machine Learning, External Links: Link Cited by: Appendix B, §4. Rakhlin and Sridharan (2013) S. Rakhlin and K. Sridharan Optimization, learning, and games with predictable sequences. In Advances in Neural Information Processing Systems, C.J. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Weinberger (Eds.), Vol. 26, p. . External Links: Link Cited by: Appendix B. Ren et al. (2025) A. Z. Ren, J. Lidard, L. L. Ankile, A. Simeonov, P. Agrawal, A. Majumdar, B. Burchfiel, H. Dai, and M. Simchowitz Diffusion policy policy optimization. In The Thirteenth International Conference on Learning Representations, External Links: Link Cited by: Appendix B, §4. Roberson (2006) B. Roberson The colonel blotto game. Economic Theory 29 (1), p. 1–24. Cited by: §1. Schulman et al. (2017) J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. Cited by: footnote 3. Shi and Zhang (2020) Y. Shi and B. Zhang Multi-agent reinforcement learning in cournot games. In 2020 59th ieee conference on decision and control (cdc), p. 3561–3566. Cited by: §1. Sohl-Dickstein et al. (2015) J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning, p. 2256–2265. Cited by: §2.2. Song and Kingma (2021) Y. Song and D. P. Kingma How to train your energy-based models. arXiv preprint arXiv:2101.03288. Cited by: §3.1. Song et al. (2020) Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456. Cited by: §3.2. Song et al. (2021) Y. Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, External Links: Link Cited by: §4. Stein et al. (2008) N. D. Stein, A. Ozdaglar, and P. A. Parrilo Separable and low-rank continuous games. International Journal of Game Theory 37 (4), p. 475–504. Cited by: §1. Tomar et al. (2022) M. Tomar, L. Shani, Y. Efroni, and M. Ghavamzadeh Mirror descent policy optimization. In International Conference on Learning Representations, External Links: Link Cited by: §5. Wang et al. (2026) T. Wang, Y. Jiang, and D. Parkes Bundleflow: deep menus for combinatorial auctions by diffusion-based optimization. Advances in Neural Information Processing Systems 38, p. 16853–16876. Cited by: Appendix B. Wang et al. (2024) Y. Wang, L. Wang, Y. Jiang, W. Zou, T. Liu, X. Song, W. Wang, L. Xiao, J. WU, J. Duan, and S. E. Li Diffusion actor-critic with entropy regulator. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, External Links: Link Cited by: Appendix B, §4. Wang et al. (2023) Z. Wang, J. J. Hunt, and M. Zhou Diffusion policies as an expressive policy class for offline reinforcement learning. In The Eleventh International Conference on Learning Representations, External Links: Link Cited by: §1. Zhang et al. (2025) T. Zhang, C. Yu, S. Su, and Y. Wang ReinFlow: fine-tuning flow matching policy with online reinforcement learning. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, External Links: Link Cited by: Appendix B. Appendix for DiffNash: Find Nash Equilibrium in High Dimensional Continuous Action Spaces with Diffusion Policies Appendix A Limitations We focus on two-player normal-form games. Extending DiffNash to multiplayer or sequential settings, is a natural direction for future work. Appendix B Additional Related Work Equilibrium Computation Finite two-player zero-sum games admit linear-programming formulations [Dantzig, 1951], while equilibrium computation in finite general-sum games is PPAD-complete [Chen et al., 2009]. Continuous action games remain challenging because mixed strategies are distributions over infinite action spaces and best-response computation may require difficult continuous optimizaition. Prior work establishes continuous-action variants of fictitious-play [Perkins and Leslie, 2014] and develops strategy-generaion methods for zero-sum and general-sum continuous games [Adam et al., 2021, Kroupa and Votroubek, 2022]. Learning-based approaches include DeepFP, which learns generative approximate best responses [Kamra et al., 2019], randomized policy networks trained with zeroth-order equilibrium dynamics [Martin and Sandholm, 2023], and exploitability minimization with learned best responses [Martin and Sandholm, 2025] . Our approach instead parameterizes each player’s evolving mixed strategy with a diffusion model and trains it using FP and OMD based equilibrium updates [Rakhlin and Sridharan, 2013]. Diffusion and Flow-Based Models for Online RL Diffusion and flow-based policies have been widely studied for RL [Wang et al., 2024, Psenka et al., 2024, Ma et al., 2025, Ren et al., 2025]. Most closely related to our policy update, Ma et al. [2025] show that exponential reweighting of samples from the current diffusion policy implements a KL-regularized policy-improvement step, yielding Diffusion Policy Mirror Descent. Related work also applies expressive diffusion policies in cooperative multi-agent RL [Li et al., 2026], while flow-matching policies have been developed for online continuous and combinatorial control [Park et al., 2025, Zhang et al., 2025, McAllister et al., 2026, Kong et al., 2026a]. These methods optimize return or coordination rather than equilibrium against strategic opponents. In our setting, policy-improvement weights depend on utility against an evolving opponent. We additionally use ELBO differences as approximate density ratios to reuse samples across strategic iterates. Diffusion Applications in Strategic Environment Diffusion models and their variants, such as flow matching [Lipman et al., 2023], have been applied to strategic settings adjacent to equilibrium computation. For auction theory, Wang et al. [2026] uses flow matching to represent bundle distributions in combinatorial auctions, achieving higher revenue while scaling to larger numbers of items. For green security game, when opponent data is available, diffusion and flow-based models can also be used to learn opponent behavior. Kong et al. [2025] uses diffusion models to learn poacher behavior in green security games and design robust patrol strategies. Kong et al. [2026b] uses flow matching to model poacher behavior from real-world data, and Kong et al. [2026c] subsequently uses reinforcement learning to plan patrol policies against the learned poacher behavioral model. Unlike these works, we assume no prior knowledge of opponent behavior and instead seek a Nash strategy, which provides a worst-case guarantee in 2p0s games. Appendix C Training Algorithm We provide the full pseudocode for DiffNash in Algorithm 3. Algorithm 1 constructs replay entries, Algorithm 2 computes their ELBO-based importance weights, and Algorithm 3 gives the complete OMD training procedure. Self-action replay is activated only if the specified trigger fires; otherwise, training continues using fresh self-actions. Algorithm 1 PrepareSample 1: Action 0a_0, behavior parameters θk(ℓ)θ^k^( ), and diffusion schedule αt,α¯t,βt,σt2t=1T\ _t, α_t, _t, _t^2\_t=1^T. 2: Replay entry z=(0,t,ϵ,Dk(ℓ))z=(a_0,t,ε,D^k^( )). 3: Sample t∼Unif1,…,T,ϵ∼(0,I).t \1,…,T\, ε (0,I). 4: Construct t←α¯t0+1−α¯tϵ.a_t← α_ta_0+ 1- α_tε. 5: Compute Dk(ℓ)←βt22σt2αt‖sθk(ℓ)(t,t)+ϵ1−α¯t‖22.D^k^( )← _t^22 _t^2 _t \|s_θ^k^( )(a_t,t)+ ε 1- α_t \|_2^2. 6: return z←(0,t,ϵ,Dk(ℓ))z←(a_0,t,ε,D^k^( )). Algorithm 2 ComputeWeight 1: Current parameters θkθ^k, replay entry z=(0,t,ϵ,Dk(ℓ))z=(a_0,t,ε,D^k^( )) generated at iteration k(ℓ)k^( ), diffusion schedule, and maximum log-weight C>0C>0. 2: Clipped importance weight rk,k(ℓ)≤eCr^k,k^( )≤ e^C. 3: Reconstruct t←α¯t0+1−α¯tϵ.a_t← α_ta_0+ 1- α_tε. 4: Compute Dk←βt22σt2αt‖sθk(t,t)+ϵ1−α¯t‖22.D^k← _t^22 _t^2 _t \|s_θ^k(a_t,t)+ ε 1- α_t \|_2^2. 5: Compute Δ^k,k(ℓ)←T(Dk(ℓ)−Dk). ^k,k^( )← T(D^k^( )-D^k). 6: return rk,k(ℓ)←exp(minΔ^k,k(ℓ),C).r^k,k^( )← \! ( \ ^k,k^( ),C\ ). Algorithm 3 Replay-Buffer Diffusion Online Mirror Descent (DiffNash) 1: Outer iterations K, inner steps H, self-action batch size BselfB_self, opponent batch size M, initial buffer size N0N_0, opponent-buffer capacity NbufN_buf, refresh size R, OMD parameter η, self-action replay fraction ζ, trigger threshold κ, and maximum log-weight C. 2: Initialize πθi,1=Unif(i) _θ^i,1=Unif(A^i), i∈1,2i∈\1,2\; initialize each player’s opponent replay buffer and self-action history with N0N_0 samples using PrepareSample with behavior parameters θi,1θ^i,1, and record the initial self-action spread. 3: for k=1,…,Kk=1,…,K do 4: for i∈1,2i∈\1,2\ do 5: Sample opponent entries z~−i,(1:L) z^-i,(1:L) with behavior iterations k(1),…,k(L)k^(1),…,k^(L); let ~0−i,(ℓ) a_0^-i,( ) denote the clean action in z~−i,(ℓ) z^-i,( ), and compute r~−i,k,k(ℓ)←ComputeWeight(θ−i,k,z~−i,k(ℓ),C),w~(ℓ)−i←Lr~−i,k,k(ℓ)∑l=1Lr~−i,k,k(ℓ),l∈[L]. r^-i,k,k^( )← ComputeWeight(θ^-i,k, z^-i,k^( ),C), w_( )^-i← L r^-i,k,k^( ) _l=1^L r^-i,k,k^( ), l∈[L]. 6: Set bIS←⌈ζBself⌉b_IS← ζ B_self if the current self-action spread is below κ times its initial value, and bIS←0b_IS← 0 otherwise; set θi,k,0←θi,kθ^i,k,0←θ^i,k. 7: for h=1,…,Hh=1,…,H do 8: Sample 0i,(bIS+1:Bself)∼i.i.d.πθi,k,zi,(1:bIS)∼self-action history.a_0^i,(b_IS+1:B_self) i.i.d. _θ^i,k, z^i,(1:b_IS) -action history. 9: For m∈[bIS]m∈[b_IS], let 0i,k(m)a_0^i,k^(m) be the clean action in zi,k(m)z^i,k^(m), with behavior iteration k(m)k^(m), and compute ri,k,k(m)←ComputeWeight(θi,k,zi,k(m),C),w(m)i←bISri,k,k(m)∑j=1bISri,k,k(m).r^i,k,k^(m)← ComputeWeight(θ^i,k,z^i,k^(m),C), w_(m)^i← b_ISr^i,k,k^(m) _j=1^b_ISr^i,k,k^(m). 10: For m∈[Bself]m∈[B_self], estimate u^i((m)i,πθ−i,k)←1L∑ℓ=1Lw~(ℓ)−iui((m)i,~(ℓ)−i,k(ℓ)). u^i (a_(m)^i; _θ^-i,k )← 1L _ =1^L w_( )^-iu^i (a_(m)^i, a_( )^-i,k^( ) ). 11: For m∈[Bself]m∈[B_self], sample t∼Unif1,…,Tt \1,…,T\ and ϵ∼(0,I)ε (0,I), and set t,(m)i←α¯t(m)i+1−α¯tϵ.a_t,(m)^i← α_t\,a_(m)^i+ 1- α_t\,ε. 12: Take one optimizer step from θi,k,h−1θ^i,k,h-1 on ℒi,k,h(θ)=1Bself[ ^i,k,h(θ)= 1B_self [ ∑m=1bISw(m)iexp(u^i((m)i,k(m),πθ−i,k)η)(1−α¯t)‖sθ(t,(m)i,k(m),t)+ϵ1−α¯t‖22 _m=1^b_ISw_(m)^i \! ( u^i(a_(m)^i,k^(m); _θ^-i,k)η )(1- α_t) \|s_θ(a_t,(m)^i,k^(m),t)+ ε 1- α_t \|_2^2 + + ∑m=bIS+1Bselfexp(u^i((m)i,k,πθ−i,k)η)(1−α¯t)‖sθ(t,(m)i,k,t)+ϵ1−α¯t‖22], _m=b_IS+1^B_self \! ( u^i(a_(m)^i,k; _θ^-i,k)η )(1- α_t) \|s_θ(a_t,(m)^i,k,t)+ ε 1- α_t \|_2^2 ], obtaining θi,k,hθ^i,k,h. 13: end for 14: Set θi,k+1←θi,k,Hθ^i,k+1←θ^i,k,H. 15: Draw R actions from πθi,k+1 _θ^i,k+1 and apply PrepareSample with behavior parameters θi,k+1θ^i,k+1; append the resulting entries to both buffers and retain the most recent NbufN_buf opponent entries. 16: end for 17: end for 18: return Πi,K=1K∑k=1Kπθi,k,i∈1,2 ^i,K= 1K _k=1^K _θ^i,k, i∈\1,2\. Appendix D Details about Experiment Setup D.1 Nash Gap Computation For a learned strategy profile π=(π1,π2)π=(π^1,π^2), we measure the total gain available from unilateral deviations. For player i, define ϵi(π)=maxai∈iui(ai,π−i)−Ui(πi,π−i),ε^i(π)= _a^i ^iu^i(a^i;π^-i)-U^i(π^i,π^-i), (9) and compute the Nash gap as G(π1,π2)=ϵ1(π)+ϵ2(π).G(π^1,π^2)=ε^1(π)+ε^2(π). (10) Because expected utility is linear in the deviating strategy, it is sufficient to optimize over pure best responses. In the two-player zero-sum setting, writing U for player 1’s payoff, this simplifies to G(π1,π2)=maxa1∈1U(a1,π2)−mina2∈2U(π1,a2).G(π^1,π^2)= _a^1 ^1U(a^1,π^2)- _a^2 ^2U(π^1,a^2). (11) For continuous action spaces, the best-response optimization is computed with a game-specific evaluation oracle. For the forest game, following Kamra et al. [2019], we use evolutionary search with a population of 50 candidates for 20 iterations. At each iteration, the top 10 candidates are retained and four Gaussian-perturbed mutations are generated from each survivor. Candidate utilities are estimated against 100 opponent samples during the search, and the resulting best response is reevaluated against 1,000 samples. When estimating the individual unilateral gains in (10), the utility of the learned profile is estimated from 2,000 sampled action pairs. For symmetric Colonel Blotto, we follow the discretized best-response procedure of Adam et al. [2021]. Final Nash gaps use a simplex grid with spacing 1/10241/1024 and an exact knapsack dynamic program, with 20,000 samples from each learned policy. For the radial-band variant, we approximate each best response from a pool of approximately 11,400 candidates, refine the best 64 candidates for 400 gradient steps, and estimate final payoffs using 8,000 policy samples per player. The same evaluation procedure is used for all methods within each game. D.2 Experimental Details of Cournot Game We consider a two-player multi-market Cournot competition game with d independent markets, where A>0A>0 is the demand intercept and b>0b>0 controls the slope of inverse demand. Player i∈1,2i∈\1,2\ chooses production quantities across d markets: i=([i]1,…,[i]d)∈[0,qmax]d.a^i=([a^i]_1,…,[a^i]_d)∈[0,q_ ]^d. In each market j, the inverse demand function is pj(j1,j2)=A−b(j1+j2),p_j(a^1_j,a^2_j)=A-b(a^1_j+a^2_j), and both firms have constant marginal cost c. Player i’s total utility is ui(i,−i)=∑j=1d(A−b(ji+j−i)−c)ji.u^i(a^i,a^-i)= _j=1^d (A-b(a^i_j+a^-i_j)-c )a^i_j. (12) We use A=10A=10, b=1b=1, c=1c=1, and qmax=(A−c)/b=9q_ =(A-c)/b=9, with d∈1,30d∈\1,30\ in the experiments. In the implementation, the policy produces a normalized action i∈[−1,1]dz^i∈[-1,1]^d, which is mapped to the feasible Cournot action by i=i+12qmax.a^i= z^i+12q_ . During optimization, we use the scaled utility ui/du^i/d so that its magnitude does not grow with the number of markets. Analytical equilibrium. The separability and quadratic structure of the Cournot game allow its Nash equilibrium to be characterized exactly. Proposition 3. The Cournot game in (12) has a unique Nash equilibrium (1,⋆,2,⋆)(a^1, ,a^2, ), and for every market j=1,…,dj=1,…,d, [1,⋆]j=[2,⋆]j=A−c3b.[a^1, ]_j=[a^2, ]_j= A-c3b. Under our parameters, [1,⋆]j=[2,⋆]j=3[a^1, ]_j=[a^2, ]_j=3 for every market. Proof. Consider an arbitrary mixed strategy π−iπ^-i of player −i-i, and let μj−i=−i∼π−i[[−i]j]μ^-i_j=E_a^-i π^-i[[a^-i]_j] denote its mean production in market j. Since the payoff is separable across markets and linear in the opponent’s production, the expected utility from a action ia^i is −i∼π−i[ui(i,−i)]=∑j=1d[(A−c−bμj−i)ji−b(ji)2].E_a^-i π^-i [u^i(a^i,a^-i) ]= _j=1^d [(A-c-bμ^-i_j)a^i_j-b(a^i_j)^2 ]. For b>0b>0, this expression is strictly concave in ia^i, so player i has a unique best response against any mixed opponent strategy. In each market, BRji(π−i)=A−c−bμj−i2b.BR^i_j(π^-i)= A-c-bμ^-i_j2b. Since 0≤μj−i≤qmax=A−cb,0≤μ^-i_j≤ q_ = A-cb, this quantity lies in [0,qmax][0,q_ ], so the action constraints do not alter the best response. A player can randomize at a Nash equilibrium only over best responses. Since the best response is unique, every Nash equilibrium must therefore be pure (all probability mass on a single action). For each market j, the equilibrium quantities satisfy 2b[1,⋆]j+b[2,⋆]j=A−c,b[1,⋆]j+2b[2,⋆]j=A−c.2b[a^1, ]_j+b[a^2, ]_j=A-c, b[a^1, ]_j+2b[a^2, ]_j=A-c. Subtracting the two equations gives [1,⋆]j=[2,⋆]j[a^1, ]_j=[a^2, ]_j, and substitution yields [1,⋆]j=[2,⋆]j=A−c3b.[a^1, ]_j=[a^2, ]_j= A-c3b. Because the markets are independent, this holds for every j=1,…,dj=1,…,d. With A=10A=10, b=1b=1, and c=1c=1, the equilibrium quantity is 33 in every market. ∎ D.3 Experimental Details of Forest Protection Game D.3.1 Environment Details We use the continuous forest security game introduced by Kamra et al. [2018] and subsequently adopted as the forest protection benchmark in DeepFP [Kamra et al., 2019]. Our environment and parameter settings follow the latter. The game is played between a defender D and an adversary A in a circular forest containing a fixed set of tree locations ⊂ℝ2T ^2. The adversary controls N adversaries, while the defender controls M stationary guards. Each adversary enters at the boundary of the forest, travels along a radial path toward the center, stops at a chosen location, and exits along the same path. Its stopping location therefore determines both its trajectory and its harvesting region. The adversary’s action is A=([A]1,…,[A]N)∈ℱN,a_A=([a_A]_1,…,[a_A]_N) ^N, where ℱF is the unit disk and [A]k[a_A]_k is the stopping location of the k-th adversary. The defender selects the guard locations D=([D]1,…,[D]M)∈ℱM.a_D=([a_D]_1,…,[a_D]_M) ^M. where [D]j[a_D]_j is the location of guard j. Locations are represented using normalized polar coordinates (r,θ)(r,θ), so the adversary and defender action dimensions are 2N2N and 2M2M, respectively. An adversary is captured when its path passes within guard radius RgR_g of at least one defender resource. Let Pk([A]k)P_k([a_A]_k) denote the radial path followed by the adversary k. Its capture indicator is Ck(D,A)=[minj∈1,…,Mminp∈Pk([A]k)∥p−[D]j∥2≤RD].C_k(a_D,a_A)=I [ _j∈\1,…,M\ _p∈ P_k([a_A]_k) p-[a_D]_j _2≤ R_D ]. An uncaptured adversary harvests trees lying within chopping radius RAR_A of its stopping location. Define Hk([A]k)=t∈:∥t−[A]k∥2≤RA.H_k([a_A]_k)= \t : t-[a_A]_k _2≤ R_A \. Let Nstolen(D,A)N_stolen(a_D,a_A) denote the number of trees successfully removed by the uncaptured adversaries, with overlapping harvest regions combined according to the game simulator. A captured adversary loses its harvest and incurs a penalty rpen>0r_pen>0. The adversary’s reward is uA(aD,aA)=Nstolen(aD,aA)−rpen∑k=1NCk(aD,aA),u_A(a_D,a_A)=N_stolen(a_D,a_A)-r_pen _k=1^NC_k(a_D,a_A), and the defender receives the negative reward, uD(aD,aA)=−uA(aD,aA)u_D(a_D,a_A)=-u_A(a_D,a_A). Following the DeepFP experimental setup [Kamra et al., 2019], we use RD=0.1,RA=0.04,rpen=4.0.R_D=0.1, R_A=0.04, r_pen=4.0. The reward landscape is highly nonconvex and contains large flat regions. In particular, moving a guard without changing which trajectories it intercepts does not change the defender’s reward. Heterogeneous tree densities also create spatially separated high-value regions, so effective mixed strategies may place probability mass on several distinct guard or adversary configurations. The game therefore tests both the expressiveness of the learned strategy distribution and its ability to improve when informative utility gradients are sparse. Forest instances. We use four forest layouts, denoted F1–F4 throughout the paper. F1 and F2 contain 4,000 and 6,000 trees, respectively, and are evaluated with M=N∈1,3M=N∈\1,3\. F3 and F4 are denser layouts containing 6,200 and 7,100 trees and are evaluated with M=N=5M=N=5. Thus, the six configurations reported in the main experiments are (F1,1),(F1,3),(F2,1),(F2,3),(F3,5),(F4,5).(F1,1),\;(F1,3),\;(F2,1),\;(F2,3),\;(F3,5),\;(F4,5). The interaction and payoff structure is held fixed across instances; only the spatial tree layout and number of resources change. This changes which attack locations are valuable and how many spatially separated regions must be represented by the equilibrium strategy. Scaling the action space. Each guard and adversary is represented by a polar coordinate (r,θ)(r,θ), so the defender and attacker action spaces have dimensions 2M2M and 2N2N, respectively. Latent actions in [−1,1][-1,1] are mapped to r∈[0,1]r∈[0,1] and θ∈[0,2π]θ∈[0,2π]. Consequently, the headline experiments increase each player’s action dimension from 2 at M=N=1M=N=1, to 6 at M=N=3M=N=3, and 10 at M=N=5M=N=5. Although resources of the same type are interchangeable in the payoff, we use the direct ordered representation and do not explicitly enforce permutation symmetry. D.3.2 Implementation Details Compared methods. We compare DiffNash with DeepFP [Kamra et al., 2019], DiffFP [Karthikeyan and Pant, 2025], and Neural OMD. DeepFP uses neural fictitious play with a learned attacker and geometric defender best response. DiffFP replaces the neural policy with a diffusion best response while retaining fictitious-play updates. Neural OMD uses the same OMD training loop as DiffNash but represents the policy with a squashed diagonal Gaussian, isolating the contribution of the diffusion policy. DiffNash combines a diffusion policy with OMD, opponent replay, and self-action importance sampling. Training details. All headline methods are trained for 40,000 gradient updates per player. DiffNash and Neural OMD use 2,000 outer OMD iterations with 20 inner updates per iteration. DiffFP uses 100 fictitious-play iterations with 400 best-response updates per iteration. Unless otherwise noted, we run five seeds. OMD methods use batches of 256 player actions and 128 opponent replay samples. DiffNash maintains a FIFO opponent replay buffer of size 1,024 and draws 1% of each player batch from its full self-action history. DiffNash and Neural OMD use Adam Kingma and Ba [2014] with learning rate 3×10−43× 10^-4, decayed to 3×10−53× 10^-5 during training. The diffusion policies use T=20T=20 denoising steps. All experiments are run on CPU using 8 cores and 16 GB of memory. Evaluation. We report defender exploitability, defined as the best attacker reward against the learned time-averaged defender, together with the two sided Nash gap. Best responses are approximated using evolutionary search on the reference simulator. The search uses a population of 50 candidates for 20 iterations, retaining the top 10 and generating four mutations per survivor. Each candidate is evaluated against 100 sampled opponent actions, followed by a final evaluation against 1,000 samples. Self-play utility for the Nash gap is estimated from 2,000 sampled action pairs. We evaluate only the final checkpoint and use the complete time-averaged strategy accumulated during training. Reported uncertainty is the 95% Student-t confidence interval across seeds. Hyperparameters. DiffNash uses a DACER-style denoiser with three 256-unit hidden layers and Mish activations. The diffusion-time embedding is a 16-dimensional sinusoidal encoding followed by a two-layer MLP. We use OMD step size η=1η=1, normalize utility-derived energies within each batch, clip their logits to [−3,3][-3,3], and renormalize the resulting weights to mean one. Opponent importance ratios are upper-clipped at 10. Self-action replay constitutes 1% of the training batch and is activated once the standard deviation of the current action samples falls below 10% of its initial value. D.4 Experimental Details of Continuous Colonel Blotto D.4.1 Environment Details We follow the continuous Colonel Blotto formulation of Adam et al. [2021]. Two players simultaneously allocate a unit budget across N battlefields. Both players have the action space 1=2===([]1,…,[]N)∈ℝ+n:∑j=1N[]j=1.A^1=A^2=A= \a=([a]_1,…,[a]_N) _+^n: _j=1^N[a]_j=1 \. Let 1,2∈a^1,a^2 denote the allocations selected by players 11 and 22, respectively. The symmetric case and radial band case of Table 2 differ in the utility function of the agent, which we detail below. Symmetric To construct the symmetric case of Table 2, we define the following battlefield contest function: ℓc(z)=−1,z≤−c,z/c,−c<z<c,1,z≥c, _c(z)= cases-1,&z≤-c,\\[2.84526pt] z/c,&-c<z<c,\\[2.84526pt] 1,&z≥ c, cases , where ωj _j is battlefield j’s positive weight and c>0c>0 controls the allocation advantage required to obtain the full value of a battlefield. Player 11’s utility is u1(1,2)=∑j=1nωjℓc(j1−j2),u^1(a^1,a^2)= _j=1^n _j\, _c(a^1_j-a^2_j), and player 22 receives u2(1,2)=−u1(1,2).u^2(a^1,a^2)=-u^1(a^1,a^2). Thus, player 11 receives ωj _j when it allocates at least c more than player 22 to battlefield j, receives −ωj- _j when it allocates at least c less, and receives a payoff that varies linearly with the allocation difference inside this margin. Our symmetric Blotto experiment uses the symmetric N-battlefield setting ωj=1for all j,c=116. _j=1\ for all j, c= 116. for N=6,10N=\6,10\. As c→0c→ 0, ℓc _c approaches the sign function used in the classical continuous Colonel Blotto game. The game is symmetric and zero-sum, so its equilibrium value is zero. Because the budget constraint couples all six coordinates, increasing the allocation to one battlefield reduces the allocation available to the others. Equilibrium play therefore requires coordinated allocations rather than independent decisions at individual battlefields. This makes the game a useful test of whether a policy class can represent a broad mixed strategy. Radial-band instances. To construct the radial band case of Table 2, we first specify precisely what the radial band is. Define the uniform allocation ¯ a which allocates 1N 1N budget to each battlefield, and define 1e_1 be the allocation which allocates all battles to the first battlefield. Now define: vrad=1−¯v_rad=e_1- a, center line γrad(τ)=¯+τvrad _rad(τ)= a+τ v_rad, and width ωrad(τ)=min0.06,0.20(1−τ) _rad(τ)= \0.06,0.20(1-τ)\. The radial interval is [τmin,τmax][ _ , _ ], with τmin=0.12,n=6,0.15,n=10, _ = cases0.12,&n=6,\\ 0.15,&n=10, cases and τmax=0.995 _ =0.995. γrad(τ) _rad(τ) and ωrad(τ) _rad(τ) together define the shape of the radial band. The next step we define how probability is distributed within the band. We put 60%60\% of the probability along the interior portion and 40%40\% closer to the vertex. To achieve that we first sample First sample τ∼Unif[τmin,0.80),with probability 0.6,Unif[0.80,τmax],with probability 0.4.τ casesUnif[ _ ,0.80),&with probability 0.6,\\ Unif[0.80, _ ],&with probability 0.4. cases We also add some perpendicular displacement to make the distribution has width. Formally, given τ, draw the displacement ξ uniformly from ξ∈ℝn:⊤ξ=0,vrad⊤ξ=0,∥ξ∥2≤wrad(τ),γrad(τ)+ξ∈ΔN−1. \ξ ^n:1 ξ=0, v_rad ξ=0, \|ξ\|_2≤ w_rad(τ), _rad(τ)+ξ∈ ^N-1 \. The first constraint make sure the resulting allocation does not exceed the budget, and the second make sure the displacement is orthogonal to the radial band. Define the target distribution μrad _rad as the law of γrad+ξ _rad+ξ. Before we specify the payoff which makes μrad _rad the equilibrium strategy, we first define the following notations. For any allocation, we first locate its clipped projection onto the centerline, measure its perpendicular distance, and finally penalize the portion that exceeds the allowed width. Define the projected radial coordinate τ^() τ(a) and penalty Vrad()V_rad(a) by τ^()=minτmax,maxτmin,⟨−¯,vrad⟩‖vrad‖22, τ(a)= \ _ , \ _ , - a,\,v_rad \|v_rad\|_2^2 \ \, Vrad()=−15[‖−γrad(τ^())‖2−wrad(τ^())]+.V_rad(a)=-15 [ \|a- _rad ( τ(a) ) \|_2-w_rad ( τ(a) ) ]_+. Here Vrad≤0V_rad≤ 0, and every target sample does not receive penalty. We also define the following Gaussian kernel K(⋅,⋅)K(·,·) which measure the similarity between allocations: K(a1,a2)=exp(−‖a1−a2‖222σK2).K(a^1,a^2)= (- \|a^1-a^2\|_2^22 _K^2 ). , with bandwidth σK=0.08 _K=0.08. We define a centered version of the Gaussian kernel where we subtract each allocation’s average similarity to the target radial shape μrad _rad, then add back the target’s average similarity to itself. Formally, for independent samples ,′∼μradb,b _rad, define K¯(1,2)=K(1,2)−K(1,)−bK(,2)+,′K(,′). K(a^1,a^2)=K(a^1,a^2)-E_bK(a^1,b)-E_bK(b,a^2)+E_b,b K(b,b ). Set player 11’s utilities to u1(1,2)=K¯rad(1,2)+Vrad(1)−Vrad(2)u^1(a^1,a^2)= K_rad(a^1,a^2)+V_rad(a^1)-V_rad(a^2) and set u2(1,2)=−u1(1,2)u^2(a^1,a^2)=-u^1(a^1,a^2). Proposition 4 (Radial Band is a Nash equilibrium). (μrad,μrad)( _rad, _rad) is the Nash equilibrium of the game we described above. Proof. Observe that for every 1,2∈ΔN−1a^1,a^2∈ ^N-1, ∼μrad[K¯(1,)]=0,∼μrad[K¯(,2)]=0,E_b _rad [ K(a^1,b) ]=0, _b _rad [ K(b,a^2) ]=0, by the definition of the centered kernel. Since μrad _rad is supported inside the corridor, Vrad()=0V_rad(b)=0 almost surely for ∼μradb _rad, while Vrad()≤0V_rad(a)≤ 0 for every feasible action. Let U1U^1 denote the expected utility induced by u1u^1. For any pure deviations 1a^1 and 2a^2, we therefore have U1(1,μrad)=Vrad(1)≤0,U1(μrad,2)=−Vrad(2)≥0.U^1(a^1, _rad)=V_rad(a^1)≤ 0, U^1( _rad,a^2)=-V_rad(a^2)≥ 0. Moreover, U1(μrad,μrad)=0.U^1( _rad, _rad)=0. Thus, U1(1,μrad)≤U1(μrad,μrad)≤U1(μrad,2)U^1(a^1, _rad)≤ U^1( _rad, _rad)≤ U^1( _rad,a^2) for all 1,2∈Δn−1a^1,a^2∈ ^n-1. These inequalities extend to mixed deviations by linearity of expected utility. Hence (μrad,μrad)( _rad, _rad) is an exact Nash equilibrium with value zero for both N=6N=6 and N=10N=10. ∎ In implementation, we replace μrad⋆ _rad by a fixed empirical distribution and compute all kernel-centering terms from the same bank. The same argument then applies exactly to the resulting finite game. Visualization. For visualization, six-dimensional allocations are projected onto a regular hexagon using barycentric coordinates. Let []1,…,[]N∈ℝN[a]_1,…,[a]_N ^N denote the hexagon vertices. An allocation a is plotted at p()=∑j=1N[]jej.p(a)= _j=1^N[a]_je_j. (13) Under this projection, the uniform allocation appears at the hexagon center. Allocating the entire budget to battlefield j would map directly to vertex eje_j. The same projection is used for both the standard Blotto and radial-band games, which allows their learned strategy distributions to be compared directly. D.4.2 Implementation Details Reference solutions. For the symmetric Colonel Blotto instances, we compute reference equilibria with the grid-based double-oracle (DO) solver proposed in Adam et al. [2021]. For both N=6N=6 and N=10N=10, we discretize allocations with grid spacing 1/2561/256 and run DO to tolerance 10−610^-6, using an exact knapsack dynamic program to compute best responses over the discretized action space. For N=6N=6, DO terminates after 2121 iterations with a 4242-action support and Nash gap 4.31×10−144.31× 10^-14; for N=10N=10, it terminates after 1212 iterations with a 5050-action support and gap 1.33×10−141.33× 10^-14. Both instances use c=1/16c=1/16. This solver relies on a game-specific best-response oracle: discretization makes exact best responses tractable for these Blotto instances, but such an oracle is generally unavailable in high-dimensional continuous-action games, where best-response optimization may itself be difficult and uniform discretization scales poorly with dimension. We therefore use DO only to construct reference equilibria for these structured symmetric instances, rather than as a general-purpose baseline. For the radial-band instances, the reference equilibrium is known by construction. We instantiate it using a fixed empirical distribution μ^n⋆ μ _n with K=1200K=1200 anchors, consisting of 720720 samples from the interior portion of the corridor and 480480 from the near-extreme portion. The anchor bank is sampled once for each game instance and then frozen, and all kernel-centering terms are computed from the same bank. Thus, μ^n⋆ μ _n is an exact equilibrium of the implemented finite-anchor game. We use rmin=0.12r_ =0.12 for N=6N=6 and rmin=0.15r_ =0.15 for N=10N=10, with rsplit=0.80r_split=0.80 and rmax=0.995r_ =0.995. The corresponding reference Nash gap is exactly zero. We additionally report the finite-sample evaluation floors, 0.0142130.014213 for N=6N=6 and 0.0247410.024741 for N=10N=10. Compared Methods. We compare DiffNash against DeepFP, DiffFP, and Neural OMD. For DeepFP, we use the original implementation provided by the authors; their codebase is not publicly released. For DiffFP, no implementation is available, so we reimplement the method from the paper, using diffusion policies within the fictitious-play update and its Q-guided action refinement procedure. We additionally introduce Neural OMD as a controlled baseline to separate the effect of the OMD update from the expressive diffusion used by DiffNash. Neural OMD uses the same OMD training structure and optimization budget as DiffNash, but replaces the diffusion policy with a squashed diagonal-Gaussian policy parameterized by a two-layer MLP with hidden dimensions (256,256)(256,256) and tanh activations. Because this policy admits a tractable density, the mirror-descent regularizer is implemented using the analytic KL divergence between consecutive policies, with coefficient 1.01.0. Neural OMD is trained on-policy using fresh samples at each outer iteration and doe not use a replay buffer for importance weighting. For a given seed, all methods are evaluated on the same game instance and frozen anchor bank. The target distribution, kernel centering terms, and payoff parameters are therefore identical across methods. Training details. All four experiments use five random seeds 00-44. DiffNash and Neural OMD each use 20002000 outer iterations with 2020 gradient updates per iteration, for a total of 40,00040,000 updates per player. Both use Adam with a learning rate annealed from 3×10−43× 10^-4 to 3×10−53× 10^-5, a policy batch size of 256256, and 128128 opponent samples. DiffNash parameterizes each policy with a 2020-step diffusion model with three hidden layers of width 256256 and a linear noise schedule scaled by 0.8. Its replay buffer if FIFO with capacity 10241024, with 3232 new samples added per outer iteration. Opponent samples are corrected using importance weights capped at 1010. For own-policy correction, 33 of the 256256 samples in each batch are importance weighted, corresponding to approximately 1.17%1.17\% of the batch. DiffFP uses a 2020-step diffusion model with hidden dimensions (256,256)(256,256) and a 3232-dimensional time embedding. We train it for 100100 fictitious-play iterations, with 400400 best-response updates per iteration, again giving 40,00040,000 gradient updates. It uses Adam [Kingma and Ba, 2014] with learning rate 3×10−43× 10^-4, gradient-norm clipping at 2.02.0, training batches of 256256, and rollout batches of 3232. Its replay buffer has capacity 200,000200,000, with a pool of 100100 historical policies, and the best-response policy is reinitialized at each fictitious-play iteration. DiffFP does not use importance sampling. DeepFP uses the authors’ TensorFlow 1 implementation, with a two-layer (128,128)(128,128) policy network followed by a softmax output. We use Adam with learning rate 10−310^-3 and train for 400,000400,000 games, applying a policy update every 1010 games. Evaluation We measure convergence using the two-player zero-sum Nash gap G(p,q)=maxxU(x,q)−minyU(p,y),G(p,q)= _xU(x,q)- _yU(p,y), which is the sum of the two players’ unilateral improvement incentives and is zero at equilibrium. For symmetric Blotto, we follow the double-oracle evaluation of Adam et al. [2021], computing best responses exactly on a discretized simplex with a knapsack dynamic program. Reference solutions use grid spacing 1/2561/256, while final Nash gaps use a finer 1/10241/1024 grid and 20,00020,000 policy samples per player. For the radial-band games, we approximate each best response using a pool of approximately 11,40011,400 candidates followed by 400400 gradient-refinement steps on the best 6464, and estimate payoffs using 8,0008,000 samples per player. We additionally report recovery of the known radial equilibrium, including near-extreme mass (0.400.40 at equilibrium) and active versus inactive radial mass. All metrics are aggregated over five seeds using the same evaluator for every method within each game. Appendix E Missing Proofs E.1 Proof of Proposition 1 Proposition 1 (GWFP and OMD Loss). By choosing appropriate gOMD(t)g_OMD(a_t) and gGWFP(t)g_GWFP(a_t) and substituting them into Equation 6, we can equivalently write their loss functions as ℒOMD(θ;t)=0∼πi,k[t∼qt|0(⋅∣0)[exp(1ηui(0;π−i,k))‖sθ(t,t)−∇tlogqt|0(t∣0)‖22].]L_OMD(θ;t)=E_a_0 π^i,k [E_a_t q_t|0(· _0) [ ( 1ηu^i(a_0;π^-i,k) ) \|s_θ(a_t,t)- _a_t q_t|0(a_t _0) \|_2^2 ]. ] (14) andℒGWFP(θ;t)=t∼ht,~0∼ϕ0|t[exp(1ηkui(~0;Π−i,k))‖sθ(t,t)−∇tlogϕ0|t(~0∣t)‖22],and _GWFP(θ;t)=E_ subarrayca_t h_t,\\ a_0 _0|t subarray [ ( 1η^ku^i( a_0; ^-i,k) ) \|s_θ(a_t,t)- _a_t _0|t( a_0 _t) \|_2^2 ], (15) where hth_t is a density which is positive almost anywhere and ϕ0|t(~0∣t)≔(~0,tα¯t,1−α¯tα¯tI). _0 t( a_0 _t) \! ( a_0; a_t α_t, 1- α_t α_tI ). For every t∈(0,T]t∈(0,T], the minimizers of Equations (14) and (15) over s are the score functions of πt,OMDi,k+1(at)π^i,k+1_t,OMD(a_t) and πt,GWFPi,k+1(at)π^i,k+1_t,GWFP(a_t), respectively. Proof. Fix player i, equilibrium-solving iteration k, and diffusion time t∈(0,T]t∈(0,T]. Recall that qt|0(t∣0)=(t,α¯t0,(1−α¯t)I).q_t|0(a_t _0)=N (a_t; α_ta_0,(1- α_t)I ). For either solver∈GWFP,OMDsolver∈\GWFP,OMD\, define πt,solveri,k+1(t)=∫πsolveri,k+1(0)qt|0(t∣0)d0.π^i,k+1_t,solver(a_t)= π^i,k+1_solver(a_0)q_t|0(a_t _0)\,da_0. We first recall the denoising score-matching identity. For any target density ρ0 _0 and its noised marginal ρt(t)=∫ρ0(0)qt|0(t∣0)d0, _t(a_t)= _0(a_0)q_t|0(a_t _0)\,da_0, differentiating under the integral gives ∇tlogρt(t) _a_t _t(a_t) =∫ρ0(0)qt|0(t∣0)∇tlogqt|0(t∣0)d0ρt(t) = _0(a_0)q_t|0(a_t _0) _a_t q_t|0(a_t _0)\,da_0 _t(a_t) =0∼ρ0|t(⋅∣t)[∇tlogqt|0(t∣0)]. =E_a_0 _0|t(· _t) [ _a_t q_t|0(a_t _0) ]. (16) For any nonnegative function wt(t)w_t(a_t) for which the expectations are finite, ∫wt(t)ρt(t)‖sθ(t,t)−∇tlogρt(t)‖22dt w_t(a_t) _t(a_t) \|s_θ(a_t,t)- _a_t _t(a_t) \|_2^2\,da_t =∬wt(t)ρ0(0)qt|0(t∣0)‖sθ(t,t)−∇tlogqt|0(t∣0)‖22d0dt+Ct, = w_t(a_t) _0(a_0)q_t|0(a_t _0) \|s_θ(a_t,t)- _a_t q_t|0(a_t _0) \|_2^2\,da_0\,da_t+C_t, (17) where CtC_t does not depend on θ. This follows directly from Equation (16) by expanding the squared norms. OMD. From Equation (4), the OMD update is πOMDi,k+1(0)=1ZOMDi,kπi,k(0)exp(1ηui(0,π−i,k)).π^i,k+1_OMD(a_0)= 1Z^i,k_OMDπ^i,k(a_0) ( 1ηu^i(a_0;π^-i,k) ). (18) In Equation (6), set pt=πt,OMDi,k+1p_t=π^i,k+1_t,OMD and choose gOMD(t)=ZOMDi,kπt,OMDi,k+1(t).g_OMD(a_t)=Z^i,k_OMDπ^i,k+1_t,OMD(a_t). (19) Substituting both ptp_t and gOMDg_OMD into Equation (6) gives ℒgOMD(θ;t)=ZOMDi,k∫πt,OMDi,k+1(t)‖sθ(t,t)−∇tlogπt,OMDi,k+1(t)‖22dt.L^g_OMD(θ;t)=Z_OMD^i,k _t,OMD^i,k+1(a_t) \|s_θ(a_t,t)- _a_t _t,OMD^i,k+1(a_t) \|_2^2\,da_t. Then apply Equation 17 with ρ0=πOMDi,k+1,ρt=πt,OMDi,k+1,wt(t)=ZOMDi,k. _0= _OMD^i,k+1, _t= _t,OMD^i,k+1, w_t(a_t)=Z_OMD^i,k. Then ℒgOMD(θ;t)=ZOMDi,k∬πOMDi,k+1(0)qt|0(t∣0)‖sθ(t,t)−∇tlogqt|0(t∣0)‖22d0dt+CtOMD,L^g_OMD(θ;t)=Z_OMD^i,k _OMD^i,k+1(a_0)q_t|0(a_t _0) \|s_θ(a_t,t)- _a_t q_t|0(a_t _0) \|_2^2\,da_0\,da_t+C_t^OMD, where CtOMDC_t^OMD does not depend on θ. Substituting the OMD update and cancelling ZOMDi,kZ_OMD^i,k yields ℒgOMD(θ,t) ^g_OMD(θ;t) =∬πi,k(0)qt|0(t∣0)exp(1ηui(0,π−i,k)) = π^i,k(a_0)q_t|0(a_t _0) ( 1ηu^i(a_0;π^-i,k) ) ×‖sθ(t,t)−∇tlogqt|0(t∣0)‖22d0dt+CtOMD × \|s_θ(a_t,t)- _a_t q_t|0(a_t _0) \|_2^2\,da_0\,da_t+C_t^OMD =0∼πi,k[t∼qt|0(⋅∣0)[exp(1ηui(0;π−i,k))‖sθ(t,t)−∇tlogqt|0(t∣0)‖22]]+CtOMD, =E_a_0 π^i,k [E_a_t q_t|0(· _0) [ ( 1ηu^i(a_0;π^-i,k) ) \|s_θ(a_t,t)- _a_t q_t|0(a_t _0) \|_2^2 ] ]+C_t^OMD, which is Equation (14) up to an additive constant independent of θ. GWFP. From Equation (5), the GWFP target is πGWFPi,k+1(0)=1ZGWFPi,kexp(1ηkui(0,Π−i,k)),π^i,k+1_GWFP(a_0)= 1Z^i,k_GWFP ( 1η^ku^i(a_0; ^-i,k) ), (20) where ZGWFPi,kZ^i,k_GWFP is the corresponding normalization constant. In Equation (6), set pt=πt,GWFPi,k+1p_t=π^i,k+1_t,GWFP and choose gGWFP(t)=α¯tdi/2ZGWFPi,kht(t)πt,GWFPi,k+1(t),g_GWFP(a_t)= α_t^d_i/2Z^i,k_GWFPh_t(a_t)π^i,k+1_t,GWFP(a_t), (21) where did^i is player i’s action dimension. Substituting both ptp_t and gGWFPg_GWFP into Equation (6) and applying Equation (17) gives ℒgGWFP(θ,t) ^g_GWFP(θ;t) =α¯tdi/2∬ht(t)exp(1ηkui(0,Π−i,k))qt|0(t∣0) = α_t^d_i/2 h_t(a_t) ( 1η^ku^i(a_0; ^-i,k) )q_t|0(a_t _0) ×‖sθ(t,t)−∇tlogqt|0(t∣0)‖22d0dt+CtGWFP. × \|s_θ(a_t,t)- _a_t q_t|0(a_t _0) \|_2^2\,da_0\,da_t+C_t^GWFP. (22) Now define ϕ0|t(0∣t)=(0,tα¯t,1−α¯tα¯tI). _0|t(a_0 _t)=N (a_0; a_t α_t, 1- α_t α_tI ). Comparing the Gaussian densities gives ϕ0|t(0∣t)=α¯tdi/2qt|0(t∣0), _0|t(a_0 _t)= α_t^d_i/2q_t|0(a_t _0), (23) and therefore ∇tlogϕ0|t(0∣t)=∇tlogqt|0(t∣0)=−t−α¯t01−α¯t. _a_t _0|t(a_0 _t)= _a_t q_t|0(a_t _0)=- a_t- α_ta_01- α_t. (24) Applying Equations (23) and (24) to Equation (22) yields ℒgGWFP(θ,t) ^g_GWFP(θ;t) =t∼ht~0∼ϕ0|t(⋅∣t)[exp(1ηkui(~0;Π−i,k)) =E_ subarrayca_t h_t\\ a_0 _0|t(· _t) subarray [ ( 1η^ku^i( a_0; ^-i,k) ) . ×‖sθ(t,t)−∇tlogϕ0|t(~0∣t)‖22]+CtGWFP, 76.82243pt .× \|s_θ(a_t,t)- _a_t _0|t( a_0 _t) \|_2^2 ]+C_t^GWFP, (25) which is the GWFP loss up to an additive constant independent of θ. ∎ Finally, Gaussian corruption implies πt,GWFPi,k+1(t)>0π^i,k+1_t,GWFP(a_t)>0 and πt,OMDi,k+1(t)>0π^i,k+1_t,OMD(a_t)>0 for t>0t>0. Because hth_t is strictly positive on its support, both gGWFPg_GWFP and gOMDg_OMD are positive almost everywhere. Equation (6) is therefore globally minimized precisely when sθ(t,t)=∇tlogπt,solveri,k+1(t)s_θ(a_t,t)= _a_t π^i,k+1_t,solver(a_t) almost everywhere. This proves the result for both solver∈GWFP,OMDsolver∈\GWFP,OMD\. E.2 Proof of Proposition 2 See 2 Proof. Fix 0a_0. By the ELBO decomposition in Ho et al. [2020], ELBOθ(0)= _θ(a_0)= q(1∣0)[logpθ(0∣1)]⏟Reconstruction term E_q(a_1 _0) [ p_θ(a_0 _1) ]_Reconstruction term −∑t=2Tq(t∣0)[DKL(q(t−1∣t,0)∥pθ(t−1∣t))]⏟Denoising KL terms - _t=2^TE_q(a_t _0) [D_KL (q(a_t-1 _t,a_0)\,\|\,p_θ(a_t-1 _t) ) ]_Denoising KL terms −DKL(q(T∣0)∥p(T))⏟Terminal prior KL term. - D_KL\! (q(a_T _0)\,\|\,p(a_T) )_Terminal prior KL term. (26) Under the forward process, t=α¯t0+1−α¯tϵ,ϵ∼(0,I).a_t= α_t\,a_0+ 1- α_t\,ε, ε (0,I). This implies ϵ=t−α¯t01−α¯t.ε= a_t- α_t\,a_0 1- α_t. (27) Proof Strategy We first express the reconstruction term and each denoising KL term as an expectation of D(θ,a0,t,ϵ)D(θ;a_0,t,ε), with additive constants independent of θ. We then subtract the two ELBOs: the shared terminal prior KL term and additive constants cancel, leaving a difference of expected D terms from which we construct an unbiased estimator. Denoising KL Terms Our goal here is to prove that DKL(q(t−1∣t,0)∥pθ(t−1∣t))D_KL (q(a_t-1 _t,a_0)\,\|\,p_θ(a_t-1 _t) ) is equivalent to D(θ,a0,t,ϵ)D(θ;a_0,t,ε) up to a constant that does not depend on θ. For t≥2t≥ 2, citing Equation 6 in Ho et al. [2020] we know that: q(t−1∣t,0)=(t−1,μ~t(t,0),β~tI),q(a_t-1 _t,a_0)=N\! (a_t-1; μ_t(a_t,a_0), β_tI ), (28) where β~t=1−α¯t−11−α¯tβt β_t= 1- α_t-11- α_t _t and μ~t(t,0)=α¯t−1βt1−α¯t0+αt(1−α¯t−1)1−α¯tt. μ_t(a_t,a_0)= α_t-1 _t1- α_ta_0+ _t(1- α_t-1)1- α_ta_t. Use Equation 27 and α¯t=αtα¯t−1 α_t= _t α_t-1, we can rewrite this mean μ~t(t,0) μ_t(a_t,a_0) in terms of ϵε: 1αt(t−βt1−α¯tϵ) 1 _t (a_t- _t 1- α_tε ) =1αt(t−βt1−α¯t(t−α¯t0)) = 1 _t (a_t- _t1- α_t (a_t- α_ta_0 ) ) =1−α¯t−βtαt(1−α¯t)t+βtα¯tαt(1−α¯t)0 = 1- α_t- _t _t(1- α_t)a_t+ _t α_t _t(1- α_t)a_0 =αt(1−α¯t−1)1−α¯tt+βtα¯t−11−α¯t0 = _t(1- α_t-1)1- α_ta_t+ _t α_t-11- α_ta_0 =μ~t(t,0). = μ_t(a_t,a_0). (29) To obtain the third equality above, we use the identity 1−α¯t−βt=αt(1−α¯t−1)1- α_t- _t= _t(1- α_t-1). Now we focus on the the reverse transition which has fixed covariance pθ(t−1∣t)=(t−1,μθ(t,t),σt2I)p_θ(a_t-1 _t)=N\! (a_t-1; _θ(a_t,t), _t^2I ) with score-parameterized mean μθ(t,t)=1αt(t+βtsθ(t,t)). _θ(a_t,t)= 1 _t (a_t+ _ts_θ(a_t,t) ). Subtracting the two means gives μ~t(t,0)−μθ(t,t) μ_t(a_t,a_0)- _θ(a_t,t) =1αt[t−βt1−α¯tϵ−t−βtsθ(t,t)] = 1 _t [a_t- _t 1- α_tε-a_t- _ts_θ(a_t,t) ] =−βtαt(sθ(t,t)+ϵ1−α¯t). =- _t _t (s_θ(a_t,t)+ ε 1- α_t ). (30) Consequently, ‖μ~t−μθ‖22=βt2αt‖sθ(t,t)+ϵ1−α¯t‖22. \| μ_t- _θ \|_2^2= _t^2 _t \|s_θ(a_t,t)+ ε 1- α_t \|_2^2. (31) Recall d is the dimension of ta_t. Now we can focus on the denoising KL terms. Note the KL between these two Gaussians is DKL(q(t−1∣t,0)∥pθ(t−1∣t))=12[dβ~tσt2−d+dlogσt2β~t+‖μ~t−μθ‖22σt2]. D_KL\! (q(a_t-1 _t,a_0)\,\|\,p_θ(a_t-1 _t) )= 12 [d β_t _t^2-d+d _t^2 β_t+ \| μ_t- _θ \|_2^2 _t^2 ]. Plug in the expression of mean difference in Equation 31 we obtain DKL(q∥pθ)=d2(β~tσt2−1+logσt2β~t)+βt22σt2αt⏟Ct‖sθ(t,t)+ϵ1−α¯t‖22=ct+D(θ;0,t,ϵ). D_KL(q\|p_θ)= d2 ( β_t _t^2-1+ _t^2 β_t )+ _t^22 _t^2 _t_C_t \|s_θ(a_t,t)+ ε 1- α_t \|_2^2=c_t+D(θ;a_0,t,ε). We denote Ct=βt22σt2αtC_t= _t^22 _t^2 _t, a constant that does not depend on θ. This finishes our goal that the denoising KL term is equivalent to D(θ,a0,t,ϵ)D(θ;a_0,t,ε). Reconstruction Term We next consider the reconstruction term at t=1t=1. Since α¯1=α1 α_1= _1 and 1−α¯1=β11- α_1= _1, the forward sample satisfies 1=α10+β1ϵa_1= _1a_0+ _1ε Therefore, we can rewrite 0=1−β1ϵα1.a_0= a_1- _1ε _1. Subtracting the reverse-model mean μθ(1,1) _θ(a_1,1) gives 0−μθ(1,1) _0- _θ(a_1,1) =1−β1ϵ−1−β1sθ(1,1)α1 = a_1- _1ε-a_1- _1s_θ(a_1,1) _1 =−β1α1(sθ(1,1)+ϵβ1). =- _1 _1 (s_θ(a_1,1)+ ε _1 ). (32) For the Gaussian decoder pθ(0∣1)=(0,μθ(1,1),σ12I)p_θ(a_0 _1)=N(a_0; _θ(a_1,1), _1^2I) with fixed σ12>0 _1^2>0, by plugging in 32, we obtain logpθ(0∣1) p_θ(a_0 _1) =−d2log(2πσ12)−‖0−μθ(1,1)‖222σ12 =- d2 (2π _1^2)- \|a_0- _θ(a_1,1)\|_2^22 _1^2 =−d2log(2πσ12)−β122σ12α1⏟C1‖sθ(1,1)+ϵ1−α¯1‖22 =- d2 (2π _1^2)- _1^22 _1^2 _1_C_1 \|s_θ(a_1,1)+ ε 1- α_1 \|_2^2 =−d2log(2πσ12)−D(θ,0,1,ϵ). =- d2 (2π _1^2)-D(θ;a_0,1,ε). (33) The coefficient C1C_1 has the same form as Ct=βt2/(2σt2αt)C_t= _t^2/(2 _t^2 _t) for t≥2t≥ 2. Combining the Reconstruction Term and the denoising KL term Because the two policy iterates share the same forward schedule, reverse-variance schedule, and terminal prior, the same const(0)const(a_0) applies to both. Consequently, the constant terms get cancelled and we obtain ELBOk(0)−ELBOk(ℓ)(0) ^k(a_0)-ELBO^k^( )(a_0) =∑t=1Tϵ[D(θk(ℓ),0,t,ϵ)−D(θk,0,t,ϵ)]. = _t=1^TE_ε [D(θ^k^( );a_0,t,ε)-D(θ^k;a_0,t,ε) ]. (34) Finally, because t∼Unif1,…,Tt \1,…,T\, t,ϵ[Δ^k,k(ℓ)] _t,ε[ ^k,k^( )] =Tt,ϵ[D(θk(ℓ),0,t,ϵ)−D(θk,0,t,ϵ)] =T\,E_t,ε [D(θ^k^( );a_0,t,ε)-D(θ^k;a_0,t,ε) ] =∑t=1Tϵ[D(θk(ℓ),0,t,ϵ)−D(θk,0,t,ϵ)] = _t=1^TE_ε [D(θ^k^( );a_0,t,ε)-D(θ^k;a_0,t,ε) ] =ELBOk(ℓ)(0)−ELBOk(0). =ELBO^k^( )(a_0)-ELBO^k(a_0). Thus, Δ^k,k(ℓ) ^k,k^( ) is an unbiased estimator of the ELBO difference. ∎ Appendix F Additional Experimental Results F.1 Additional Forest Game Visualizations Figure 7 provides the complete strategy visualization for all six forest configurations reported in Table 1. Although the payoff rule is unchanged, each tree layout induces a different spatial equilibrium. Figure 7: Learned strategies across all forest security games. Green points denote trees, blue points defender actions, and red points attacker actions. Rows vary both the forest geometry and the number of resources m=nm=n. DiffNash obtains the lowest mean Nash gap in all six configurations; Figure 4 highlights two representative multi-resource cases in the main text. F.2 10-dimensional Continuous Colonel Blotto Figure 8 shows the learned strategy distributions for the n=6n=6 and n=10n=10 Blotto experiments reported in Table 2. Figure 8: Continuous Colonel Blotto strategy distributions. Representative learned strategies projected into the polygonal action-space visualization. Columns show the oracle reference and each method; rows correspond to symmetric and radial-band Blotto for n=6n=6 and n=10n=10, respectively. Green denotes equilibrium samples and purple the learned strategy. F.3 General-Sum Forest Protection We also consider a general-sum variant of the F2 forest protection game found in Kamra et al. [2019]. We retain the original F2 geometry and interaction mechanics, but assign each tree separate ecological and economic values. This creates different objectives for the defender and poacher without changing the continuous action space. Let vtDv_t^D and vtPv_t^P denote the ecological and economic values of tree t, respectively. For each adversary logger l, let al=1a_l=1 indicate that it is captured by at least one guard and let st,ls_t,l denote its share of tree t when multiple adversary target the same tree. The payoffs are uD u^D =∑l[−(1−al)∑tst,lvtD+4al], = _l [-(1-a_l) _ts_t,lv_t^D+4a_l ], (35) uP u^P =∑l[(1−al)∑tst,lvtP−4al]. = _l [(1-a_l) _ts_t,lv_t^P-4a_l ]. (36) where ala_l indicates whether adversary l is captured and s(t,lCLOSEs_(t,l is its share of tree t. The original zero-sum game is recovered when vtD=vtP=1v_t^D=v_t^P=1. We construct the two value fields to favor different parts of the forest. For a tree t, let rtr_t denote its radial distance from the center and let ktk_t be the number of other trees within Euclidean distance 0.050.05. We define v~tD=exp(−rt/0.3)1+kt,v~tP=exp(−1−rt0.3)(1+kt). v_t^D= (-r_t/0.3)1+k_t, v_t^P= (- 1-r_t0.3 )(1+k_t). (37) and independently rescale and clip each field to [0.25,4][0.25,4] while preserving mean 1, so that the overall payoff scale is comparable to the original game. Thus, the defender places greater value on isolated interior trees, while the poacher focuses more on dense patches near the boundary. On F2, the two value fields have correlation −0.52-0.52. Figure 9 compares DeepFP, Neural OMD, DiffFP, and DiffNash for m=n∈1,3m=n∈\1,3\. DiffNash attains the lowest mean Nash gap, while DeepFP and DiffFP remain more exploitable. Neural OMD performs better than the FP-based methods, but exhibits higher variance than DiffNash. The visualized seed also shows qualitatively different spatial strategies across methods. Figure 9: General-sum Forest Game. Learned strategies on the general-sum F2 forest game for m=n∈1,3m=n∈\1,3\. Metrics are mean ± 95% confidence intervals over three seeds. F.4 Importance-sampling Forest ablation Figure 10: Importance-sampling ablation on F2. Removing own-policy importance sampling substantially degrades DiffNash, whereas opponent IS has little effect on convergence when own IS is omitted. We use F2, where m=n=3m=n=3, to isolate the roles of the two importance-sampling corrections in DiffNash. In particular, we separate the correction applied to replayed samples from the player’s own past policies (own IS) from the correction used when evaluating samples from past opponent policies (opponent IS). Figure 10 shows that the two corrections have different effects. Removing own IS substantially degrades performance, increasing the Nash gap from 300.7±126.4300.7± 126.4 for the full DiffNash method to 797.0±153.4797.0± 153.4 with opponent IS alone. In contrast, opponent IS has little effect on convergence when own IS is omitted: the Nash gap is 787.1±134.6787.1± 134.6 without either correction and 797.0±153.4797.0± 153.4 with opponent IS alone. This is consistent with the Cournot ablation in the main paper (Figure 6, where opponent IS likewise has little effect on convergence performance, but reduces score-network calls to roughly 66%66\% of those required without opponent IS. Together, these results suggest that own IS provides the primary optimization benefit by correcting the distribution shift induced by reusing the player’s historical diffusion samples, while opponent IS mainly improves computational efficiency. F.5 Forest Game Hyperparameter Sweep We fix other hyperparamters and change self action rate ζ in Figure 11. Figure 11 shows that lower self-action IS rates generally yield smaller mean Nash gaps. Because importance weights are clipped, including too many samples from past iterations may bias the OMD loss in Equation 7. Empirically, a 1%1\% self-action IS rate suffices to stabilize training. In Table 3, we study sensitivity to the own-action batch size, opponent batch size, and trigger threshold κ on F2 and F3 over five seeds. Trigger κ : The default κ=10%κ=10\% achieves a slightly lower Nash gap on F2 than the other trigger values; otherwise, no trigger value consistently outperforms the others, demonstrating that DiffNash is relatively robust to this hyperparameter. Own Buffer Size : Increasing the own-action batch size to 512512 degrades exploitability by 25.9%25.9\% and the Nash gap by 28.2%28.2\% relative to the default, suggesting that a moderate own-action batch size combined with self action importance sampling is more robust. In F2, the default batch size modestly outperforms a batch size of 128. Opponent Buffer Size : The default opponent batch size wins in all but one setting in terms of defender exploitability and Nash gap. Table 3: Hyperparameter sensitivity on F2 (m=n=3) and F3 (m=n=5). Each row changes one parameter from the default DiffNash configuration (own batch 256, opponent batch 128, ζ=1%ζ=1\%, κ=10%κ=10\%). We report defender exploitability ExploitDExploit_D and Nash gap G; lower is better. We add 95%95\% confidence interval to each value. F2 F3 Setting Value ExploitDExploit_D GapGap ExploitDExploit_D GapGap Default – 322.5322.5 ± 110.1 300.7300.7 ± 126.4 559.7559.7 ± 76.4 528.2528.2 ± 142.7 Own batch 128 388.9 ± 62.0 358.7 ± 88.2 553.5553.5 ± 250.5 510.5 ± 271.8 512 441.5 ± 53.0 431.5 ± 66.7 669.2 ± 87.8 631.4 ± 75.8 Opponent batch 64 442.0 ± 179.8 419.5 ± 200.5 679.8 ± 289.7 667.2 ± 292.1 256 397.9 ± 129.4 385.5 ± 141.3 576.7 ± 209.6 485.0 ± 246.6 Trigger κ 5%5\% 372.2 ± 39.9 361.9 ± 51.6 576.7 ± 94.7 531.8 ± 126.4 15%15\% 373.0 ± 93.5 366.2 ± 81.2 570.3 ± 189.1 502.0 ± 245.7 Figure 11: Self-action IS ratio sensitivity. Nash gap on F2 (m=n=3) as the self-action IS ratio varies. Lower is better. Appendix G Computing Infrastructure Experiments were run entirely on CPU hardware, using Intel Xeon Platinum 8268 nodes (48 cores, 192 GB RAM) and 32-vCPU cloud instances.