Paper deep dive
GARIP: A Running-Average Moving Reference for Last-Iterate Self-Play in Two-Player Zero-Sum Games
Can Savcı
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 95%
Last extracted: 7/9/2026, 6:49:31 AM
Summary
The paper introduces GARIP, a self-play method for two-player zero-sum games that anchors updates to a running average reference. It proves that the running average uniquely minimizes peak lag among causal convex averages, preventing policy collapse. GARIP achieves local last-iterate convergence with a constant anchor strength, matches R-NaD's peak performance, and demonstrates superior hyperparameter robustness across matrix games, board games, and deep reinforcement learning.
Entities (12)
Relation Signals (8)
GARIP → appliedto → Zero-Sum Game
confidence 99% · Self-play with naive gradient ascent cycles in two-player zero-sum games.
GARIP → usesreference → Running Average
confidence 97% · GARIP anchors to the running average to restore last-iterate convergence.
GARIP → solves → Last-Iterate Convergence
confidence 96% · We prove local last-iterate convergence at constant anchor strength.
Running Average → minimizes → Peak Lag
confidence 95% · Among causal convex averages of a fixed mean lag the running average uniquely minimizes that peak.
R-NaD → usesreference → Periodic Snapshot
confidence 94% · R-NaD uses a periodic snapshot reference, reset to the current policy every K steps.
GARIP → implementedin → JAX
confidence 93% · All experiments are pure JAX and reproducible.
GARIP → matchesperformanceof → R-NaD
confidence 93% · GARIP matches R-NaD's peak performance on matrix games and board games.
DeepNash → uses → R-NaD
confidence 92% · R-NaD a periodic snapshot (the engine of DeepNash).
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Self-play with naive gradient ascent cycles in two-player zero-sum games: the last iterate orbits the equilibrium. Modern methods restore last-iterate convergence by regularizing toward a reference policy -- MMD a fixed one (reaching only the regularized equilibrium), R-NaD a periodic snapshot (the engine of DeepNash). We study GARIP, which anchors to the running average, and isolate what the choice of reference controls. Our central result is a mechanism: collapse tracks the peak lag of the reference, and among causal convex averages of a fixed mean lag the running average (flat profile, peak $=$ mean) uniquely minimizes that peak, while a snapshot's sawtooth has peak $= 2\times$ mean (a one-line theorem). Two consequences follow. Convergence: we prove local last-iterate convergence at constant anchor strength -- the anchor scales the base map's rotation by $1-\beta$, crossing the stability boundary and turning a recurrent base into a contraction (global convergence is conjectured at small $\beta$; we characterize a large-$\beta$ consensus failure). Robustness: GARIP matches R-NaD's peak performance -- on matrix games, the Coin Game, and the board games Connect Four/Othello, both moving references are far more robust than fixed-magnet and magnet-free baselines -- but is the better hyperparameter default; we report it both ways: over the full grid collapse rates are statistically indistinguishable, yet at conventional parameterizations a matched-mean-lag setting collapses in 0/40 vs 10/40 seeds (a snapshot matches it only by knowing to shorten $K$). The boundaries: an anticipatory (negative-weight) reference does better still on the stale side, and the advantage appears only where naive self-play cycles (five deep self-play loops). All experiments are pure JAX and reproducible.
Tags
Links
- Source: https://arxiv.org/abs/2606.22688v1
- Canonical: https://arxiv.org/abs/2606.22688v1
Trouble viewing inline? Open PDF directly →
Full Text
63,496 characters extracted from source content.
Expand or collapse full text
GARIP: A Running-Average Moving Reference for Last-Iterate Self-Play in Two-Player Zero-Sum Games Can Savcı Independent Researcher İzmir, Türkiye savcisavcican@gmail.com Abstract Self-play with naive gradient ascent cycles in two-player zero-sum games: the last iterate orbits the equilibrium. Modern methods restore last-iterate convergence by regularizing toward a reference policy — MMD a fixed one (reaching only the regularized equilibrium), R-NaD a periodic snapshot (the engine of DeepNash). We study GARIP, which anchors to the running average, and isolate what the choice of reference controls. Our central result is a mechanism: collapse tracks the peak lag of the reference, and among causal convex averages of a fixed mean lag the running average (flat profile, peak == mean) uniquely minimizes that peak, while a snapshot’s sawtooth has peak =2×=2× mean (a one-line theorem). Two consequences follow. Convergence: we prove local last-iterate convergence at constant anchor strength — the anchor scales the base map’s rotation by 1−β1-β, crossing the stability boundary and turning a recurrent base into a contraction (global convergence is conjectured at small β; we characterize a large-β consensus failure). Robustness: GARIP matches R-NaD’s peak performance — on matrix games, the Coin Game, and the board games Connect Four/Othello both moving references are far more robust than fixed-magnet and magnet-free baselines — but is the better hyperparameter default; we report it both ways: over the full grid collapse rates are statistically indistinguishable, yet at conventional parameterizations a matched-mean-lag setting collapses in 0/400/40 vs 10/4010/40 seeds (a snapshot matches it only by knowing to shorten K). The boundaries: an anticipatory (negative-weight) reference does better still on the stale side, and the advantage appears only where naive self-play cycles (five deep self-play loops). All experiments are pure JAX and reproducible. 1 Introduction In a two-player zero-sum game with payoff matrix A, the row player maximizes and the column player minimizes V(x,y)=x⊤AyV(x,y)=x Ay. The unique solution concept is the Nash equilibrium, and the natural learning rule — each player ascending/descending its own payoff — is self-play gradient ascent (SGA). It is a classical fact that SGA cycles: the trajectory orbits the equilibrium on a closed conservative field and the last iterate never converges, even though the time average does [12]. For modern deep multi-agent RL, where one keeps the last network and not a time average over parameters, last-iterate behavior is what matters. The optimization literature restores last-iterate convergence with extragradient [6] and optimistic / past-gradient updates [7, 8, 9, 11], and more recently with Halpern anchoring toward a reference point for accelerated last-iterate rates [13, 14, 15]. GARIP’s update is an optimistic step composed with a Halpern anchor whose anchor moves (the running average) rather than being fixed. A now-standard fix is to add a regularizer that pulls the policy toward a reference (or “magnet”) policy, breaking the conservative rotation with a contractive potential. The methods differ in which reference they use and how it moves: • MMD [2]: a fixed reference (e.g. uniform). Converges to the regularized (QRE) equilibrium; reaching true Nash needs the regularization annealed to zero. • R-NaD [1]: a periodic snapshot reference, reset to the current policy every K steps. Reaches Nash without annealing and powers DeepNash on Stratego. • NFSP / fictitious play [3]: best-respond to the average opponent; no self-anchor. We examine a fourth point in this design space: anchor to the policy’s own running average. We call the resulting method GARIP (Generative Adversarial Reciprocal Iterative Play); it grew out of a cycle-consistency reading of zero-sum play (Section 2). The running average is a moving reference, like R-NaD’s snapshot, so it too reaches Nash without annealing — but it is updated continuously rather than in discrete jumps. The contribution, made precise. A moving reference that does not anneal is R-NaD’s idea, and GARIP matches R-NaD’s peak performance — we claim no more on that axis: at each method’s tuned best the two tie across matrix games and deep RL. GARIP’s advantage is hyperparameter robustness, and it is structural, not a tuning accident. Held for K steps, R-NaD’s snapshot has a sawtooth lag whose peak (K) is twice its mean (K/2K/2); collapse is driven by that peak. A running average has a flat lag profile, and we prove (Prop. 1) that a flat profile uniquely minimizes the peak lag among causal averages at fixed mean lag — so the running average is the collapse-optimal causal reference shape, and any sawtooth is strictly dominated. Since R-NaD is parameterized by reset frequency (a mean-like quantity), conventional K under-counts its collapse-driving peak by 2×2×: ρ=10−2ρ\!=\!10^-2 and K=200K\!=\!200 share a mean lag yet collapse 0%0\% versus 25%25\% of seeds (Sec. 3). On the convergence axis we prove a local last-iterate theorem (Prop. 3): the anchor scales the base map’s rotation by 1−β1-β, crossing the stability boundary so a recurrent base becomes a contraction — the tabular form of the same anti-cycling mechanism; the global version is conjectured at small β, with a characterized large-β consensus failure. The thesis: A running-average reference matches R-NaD’s peak performance and is the better default for non-annealed self-play: under an empirical collapse-tracks-peak law it is the peak-minimizing reference among causal convex averages, so its conventional rate is collapse-free where the snapshot’s conventional reset period is not — which the snapshot recovers only by knowing to shorten K. The boundaries: the aggregate gap is a tie, an anticipatory (negative-weight) reference does better on the stale side, and the edge appears only where naive self-play cycles. 2 The GARIP method Cycle-consistency view. A zero-sum game is the adversarial half of a CycleGAN: the two players are opposed “generators” G,FG,F given by smoothed best-response maps, G(x)=softmax(−(x⊤A)/τ),F(y)=softmax((Ay)/τ),G(x)=softmax\! (\!-(x A)/τ ), F(y)=softmax\! ((Ay)/τ ), and the equilibrium is the fixed point of the round trip F∘GF\! \!G. “Be a best response to your average” is exactly cycle-consistency F(G(x¯))≈x¯F(G( x))≈ x. GARIP enforces it by anchoring each update toward the running average, which is the self-consistent reference. We are candid that this is motivation, not mechanism. The cycle residual rcycr_cyc (Alg. 1, line 5) is computed only as a convergence diagnostic and never enters the update — mechanically, GARIP is optimistic mirror ascent [9] composed with a moving Halpern anchor [13, 14], and it does not optimize a cycle-consistency loss. The CycleGAN lens explains why the running average is the right anchor (it is the F∘GF\! \!G fixed point); the name is a historical artifact of the method’s lineage and not a claim that cycle-consistency is load-bearing. Matrix-game update. Maintain iterates x,yx,y and running averages x¯,y¯ x, y. Each step takes an optimistic entropic ascent on the game value, then a constant-strength Halpern step toward the moving anchor (Algorithm 1). Algorithm 1 GARIP (matrix-game form, row player; column y symmetric) 1:Input: payoff A, step η, anchor weight β, temperature τ 2:Init: x←x← uniform, x¯←x x← x, gprev←0g^prev← 0 3:for t=1,2,…,Tt=1,2,…,T do 4: ⊳ CycleGAN generators (smoothed best responses): 5: G(x)←softmax(−(x⊤A)/τ)G(x) \! (\!-(x A)/τ ) ⊳ opponent’s BR to x 6: x^←F(G(x))=softmax((AG(x))/τ) x← F(G(x))=softmax\! ((A\,G(x))/τ ) ⊳ cycle round-trip F∘GF\! \!G 7: rcyc←∥x^−x∥r_cyc← x-x ⊳ rcyc=0⇔r_cyc\!=\!0 equilibrium 8: ⊳ Optimistic ascent, then anchor to the self-consistent reference: 9: gx←Ayg_x← Ay ⊳ game gradient (row maximizes) 10: x1/2←softmax(logx+η(2gx−gprev))x_1/2 \! ( x+η(2g_x-g^prev) ) ⊳ optimistic step 11: x←(1−β)x1/2+βx¯x←(1-β)\,x_1/2+β\, x ⊳ Halpern step, moving anchor 12: x¯←x¯+1t+1(x−x¯) x← x+ 1t+1(x- x) ⊳ x¯ x: fixed point of F∘GF\! \!G 13: gprev←gxg^prev← g_x 14:end for 15:return x (and x¯ x) (y symmetric, with descent.) Crucially β is constant — there is no annealing. Setting β=0β=0 recovers optimistic mirror ascent; replacing x¯ x by a periodic snapshot recovers R-NaD; replacing it by a fixed distribution recovers MMD. This makes the four methods a clean ablation over which reference, moving how. Deep-RL form. In function-approximation settings the same idea is PPO self-play with a λKL(πθ∥πmag)λ\,KL( _θ\,\|\, _mag) term, where the magnet πmag _mag is the running-average (Polyak) policy and the opponent is the average opponent. MMD uses a fixed magnet; R-NaD uses a periodic-snapshot magnet reset every K updates. The five methods span the opponent × magnet grid; note that GARIP (average opponent, moving magnet) and R-NaD (current opponent, snapshot magnet) differ on both axes, so their head-to-head gap is not a clean single-variable isolation of the reference. We therefore fill the full 2×22× 2 grid at the standard config (λ=0.5λ\!=\!0.5, K=200K\!=\!200, ρ=10−2ρ\!=\!10^-2, 10 seeds; exploit return, lower is more robust): GARIP (avg+moving) −11.3-11.3, R-NaD (current+snapshot) −11.1-11.1, average+snapshot −12.3-12.3, current+moving −11.9-11.9, all with 0%0\% collapse and std ≈1.5≈\!1.5. Every single-axis effect (changing the magnet, or changing the opponent) is below 1.31.3 — smaller than the seed noise. So at default settings neither the magnet nor the opponent axis matters: the GARIP–R-NaD tie is not driven by either variable. The reference choice only becomes decisive in the staleness regime (Sec. 5), which is the entire content of the robustness claim. 3 Staleness: the failure mode of every moving reference Two averagers. GARIP realizes the running average two ways, and the distinction is load-bearing. In the matrix games (Alg. 1, line 10) it is the cumulative mean, weight 1/(t+1)1/(t+1) on the newest iterate; in the deep-RL setting it is a fixed-rate exponential moving average (Polyak), θ¯←(1−ρ)θ¯+ρθ θ←(1-ρ) θ+ρ\,θ. These have different lag profiles, and the D/ρD/ρ bound below governs only the EMA. One mechanism, three references. Let D be the per-step policy drift and let a reference lag the current policy by ℓ steps. Under KL weight λ it exerts a restoring force ∼λℓD \!λ\, \,D toward an outdated target; once this overwhelms the game gradient the policy is dragged to the stale reference and collapses to an exploitable policy. The references differ only in ℓ : • periodic snapshot (R-NaD): held fixed for K steps, ℓ=Θ(K) = (K), stale force ∼λK \!λ K; • fixed-rate EMA (deep-RL GARIP): ℓ=Θ(1/ρ) = (1/ρ), stale force ∼λ/ρ \!λ/ρ; • cumulative mean (matrix GARIP): ℓ=Θ(t) = (t) grows without bound, but D vanishes as the iterates approach equilibrium, so the product stays bounded and the mean catches up — which is why the matrix form has no collapse region. Consequently the EMA used in deep RL has its own staleness axis. A slow average (ρ→0ρ\!→\!0) at non-trivial λ is exactly as stale as a long snapshot — the correspondence λK↔λ/ρλ K λ/ρ is symmetric. So a running average does not “cannot go stale”: our own sweep (Sec. 5) finds GARIP collapses at small ρ (ρ≈10−3ρ\!≈\!10^-3) for λ≥0.5λ\!≥\!0.5, just as R-NaD collapses at large K. The corrected prediction. This gives a sharper, falsifiable claim than “GARIP never collapses”: each method degrades monotonically in its own staleness product (λKλ K for R-NaD, λ/ρλ/ρ for EMA-GARIP), and the practical question is which method’s commonly used hyperparameter range sits farther from its collapse boundary. R-NaD’s default reset period is O(102–103)O(10^2--10^3), inside the regime our sweep finds dangerous; GARIP’s default ρ=10−2ρ=10^-2 is an order of magnitude from its boundary ρ≈10−3ρ\!≈\!10^-3. Section 5 measures both boundaries on a matched grid (GARIP’s ρ swept down to 1.25×10−31.25\!×\!10^-3, symmetric to R-NaD’s K=800K\!=\!800) and reports the collapse rates with Wilson confidence intervals. The prediction holds: GARIP’s collapse boundary is at ρ≈0.005ρ\!≈\!0.005 (collapse-free for ρ≥0.01ρ\!≥\!0.01), R-NaD’s at K≈100K\!≈\!100–200200 (collapsing for the larger K used at scale) — symmetric mechanisms, but GARIP’s boundary sits past its default rate and R-NaD’s inside common reset periods (Sec. 5). A fuller treatment is in the supplementary staleness_analysis.md. Why it is structural: the running average is the optimal reference shape. Model a moving reference as a convex combination of past iterates, rt=∑s≤twt,sθsr_t= _s≤ tw_t,s _s (wt,s≥0w_t,s\!≥\!0, ∑swt,s=1 _sw_t,s\!=\!1). Under local drift ∥θt+1−θt∥≈D _t+1- _t \!≈\!D, its lag is ℓt=∥θt−rt∥/D=∑swt,s(t−s) _t= _t-r_t /D= _sw_t,s(t-s), the weighted mean age, and it exerts a stale force ∼λℓtD \!λ\, _t\,D. The load-bearing physics is empirical: collapse tracks the peak force — the GARIP and R-NaD peak-lag curves overlay (Fig. 1), so a flat reference and a snapshot peaking at the same lag collapse identically and the reset discontinuity contributes nothing; only the peak magnitude matters. Granting that empirical law, which reference shape minimizes the peak is a one-line theorem. Proposition 1 (The running average minimizes peak lag among causal averages). Among all causal convex references rt=∑s≤twt,sθsr_t= _s≤ tw_t,s _s (wt,s≥0,∑swt,s=1w_t,s\!≥\!0,\ _sw_t,s\!=\!1) with a given time-averaged lag ℓ¯ , the peak lag obeys ℓmax≥ℓ¯ _ ≥ , with equality iff the lag profile ℓt\ _t\ is constant in t. Hence, at comparable per-step drift D (so the collapse-safe weight scales as λ⋆∝1/(ℓmaxD)λ \! \!1/( _ D)), the constant-lag profile is the unique safest shape at matched mean lag. The running average realizes it — the EMA’s lag is stationary at (1−ρ)/ρ≈1/ρ(1-ρ)/ρ\!≈\!1/ρ — whereas the periodic snapshot is a sawtooth with ℓmax=2ℓ¯ _ =2 , strictly dominated by a factor 22. (Proof in App. A.) Remark 1 (Scope: non-anticipatory averages). Optimality is within causal convex averages. Extrapolating references (negative/Anderson-style weights) admit ℓ<0 \!<\!0 and can lower the peak force further — and indeed an anticipatory double-EMA magnet measurably widens the safe basin on the stale side (App. C), confirming the running average is optimal only in the causal convex class (the double EMA is still causal — only past iterates — but not a convex combination). It is not free: the same anticipation overshoots at fast averaging rates, so it complements rather than dominates GARIP. So, granting the empirical peak-force law, the running average is the peak-minimizing reference shape among causal averages, and a sawtooth’s peak is twice its mean. Whether this matters depends on parameterization. R-NaD is parameterized by reset frequency K, a mean-like quantity, so a K chosen to feel “moderately stale” carries a collapse-driving peak twice as large; the EMA’s peak equals its mean. At matched peak lag the two tie — one could set K≈100K\!≈\!100 and recover safety — so the gap is a property of the conventional parameterization, not of the methods in the abstract. Concretely, at the standard ρ=10−2ρ\!=\!10^-2 and K=200K\!=\!200 (same mean lag 100100), GARIP collapses in 0/400/40 seeds (including the stress λ=2λ\!=\!2) versus R-NaD’s 10/4010/40. The lone outlier in Fig. 1, an extreme ρ=1.25×10−3ρ\!=\!1.25\!×\!10^-3, is a never-used EMA so slow its permanent lag exceeds the snapshot’s transient peak. Figure 1: Collapse rate vs. effective reference lag (deep-RL sweep, 10 seeds). The EMA’s lag is flat (1/ρ1/ρ); the snapshot’s is a sawtooth (peak K, mean K/2K/2). Each point averages the same four λ, so the overlay is not a λ artifact. The GARIP and R-NaD peak-lag curves overlay at the matched points (ℓ=100,200,400 \!=\!100,200,400) — collapse is peak-driven, and the reset discontinuity adds nothing — so at matched mean lag R-NaD (dashed) collapses earlier. ρ=10−2ρ\!=\!10^-2 and K=200K\!=\!200 share mean lag 100100 yet collapse 0%0\% vs 25%25\%. 4 Tabular dynamics: fixed points and the unbiased anchor The robustness theorem (Prop. 1) is the load-bearing theory; this section adds the matrix-game dynamics that complete the picture — why GARIP reaches Nash without annealing. On the convergence axis it gives a local last-iterate theorem (Prop. 3: the anchor crosses the stability boundary) and leaves only the global statement conjectured; we are explicit about that boundary. On a matrix game A the row plays x∈Δmx∈ _m against y∈Δny∈ _n for value x⊤Ayx \!Ay. GARIP’s row update (column symmetric), with step η and constant anchor β∈(0,1)β∈(0,1), is gt g_t =Ayt, =Ay_t, xt+12 x_t+ 12 =softmax(logxt+η(2gt−gt−1)), =softmax\! ( x_t+η(2g_t-g_t-1) ), xt+1 x_t+1 =(1−β)xt+12+βx¯t, =(1-β)\,x_t+ 12+β\, x_t, x¯t+1 x_t+1 =x¯t+1t+2(xt+1−x¯t). = x_t+ 1t+2 (x_t+1- x_t ). The half-step is optimistic multiplicative weights (OMWU); the anchor is a Halpern step toward the running average x¯t x_t. Two facts are exact, and together they explain why GARIP reaches Nash without annealing. Lemma 1 (The anchor contracts deviation from the average). For every t, xt+1−x¯t=(1−β)(xt+12−x¯t)\;x_t+1- x_t=(1-β)\,(x_t+ 12- x_t), hence ∥xt+1−x¯t∥=(1−β)∥xt+12−x¯t∥ x_t+1- x_t =(1-β)\, x_t+ 12- x_t . The anchor shrinks the gap to the instantaneous anchor x¯t x_t by exactly 1−β1-β each step (the running average itself then moves by x¯t+1−x¯t=O(1/t) x_t+1- x_t=O(1/t), which the identity does not net out — contraction is toward the current anchor, not a fixed point). This is the oscillation damping of Sec. 3 made exact, and it carries no fixed-point bias: Proposition 2 (Fixed points are Nash; the anchor is unbiased). (i) Every Nash equilibrium of A is a fixed point of GARIP, with the anchor inactive there. (i) Conversely, every interior fixed point is a Nash equilibrium. (i) Unlike MMD, whose fixed magnet m exerts a nonzero pull at Nash — making its fixed point the magnet-regularized (quantal-response) equilibrium, O(η)O(η) off Nash — GARIP’s self-referential anchor vanishes at any fixed point, so no annealing is needed to remove a magnet bias. (Proof in App. A.) Remark 2 (The β→0β\!→\!0 core). At β=0β=0 GARIP is exactly OMWU, which converges to Nash in the last iterate at a linear rate in bilinear games [10, 9]. The anchor is the only addition. We can now prove last-iterate convergence locally — and, more importantly, show analytically that the anchor is what crosses the stability boundary. Proposition 3 (Local last-iterate convergence; the anchor crosses the stability boundary). Let x⋆x be an interior Nash of an m×nm× n zero-sum game. The base half-step linearizes there into independent modes; let M=μeiωM=μ\,e^iω be a mode (an eigenvalue of the base operator). For the un-anchored base (β=0β\!=\!0), non-optimistic MWU is a rotation with μ≥1μ≥ 1 — its last iterate recurs (orbits/spirals out) and does not converge — while optimism gives μ<1μ<1. Because the anchor and the running average act identically on every coordinate (scalar operators), they commute with the base operator; in the basis diagonalizing it the full linearized GARIP map block-diagonalizes, per mode, into J(M)=((1−β)Mβρ(1−β)M1−ρ(1−β)),detJ=(1−ρ)(1−β)M, gatheredJ(M)= pmatrix(1-β)M&β\\[1.0pt] ρ(1-β)M&1-ρ(1-β) pmatrix,\\[3.0pt] J=(1-ρ)(1-β)M, gathered so ρ(full)=maxkρ(J(Mk))ρ(full)= _kρ (J(M_k) ) over the modes MkM_k. Each block has a fast policy eigenvalue of modulus (1−β)μ(1-β)μ and a slow average eigenvalue near 11; for β∈(0,1)β∈(0,1), ρ∈(0,1]ρ∈(0,1] and small η both lie strictly inside the unit disk, so x⋆x is locally asymptotically stable in the last iterate — with no annealing. Crucially the anchor scales each base mode by exactly 1−β1-β (Lemma 1): it turns a recurrent rotation (μ=1⇒(1−β)μ<1μ\!=\!1 (1-β)μ<1) into a contraction, so it induces last-iterate convergence for the non-optimistic base — the tabular form of the deep-RL collapse-prevention mechanism (§3). (Proof in App. A: per-mode block-diagonalization — exact for the one-step base on any m×nm× n game, confirmed numerically — plus the first-order slow-mode expansion, whose coefficient is complex because M is a rotation.) What stays open is the global statement. The right potential-level tool is exact: Lemma 2 (Potential-level anchor contraction). Write Φt=KL(x⋆∥xt) _t=KL(x \|x_t), Φt+12=KL(x⋆∥xt+12) _t+ 12=KL(x \|x_t+ 12), Ψt=KL(x⋆∥x¯t) _t=KL(x \| x_t). Because q↦KL(x⋆∥q)q (x \|q) is convex, Φt+1≤(1−β)Φt+12+βΨt,Ψt≤1t∑s<tΦs. _t+1≤(1-β)\, _t+ 12+β\, _t, _t≤ 1t _s<t _s. (Proof in App. A, by convexity of KLKL.) Conjecture 1 (Global last iterate, at small β — with a failure mode at large β). For small enough β and η, GARIP’s last iterate converges to Nash from any interior start. Reduction (two open steps collapse to one). If the OMWU half-step is potential-non-increasing, Φt+12≤Φt−Dt _t+ 12≤ _t-D_t (Dt≥0D_t≥ 0; [10]), Lemma 2 gives Φt+1≤(1−β)Φt+βΨt−(1−β)Dt _t+1≤(1-β) _t+β _t-(1-β)D_t, so since 1−β<11-β<1, Ψt=KL(x⋆∥x¯t)→0⟹Φt→0: _t=KL(x \| x_t)→ 0\ \ _t→ 0: global last iterate follows from a single estimate, x¯t→ x_t→ Nash. But that estimate is exactly where the constant anchor can fail, and we found it does. In long-horizon sweeps (many random and near-boundary starts; repo script), at the default small β the last iterate descends to Nash on every game we tried — but slowly on large games (a Cesàro-limited ∼t−1/3 t^-1/3 rate, not acceleration). The premature-consensus regime. At larger β on larger interior-Nash games the anchor instead drives xtx_t and x¯t x_t into premature consensus (‖xt−x¯t‖→0\|x_t- x_t\|→ 0 with exploitability frozen at ∼10−1 10^-1, so x¯t↛ x_t → Nash), and the iterate stalls. The boundary is orderly — the critical anchor β⋆(d)β (d) shrinks with game size, and the default β=0.02β\!=\!0.02 sits below it through d≈10d\!≈\!10 (App. D). Pure OMWU (β=0β\!=\!0) reaches Nash to ∼10−7 10^-7 on these games, so on an already-convergent base a strong anchor over-regularizes — the tabular face of the Animal-Shogi lesson (§5.4). This is a basin-of-attraction failure, not an undercut of no-annealing: Prop. 2 (fixed points are Nash, anchor unbiased) is about where fixed points sit and is untouched; premature consensus is about which basin a large anchor steers into. The recommended deep-RL point is unaffected — there collapse is reference staleness, not anchor strength, and the anchor is load-bearing. A global proof for small β and a sharp β⋆(d)β (d) law are open. The collapse law has a derivable discrete (not continuous) origin — a linear one-mode map overshoots through the peak-lag state while the continuous-time delay only damps; we defer the derivation to App. B. 5 Experiments All code is pure JAX; matrix and poker exploitability are exact (LP-free NashConv); the deep-RL proxy freezes the policy and trains a fresh best-response PPO agent. Unless noted, results are over 10 seeds. 5.1 Matrix games: last-iterate convergence Table 1: Matrix games: last-iterate exploitability (10 seeds, 5000 steps, no annealing). Lower is better; bold marks the best method on the non-trivial random games. GARIP and R-NaD both reach near-Nash; MMD plateaus at the QRE; SGA never converges. game GARIP (ours) R-NaD MMD Optimistic MD Fictitious play Self-play GA rps 0.0003 0.0012 0.0000 0.0004 0.0247 1.999 matching pennies 0.0001 0.0000 0.0000 0.0000 0.0200 1.998 random 10×1010× 10 0.0028 0.0944 0.1207 0.0634 0.0283 2.522 random 12×612× 6 0.0052 0.0617 0.1400 0.0250 0.0169 2.101 Table 1 shows that at each method’s default configuration GARIP reaches near-Nash on every game, R-NaD’s default trails on the random games (and only matches GARIP once (α,K)(α,K) is tuned), and MMD plateaus at the regularized equilibrium. The exploitability curves and the RPS simplex trajectory (App. E) visualize the mechanism — SGA spirals outward forever while GARIP walks straight into Nash and stays. 5.2 Poker: Kuhn and Leduc On two imperfect-information benchmarks with exact exploitability — neural Kuhn (MLP through the differentiable tree) and tabular Leduc (288 infosets, exact CFR/BR) — GARIP is the best regularized method at default tuning: Kuhn 0.0760.076 vs R-NaD 0.300.30 and MMD 0.540.54 (approaching exact-BR fictitious play, 0.0430.043), and Leduc 0.0740.074 vs R-NaD 0.230.23 and MMD 0.550.55, below CFR’s 0.1160.116 at a matched 30003000-iteration budget (a reference point, not a converged baseline). These are default-vs-default configs — we did not separately tune R-NaD’s (α,K)(α,K) on poker, and a tuned R-NaD would likely close much of the gap, exactly as on the matrix games — so this is consistent with the tie-at-each-tuned-best finding, not a tuned-best poker win. Full tables and curves: App. E. 5.3 Deep-RL self-play: Coin Game We leave exact-solver territory with a deep-RL self-play loop on the Coin Game, a canonical 2-player testbed vendored from JaxMARL (dependencies stripped to avoid version coupling) and run zero-sum. A shared PPO actor-critic plays both sides; the proxy exploitability freezes the policy and trains a fresh best-response agent (lower / more negative is more robust). Table 2 shows GARIP (−11.29-11.29) and R-NaD (−11.07-11.07) tie and are far more robust than the fixed magnet (MMD, +7.83+7.83) and the magnet-free baselines (Figure 2). The moving reference matters; the specific moving reference does not, for peak performance. Table 2: Coin Game deep-RL self-play: proxy exploit return (10 seeds; lower is more robust) at the default config. The five methods span the opponent×magnet grid; at this config the four magnet/opponent combinations tie (Sec. 2), so the gap to MMD/naive is the magnet’s presence, not its type. method opponent / magnet exploit return GARIP avg / running-avg −11.29±1.51-11.29± 1.51 R-NaD current / snapshot −11.07±1.49-11.07± 1.49 naive self-play current / none −1.72±2.22-1.72± 2.22 fictitious self-play avg / none −0.04±2.84-0.04± 2.84 MMD current / fixed +7.83±0.93+7.83± 0.93 Figure 2: Coin Game proxy exploitability vs. training (10 seeds). GARIP and R-NaD reach the lowest (most robust) exploit return; MMD’s fixed magnet is worst. 5.4 Four more deep environments: board games To test whether the deep-RL finding is Coin-Game-specific, we add four structurally unrelated deep envs: the turn-based, perfect-information, strictly zero-sum board games Connect Four, Othello, Animal Shogi and Hex (via pgx). Board games sidestep the proxy metric’s main weakness — a policy cannot look robust by being passive, because every game terminates with a win/lose/draw, and the action distribution is the strategy (so the magnet regularizes exactly what is scored). A shared actor-critic plays both colours (the observation is egocentric); the same opponent×magnet ablation applies. We reduce turn-based self-play to a single-agent problem from the learner’s view (learner moves, then the opponent replies), and measure robustness by the win-rate of a freshly trained best-responder against the frozen policy (lower == harder to exploit; stochastic play with random openings so the rate grades exploitability). 8 seeds; 10001000 self-play + 400+\,400 BR updates (Hex: 10,00010,000 self-play). These four games map the scope of the advantage and confirm a prediction the mechanism makes — a moving reference helps exactly where naive self-play cycles (Table 4, Figure 3): • Connect Four and Othello replicate the Coin Game ordering cleanly. The moving-reference methods are far more robust than the fixed-magnet and magnet-free baselines — on Connect Four GARIP (0.430.43) and R-NaD (0.440.44) tie below 0.50.5 (a budget-matched exploiter loses to them) while the rest are exploited 0.760.76–0.860.86; on Othello GARIP (0.55±0.060.55\!±\!0.06) significantly edges R-NaD (0.67±0.070.67\!±\!0.07; 8 seeds, p<0.01p\!<\!0.01), both far below 0.860.86–0.940.94. • Animal Shogi is the predicted control, and we confirmed the mechanism directly rather than inferring it. Here naive self-play is hardest to exploit (0.250.25), exactly as the cycling account predicts for a game that does not cycle, and two diagnostics establish it. (i) Its exploitability decreases monotonically over training (0.52→0.35→0.280.52\!→\!0.35\!→\!0.28 at 250/500/1000250/500/1000 updates) — it converges rather than cycles (on Connect Four naive instead stays at 0.790.79). (i) It beats the regularized policies head-to-head (naive>\,>\,GARIP 0.59/0.400.59/0.40, naive>\,>\,R-NaD 0.67/0.320.67/0.32; each over n=2048n\!=\!2048 games, so ±0.02± 0.02 at 95%95\%), so it is the genuinely stronger player. This small (3×43× 4) piece-drop game does not induce the last-iterate cycling that regularization cures, so the magnet only handicaps playing strength. Even so, GARIP remains the best regularizer: GARIP>\,>\,R-NaD head-to-head (0.60/0.390.60/0.39) and in exploitability (0.43<0.52<0.870.43<0.52<0.87 for GARIP/R-NaD/MMD). • Hex marks the learner’s ceiling, not a counterexample. A 10,00010,000-update stress test establishes it: every method stays 0.890.89–0.960.96 exploitable because an 11×1111× 11 connection game is beyond what this search-free PPO setup learns robustly, so the metric has nothing to separate (R-NaD/GARIP are marginally lowest). This bounds the learner, and the obvious next step is a stronger one (search/AlphaZero-style). This is a mechanism test, not a self-correlation: the diagnostic is the shape of naive’s training curve — the learning dynamics, causally prior to any endpoint. Naive’s exploitability descends monotonically on Animal Shogi (0.52→0.35→0.280.52\!→\!0.35\!→\!0.28, each step lower — converging dynamics) but is flat or stuck on Connect Four (0.90→0.820.90\!→\!0.82), Othello (0.97→0.930.97\!→\!0.93) and Hex (0.93→0.930.93\!→\!0.93). Read Table 3 as dynamics (cause) → advantage (effect): the moving reference helps iff naive’s dynamics fail to converge and the game is learnable, and all six environments comply — the single game whose naive dynamics descend (Animal Shogi) is the single game where regularization does not help. GARIP is the best regularizer wherever the comparison is meaningful. Table 3: Dynamics (cause) → advantage (effect). The shape of naive self-play’s training curve, and whether the moving-reference advantage then appears. The single environment whose naive dynamics descend is the single one without an advantage. (Endpoint metric is per-env: last-iterate exploitability for matrix/Coin Game, best-response win-rate for board games; lower == more robust.) environment naive dynamics bestreg_reg vs naive advantage matrix (random) orbits 0.0030.003 vs 2.02.0 yes Coin Game oscillates −11.3-11.3 vs −1.7-1.7 yes Connect Four flat (0.90→0.820.90\!→\!0.82) 0.430.43 vs 0.790.79 yes Othello flat (0.97→0.930.97\!→\!0.93) 0.550.55 vs 0.940.94 yes Animal Shogi descends (0.52→0.280.52\!→\!0.28) naive wins (0.250.25) no Hex stuck (unlearnable) all ∼0.9 \!0.9 no Table 4: Board-game self-play robustness: best-response win-rate vs. the frozen policy (mean±std_ over 8 seeds; lower == harder to exploit). Bold == most robust per game. game GARIP R-NaD MMD naive fict. Connect Four 0.43±.060.43_±.06 0.44±.050.44_±.05 0.76±.020.76_±.02 0.79±.090.79_±.09 0.86±.050.86_±.05 Othello 0.55±.060.55_±.06 0.67±.070.67_±.07 0.92±.010.92_±.01 0.94±.020.94_±.02 0.86±.060.86_±.06 Animal Shogi 0.43±.040.43_±.04 0.52±.060.52_±.06 0.87±.010.87_±.01 0.25±.070.25_±.07 0.58±.120.58_±.12 Hex (10k) 0.92±.040.92_±.04 0.89±.030.89_±.03 0.96±.010.96_±.01 0.93±.030.93_±.03 0.96±.010.96_±.01 Figure 3: Best-response win-rate across four board games (8 seeds; GARIP red, R-NaD purple; lower == more robust). Moving-reference methods dominate on Connect Four and Othello; Animal Shogi does not cycle (naive wins, but GARIP is the best regularizer); Hex saturates (all ∼ 0.9, under-trained). 5.5 Hyperparameter robustness We now sweep each method’s hyperparameter grid and score every configuration. On a 5×5 matrix-game grid (App. F), GARIP’s median exploitability is 0.0310.031 vs R-NaD’s 0.3320.332, and 56% of GARIP configs converge vs R-NaD’s 20%. R-NaD only works in a narrow band α≤0.5α\!≤\!0.5 and is sensitive to its reset period K; GARIP has no reset period and a much wider basin. And in deep RL — a basin, not immunity. We sweep the Coin Game on a matched grid: GARIP over λ∈0.25,0.5,1,2×ρ∈1.25,2.5,5,10,20,50×10−3λ\!∈\!\0.25,0.5,1,2\×ρ\!∈\!\1.25,2.5,5,10,20,50\\!×\!10^-3 (the two slowest ρ push the EMA lag ∼1/ρ \!1/ρ into GARIP’s own stale regime, symmetric to R-NaD’s K=800K\!=\!800), R-NaD over λ×K∈100,200,400,800λ× K\!∈\!\100,200,400,800\, 10 seeds. Over the full grids the aggregate collapse rates are statistically indistinguishable — GARIP 25.4%25.4\% [20.3,31.3][20.3,31.3], R-NaD 31.9%31.9\% [25.2,39.4][25.2,39.4] (Wilson 95% CIs) — precisely because GARIP’s grid now contains configs that collapse. The advantage is where each collapse boundary sits relative to the rate one actually uses (Table 5, Figure 4): GARIP collapses only at slow averaging (ρ≤0.005ρ\!≤\!0.005) and is essentially collapse-free across ρ≥10−2ρ\!≥\!10^-2 (1/1201/120 runs — the single failure is one seed at the fastest rate ρ=0.05ρ\!=\!0.05, i.e. too-fast tracking, not staleness), whereas R-NaD already collapses 25%25\% at the commonly-used K=200K\!=\!200 and 55%55\% at K=800K\!=\!800, safe only at the short K=100K\!=\!100. GARIP’s failure region thus lies an order of magnitude slower than the standard Polyak rate; R-NaD’s overlaps the reset periods used at scale. This is the basin-width advantage — not collapse-immunity (an earlier draft’s 7%7\% figure was a grid artifact from a ρ range that stopped short of the stale regime, corrected here). Exact-metric confirmation on Leduc. Finally, the staleness prediction — R-NaD degrades monotonically in αKα K — is confirmed on a second environment with an exact metric (App. F). R-NaD’s exploitability rises monotonically from 0.24 (low α, low K) to 2.01 (high α, high K) and never reaches near-Nash on Leduc (no config below 0.1), because the stale magnet leaves a residual bias even at its best. GARIP reaches 0.04. Table 5: Deep-RL collapse rate (exploit return >0>0) vs. staleness depth (Coin Game, 10 seeds, 40 runs per column entry). Both methods collapse at their stale end; GARIP’s boundary lies past its default ρ=10−2ρ\!=\!10^-2, R-NaD’s lies inside commonly-used K. Aggregate rates over the full grids are indistinguishable (GARIP 25%25\%, R-NaD 32%32\%); the boundary is the story. GARIP ρ collapse R-NaD K collapse 0.00125 0.75 100 0.00 0.0025 0.50 200 0.25 0.005 0.25 400 0.47 0.01† 0.00 800 0.55 0.02 0.00 — — 0.05 0.03 — — † GARIP’s default Polyak rate. Figure 4: Deep-RL hyperparameter sensitivity (10 seeds; GARIP λ×ρλ×ρ, R-NaD λ×Kλ× K). Both methods have a stale-collapse region (red): R-NaD at large K, GARIP at small ρ (the two leftmost columns). The point is location, not absence — GARIP’s red region sits at averaging rates ≤0.005≤\!0.005, past its default ρ=0.01ρ\!=\!0.01, whereas R-NaD’s reaches the commonly-used K=200K\!=\!200. Matrix-grid and Leduc sweeps (App. F) confirm the same picture with an exact metric. 6 Honesty and limitations We are deliberately careful about what GARIP does not do (extended version, App. G). • No peak-performance claim; not collapse-immune. GARIP only ties R-NaD at each method’s tuned best; the contribution is a wider safe basin, not lower exploitability. It has its own staleness collapse — a slow EMA (ρ≈10−3ρ\!≈\!10^-3, λ≥0.5λ\!≥\!0.5) goes exploitable, symmetric to R-NaD’s large-K failure — so the earlier “no collapse region” claim was wrong and we corrected it after a fair symmetric-ρ sweep; the default ρ=10−2ρ\!=\!10^-2 stays an order of magnitude clear. • Proxy metric and scope. Exact metrics cover matrix games, Kuhn and Leduc; deep RL uses a best-response proxy that can reward passivity on sparse-reward spatial games (so we use turn-based board games, which always terminate). The search-free learner does not reach robust play on 11×1111× 11 Hex, and Animal Shogi (non-cycling) needs no regularizer — both honest boundaries, not counterexamples. • Which R-NaD. We test fixed-(λ,K)(λ,K) R-NaD (standalone regularizer); DeepNash’s outer schedule that anneals across convergence is untested, and we expect it trades staleness for annealing-sensitivity. • Local last iterate proven; global conjectured, with a failure mode. Prop. 3 gives local convergence and Lemma 2 reduces the global version to x¯t→ x_t\!→ Nash — which our sweeps show fails at large β via premature consensus (Conjecture 1). This is a basin-of-attraction failure, not a fixed-point/annealing one: Prop. 2’s unbiasedness is untouched, and the deep-RL operating point has no large-β analog. We claim no acceleration — on an already-convergent base the anchor is a cost. 7 Conclusion GARIP anchors self-play to the policy’s running average — a moving reference that, like R-NaD’s periodic snapshot, gives last-iterate Nash without annealing, and matches R-NaD’s peak performance. On robustness the two differ in their reference’s lag profile: collapse empirically tracks the peak stale force, and among causal averages the flat profile (the running average) is the peak-minimizer (Prop. 1), while a snapshot’s peak is twice its mean. This makes the running average the better default: over the full grid the aggregate collapse rates are indistinguishable, but at conventional parameterizations — R-NaD’s mean-like K under-counting its peak — the standard ρ=10−2ρ\!=\!10^-2 and K=200K\!=\!200 collapse in 0/400/40 vs 10/4010/40 seeds (including the λ=2λ\!=\!2 stress). The two tie at matched peak lag, so a snapshot recovers safety by shortening K — but GARIP gives the safe parameterization for free, where R-NaD asks the user to know to do it. Its boundaries: an anticipatory (negative-weight) reference does better still on the stale side (App. C), and the edge over R-NaD appears only where naive self-play cycles (Coin Game, Connect Four, Othello), not where it converges (Animal Shogi) or where no method does (Hex). Within that scope, the running average is the reference we would reach for, and the lag-profile lens says why. Code Availability All code (the GARIP method and baselines, the tabular/poker/deep-RL experiments, the local-stability and per-mode verification scripts, the premature-consensus sweep, and the LaTeX source) is available at https://github.com/highcansavci/garip. Supplementary material (proofs, the discrete-dynamics derivation, the anticipatory-reference study, and the β⋆(d)β (d) phase boundary) is bundled in the appendices there. Use of AI-Assisted Technologies The author used Claude (Anthropic), including Claude Code, to assist with the experimental code, manuscript review and revision, and formatting. All AI-assisted content was verified by the author, who is solely responsible for the methodology, results, and conclusions of this paper. References [1] J. Perolat et al. Mastering the game of Stratego with model-free multiagent reinforcement learning. Science, 378(6623), 2022. (R-NaD / DeepNash.) [2] S. Sokota et al. A unified approach to reinforcement learning, quantal response equilibria, and two-player zero-sum games. ICLR, 2023. (Magnetic Mirror Descent.) [3] J. Heinrich and D. Silver. Deep reinforcement learning from self-play in imperfect-information games. arXiv:1603.01121, 2016. (NFSP.) [4] N. Brown, A. Lerer, S. Gross, T. Sandholm. Deep counterfactual regret minimization. ICML, 2019. [5] J.-Y. Zhu, T. Park, P. Isola, A. Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. ICCV, 2017. (CycleGAN.) [6] G. M. Korpelevich. The extragradient method for finding saddle points and other problems. Ekonomika i Matematicheskie Metody, 12, 1976. [7] L. D. Popov. A modification of the Arrow–Hurwicz method for search of saddle points. Mathematical Notes, 28(5), 1980. (Optimistic/past-gradient update.) [8] A. Rakhlin and K. Sridharan. Online learning with predictable sequences. COLT, 2013. (Optimism / predictable sequences.) [9] C. Daskalakis, A. Ilyas, V. Syrgkanis, H. Zeng. Training GANs with optimism. ICLR, 2018. (OGDA, last-iterate in bilinear games.) [10] C.-Y. Wei, C.-W. Lee, M. Zhang, H. Luo. Linear last-iterate convergence in constrained saddle-point optimization. ICLR, 2021. (OMWU linear last-iterate.) [11] P. Mertikopoulos, B. Lecouat, H. Zenati, C.-S. Foo, V. Chandrasekhar, G. Piliouras. Optimistic mirror descent in saddle-point problems: going the extra (gradient) mile. ICLR, 2019. [12] N. Golowich, S. Pattathil, C. Daskalakis, A. Ozdaglar. Last iterate is slower than averaged iterate in smooth convex-concave saddle point problems. COLT, 2020. (Last-iterate vs. averaged.) [13] B. Halpern. Fixed points of nonexpanding maps. Bull. Amer. Math. Soc., 73, 1967. (Halpern anchoring iteration.) [14] T. Yoon and E. K. Ryu. Accelerated algorithms for smooth convex-concave minimax problems with O(1/k2)O(1/k^2) rate on squared gradient norm. ICML, 2021. (Halpern-anchored acceleration for minimax.) [15] Y. Cai, A. Oikonomou, W. Zheng. Finite-time last-iterate convergence for learning in multi-player games. NeurIPS, 2022. Appendix A Proofs Proof of Prop. 1 (peak-lag minimality). ℓmax=maxtℓt≥T−1∑tℓt=ℓ¯ _ = _t _t≥ T^-1\! _t _t= , with equality iff all ℓt _t coincide. The EMA’s weights wt,s=ρ(1−ρ)t−sw_t,s=ρ(1-ρ)^t-s give the stationary mean age (1−ρ)/ρ(1-ρ)/ρ, constant in t; a snapshot held for K steps has ℓt∈0,…,K−1 _t∈\0,…,K\!-\!1\, so ℓ¯=K−12 = K-12 and ℓmax=K−1=2ℓ¯ _ =K\!-\!1=2 . ∎ Proof of Lemma 1 (anchor contraction). xt+1−x¯t=(1−β)xt+12+βx¯t−x¯t=(1−β)(xt+12−x¯t)x_t+1- x_t=(1-β)x_t+ 12+β x_t- x_t=(1-β)(x_t+ 12- x_t). ∎ Proof of Prop. 2 (fixed points are Nash). At a fixed point xt+1=xt=:x_t+1=x_t=:x with x¯t=x x_t=x, Lemma 1 forces xt+12=x_t+ 12=x: the OMWU half-step fixes x. With gt=gt−1=Ayg_t=g_t-1=Ay this is x∝x⊙exp(ηAy)x x (η\,Ay), so (Ay)i(Ay)_i is constant on supp(x)supp(x). For interior x this holds for all i, i.e. y makes the row player indifferent; symmetrically x makes the column player indifferent, so (x,y)(x,y) is Nash, giving (i); a Nash satisfies these on its support with the anchor inactive, giving (i). For (i), the magnet contribution at the fixed point is β(x¯t−xt+12)=0β( x_t-x_t+ 12)=0. ∎ Proof of Prop. 3 (local last-iterate convergence). The anchor xt+1=(1−β)xt+12+βx¯tx_t+1=(1-β)x_t+ 12+β x_t and average x¯t+1=(1−ρ)x¯t+ρxt+1 x_t+1=(1-ρ) x_t+ρ x_t+1 are coordinate-wise identical scalars, so they commute with the base operator and the linearization splits per base mode M, with xt+12−x⋆=M(xt−x⋆)x_t+ 12-x =M(x_t-x ); the (1−β)(1-β) factor on the policy row is Lemma 1, and detJ=(1−β)M[(1−ρ)+ρβ]−βρ(1−β)M=(1−ρ)(1−β)M J=(1-β)M[(1-ρ)+ρβ]-βρ(1-β)M=(1-ρ)(1-β)M. Fast mode: at ρ=0ρ=0 the eigenvalues are exactly (1−β)M(1-β)M (modulus (1−β)μ(1-β)μ) and the marginal averaging mode 11; the former carries the mechanism — a recurrent base (μ=1μ=1) is strictly contracted to (1−β)<1(1-β)<1. Slow mode: expanding the marginal root λ=1+ρcλ=1+ρ\,c in the characteristic polynomial λ2−(trJ)λ+detJλ^2-(tr\,J)λ+ J gives, to first order, λslow=1−ρ(1−β)(1−M)1−(1−β)M+O(ρ2), _slow=1-ρ\, (1-β)(1-M)1-(1-β)M+O(ρ^2), so |λslow|=1−ρReK+O(ρ2)| _slow|=1-ρ\,Re\,K+O(ρ^2) with K=(1−β)(1−M)/(1−(1−β)M)K=(1-β)(1-M)/(1-(1-β)M). Writing M=1−δ+iωM=1-δ+iω (δ,ω=Θ(η)δ,ω= (η), δ>0δ>0), K→(1−β)δ/β>0K→(1-β)δ/β>0 as η→0η→ 0, so ReK>0Re\,K>0 and |λslow|<1| _slow|<1 for small η. (The naive real form 1−ρ(1−μ)(1−β)1-ρ(1-μ)(1-β) is not correct — M is a rotation, so the coefficient is complex.) For the Cesàro average of Sec. 4 (weight 1/(t+2)→01/(t+2)\!→\!0), ut:=xt−x¯tu_t:=x_t- x_t obeys ut+1=(1−β)Mut+(1−β)(M−I)δ¯tu_t+1=(1-β)M\,u_t+(1-β)(M-I) δ_t with δ¯t:=x¯t−x⋆ δ_t:= x_t-x slowly varying; ut→0u_t→ 0 geometrically, and the only frozen anchor consistent with u=0u\!=\!0 solves (M−I)δ¯=0(M-I) δ=0, i.e. δ¯=0 δ=0 since 1∉spec(M)1 (M) — so x¯t→x⋆ x_t→ x , closing step (a) locally. For the one-step base (MWU — the recurrent, deep-RL regime) the anchor and average are scalars on the entire mode space, so the block-diagonalization is exact for any m×nm× n game and ρ(full)=maxkρ(J(Mk))ρ(full)= _kρ(J(M_k)) holds identically (confirmed to finite-difference precision, <10−10<\!10^-10, across sizes up to 8×88× 8 with 1010 random payoffs each). GARIP’s optimistic half-step carries a lagged gradient the anchor does not touch, enlarging each per-mode block from 2×22× 2 to 3×33× 3; the structure — a fast mode scaled by 1−β1-β, a slow average mode near 11 — is unchanged, and the 2×22× 2 form agrees with the full spectrum to <4×10−4<\!4× 10^-4. ∎ Proof of Lemma 2 (potential-level contraction). xt+1=(1−β)xt+12+βx¯tx_t+1=(1-β)x_t+ 12+β x_t with convexity gives the first bound; x¯t=1t∑s<txs x_t= 1t _s<tx_s with the same convexity (Jensen) gives the second. ∎ Appendix B A discrete-dynamics mechanism for the peak-force collapse law This appendix expands the footnote-level claim that the collapse law has a derivable (discrete, not continuous) origin. The collapse law is not pure phenomenology; it has a derivable origin, just not a continuous one. Linearizing a 22-state game in continuous time, the stale magnet is a delayed term z˙=μ0z+λ(z(t−τ)−z(t)) z= _0z+λ(z(t\!-\!τ)-z(t)) on the converging spiral μ0=−δ±iω _0=-δ± iω; its characteristic equation μ=μ0+λ(e−μτ−1)μ= _0+λ(e^-μτ-1) has no imaginary-axis root (ReRe at μ=iνμ=iν is −δ+λ(cosντ−1)≤−δ<0-δ+λ( ντ-1)≤-δ<0), so in the continuous limit the magnet damps — Nash never loses stability. The discrete map, which is what runs, does collapse. With one mode zt+1=(1−β)Mzt+βrtz_t+1=(1-β)Mz_t+β\,r_t, M=(1−δ)eiωM=(1-δ)e^iω (|M|<1|M|<1), the past iterate zt−ℓ=M−ℓztz_t- =M^- z_t had larger amplitude |M|−ℓ>1|M|^- \!>\!1. A snapshot reaches the single peak-lag state M−KM^-K (full amplification |M|−K≈eδK|M|^-K\!≈\!e^δ K), so a strong/slow magnet drives the step gain g=(1−β)M+βM−Kg=(1-β)M+β M^-K past |g|=1|g|\!=\!1 — a discrete overshoot, governed by the peak reach; the EMA smears its reach over lags (∑sρ(1−ρ)sM−s _sρ(1-ρ)^sM^-s), so its gain is bounded by the flat profile, not a deep peak. This reproduces the peak-force law and its peak-vs-mean asymmetry. The toy in fact over-predicts — an eδKe^δ K gap, exponential in lag, sharper than the roughly-linear empirical curves of Fig. 1; the gap between the two is exactly the phase cancellation and finite-step saturation the linear one-mode model omits, so the over-prediction is a consistency check, not a contradiction. Only the exact threshold is therefore phase-dependent, which is why we calibrate the law empirically (Fig. 1) and prove the reference-shape optimality on top of it (Prop. 1). Appendix C Beyond causal averages: an anticipatory reference Proposition 1 is optimal only within causal averages, and Remark 1 flagged that an extrapolating reference (negative weights, lower effective lag) might beat it. We tested it directly. A double-EMA magnet (1+γ)θ¯−γθ¯(1+γ)\, θ-γ\, θ (θ¯ θ the EMA of θ¯ θ, gain γ=1γ=1) leads the policy — a negative-weight reference, still causal but outside the convex class — which we swept against plain GARIP on the deep-RL (λ,ρ)(λ,ρ) collapse grid (Table 6; 1818 runs/cell over λ∈0.5,1,2λ\!∈\!\0.5,1,2\). Table 6: Collapse rate (exploit return >0>0) vs. averaging rate ρ for the causal running average (GARIP) and the anticipatory double-EMA magnet (Coin Game; lower == safer, bold). Anticipation widens the safe basin on the stale (slow-ρ) side but overshoots at fast ρ. Here each cell is 1818 runs (λ∈0.5,1,2×6λ\!∈\!\0.5,1,2\× 6 seeds), not the 4040 of Table 5 (which also includes the safe λ=0.25λ\!=\!0.25), so GARIP’s rates run higher; Wilson 95%95\% CIs are wide (∼±0.2 \!± 0.2), e.g. the ρ=0.05ρ\!=\!0.05 overshoot is 2/182/18, CI [0.03,0.33][0.03,0.33]. ρ GARIP (causal avg.) extrap. (anticipatory) 0.001250.00125 1.001.00 0.560.56 0.00250.0025 0.670.67 0.000.00 0.0050.005 0.330.33 0.000.00 0.010.01 (default) 0.000.00 0.000.00 0.020.02 0.000.00 0.000.00 0.050.05 0.000.00 0.110.11 The prediction holds, with a twist. Anticipation subtracts lag, so it widens the safe basin exactly on the stale side: collapse falls 67%→0%67\%\!→\!0\% at ρ=0.0025ρ=0.0025 and 33%→0%33\%\!→\!0\% at ρ=0.005ρ=0.005, pushing the boundary from ρ≈0.005ρ\!≈\!0.005 down to ρ≈0.0025ρ\!≈\!0.0025. But over-anticipating a fast average overshoots — at ρ=0.05ρ=0.05 the extrapolated magnet newly collapses (0%→11%0\%\!→\!11\%). The two are therefore complementary, not dominating: plain GARIP is safe for ρ∈[0.01,0.05]ρ\!∈\![0.01,0.05], the anticipatory magnet for ρ∈[0.0025,0.02]ρ\!∈\![0.0025,0.02], and GARIP’s default ρ=0.01ρ=0.01 sits safely in both. So Prop. 1 is genuinely breakable by leaving the convex class (negative weights) — exactly as scoped — and the peak-lag picture predicts both halves (anticipation helps when lag is large, hurts when it is already small): the running average is the robust default, anticipation is the right tool when one is forced to slow averaging, and an adaptive magnet that tunes γ to the operating lag is a natural next step. Appendix D The premature-consensus phase boundary At a constant anchor the reduction of Conjecture 1 (global last iterate ⇐x¯t→ x_t\!→ Nash) can fail: a strong anchor drives xtx_t and x¯t x_t into agreement before the average reaches Nash, freezing the iterate at a non-Nash consensus. We map the boundary on antisymmetric d×d× d games (interior Nash), classifying a run as stalled when exploitability is frozen high while ‖xt−x¯t‖→0\|x_t- x_t\|\!→\!0. The critical anchor β⋆(d)β (d) above which a game stalls falls with size: d ≤4≤ 4 66 88 1010 1212 β⋆(d)β (d) none (≤0.4≤ 0.4) 0.20.2 0.10.1 0.050.05 0.020.02 So β⋆β shrinks as the game grows and the default β=0.02β\!=\!0.02 sits below the boundary through d≈10d\!≈\!10. Pure OMWU (β=0β\!=\!0) reaches Nash to ∼10−7 \!10^-7 on these games: on an already-convergent base a strong anchor over-regularizes — the tabular face of the Animal-Shogi lesson (a magnet is unnecessary on a non-cycling game). The deep-RL operating point has no analog: there collapse is reference staleness (slow ρ / long K), not anchor strength, and the anchor is load-bearing rather than a liability. Appendix E Matrix curves, poker details, and the RPS trajectory Figure 5: Last-iterate exploitability vs. iteration on four matrix games (10 seeds, shaded IQR). GARIP (red) reaches the lowest exploitability on the random games; MMD plateaus at the QRE; self-play gradient ascent stays at ∼ 2. Figure 6: Row-player strategy trajectory on the RPS simplex. Naive self-play gradient ascent (grey) orbits the boundary; GARIP (red) spirals into Nash (star). Kuhn (neural). Each player is a small MLP over information-set features, trained through the differentiable game tree; exploitability is exact (brute force over the 64 pure strategies). Using the unified magnet framework (Table 7), GARIP (0.076) is the best regularized method, ahead of R-NaD (0.30) and MMD (0.54), and approaches exact-best-response fictitious play (0.043). These are each method’s default config — we did not separately tune R-NaD’s (α,K)(α,K) on the poker games, so the gap is default-vs-default and a tuned R-NaD would likely close much of it, exactly as it does on the matrix games; we do not claim a tuned-best poker win. Table 7: Neural Kuhn poker: average-strategy exact exploitability (10 seeds). Same ordering throughout: moving >> snapshot >> fixed magnet. method exploitability fictitious play (exact BR) 0.043 GARIP (moving magnet) 0.076 R-NaD (periodic snapshot) 0.301 naive self-play 0.373 MMD (fixed magnet) 0.538 Leduc (tabular, exact). Leduc has 288 information sets and an exact recursive best response and CFR solver (game value lands on the known −0.0856-0.0856). Because raw-gradient neural self-play does not converge for the magnet baselines here — the reach/credit-assignment failure the counterfactual target was built to fix — we run the methods tabularly with mirror ascent on exact counterfactual values. GARIP is fastest and lowest at this iteration budget (Table 8, Figure 7). It nominally sits below CFR at 3000 iterations, but we do not read this as beating CFR: CFR’s averaged strategy converges slowly early (O(1/T)O(1/ T)), and a mirror-ascent step is not the same unit of compute as a CFR iteration, so CFR is a reference point, not a baseline GARIP outperforms — with more sweeps it keeps falling to ∼ 0.04. As on Kuhn, these are default configs and R-NaD was not separately tuned here. Table 8: Leduc hold’em: average-strategy exact exploitability (8 seeds, 3000 iterations). Default configs; CFR is shown at a matched iteration count as a reference point (not converged — it falls to ∼ 0.04 with more sweeps), not as a baseline GARIP outperforms. method exploitability GARIP (moving magnet) 0.074 CFR (reference) 0.116 R-NaD (periodic snapshot) 0.227 MMD (fixed magnet) 0.552 naive self-play 0.741 Figure 7: Leduc hold’em: exact exploitability of the average strategy (8 seeds). GARIP converges fastest; MMD’s fixed magnet plateaus at the QRE; naive self-play cycles. Appendix F Additional robustness sweeps The matrix-grid and Leduc confirmations of the Coin Game sensitivity picture (§5.5), deferred from the main text. Table 9: Robustness over the 5×55× 5 matrix grid (GARIP (η,β)(η,β), R-NaD (α,K)(α,K)); lower median / IQR / worst-case exploitability and a higher converged fraction indicate a wider safe basin. over the grid GARIP (η,β)(η,β) R-NaD (α,K)(α,K) median exploitability 0.031 0.332 spread (IQR) 0.126 0.450 worst config 0.772 1.209 configs converged (<0.05<\!0.05) 56% 20% Table 10: Robustness over the Leduc grid, exact exploitability (6 seeds). R-NaD never reaches near-Nash; GARIP’s only failure is generic large-step instability. Leduc GARIP (η,β)(η,β) R-NaD (α,K)(α,K) median 0.080 0.427 configs near Nash (<0.1<\!0.1) 64% 0% worst config 2.21 2.70 Figure 8: Matrix-game hyperparameter sensitivity. GARIP (η,β)(η,β) has a wide low-exploitability basin; R-NaD (α,K)(α,K) degrades sharply outside a narrow band. Figure 9: Leduc collapse heatmaps (exact exploitability, 6 seeds). R-NaD shows the predicted monotonic αKα K gradient and never reaches near-Nash; GARIP’s only failure is generic large-step-size instability (η=0.5η\!=\!0.5). Alternative references we tested. Beyond the causal running average and the anticipatory double-EMA (App. C), we tried two further mechanisms and report them for completeness: a reciprocal-cycle reference (anchor to the double quantal-best-response F(G(x))F(G(x)), the literal CycleGAN cycle term) and a drift-adaptive averaging rate. Neither improved on the running average — the reciprocal cycle inherits the game’s rotation and destabilizes, and adaptive anticipation either leaves the simplex or overshoots — consistent with Prop. 1: among causal references the running average is already peak-lag optimal, and leaving that class buys only the narrow stale-side gain of App. C. Appendix G Extended honesty and limitations The full version of the limitations summarized in the main text. • No peak-performance claim. GARIP only ties R-NaD at each method’s tuned best (matrix games and Coin Game). The contribution is a wider safe basin (collapse boundary past the default rate), not lower exploitability and not collapse-immunity. • GARIP has its own staleness collapse. A slow EMA (ρ≈10−3ρ\!≈\!10^-3) at λ≥0.5λ\!≥\!0.5 goes exploitable, symmetric to R-NaD’s large-K failure (Sec. 3); GARIP is not collapse-free. Its default ρ=10−2ρ\!=\!10^-2 is robust, and the collapse region lies an order of magnitude away — but the “no collapse region” claim of an earlier draft was wrong, and we corrected it after a fair (symmetric-ρ) sweep. Separately, the largest matrix step size η=0.5η\!=\!0.5 blows up on Leduc — generic step-size instability, a single isolated row. • Scope and the proxy metric. Exact metrics cover matrix games, Kuhn and Leduc; the deep-RL evidence uses a fixed-budget best-response proxy. This proxy is delicate: on the simultaneous-move Coin Game it needs a strong enough best responder, and on a sparse-reward spatial game we found it can reward passivity (a non-engaging policy is unexploitable by construction), so we did not use such games. Turn-based board games are the fix — they always terminate with a result, so the metric cannot be gamed by inaction — which is why the board panel (Sec. 5.4) is the primary deep evidence. Its limits are honest: the search-free PPO learner does not reach robust play on 11×1111× 11 Hex even at 10,00010,000 updates (the metric saturates), and on a small non-cycling game (Animal Shogi) regularization is simply unnecessary. Stronger learners (search/AlphaZero-style) and Stratego-scale games remain future work. • Which R-NaD is under test. We test fixed-(λ,K)(λ,K) R-NaD: a periodic snapshot reset every K steps under a constant KL weight λ — the form used as a standalone self-play regularizer, and the clean point of comparison for “which moving reference.” The full R-NaD behind DeepNash additionally runs an outer schedule that updates the regularization policy and anneals the reward transform across convergence, partly to manage exactly the staleness we exploit. Our critique is of the fixed-schedule instantiation; whether it survives the scheduled version is open. We expect the schedule trades staleness for annealing-sensitivity — a fixed K becomes a schedule to tune, which is itself a hyperparameter-robustness cost — but we have not measured this, and say so plainly. • Local last iterate proven; global is conjectured. Sec. 4 proves the structural facts — the anchor contracts deviation from the average by exactly 1−β1-β (Lemma 1), and is unbiased so that fixed points are Nash, unlike MMD’s QRE-biased fixed magnet (Prop. 2) — and now a local last-iterate theorem (Prop. 3): the anchor scales the base’s rotation by 1−β1-β, crossing the stability boundary and turning a recurrent base into a contraction, so Nash is locally asymptotically stable in the last iterate at β>0β\!>\!0. The potential-level reduction (Lemma 2) collapses the global version to a single estimate, x¯t→ x_t→ Nash — but our long-horizon sweeps show this can fail (App. D): at larger β on larger interior-Nash games the anchor drives a premature consensus (xt≈x¯tx_t\!≈\! x_t at a non-Nash point) and the last iterate stalls, while pure OMWU (β=0β\!=\!0) reaches Nash on the same games. So we do not claim global convergence or acceleration: the anchor is a robustness device for cycling bases, not a last-iterate accelerator, and on an already-convergent base it is a cost. This is a basin-of-attraction failure at large β, not a fixed-point or annealing issue — Prop. 2’s unbiasedness (no annealing needed) is untouched, and the deep-RL operating point has no large-β analog (there collapse is reference staleness, not anchor strength). • Earlier mis-report, corrected. An initial 2-seed, narrow-grid run suggested the opposite robustness conclusion; it under-sampled R-NaD’s stale-reference failure region. The 10-seed wider-grid results here supersede it.