Paper deep dive

Robust Nash Alignment under Preference Uncertainty

Shihab Ahmed, Debamita Ghosh, David Tang, Yudan Wang, Alvaro Velasquez, Yue Wang

Year: 2026Venue: arXiv preprintArea: cs.AIType: PreprintEmbeddings: 140

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 94%

Last extracted: 10/2/2026, 9:34:25 AM

Summary

The paper introduces Robust Nash Alignment, a game-theoretic framework for aligning large language models (LLMs) under uncertain pairwise preferences. Unlike standard methods that optimize against a single preference model, this approach formulates alignment as a robust game where the learner maximizes the worst-case win rate against an adversarial competitor and any preference kernel within an ambiguity set. To solve this computationally challenging problem, the authors propose a four-player primal-dual proxy game and an optimistic mirror descent-ascent algorithm, proving convergence guarantees and demonstrating improved robustness in tabular games and LLM alignment tasks.

Entities (10)

Arizona State University · organization · 99%University of Central Florida · organization · 99%Robust Nash Alignment · method · 98%Large Language Models · technology · 97%Four-Player Primal-Dual Proxy Game · model · 96%Optimistic Mirror Descent-Ascent · algorithm · 95%Ambiguity Set · concept · 93%Nash learning from human feedback · method · 92%Reinforcement Learning from Human Feedback · method · 90%Direct Preference Optimization · method · 88%

Relation Signals (7)

Yudan Wang → affiliatedwith → Arizona State University

confidence 99% · Yudan Wang Affiliation: Arizona State University

Shihab Ahmed → affiliatedwith → University of Central Florida

confidence 99% · Shihab Ahmed Affiliation: University of Central Florida

Robust Nash Alignment → addresses → Preference Uncertainty

confidence 95% · To address these issues, we propose Robust Nash Alignment, a game-theoretic framework for alignment to uncertain pairwise preferences.

Robust Nash Alignment → uses → Four-Player Primal-Dual Proxy Game

confidence 94% · we introduce a four-player primal-dual proxy game involving the leader policy, follower policy, adversarial kernel, and dual variable

Four-Player Primal-Dual Proxy Game → solvedby → Optimistic Mirror Descent-Ascent

confidence 93% · develop a single-loop optimistic mirror descent-ascent algorithm for it.

Robust Nash Alignment → appliedto → Large Language Models

confidence 90% · Experiments on controlled tabular games and LLM alignment with uncertain preference further validate the convergence theory

Robust Nash Alignment → improvesover → Nash learning from human feedback

confidence 85% · Unlike standard NLHF, which optimizes against a single nominal kernel, our objective maximizes the worst-case game value

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Preference-based alignment methods typically optimize against a single preference model, and can therefore be brittle when pairwise preferences are uncertain: noisy, heterogeneous, or shift after deployment. To address these issues, we propose Robust Nash Alignment, a game-theoretic framework for alignment to uncertain pairwise preferences. Our formulation has a major learner seeking a policy with a large worst-case win rate against both an adversarial competitor and any preference kernel lying in an ambiguity set around a nominal preference. When the ambiguity set captures the uncertainty in preferences, the resulting robust objective of the game directly yields a certified lower bound on worst-case performance. However, we note this problem is computationally challenging to optimize, and to address this, we introduce a four-player primal-dual proxy game involving the leader policy, follower policy, adversarial kernel, and dual variable, and develop a single-loop optimistic mirror descent-ascent algorithm for it. We show that the proxy always lower-bounds the truncated hard-constrained objective, quantify the proxy-to-hard gap, and characterize an exactness condition under which the proxy recovers the robust objective. We then prove an \(\mathcal{O}(1/\sqrt{T})\) average-iteration convergence for the proxy-game duality gap, which implies a near-optimal robust policy for the original robust objective. Experiments on controlled tabular games and LLM alignment with uncertain preference further validate the convergence theory and show improved performance over nominal baselines.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

136,434 characters extracted from source content.

Expand or collapse full text

Robust Nash Alignment under Preference Uncertainty Shihab Ahmed Affiliation: University of Central Florida Email: shihab.ahmed@ucf.edu Debamita Ghosh Affiliation: University of Central Florida Email: debamita.ghosh@ucf.edu David Tang Affiliation: University of Central Florida Email: david.tang@ucf.edu Yudan Wang Affiliation: Arizona State University Email: ywan1645@asu.edu Alvaro Velasquez Affiliation: University of Colorado Boulder Email: alvaro.velasquez@colorado.edu Yue Wang Affiliation: University of Central Florida Email: yue.wang@ucf.edu Abstract Preference-based alignment methods typically optimize against a single preference model, and can therefore be brittle when pairwise preferences are uncertain: noisy, heterogeneous, or shift after deployment. To address these issues, we propose Robust Nash Alignment, a game-theoretic framework for alignment to uncertain pairwise preferences. Our formulation has a major learner seeking a policy with a large worst-case win rate against both an adversarial competitor and any preference kernel lying in an ambiguity set around a nominal preference. When the ambiguity set captures the uncertainty in preferences, the resulting robust objective of the game directly yields a certified lower bound on worst-case performance. However, we note this problem is computationally challenging to optimize, and to address this, we introduce a four-player primal-dual proxy game involving the leader policy, follower policy, adversarial kernel, and dual variable, and develop a single-loop optimistic mirror descent-ascent algorithm for it. We show that the proxy always lower-bounds the truncated hard-constrained objective, quantify the proxy-to-hard gap, and characterize an exactness condition under which the proxy recovers the robust objective. We then prove an ⁡(1/T)O(1/ T) average-iteration convergence for the proxy-game duality gap, which implies a near-optimal robust policy for the original robust objective. Experiments on controlled tabular games and LLM alignment with uncertain preference further validate the convergence theory and show improved performance over nominal baselines. 1 Introduction Aligning large language models (LLMs) with preferences is a central challenge in modern post-training. The dominant paradigm, reinforcement learning from human feedback (RLHF) [9, 41, 33], collects pairwise comparisons, fits a scalar reward model, and optimizes policy against that reward with a proximity constraint to a reference model. More recent direct alignment methods, including Direct Preference Optimization (DPO), SLiC-HF, Contrastive Preference Learning (CPL), and Soft Preference Optimization (SPO) [35, 55, 15, 40], bypass the explicit reward model by optimizing preference-derived surrogate losses directly. Despite their practical success, all of these methods share a common structural assumption: they model preferences through a latent scalar reward function, typically via the Bradley-Terry (BT) [45, 4] or Thurstone framework. This assumption is, however, restrictive. Preferences may be context-dependent, cyclic, or heterogeneous across annotators, and cannot always be represented by a single scalar reward [3, 29, 54, 52, 8, 14]. A growing line of work addresses this limitation by modeling preferences directly through pairwise comparison probabilities and framing alignment as a two-player zero-sum game. In particular, Nash Learning from Human Feedback (NLHF) and its extensions [29, 54, 52, 5] seek a policy that is hard to beat in the induced preference game, rather than one that maximizes a learned scalar reward. This general-preference perspective resolves the modeling mismatch: it can represent non-transitive, cyclic, and population-heterogeneous preferences that no scalar reward can capture. However, it introduces a different and equally important challenge that has been largely overlooked: the preference kernel itself is uncertain. The true pairwise preference probability P⁡(yi≻yj∣x)P(y_i y_j x) is never observed or fixed. In practice, it must be estimated from finite, noisy, and potentially biased comparisons, often aggregated across heterogeneous annotator populations and changing deployment environments [9, 41, 33, 43]. Existing general-preference methods, including NLHF, treat the learned preference kernel as a fixed ground truth and optimize against this single point estimate. This can be brittle: small errors in preference estimation, misspecification of the comparison model, or deployment-time shifts in who provides feedback may materially change which responses the policy favors [36, 38, 8, 14]. We refer to this vulnerability as the fragile consensus problem—a policy that appears optimal under the nominal preference data may concentrate on responses whose advantages rest on slim margins that are statistically indistinguishable from noise, and a small perturbation to the preference kernel can reverse the dominance ordering entirely. To address this challenge, we propose to treat the preference kernel as fundamentally uncertain and to optimize for the worst case over a set of plausible kernels. Inspired by the literature on distributionally robust optimization [36, 20, 31], where ambiguity sets, divergence balls, and worst-case guarantees are standard tools for hedging estimation error and distribution shift, we formulate alignment under uncertain preferences as a robust two-player game over a pairwise preference kernel. At each prompt, the main player seeks to maximize a Kullback-Leibler (KL)-regularized win rate, while an adversarial player simultaneously chooses the hardest competitor and the worst-case preference kernel from an ambiguity set centered at the nominal estimate. The resulting objective directly provides a lower bound on the policy’s alignment performance under all plausible preference kernels in the ambiguity set, effectively addressing the fragile consensus problem. Our contributions are summarized as follows. Robust preference-game formulation. We formulate alignment under preference uncertainty as a robust game over a pairwise preference kernel with a Bernoulli-KL ambiguity set. Unlike standard NLHF, which optimizes against a single nominal kernel, our objective maximizes the worst-case game value over all kernels within a prescribed divergence ball. We show that this formulation yields a principled worst-case performance guarantee: the learned policy’s win rate is lower-bounded against every plausible preference kernel in the ambiguity set, improving the robustness and reliability of preference-based alignment. Four-player algorithm with convergence guarantees. To avoid the computational burden of solving the hard-constrained distributionally robust problem directly, we pass to its Lagrangian as a proxy and derive a four-player primal-dual formulation involving the leader policy, follower policy, adversarial preference kernel, and dual variable. We further provide a detailed duality and convergence analysis: we characterize when the four-player game exactly or approximately recovers the original hard-constrained robust objective, and we prove that the proposed algorithm converges to a saddle point of the proxy robust game at a rate of ⁡(1/T)O(1/ T). Under the stated exactness conditions, this yields a certified worst-case game-value guarantee. Experimental validation. We validate our algorithm on controlled tabular games, a best-of-m reranking task on IMDb continuations with heterogeneous annotator groups, and a summarization task on TL;DR using LLMs. Across all settings, the robust policy achieves substantially higher worst-case win rates than the non-robust NashMD baseline, and performs better alignment under perturbed preferences from the ambiguity set in the IMDb task, and enjoys better generalizability to external LLM judges in the summarization task, while maintaining comparable nominal performance. 2 Preliminaries Notation. Let X denote the prompt space, and let D0D_0 be the deployment-time prompt distribution. For each prompt x∈x , let Y⁡(x)=y1,…,ymY(x)=\y_1,…,y_m\ be the finite response set. A policy is a mapping π:→Δ⁡(Y)π:X→ (Y), where Δ⁡(Y⁡(x)) (Y(x)) denotes the simplex over responses for prompt x. Given x, a response y is sampled from π(⋅∣x)π(· x). We also fix a reference policy πref(⋅∣x) _ ref(· x) with full support, which can be derived from Supervised Fine-Tuning (SFT), and a KL regularization parameter τ>0τ>0. Reward-based alignment. A standard approach to preference-based alignment assumes that pairwise human preferences are generated from an underlying scalar reward function R⋆(x,y)R (x,y). Under the classical Bradley-Terry / logistic preference model [4, 9, 33], the probability that response y is preferred to y′y under prompt x is ℙ⁡(y≻y′∣x)=exp⁡(R⋆(x,y))exp⁡(R⋆(x,y))+exp⁡(R⋆(x,y′)).P(y y x)= (R (x,y)) (R (x,y))+ (R (x,y )). (1) Given a dataset with pairwise responses and preferences, one can first learn a reward model R through regression, and maximize the classical KL-regularized RLHF objective JRB(π;R):=x∼D0[y∼π(⋅∣x)[R(x,y)]−τDKL(π(⋅∣x)∥πref(⋅∣x))].J_ RB(π;R):=E_x D_0 [E_y π(· x)[R(x,y)]-τ D_KL (π(· x)\,\|\, _ ref(· x) ) ]. (2) The reward-based alignment problem is then πR⋆∈arg⁡maxπ∈ΠJRB(π,R), _R ∈ _π∈ J_ RB(π;R), where Π is the policy class of interest. The KL term discourages the learned policy from drifting arbitrarily far from the reference policy [41, 33, 35], and it results in a closed-form solution to (2) as πR⋆(y∣x)∝πref(y∣x)exp⁡(1τR(x,y)), _R (y x) _ ref(y x) ( 1τR(x,y) ), (3) which underlies the derivation of the direct preference optimization method [35]. General preference alignment. A limitation of (1) is that it assumes all pairwise preferences are induced by a single scalar reward function. More generally, one can directly treat pairwise preference probabilities as primitive objects, without assuming the latent reward [29, 52, 54]. Definition 1. A general preference oracle is a mapping P:×Y×Y→[0,1]P:X× Y× Y→[0,1] such that, for every prompt x∈x and responses y,y′∈Y⁡(x)y,y ∈ Y(x), there exists a preference z∼Ber⁡(P⁡(y≻y′∣x)),z (P(y y x) ), where z=1z=1 indicates that y is preferred to y′y , and z=0z=0 indicates the opposite. Assumption 1 (Antisymmetric preference model). For any fixed prompt x, define Px(y≻y′):=P⁡(y≻y′∣x)P_x(y y ):=P(y y x). We assume that the preference probabilities are antisymmetric, i.e., Px(yi≻yj)+Px(yj≻yi)=1,∀yi≠yj.P_x(y_i y_j)+P_x(y_j y_i)=1, ∀\,y_i≠ y_j. (4) Given two policies π,ν∈Ππ,ν∈ , Nash learning from human feedback (NLHF) considers the KL-regularized game payoff under the general preference kernel P: J⁡(π,ν,P) J(π,ν;P) :=x∼D0[y∼π(⋅∣x),y′∼ν(⋅∣x)[Px(y≻y′)]−τDKL(π(⋅∣x)∥πref(⋅∣x)) :=E_x D_0 [E_y π(· x),\;y ν(· x) [P_x(y y ) ]-τ D_KL (π(· x)\,\|\, _ ref(· x) ) +τDKL(ν(⋅∣x)∥πref(⋅∣x))]. +τ D_KL (ν(· x)\,\|\, _ ref(· x) ) ]. (5) Here, π is the max-player and ν is the min-player. The interpretation is that both players seek a policy that is consistently preferred to any competitor, while both players are regularized toward the same reference policy. The nominal general-preference alignment problem is therefore formulated as a regularized minimax game (π⋆,ν⋆)∈arg⁡maxπ∈Πargminν∈Π⁡J⁡(π,ν,P).(π ,ν )∈ _π∈ _ν∈ J(π,ν;P). (6) Due to the symmetric and convex-concave game structure, (6) has a unique symmetric Nash equilibrium (π⋆,π⋆)(π ,π ) [29]. NLHF then aims to find this policy π⋆π as the alignment policy. 3 Robust Game Formulation for Uncertain Preference Alignment In this section, we introduce our formulation of a robust game to address uncertainty in preferences. For notational simplicity, we fix a prompt and suppress the prompt index. The global prompt-averaged objective is obtained by taking expectations under a prompt-rectangular ambiguity model. Let Y=1,…,mY=\1,…,m\ be a finite response set, and d=(m2)d= m2, and set E:=(i,j):1≤i<j≤m.E:=\(i,j):1≤ i<j≤ m\. For p=(pij)(i,j)∈E∈[0,1]dp=(p_ij)_(i,j)∈ E∈[0,1]^d, we define the preference kernel that satisfies Assumption 1 as Pp(i≻j)=pij,i<j,1−pji,i>j,0.5,i=j.P_p(i j)= casesp_ij,&i<j,\\ 1-p_ji,&i>j,\\ 0.5,&i=j. cases We model the potential uncertainty in the preference kernel through the Bernoulli-KL divergence. Namely, centered at a nominal kernel p⋆∈(0,1)dp ∈(0,1)^d, the Bernoulli-KL divergence is defined as DKLBern(p∥p⋆):=∑(i,j)∈E[pijlogpijpij⋆+(1−pij)log1−pij1−pij⋆],D_KL^Bern(p\|p ):= _(i,j)∈ E [p_ij p_ijp _ij+(1-p_ij) 1-p_ij1-p _ij ], (7) measuring the difference between the two kernels. We then set the ambiguity set as :=p:DKLBern(p∥p⋆)≤ρ, :=\p:D_KL^Bern(p\|p )≤ρ\, (8) where ρ is some uncertainty radius, quantifying the uncertainty level. Namely, the ambiguity set contains the kernels whose differences from the nominal one are bounded by ρ. Remark 1. The ambiguity set P captures preference kernels that are close to a nominal estimate p⋆p , and thus reflects uncertainty arising from finite, noisy, or heterogeneous preference data. The radius ρ is generally chosen by domain experts or users, reflecting the users’ consideration level of uncertainty. For instance, in offline preference-based learning, one may estimate p⋆p from a dataset of pairwise comparisons and choose the radius ρ via concentration inequalities [57] so that P contains all statistically plausible kernels; When preferences are aggregated across different users or annotator populations, P can model the resulting variability in pairwise comparisons. Unlike standard robust optimization settings where uncertainty sets are often rectangular and decouple across components [48, 20], our set P induces coupling across all pairwise comparisons. As a result, the set is inherently non-rectangular, which significantly complicates the problem and is a key source of computational hardness in our formulation. Objective. With the ambiguity set defined in (8), we propose our robust formulation as maxπ∈Δm⁡minν∈Δmminp∈⁡J⁡(π,ν,p), _π∈ _m _ν∈ _m _p J(π,ν;p), (9) where J⁡(π,ν,p)J(π,ν;p) is the game value under the specific kernel p: J(π,ν;p):=∑i=1m∑j=1mπiνjPp(i≻j)−τDKL(π∥πref)+τDKL(ν∥πref).J(π,ν;p):= _i=1^m _j=1^m _i _j\,P_p(i j)-τ D_KL(π\| _ref)+τ D_KL(ν\| _ref). (10) Namely, we enable the adversarial player to further choose a preference kernel as part of its strategy, and the main player needs to maximize its winning rate under the worst-case, i.e., the least favorable preference kernel. This consideration of the worst-case then provides a lower bound guarantee. Namely, if π⋆π solves (9), then for any p∈p and any adversarial policy ν, one has (ignoring the regularizer) an optimized lower bound on the winning rate under p and against ν: Pp(π⋆≻ν)≥minp∈,ν⁡Pp(π⋆≻ν)=maxπ⁡minp∈,νPp(π≻ν)= optimal value of (9). P_p(π ν)≥ _p ,νP_p(π ν)= _π _p ,νP_p(π ν)= optimal value of eq:obj. (11) Therefore, the maximizer π⋆π of our robust value (9) provides a baseline performance of uncertain alignment, thus improving the robustness and reliability against preference uncertainty. 4 Four-Player Proxy Game In this section, we aim to design a concrete algorithm to solve the robust game (9). For standard, non-robust NLHF, where the kernel is fixed and the only strategic interaction is between the two players, the optimization landscape is that of a regularized two-player game, which can be handled by no-regret and optimistic first-order methods. However, our robust formulation is qualitatively more difficult. The major challenge arises from the fact that, even for a fixed leader policy π, the inner value gH(π):=minν,p⁡J⁡(π,ν,p)g_H(π):= _ν,pJ(π,ν;p) is difficult to compute exactly because ν and p are coupled through the win-rate term. If one first fixes π,pπ,p and eliminates the adversarial policy ν⋆ν using the Gibbs response, the reduced objective becomes minp∈−τDKL(π∥πref)−τlog∑j=1mπref(j)exp(−b⁡(j,π,p)τ), _p \-τ D_KL(π\| _ ref)-τ _j=1^m _ ref(j) (- b(j;π,p)τ ) \, (12) where b⁡(j,π,p)=∑i=1mπ⁡(i∣x)Pp(i≻j)b(j;π,p)= _i=1^mπ(i x)P_p(i j) is affine in p. Since the log-sum-exp is convex and appears with a negative sign, the reduced objective is concave in p. However, minimizing a concave function over a convex uncertainty set is NP-hard in general [18]. On the other hand, if one fixes π,νπ,ν first and solves for the worst-case kernel p⋆p , since the uncertainty set P is defined with correlation among response pairs, P is not rectangular over all (i,j)(i,j) pairs; thus, solving the distributionally robust optimization problem can still be inefficient or NP-hard [48]. Moreover, standard NLHF involves only two strategic blocks (π,ν)(π,ν), yet our robust game has another player controlling p. Although one can combine the two minimizing players as a single adversary who takes the joint action (ν,p)(ν,p), it breaks the standard two-player saddle problem structure in NLHF, losing all of its properties like the minimax theorem or unique Nash equilibrium [29]. The resulting game need not be monotone in the Euclidean sense, which makes naive simultaneous gradient methods unstable. Due to these challenges, we develop a proxy game formulation that is easier to solve and develop detailed studies on the connection between the proxy and the original robust game. For regularity, we first assume the main player policy is constrained to the floor-truncated simplex Πϵ:=π∈Δm:πi≥ϵπref,i∀i _ε:= \π∈ _m:\ _i≥ε\, _ref,i\ \ ∀ i \ for some positive value ϵε. Specifically, one can choose ϵε arbitrarily small to approximately recover the probability simplex Δm _m. For each π∈Πϵπ∈ _ε, denote the original inner problem value with the hard constraint p∈p as gH(π,ρ):=minν∈Δm⁡minp∈⁡J⁡(π,ν,p),g_H(π;ρ):= _ν∈ _m _p J(π,ν;p), (13) and the corresponding optimal robust value as V⁡(ρ):=maxπ∈Πϵ⁡gH(π,ρ).V(ρ):= _π∈ _εg_H(π;ρ). (14) We further define an interior box Pδ:=[δ,1−δ]dP_δ:=[δ,1-δ]^d. with some δ∈(0,1/2)δ∈(0,1/2) due to regularity, and consider the truncated hard objective: gHδ(π,ρ):=minν∈Δm⁡minp∈Pδ∩⁡J⁡(π,ν,p),g_H^δ(π;ρ):= _ν∈ _m _ subarraycp∈ P_δ subarrayJ(π,ν;p), (15) and denote its optimal value as Vδ(ρ):=maxπ∈Πϵ⁡gHδ(π,ρ).V^δ(ρ):= _π∈ _εg_H^δ(π;ρ). (16) We first show that the truncation error is explicit and uniformly controlled. Theorem 1 (Uniform approximation by the truncated hard objective). Assume p⋆∈Pδp ∈ P_δ. Then, for every π∈Πϵπ∈ _ε, 0≤gHδ(π,ρ)−gH(π,ρ)≤δ0≤ g_H^δ(π;ρ)-g_H(π;ρ)≤δ and 0≤Vδ(ρ)−V⁡(ρ)≤δ.0≤ V^δ(ρ)-V(ρ)≤δ. Moreover, if πδ⋆∈argmaxπ∈Πϵ⁡gHδ(π,ρ) _δ ∈ _π∈ _εg_H^δ(π;ρ), then gH(πδ⋆,ρ)≥V⁡(ρ)−δ.g_H( _δ ;ρ)≥ V(ρ)-δ. This theorem shows that the error introduced by the regularity relaxation is bounded by δ. Thus, one can choose δ small enough to ensure regularity and approximate the original solution arbitrarily. Therefore, we will focus on solving this problem, arg⁡maxπ∈ΠϵgHδ(π,ρ) _π∈ _εg_H^δ(π;ρ). Since gHδg^δ_H is a hard-constrained problem, which can be inefficient to solve directly, we introduce a Lagrangian dual variable and consider the Lagrangian function. Specifically, with an additional dual variable γ, we define and consider the four-player payoff as F(π,γ,ν,p):=J(π,ν;p)+γ(DKLBern(p∥p⋆)−ρ).F(π,γ,ν,p):=J(π,ν;p)+γ (D_KL^Bern(p\|p )-ρ ). (17) Moreover, fix a compact positive interval Γ:=[γmin,γmax]⊂(0,∞) :=[ _ , _ ]⊂(0,∞) and β:=ν∈Δm:νj≥βπref,j∀jN_β:= \ν∈ _m:\ _j≥β\, _ref,j\ \ ∀ j \ for β∈(0,1)β∈(0,1), we define the proxy objective as gproxyδ(π,ρ):=maxγ∈Γ⁡minν∈β,p∈Pδ⁡F⁡(π,γ,ν,p),g_proxy^δ(π;ρ):= _γ∈ _ν _β,\;p∈ P_δF(π,γ,ν,p), (18) and its optimal value as Vproxyδ(ρ):=maxπ∈Πϵ⁡gproxyδ(π,ρ).V_proxy^δ(ρ):= _π∈ _εg_proxy^δ(π;ρ). (19) We then study the proxy’s connection to the original problem. Firstly, we introduce a regularity assumption. Assumption 2. We assume the following regularization conditions hold: (1) p⋆∈(δ,1−δ)dp ∈(δ,1-δ)^d; (2) 0<β≤e−1/τ0<β≤ e^-1/τ; (3) Γ=[γmin,γmax]⊂(0,∞) =[ _ , _ ]⊂(0,∞) with γmax≥1/ρ _ ≥ 1/ρ; (4). τγmin≥14τ _ ≥ 14. Assumption 2-(1)–(3) are mild regularity conditions that can be enforced by suitable algorithmic choices. Assumption 2-(4) is a curvature condition needed for the proxy-game convergence analysis. We then derive the following results for the connections. Theorem 2. Under (1)-(3) of Assumption 2, for every π∈Πϵπ∈ _ε, the proxy value always lower bounds the hard-constrained value: gHδ(π,ρ)=minν∈β⁡maxγ∈[0,γmax]minp∈Pδ⁡F⁡(π,γ,ν,p)≥gproxyδ(π,ρ).g_H^δ(π;ρ)= _ν _β _γ∈[0, _ ] _p∈ P_δF(π,γ,ν,p)≥ g_proxy^δ(π;ρ). (20) If we additionally assume (4) of Assumption 2, then for every π∈Πϵπ∈ _ε, 0≤gHδ(π,ρ)−gproxyδ(π,ρ)≤ργmin, and 0≤Vδ(ρ)−Vproxyδ(ρ)≤ργmin.0≤ g_H^δ(π;ρ)-g_proxy^δ(π;ρ)≤ρ _ , and 0≤ V^δ(ρ)-V_proxy^δ(ρ)≤ρ _ . (21) Moreover, if Assumption 3 (in Appendix) additionally holds, then the two objectives are equivalent: gHδ(π,ρ)=gproxyδ(π,ρ), and Vδ(ρ)=Vproxyδ(ρ).g_H^δ(π;ρ)=g_proxy^δ(π;ρ), and V^δ(ρ)=V_proxy^δ(ρ). (22) This theorem characterizes the comprehensive connections between our proxy value gHδg^δ_H and the hard-constrained ones. Firstly, we showed that under the mildest conditions (1)-(3) of Assumption 2, the proxy value is always a lower bound of gHδg^δ_H. Thus, any solution to the proxy also provides an optimized lower bound to the hard-constrained problem. Moreover, if Assumption 2-(4) holds, the error between gHδg^δ_H and gproxyδg_proxy^δ is then bounded by ργminρ _ . Thus, one can set the values of γmin _ and the temperature τ to satisfy Assumption 2-(4) and to guarantee that the proxy is an accurate approximation of the original problem: recall the error between gproxyδg_proxy^δ and the original problem gHg_H is up to δ+ργminδ+ρ _ , which can be properly small. Moreover, Assumption 3 is a stronger exactness condition used only to upgrade the proxy guarantee to the truncated hard-constrained objective. If it holds, which assumes a uniformly lower bound of the maximizer γ⋆γ under any (π,ν)(π,ν), the error ργminρ _ diminishes and the proxy is exactly equivalent to the hard-constrained gHδg^δ_H. The advantage of studying the proxy is that the proxy has a much better structure and is easier to design convergent algorithms. Specifically, the hard-constrained problem gHδg^δ_H is a min−max−min - - problem per (20), and maximizing over π further makes it a four-layer max−min−max−min - - - problem, which is significantly challenging and unstable to solve. The proxy exchanges the order of maxγ _γ and minν _ν, making maxπ⁡gproxyδ _πg_proxy^δ a bi-level maxπ,γ−minp,ν _π,γ- _p,ν problem. Such a max−min - is much easier to solve, and we can view it as a two-player zero-sum game and seek for its saddle point. In the next section, we design a concrete algorithm based on this understanding, and develop its convergence analysis. 5 Algorithm Design and Convergence Analysis In this section, we develop our concrete algorithm for solving the proxy game. Specifically, we treat each parameter as a player, and develop a four-player gradient-based algorithm. As we discussed previously, the structure of the proxy enables us to view it as a max−min - problem. Also, since the robust games are non-monotone and naive simultaneous gradient approaches can be unstable, we thus propose to adapt optimistic or extra-gradient approaches [7, 37, 42, 27, 28, 47]. Specifically, we aim to solve the proxy max−min - problem maxx∈X⁡miny∈Y⁡F⁡(x,y),x=(π,γ)∈X:=Πϵ×Γ,y=(ν,p)∈Y:=β×Pδ, _x∈ X _y∈ YF(x,y), x=(π,γ)∈ X:= _ε× ,\ y=(ν,p)∈ Y:=N_β× P_δ, by a single-loop optimistic mirror descent/ascent method. We use entropy geometry for the simplex blocks, Euclidean geometry for γ, and binary entropy geometry for p. The associated Bregman divergences are: Dπ(π,π′)=DKL(π∥π′),D_π(π,π )=D_KL(π\|π ), Dγ(γ,γ′)=12(γ−γ′)2D_γ(γ,γ )= 12(γ-γ )^2, Dν(ν,ν′)=DKL(ν∥ν′)D_ν(ν,ν )=D_KL(ν\|ν ), and Dp(p,p′)=DKLBern(p∥p′).D_p(p,p )=D_KL^Bern(p\|p ). At iteration t, let zt=(πt,γt,νt,pt)∈X×Y,gt∙:=∇∙F(zt),z_t=( _t, _t, _t,p_t)∈ X× Y, g_t := _ F(z_t), and define g^t∙:=2gt∙−gt−1∙, g_t :=2g_t -g_t-1 , with ∙∈π,γ,ν,p. ∈\π,γ,ν,p\. Then, we update each player through mirror descent (exact update rules are deferred to Appendix (25)-(28)) and present our algorithm in Algorithm 1. Since the mirror maps are separable across blocks, these coordinate-wise updates are exactly optimistic mirror descent/ascent on the grouped saddle problem maxx∈X⁡miny∈Y⁡F⁡(x,y) _x∈ X _y∈ YF(x,y). Algorithm 1 Four-player optimistic gradient descent/ascent (OGDA) 1: step size η>0η>0, iteration budget T, initial point z1=(π1,γ1,ν1,p1)∈X×Yz_1=( _1, _1, _1,p_1)∈ X× Y 2: Compute g1∙=∇∙F(z1)g_1 = _ F(z_1) for all blocks, and set g0∙←g1∙g_0 ← g_1 3: for t=1,…,Tt=1,…,T do 4: Form optimistic gradients g^t∙=2gt∙−gt−1∙ g_t =2g_t -g_t-1 5: Update (πt+1,γt+1,νt+1,pt+1)( _t+1, _t+1, _t+1,p_t+1) by (25)-(28) 6: if t<Tt<T then 7: Compute gt+1∙=∇∙F(zt+1)g_t+1 = _ F(z_t+1) for all blocks 8: end if 9: end for 10: Return the averaged iterate z¯T=(π¯T,γ¯T,ν¯T,p¯T):=1T∑t=1T(πt,γt,νt,pt). z_T=( π_T, γ_T, ν_T, p_T):= 1T _t=1^T( _t, _t, _t,p_t). We then develop the convergence analysis of our algorithm in the following theorem. Theorem 3 (Convergence and robust guarantee of four-player OGDA). Assume Assumption 2, and run Algorithm 1 with the constant step size η=18Lδ,η= 18L_δ, where Lδ=⁡(1/δ)L_δ=O(1/δ) is the constant from Lemma 3. Define DXdiam≜sup(π,γ)∈XDKL(π∥π1)+12(γ−γ1)2,D_X^diam _(π,γ)∈ X \D_KL(π\| _1)+ 12(γ- _1)^2 \, and DYdiam≜sup(ν,p)∈YDKL(ν∥ν1)+DKLBern(p∥p1).D_Y^diam _(ν,p)∈ Y \D_KL(ν\| _1)+D_KL^Bern(p\|p_1) \. Then the duality gap of the averaged iterate z¯T=(x¯T,y¯T)=((π¯T,γ¯T),(ν¯T,p¯T)) z_T=( x_T, y_T)= (( π_T, γ_T),( ν_T, p_T) ) satisfies Gap⁡(z¯T):=maxx′∈X⁡F⁡(x′,y¯T)−miny′∈Y⁡F⁡(x¯T,y′)≤8Lδ(DXdiam+DYdiam)T, ( z_T):= _x ∈ XF(x , y_T)- _y ∈ YF( x_T,y )≤ 8L_δ (D_X^diam+D_Y^diam )T, and the averaged main policy π¯T π_T satisfies V⁡(ρ)−gH(π¯T,ρ)≤8Lδ(DXdiam+DYdiam)T+ργmin+δ.V(ρ)-g_H( π_T;ρ)≤ 8L_δ (D_X^diam+D_Y^diam )T+ρ\, _ +δ. Furthermore, setting δ=⁡(1/T)δ=O(1/ T), yields an ⁡(1T+ργmin)O ( 1 T+ρ\, _ )-optimal policy for the original robust game. Additionally, under Assumption 3, π¯T π_T improves to ⁡(1/T)O (1/ T )-optimality. Our OGDA algorithm thus converges to a saddle point of the proxy game, which is also a near-optimal solution to our robust objective (9). Therefore, our algorithm effectively solves our robust alignment problem, enhancing the robustness and reliability of Nash learning based alignment. A detailed comparison of our approach with standard NLHF is developed in Appendix C. 6 Experiment We then evaluate our four-player OGDA algorithm across three settings of increasing scale: (i) a synthetic tabular game that isolates the fragile-consensus phenomenon, (i) a best-of-m reranking task on the IMDb dataset with heterogeneous annotator groups, and (i) a summarization task with LLM-generated responses and external LLM judgments. 6.1 Tabular Game We first consider a collection of independently generated tabular prompts, each with a finite response set of size m=3m=3. For each prompt x, we construct a fragile-consensus preference kernel by drawing a slim margin sx∼Unif⁡[0.015,0.040]s_x [0.015,0.040] and a solid margin Sx∼Unif⁡[0.25,0.40]S_x [0.25,0.40], and setting Px⋆=(1/21/2+sx1/2−Sx1/2−sx1/21/2+Sx1/2+Sx1/2−Sx1/2).P _x\;=\; pmatrix1/2&1/2+s_x&1/2-S_x\\[3.0pt] 1/2-s_x&1/2&1/2+S_x\\[3.0pt] 1/2+S_x&1/2-S_x&1/2 pmatrix. (23) We then apply a random permutation of the response labels so that the identity of the fragile action varies across prompts. This construction produces a cyclic preference game with one easily reversible edge (KL cost ≈2sx2≤0.003≈ 2s_x^2≤ 0.003) and two expensive-to-reverse edges (KL cost ≈2Sx2/(Sx(1−Sx))≥0.5≈ 2S_x^2/(S_x(1-S_x))≥ 0.5), which is precisely the regime in which robustness matters most: the adversary can flip the cheap edge within a small budget ρ, reversing the dominance ordering, while the expensive edges are untouchable. We use a uniform reference policy and set τ=0.05τ=0.05. We compare: (i) NashMD, the nominal two-player mirror-descent baseline operating on the fixed kernel Px⋆P _x; (i) Four-Player OGDA with training radii ρ∈0.10,0.20,0.40ρ∈\0.10,0.20,0.40\; and (i) the optimal robust value V⁡(ρ=0.2)V(ρ=0.2), computed via low-dimensional global search. Both NashMD and OGDA are initialized at πref _ref and trained for 2000 iterations. At each step t, we evaluate gH(πt,ρ=0.2)g_H( _t;ρ=0.2)—the worst-case game value under the Bernoulli-KL ball of radius 0.20.2. Figure 1 (left) shows the convergence of gH(πt,0.2)g_H( _t;0.2) over training iterations. NashMD converges to a fixed value that is strictly below the optimal robust value V⁡(0.2)V(0.2), confirming that the nominal Nash policy is suboptimal for the robust objective. The four-player OGDA at ρ=0.2ρ=0.2 converges to a value close to V⁡(0.2)V(0.2), validating the convergence guarantee. At ρ=0.1ρ=0.1 (under-budgeted), the algorithm converges but to a lower value, since it optimizes for a smaller perturbation set. At ρ=0.4ρ=0.4 (over-budgeted), the policy is more conservative and also falls below the ρ=0.2ρ=0.2 optimum. Figure 1 (right) shows gH(π,ρeval)g_H(π; _eval) as a function of the evaluation radius for the final policies. The four-player OGDA policies uniformly dominate NashMD at every positive ρeval _eval, with the advantage growing as the evaluation radius increases. This further confirms the enhanced robustness of our approach against preference uncertainty. Figure 1: Experiments under Tabular Games 6.2 Best-of-m IMDb Continuation Task We next validate our algorithm on a best-of-m reranking task, where candidate responses are generated by a pretrained LLM and preference kernels are constructed from heterogeneous annotator groups. Details of our training procedure, hyperparameters, and additional results are deferred to Appendix E. For each of N=8000N=8000 prompts from the IMDb review dataset [23], we pre-generate m=8m=8 candidate continuations using google/flan-t5-large [11]. We generate the preferences based on three annotator groups: Comprehensive/analytical (p1)(p_1) prefers detailed, well-structured responses (favors length 300–600 characters); Efficient/concise (p2)(p_2) prefers short, direct responses (favors length 50–120 characters); Balanced/nuanced (p3)(p_3) prefers hedged, multi-perspective responses (favors hedging keywords). We set the nominal kernel as the uniform mixture of the three preferences: Pp⋆(a≻b)=(∑i=13Pi(a≻b))/3P_p (a b)=(Σ^3_i=1P_i(a b))/3, and construct the uncertainty set centered at it as ρeval≜(w1,w2,w3)∈Δ3:DKL((w1,w2,w3)||(1/3,1/3,1/3))≤ρevalP_ _eval \(w_1,w_2,w_3)∈ _3:D_KL((w_1,w_2,w_3)||(1/3,1/3,1/3))≤ _eval\ to model preference uncertainty. We then train and compare the robust policy πR _R (trained at radius ρtrain _train) against the NashMD baseline πN _N by their pairwise win rate under perturbed kernels. For each evaluation radius ρeval _eval, we sample M=600M=600 kernels Pjj=1M\P_j\_j=1^M uniformly from ρevalP_ _eval and compute the win rate: WinRate⁡(πR vs πN,ρeval):=|j∈[M]:Ppj(πR≻πN)≥0.5|/M,WinRate( _R vs _N;\, _eval)\;:=\; |\j∈[M]:P_p_j( _R _N)≥ 0.5\ | /M, (24) where Ppj(πR≻πN)=∑i,kπR(i)πN(k)pj(i≻k)P_p_j( _R _N)= _i,k _R(i)\, _N(k)\,p_j(i k) is the probability that a response drawn from πR _R is preferred over one drawn from πN _N under kernel pjp_j. A win rate above 50%50\% means the robust policy is preferred to NashMD under the majority of plausible kernels. Table 1 reports the win rate of the robust policy against NashMD, averaged over all 80008000 prompts, for training radii ρtrain∈0.01,0.02,0.05,0.10,0.15 _train∈\0.01,0.02,0.05,0.10,0.15\ and evaluation radii ρeval∈0.05,0.10,0.20,0.30,0.50 _eval∈\0.05,0.10,0.20,0.30,0.50\. Table 1: Win rate against NashMD under random kernel perturbations, averaged over N=8000N=8000 IMDb prompts with m=8m=8 candidates. Each entry reports WinRate⁡(πR vs πN,ρeval)WinRate( _R vs _N;\, _eval) from (24). Values above 0.500.50 indicate the robust policy is preferred under the majority of sampled kernels. Evaluation radius ρeval _eval ρtrain _train 0.050.05 0.100.10 0.200.20 0.300.30 0.500.50 0.01 0.986 0.967 0.927 0.894 0.844 0.02 0.985 0.965 0.925 0.891 0.841 0.05 0.984 0.961 0.918 0.883 0.832 0.10 0.978 0.949 0.901 0.866 0.814 0.15 0.934 0.894 0.844 0.810 0.766 As the results shown, our robust policy outperforms standard NashMD uniformly, indicating that the robust policy is preferred under most sampled kernels that are different from the nominal training preference. Therefore, our method substantially improves robustness to preference uncertainty and out-of-distribution generalization. 6.3 Summarization Task with LLMs We further evaluate our performance on the TL;DR summarization task [41]. The nominal preference kernel is a non-Bradley-Terry (non-BT) pairwise judge, Pϕpair(ya≻yb∣x)=σ⁡(Lpair(x,ya,yb))P^pair_φ(y_a y_b x)=σ (L^pair(x,y_a,y_b) ), instantiated with PairRM [21]: since each candidate’s score is conditioned on its opponent, the kernel is not constrained to admit any scalar reward decomposition. On real model outputs, the closest Bradley-Terry fit to this kernel leaves a mean residual of 0.3840.384 logits per edge (3.103.10 percentage points), with 1.4%1.4\% of triples outright cyclic, confirming the nominal kernel used here is measurably non-BT (construction and diagnostic in Appendix F.6). To evaluate robustness under preference uncertainty, we use three external LLMs from different model families: DeepSeek-R1-Distill-Qwen-32B [12], Gemma-4-31B [44], and Nemotron-3-Nano-Omni [32], as judgment at evaluation time. These external LLMs are instructed to judge the summary with focus on accuracy, coherence, conciseness, and helpfulness to the reader. Since each external judge will induce its own pairwise preference (from their own reasoning) that differs from the nominal one, these judgments simulate a deployment scenario where the user population’s preferences diverge from the training-time annotation distribution. Details of the experiments are deferred to Appendix F. The results are presented in Table 2. Most external judges (five of six configurations) yield a win rate above 0.500.50 over NashMD, confirming that the robust policy remains competitive under this non-BT nominal kernel even though, unlike prior work, our framework does not require a Bradley-Terry reward model. Appendix F.7 reports additional results with a Bradley-Terry nominal kernel across both fine-tuning backbones. Table 2: Win rate against NashMD under external LLM judges, evaluated on 10001000 TL;DR prompts, with a non-BT (PairRM) nominal kernel. Fine-Tuning Model ρtrain _train Win Rate under Evaluation Model DeepSeek-R1 Gemma-4 Nemotron-3 Qwen-1.5B 0.01 0.708 0.577 0.492 0.05 0.625 0.615 0.732 Summary. Across all three settings, the experiments confirm our central theoretical prediction: the four-player OGDA algorithm produces policies that are strictly more robust than the non-robust NashMD baseline, as the preference differs from the training ones. 7 Conclusion We addressed the sensitivity of preference-based alignment to uncertainty in the pairwise preference kernel by proposing Robust Nash Alignment, which optimizes the worst-case win rate over a divergence-based ambiguity set and provides a principled robustness guarantee. To overcome the computational challenges of the resulting hard-constrained robust objective, we introduced a tractable four-player primal-dual proxy game and established its approximation properties and convergence via a single-loop optimistic mirror descent-ascent algorithm. Empirically, our method improves robustness to preference perturbations across tabular and LLM settings while maintaining strong nominal performance. Overall, our results demonstrate that explicitly modeling preference uncertainty is a principled and practical approach to achieving reliable Nash-style alignment. References [1] M. Aghassi and D. Bertsimas. Robust game theory. Mathematical programming, 107(1):231–273, 2006. [2] S. M. Ananthanarayanan and C. Kroer. Computing the Optimal Distributionally-Robust Strategy to Commit To. arXiv preprint arXiv:2209.07647, 2022. [3] M. G. Azar, Z. D. Guo, B. Piot, R. Munos, M. Rowland, M. Valko, and D. Calandriello. A General Theoretical Paradigm to Understand Learning from Human Preferences. In International Conference on Artificial Intelligence and Statistics, pages 4447–4455. PMLR, 2024. [4] R. A. Bradley and M. E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324–345, 1952. [5] D. Calandriello, D. Guo, R. Munos, M. Rowland, Y. Tang, B. A. Pires, P. H. Richemond, C. L. Lan, M. Valko, T. Liu, et al. Human Alignment of Large Language Models through Online Preference Optimisation. arXiv preprint arXiv:2403.08635, 2024. [6] S. Chakraborty, J. Qiu, H. Yuan, A. Koppel, F. Huang, D. Manocha, A. S. Bedi, and M. Wang. MaxMin-RLHF: Alignment with Diverse Human Preferences. arXiv preprint arXiv:2402.08925, 2024. [7] C.-K. Chiang, T. Yang, C.-J. Lee, M. Mahdavi, C.-J. Lu, R. Jin, and S. Zhu. Online Optimization with Gradual Variations. In Conference on Learning Theory, pages 6–1. JMLR Workshop and Conference Proceedings, 2012. [8] K. Chidambaram, K. V. Seetharaman, and V. Syrgkanis. Direct Preference Optimization with Unobserved Preference Heterogeneity: The Necessity of Ternary Preferences. arXiv preprint arXiv:2510.15716, 2025. [9] P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei. Deep Reinforcement Learning from Human Preferences. Advances in neural information processing systems, 30, 2017. [10] X. Chu, Z. Zhang, T. Jia, and Y. Jin. Stackelberg Self-Annotation: A Robust Approach to Data-Efficient LLM Alignment. arXiv preprint arXiv:2502.18099, 2025. [11] H. W. Chung, L. Hou, S. Longpre, B. Zoph, Y. Tay, W. Fedus, E. Li, X. Wang, M. Dehghani, S. Brahma, A. Webson, S. S. Gu, Z. Dai, M. Suzgun, X. Chen, A. Chowdhery, S. Narang, G. Mishra, A. Yu, V. Zhao, Y. Huang, A. Dai, H. Yu, S. Petrov, E. H. Chi, J. Dean, J. Devlin, A. Roberts, D. Zhou, Q. V. Le, and J. Wei. Scaling Instruction-Finetuned Language Models, 2022. [12] DeepSeek-AI. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning, 2025. [13] K. Ethayarajh, W. Xu, N. Muennighoff, D. Jurafsky, and D. Kiela. KTO: Model Alignment as Prospect Theoretic Optimization. arXiv preprint arXiv:2402.01306, 2024. [14] H. Furuta, K.-H. Lee, S. S. Gu, Y. Matsuo, A. Faust, H. Zen, and I. Gur. Geometric-Averaged Preference Optimization for Soft Preference Labels. Advances in Neural Information Processing Systems, 37:57076–57114, 2024. [15] J. Hejna, R. Rafailov, H. Sikchi, C. Finn, S. Niekum, W. B. Knox, and D. Sadigh. Contrastive Preference Learning: Learning from Human Feedback without RL. arXiv preprint arXiv:2310.13639, 2023. [16] I. Hong, Z. Li, A. Bukharin, Y. Li, H. Jiang, T. Yang, and T. Zhao. Adaptive Preference Scaling for Reinforcement Learning with Human Feedback. Advances in Neural Information Processing Systems, 37:107249–107269, 2024. [17] J. Hong, N. Lee, and J. Thorne. ORPO: Monolithic Preference Optimization without Reference Model. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 11170–11189, 2024. [18] R. Horst and H. Tuy. Global Optimization: Deterministic Approaches. Springer Science & Business Media, 2013. [19] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen. LoRA: Low-Rank Adaptation of Large Language Models, 2021. [20] G. N. Iyengar. Robust Dynamic Programming. Mathematics of Operations Research, 30(2):257–280, 2005. [21] D. Jiang, X. Ren, and B. Y. Lin. LLM-Blender: Ensembling large language models with pairwise ranking and generative fusion. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14165–14178, Toronto, Canada, 2023. Association for Computational Linguistics. [22] Y. Liu, H. Xu, S.-J. S. Yang, and J. Zhang. Distributionally robust equilibrium for continuous games: Nash and Stackelberg models. European Journal of Operational Research, 265(2):631–643, 2018. [23] A. L. Maas, R. E. Daly, P. T. Pham, D. Huang, A. Y. Ng, and C. Potts. Learning Word Vectors for Sentiment Analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 142–150, Portland, Oregon, USA, June 2011. Association for Computational Linguistics. [24] D. Mandal, P. Sasnauskas, and G. Radanovic. Distributionally Robust Reinforcement Learning with Human Feedback. arXiv preprint arXiv:2503.00539, 2025. [25] J. Mcmahan, G. Artiglio, and Q. Xie. Roping in uncertainty: Robustness and regularization in markov games. In International Conference on Machine Learning, pages 35267–35295. PMLR, 2024. [26] Y. Meng, M. Xia, and D. Chen. SimPO: Simple Preference Optimization with a Reference-Free Reward . Advances in Neural Information Processing Systems, 37:124198–124235, 2024. [27] P. Mertikopoulos, B. Lecouat, H. Zenati, C.-S. Foo, V. Chandrasekhar, and G. Piliouras. Optimistic mirror descent in saddle-point problems: Going the extra (gradient) mile. arXiv preprint arXiv:1807.02629, 2018. [28] A. Mokhtari, A. Ozdaglar, and S. Pattathil. A Unified Analysis of Extra-gradient and Optimistic Gradient Methods for Saddle Point Problems: Proximal Point Approach. In International Conference on Artificial Intelligence and Statistics, pages 1497–1507. PMLR, 2020. [29] R. Munos, M. Valko, D. Calandriello, M. G. Azar, M. Rowland, Z. D. Guo, Y. Tang, M. Geist, T. Mesnard, C. Fiegel, et al. Nash Learning from Human Feedback. In Forty-first International Conference on Machine Learning, 2024. [30] A. Nayak, T. Yang, O. Yagan, G. Joshi, and Y. Chi. Achieving Logarithmic Regret in KL-Regularized Zero-Sum Markov Games. arXiv preprint arXiv:2510.13060, 2025. [31] A. Nilim and L. El Ghaoui. Robust control of markov decision processes with uncertain transition matrices. Operations Research, 53(5):780–798, 2005. [32] NVIDIA, :, A. S. Deshmukh, K. Chumachenko, T. Rintamaki, M. Le, T. Poon, D. M. Taheri, I. Karmanov, G. Liu, J. Seppanen, A. Goel, M. Ranzinger, G. Heinrich, G. Chen, L. Voegtle, P. Fischer, T. Roman, K. Sapra, C. McCarthy, S. Zhang, F. Liu, H. Ye, Y. Dong, M. Liu, Y. Peng, P. Zelasko, Z. Chen, N. R. Koluguri, N. Tadevosyan, L. Grigoryan, E. H. Asl, P. Biswas, L. Tavabi, Y. Su, Z. Yu, P. Jin, A. Milesi, N. Haber, Y. Xu, S. Amiraslani, N. Mulepati, E. Tramel, J. Jung, X. Lu, B. Cui, J. Xu, Z. Li, S. Wang, Y. Kuang, S. Zhang, H. Yang, B. Li, H. Yin, S. Han, P. Molchanov, A. Renduchintala, C. Wang, D. Mosallanezhad, S. Singhal, L. Vega, K. Cheung, S. Ghosh, Y. Zhang, A. Bukharin, V. Srinivasan, J. Greco, A. Manoel, M. V. Segbroeck, S. Panguliri, R. Watve, D. Kakwani, S. Pachori, J. Glick, R. Sri-Tharan, A. Zaman, K. Nguyen, S. Chen, J. Fang, Q. Miao, W. Zhou, Y. Wang, Z. P. Bhat, V. Praveen, A. Jain, R. Arunachalam, T. Kornuta, A. Sharabiani, A. Shen, W. Huang, Y.-F. Wu, A. R. Ghias, H. Li, B. Yu, N. Tajbakhsh, C. Cui, W. Gao, L. Ding, T. Kong, M. Kilaru, A. Bhiwandiwalla, M. Wawrzos, D. Korzekwa, P. Ribalta, G. Chlebus, B. Nushi, E. Dobrowolska, M. J. Mikulski, K. Dhawan, S. Huang, J. Balam, Y. Wang, N. Karpov, V. Mendelev, G. Zelenfroynd, M. Mkrtchyan, Q. Miao, O. Almog, B. Pawar, R. Shivbhakta, S. Sabnis, A. Sharabiani, N. Habibi, G. Venkataramani, P. Peng, P. Rodney, S. Panev, R. Mazzarese, N. Liu, M. Fukuyama, A. Skliar, R. Waleffe, D. Riach, Y. Zou, J. Hu, H. Zhang, B. Xu, Y. Yang, Z. Ahmed, A. Milesi, C. del Mundo, C. Voegele, Z. Cheng, N. Assaf, A. Skliar, D. Afrimi, N. Bagrov, R. Zilberstein, O. Masad, E. Khvedchenia, N. Bagrov, B. Tymchenko, T. Asida, D. Afrimi, P. Mannan, V. Cui, M. Evans, K. Luna, J. Lou, P. Xu, G. Huang, N. Habibi, M. Boone, P. Thalasta, A. Adesoba, D. Yared, C. Parisien, L. Derczynski, S. Ghosh, W. Feely, M. Schaffer, R. Sri-Tharan, J. Glick, B. Simkin, G. Zelenfroynd, T. Grzegorzek, R. Garg, A. Jhunjhunwala, S. Kolchenko, F. Memarian, H. Kumar, S. Kumar, I. Hulseman, A. Shah, K. Briski, P. Subramanian, J. Conway, U. Karpas, J. P. Scowcroft, A. Surla, S. Ammireddy, E. Evans, J. Oliver, T. Balough, C.-C. Chen, S. Bhaskar, A. Rico, B. Sadeghi, S. Mard, K. Cheung, M. Price, L. Sleiman, S. Kaji, W. Helmholz, W. Quan, M. Lightstone, J. Cohen, J. Zhang, O. Kuchaiev, B. Ginsburg, J. Kautz, E. Long, M. Shoeybi, M. Patwary, O. Olabiyi, A. Tao, B. Catanzaro, and U. Karpas. Nemotron 3 nano omni: Efficient and open multimodal intelligence, 2026. [33] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. Christiano, J. Leike, and R. Lowe. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744, 2022. [34] B. Pásztor, T. K. Buening, and A. Krause. Stackelberg Learning from Human Feedback: Preference Optimization as a Sequential Game. In NeurIPS 2025 Workshop: Second Workshop on Aligning Reinforcement Learning Experimentalists and Theorists, 2025. [35] R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn. Direct Preference Optimization: Your Language Model is Secretly a Reward Model. Advances in neural information processing systems, 36:53728–53741, 2023. [36] H. Rahimian and S. Mehrotra. Distributionally Robust Optimization: A Review. arXiv preprint arXiv:1908.05659, 2019. [37] A. Rakhlin and K. Sridharan. Online Learning with Predictable Sequences. In Conference on Learning Theory, pages 993–1019. PMLR, 2013. [38] S. Sagawa, P. W. Koh, T. Hashimoto, and P. Liang. Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization. In Proc. International Conference on Learning Representations (ICLR), 2020. [39] V. Sanh, L. Debut, J. Chaumond, and T. Wolf. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. ArXiv, abs/1910.01108, 2019. [40] A. Sharifnassab, S. Salehkaleybar, S. Ghiassian, S. Kanoria, and D. Schuurmans. Soft Preference Optimization: Aligning Language Models to Expert Distributions. arXiv preprint arXiv:2405.00747, 2024. [41] N. Stiennon, L. Ouyang, J. Wu, D. Ziegler, R. Lowe, C. Voss, A. Radford, D. Amodei, and P. F. Christiano. Learning to summarize with Human Feedback. Advances in neural information processing systems, 33:3008–3021, 2020. [42] V. Syrgkanis, A. Agarwal, H. Luo, and R. E. Schapire. Fast Convergence of Regularized Learning in Games. Advances in Neural Information Processing Systems, 28, 2015. [43] Y. Tang, Z. D. Guo, Z. Zheng, D. Calandriello, R. Munos, M. Rowland, P. H. Richemond, M. Valko, B. Á. Pires, and B. Piot. Generalized Preference Optimization: A Unified Approach to Offline Alignment. arXiv preprint arXiv:2402.05749, 2024. [44] G. Team, T. Mesnard, C. Hardin, R. Dadashi, S. Bhupatiraju, S. Pathak, L. Sifre, M. Rivière, M. S. Kale, J. Love, P. Tafti, L. Hussenot, P. G. Sessa, A. Chowdhery, A. Roberts, A. Barua, A. Botev, A. Castro-Ros, A. Slone, A. Héliou, A. Tacchetti, A. Bulanova, A. Paterson, B. Tsai, B. Shahriari, C. L. Lan, C. A. Choquette-Choo, C. Crepy, D. Cer, D. Ippolito, D. Reid, E. Buchatskaya, E. Ni, E. Noland, G. Yan, G. Tucker, G.-C. Muraru, G. Rozhdestvenskiy, H. Michalewski, I. Tenney, I. Grishchenko, J. Austin, J. Keeling, J. Labanowski, J.-B. Lespiau, J. Stanway, J. Brennan, J. Chen, J. Ferret, J. Chiu, J. Mao-Jones, K. Lee, K. Yu, K. Millican, L. L. Sjoesund, L. Lee, L. Dixon, M. Reid, M. Mikuła, M. Wirth, M. Sharman, N. Chinaev, N. Thain, O. Bachem, O. Chang, O. Wahltinez, P. Bailey, P. Michel, P. Yotov, R. Chaabouni, R. Comanescu, R. Jana, R. Anil, R. McIlroy, R. Liu, R. Mullins, S. L. Smith, S. Borgeaud, S. Girgin, S. Douglas, S. Pandya, S. Shakeri, S. De, T. Klimenko, T. Hennigan, V. Feinberg, W. Stokowiec, Y. hui Chen, Z. Ahmed, Z. Gong, T. Warkentin, L. Peran, M. Giang, C. Farabet, O. Vinyals, J. Dean, K. Kavukcuoglu, D. Hassabis, Z. Ghahramani, D. Eck, J. Barral, F. Pereira, E. Collins, A. Joulin, N. Fiedel, E. Senter, A. Andreev, and K. Kenealy. Gemma: Open Models Based on Gemini Research and Technology, 2024. [45] L. L. Thurstone. Psychophysical Analysis. The American journal of psychology, 38(3):368–389, 1927. [46] D. Tiapkin, D. Calandriello, D. Belomestny, E. Moulines, A. Naumov, K. Rasul, M. Valko, and P. Menard. Proximal Point Nash Learning from Human Feedback. arXiv preprint arXiv:2505.19731, 2025. [47] C.-Y. Wei, C.-W. Lee, M. Zhang, and H. Luo. Linear Last-iterate Convergence in Constrained Saddle-point Optimization. arXiv preprint arXiv:2006.09517, 2020. [48] W. Wiesemann, D. Kuhn, and B. Rustem. Robust Markov Decision Processes. Mathematics of Operations Research, 38(1):153–183, 2013. [49] J. Wu, Y. Xie, Z. Yang, J. Wu, J. Chen, J. Gao, B. Ding, X. Wang, and X. He. Towards Robust Alignment of Language Models: Distributionally Robustifying Direct Preference Optimization. arXiv preprint arXiv:2407.07880, 2024. [50] Z. Xu, S. Vemuri, K. Panaganti, D. Kalathil, R. Jain, and D. Ramachandran. Robust LLM Alignment via Distributionally Robust Direct Preference Optimization. arXiv preprint arXiv:2502.01930, 2025. [51] A. Yang, B. Yang, B. Hui, B. Zheng, B. Yu, C. Zhou, C. Li, C. Li, D. Liu, F. Huang, G. Dong, H. Wei, H. Lin, J. Tang, J. Wang, J. Yang, J. Tu, J. Zhang, J. Ma, J. Xu, J. Zhou, J. Bai, J. He, J. Lin, K. Dang, K. Lu, K. Chen, K. Yang, M. Li, M. Xue, N. Ni, P. Zhang, P. Wang, R. Peng, R. Men, R. Gao, R. Lin, S. Wang, S. Bai, S. Tan, T. Zhu, T. Li, T. Liu, W. Ge, X. Deng, X. Zhou, X. Ren, X. Zhang, X. Wei, X. Ren, Y. Fan, Y. Yao, Y. Zhang, Y. Wan, Y. Chu, Y. Liu, Z. Cui, Z. Zhang, and Z. Fan. Qwen2 technical report. arXiv preprint arXiv:2407.10671, 2024. [52] C. Ye, W. Xiong, Y. Zhang, H. Dong, N. Jiang, and T. Zhang. Online Iterative Reinforcement Learning from Human Feedback with General Preference Model. Advances in Neural Information Processing Systems, 37:81773–81807, 2024. [53] K. Zhang, T. Sun, Y. Tao, S. Genc, S. Mallya, and T. Basar. Robust multi-agent reinforcement learning with model uncertainty. In Proc. Advances in Neural Information Processing Systems (NeurIPS), volume 33, 2020. [54] Y. Zhang, D. Yu, B. Peng, L. Song, Y. Tian, M. Huo, N. Jiang, H. Mi, and D. Yu. Iterative Nash Policy Optimization: Aligning LLMs with General Preferences via No-Regret Learning. In International Conference on Learning Representations (ICLR), 2025. General-preference RLHF as a two-player game; OMD-based last-iterate convergence; KL-regularized formulation. [55] Y. Zhao, R. Joshi, T. Liu, M. Khalman, M. Saleh, and P. J. Liu. SLiC-HF: Sequence Likelihood Calibration with Human Feedback. arXiv preprint arXiv:2305.10425, 2023. [56] R. Zhou, M. Fazel, and S. S. Du. Extragradient Preference Optimization (EGPO): Beyond Last-Iterate Convergence for Nash Learning from Human Feedback. arXiv preprint arXiv:2503.08942, 2025. [57] B. Zhu, M. Jordan, and J. Jiao. Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisons. In International Conference on Machine Learning, pages 43037–43067. PMLR, 2023. Appendix Table of Contents A Related Work . A B Main Results . B C Comparisons with Standard NLHF . C D Auxiliary Lemmas . D E IMDb Continuation Task: Experiment Details . E E.1 Nominal Preference and Annotator Groups . E.1 E.2 Optimization Algorithms . E.2 E.3 Hyperparameters . E.3 E.4 Results . E.4 F TL;DR Summarization Task: Experiment Details . F F.1 Fine-Tuning Models . F.1 F.2 Nominal Preference Proxy . F.2 F.3 Adversary Proxy . F.3 F.4 Losses . F.4 F.5 Evaluation with LLM Judges . F.5 F.6 Non-BT Nominal Kernel: Construction and Diagnostic . F.6 F.7 Additional Results with a Bradley-Terry Nominal Kernel . F.7 G LLM Judge Reasoning Examples . G Appendix A Related Work Reward-based RLHF and direct alignment. A large body of work on alignment starts from the RLHF pipeline: collect pairwise comparisons, fit a reward model, and optimize a policy against that reward subject to a regularization term that keeps the policy close to a reference model [9, 41, 33]. This paradigm has been highly influential in large-scale language-model alignment, but it typically treats the preference signal as a nominal target and often relies, either explicitly or implicitly, on a latent-utility view of pairwise comparisons. Motivated by the computational and modeling limitations of reward-model-based RLHF, a broad line of direct or reward-free preference optimization methods has emerged, including DPO, SLiC-HF, CPL, KTO, ORPO, SimPO, SPO, and adaptive preference scaling [35, 55, 15, 13, 17, 26, 40, 16]. These methods substantially improve the practical pipeline of preference optimization, but they generally continue to optimize against a fixed nominal preference estimate rather than an ambiguity set of plausible preference kernels. General-preference and game-theoretic alignment. Recent work has argued that pairwise human preferences should be modeled directly rather than compressed into a scalar reward function [3]. This viewpoint leads naturally to game-theoretic formulations of alignment. In particular, Nash Learning from Human Feedback (NLHF) formulates preference alignment as a regularized two-player game and introduces Nash-MD as a no-regret algorithm for the induced general-preference game [29]. Subsequent works have further developed this perspective, including online and offline algorithms under general preference models and policy-loss implementations based on no-regret learning [52, 54, 30, 56]. More recently, sequential leader–follower formulations of preference optimization have also begun to appear, including Stackelberg-style alignment methods and Stackelberg game variants for data-efficient alignment [34, 10]. Our work is closest in spirit to this line of research, but differs in one central respect: we do not assume that the learned preference game is exact. Instead, we explicitly model uncertainty in preference kernels and optimize against the worst kernel in an ambiguity set. This brings significant challenges and hardness to our studies, and we discuss them in detail in Section C. Soft labels, heterogeneous preferences, and population shift. Another line of work emphasizes that preference data are often soft, noisy, and heterogeneous across annotators or user groups. Recent studies incorporate soft preference labels or pairwise confidence information into direct alignment objectives [16, 40, 14], while other work explicitly models hidden annotator types or heterogeneous preference populations [8]. These papers highlight an important limitation of nominal preference optimization: even if one optimizes the nominal objective correctly, the aggregated preference signal may still be unstable because it hides substantial variation across annotators or deployment populations. Our formulation is complementary to this line of work. Rather than only modifying the loss to account for label softness or latent subgroups, we directly treat the learned pairwise preference kernel as uncertain and seek a policy with a worst-case performance guarantee over that uncertainty set. Robust alignment and distributionally robust preference optimization. Our formulation is also related to the broader literature on distributionally robust optimization and robust sequential decision making, where ambiguity sets are used to hedge estimation error and distribution shift [38, 20, 31]. In the alignment literature, recent work has started to robustify reward-based RLHF or DPO-style objectives against prompt-distribution shift, noisy preference data, or uncertain preference strength [16, 6, 49, 50, 24]. These methods are close in motivation to ours, but they robustify different objects: prompt distributions, training examples, or nominal pairwise losses. More importantly, all these works are based on the reward and BT-model assumption. By contrast, our approach directly robustifies the pairwise preference kernel itself inside a general-preference game. This distinction is important because kernel uncertainty directly captures ambiguity in preferences, which is the primitive object in general-preference alignment. Robust game theory and robust Markov games. A separate line of work studies robustness to payoff or model uncertainty directly in game-theoretic solution concepts, rather than in a single-agent decision problem. This includes robust game theory under set-based payoff uncertainty [1], distributionally robust Nash and Stackelberg equilibrium models under exogenous distributional uncertainty [22], and distributionally robust commitment to a strategy when the follower’s utility is known only up to a finite set of candidate models [2]. Robust Markov games extend this to multi-stage settings with uncertain rewards or transitions [53, 25]. Our setting differs from this literature in two respects. First, robust Markov games typically seek a robust Nash equilibrium between symmetric players, whereas our objective is asymmetric, as we have a single learned policy against a strategic follower and a shared adversarial kernel. Second, a single Bernoulli-KL budget couples every response pair within a prompt, ruling out the per-action or per-player decomposition, and the multi-stage Bellman recursion that robust Markov-game algorithms rely on. Existing robust-game and robust-Markov-game results therefore do not directly provide our proxy formulation or its convergence guarantee. Appendix B Main results B.1 Algorithm Update Rules πt+1 _t+1 =argmaxπ∈Πϵ⟨g^tπ,π⟩−1ηDKL(π∥πt), = _π∈ _ε \ g_t^π,π - 1ηD_KL(π\| _t) \, (25) γt+1 _t+1 =ProjΓ⁡(γt+ηg^tγ), = _ ( _t+η g_t^γ ), (26) νt+1 _t+1 =argminν∈β⟨g^tν,ν⟩+1ηDKL(ν∥νt), = _ν _β \ g_t^ν,ν + 1ηD_KL(ν\| _t) \, (27) pt+1 p_t+1 =argminp∈Pδ⟨g^tp,p⟩+1ηDKLBern(p∥pt). = _p∈ P_δ \ g_t^p,p + 1ηD_KL^Bern(p\|p_t) \. (28) Assumption 3 (Dual-optimum inclusion). For every (π,ν)∈Πϵ×β(π,ν)∈ _ε×N_β, the scalar dual problem maxγ≥0minp∈PδJ(π,ν;p)+γ(DKLBern(p∥p⋆)−ρ) _γ≥ 0 _p∈ P_δ \J(π,ν;p)+γ (D_KL^Bern(p\|p )-ρ ) \ admits a maximizer in the algorithmic interval Γ=[γmin,γmax] =[ _ , _ ]. B.2 Proof of Theorem 1 Proof. Fix π∈Πϵπ∈ _ε. Since the feasible set in (15) is a subset of the feasible set in (13), the lower bound gHδ(π,ρ)≥gH(π,ρ)g_H^δ(π;ρ)≥ g_H(π;ρ) is immediate. It remains to show the upper bound. Fix ν∈Δmν∈ _m, and let K(ρ):=p∈[0,1]d:DKLBern(p∥p⋆)≤ρ.K(ρ):= \p∈[0,1]^d:\ D_KL^Bern(p\|p )≤ρ \. Take any p∈K⁡(ρ)p∈ K(ρ), and clip it coordinatewise into PδP_δ: p~:=ProjPδ⁡(p). p:= _P_δ(p). Because p⋆∈Pδp ∈ P_δ, each scalar p~e p_e lies on the line segment joining pep_e and pe⋆p _e. For each coordinate e, the scalar function u↦ulog⁡upe⋆+(1−u)log⁡1−u1−pe⋆u u up_e +(1-u) 1-u1-p_e is convex and has its unique minimum at u=pe⋆u=p_e . Hence clipping toward pe⋆p _e cannot increase the coordinatewise Bernoulli-KL divergence, so DKLBern(p~∥p⋆)≤DKLBern(p∥p⋆)≤ρ.D_KL^Bern( p\|p )≤ D_KL^Bern(p\|p )≤ρ. Thus p~ p is feasible for the truncated problem. Now J⁡(π,ν,p)J(π,ν;p) depends on p only through the linear term ∑(i,j)∈Ewij(π,ν)pij. _(i,j)∈ Ew_ij(π,ν)\,p_ij. Therefore |J⁡(π,ν,p~)−J⁡(π,ν,p)| |J(π,ν; p)-J(π,ν;p)| =|∑(i,j)∈Ewij(π,ν)(p~ij−pij)|≤‖p~−p‖∞∑(i,j)∈E|wij(π,ν)|. = | _(i,j)∈ Ew_ij(π,ν)\,( p_ij-p_ij) |≤\| p-p\|_∞ _(i,j)∈ E|w_ij(π,ν)|. Since each coordinate is clipped by at most δ, we have ‖p~−p‖∞≤δ.\| p-p\|_∞≤δ. Also, ∑(i,j)∈E|wij(π,ν)| _(i,j)∈ E|w_ij(π,ν)| ≤∑(i,j)∈E(πiνj+πjνi)=∑i≠jπiνj=1−∑i=1mπiνi≤1. ≤ _(i,j)∈ E( _i _j+ _j _i)= _i≠ j _i _j=1- _i=1^m _i _i≤ 1. Hence J⁡(π,ν,p~)≤J⁡(π,ν,p)+δ.J(π,ν; p)≤ J(π,ν;p)+δ. Taking the minimum first over p∈K⁡(ρ)p∈ K(ρ), then over ν∈Δmν∈ _m, yields gHδ(π,ρ)≤gH(π,ρ)+δ.g_H^δ(π;ρ)≤ g_H(π;ρ)+δ. This proves the claim 0≤gHδ(π,ρ)−gH(π,ρ)≤δ0≤ g_H^δ(π;ρ)-g_H(π;ρ)≤δ. Maximizing over π∈Πϵπ∈ _ε gives 0≤Vδ(ρ)−V⁡(ρ)≤δ0≤ V^δ(ρ)-V(ρ)≤δ. Using the above two claims, we have V⁡(ρ)≤Vδ(ρ)=gHδ(πδ⋆,ρ)≤gH(πδ⋆,ρ)+δ.V(ρ)≤ V^δ(ρ)=g_H^δ( _δ ;ρ)≤ g_H( _δ ;ρ)+δ. Rearranging gives the claim.∎ B.3 Proof of Theorem 2 Proof. Fix π∈Πϵπ∈ _ε, and define ϕπ(ν,γ):=minp∈Pδ⁡F⁡(π,γ,ν,p). _π(ν,γ):= _p∈ P_δF(π,γ,ν,p). Step 1: reduction to βN_β and nested strong duality. For fixed (π,p)(π,p), define bj(π,p):=∑i=1mπ(i)Pp(i≻j),j=1,…,m.b_j(π,p):= _i=1^mπ(i)\,P_p(i j), j=1,…,m. Since Pp(i≻j)∈[0,1]P_p(i j)∈[0,1] and ∑iπ⁡(i)=1 _iπ(i)=1, we have bj(π,p)∈[0,1]b_j(π,p)∈[0,1] for all j. For fixed (π,p)(π,p), the follower subproblem is minν∈Δm∑j=1mν(j)bj(π,p)+τDKL(ν∥πref). _ν∈ _m \ _j=1^mν(j)\,b_j(π,p)+τ D_KL(ν\| _ ref) \. Its unique minimizer is the Gibbs distribution νjbr(π,p)=πref(j)e−bj(π,p)/τ∑ℓ=1mπref(ℓ)e−bℓ(π,p)/τ. _j^br(π,p)= _ ref(j)e^-b_j(π,p)/τ _ =1^m _ ref( )e^-b_ (π,p)/τ. Because bj(π,p)∈[0,1]b_j(π,p)∈[0,1], we have e−1/τ≤e−bj(π,p)/τ≤1e^-1/τ≤ e^-b_j(π,p)/τ≤ 1. Let Z(π,p):=∑ℓ=1mπref(ℓ)e−bℓ(π,p)/τ.Z(π,p):= _ =1^m _ ref( )e^-b_ (π,p)/τ. Since ∑ℓπref(ℓ)=1 _ _ ref( )=1 and each exponential factor is at most 11, we have Z⁡(π,p)≤1Z(π,p)≤ 1. Therefore νjbr(π,p)=πref(j)e−bj(π,p)/τZ⁡(π,p)≥πref(j)e−1/τ. _j^br(π,p)= _ ref(j) e^-b_j(π,p)/τZ(π,p)≥ _ ref(j)e^-1/τ. Under Assumption 2-(2), β≤e−1/τβ≤ e^-1/τ, this implies νbr(π,p)∈β.ν^br(π,p) _β. Hence, for every (π,p)(π,p), minν∈Δm⁡J⁡(π,ν,p)=minν∈β⁡J⁡(π,ν,p). _ν∈ _mJ(π,ν;p)= _ν _βJ(π,ν;p). (29) The same identity holds with J⁡(π,ν,p)J(π,ν;p) replaced by F⁡(π,γ,ν,p)F(π,γ,ν,p), since the dual term γ(DKLBern(p∥p⋆)−ρ)γ(D_KL^Bern(p\|p )-ρ) does not depend on ν. Now fix ν∈βν _β. The p-subproblem minp∈PδDKLBern(p∥p⋆)≤ρ⁡J⁡(π,ν,p) _ subarraycp∈ P_δ\\ D_KL^Bern(p\|p )≤ρ subarrayJ(π,ν;p) is a convex program: the objective is affine in p, the feasible set is convex, and Slater’s condition holds because Assumption 2-(1) gives p⋆∈(δ,1−δ)dp ∈(δ,1-δ)^d and DKLBern(p⋆∥p⋆)=0<ρD_KL^Bern(p \|p )=0<ρ. Therefore strong duality yields minp∈PδDKLBern(p∥p⋆)≤ρ⁡J⁡(π,ν,p)=maxγ≥0⁡minp∈Pδ⁡F⁡(π,γ,ν,p). _ subarraycp∈ P_δ\\ D_KL^Bern(p\|p )≤ρ subarrayJ(π,ν;p)= _γ≥ 0 _p∈ P_δF(π,γ,ν,p). Under Assumption 2-(3), γmax≥1/ρ _ ≥ 1/ρ, and the standard dual upper-bound argument implies that a maximizer can be chosen in [0,γmax][0, _ ]. Consequently, minp∈PδDKLBern(p∥p⋆)≤ρ⁡J⁡(π,ν,p)=maxγ∈[0,γmax]⁡minp∈Pδ⁡F⁡(π,γ,ν,p). _ subarraycp∈ P_δ\\ D_KL^Bern(p\|p )≤ρ subarrayJ(π,ν;p)= _γ∈[0, _ ] _p∈ P_δF(π,γ,ν,p). Combining this with (29), we obtain gHδ(π,ρ)=minν∈β⁡maxγ∈[0,γmax]minp∈Pδ⁡F⁡(π,γ,ν,p).g_H^δ(π;ρ)= _ν _β _γ∈[0, _ ] _p∈ P_δF(π,γ,ν,p). (30) Step 2: proof of the lower bound (20). By definition, gproxyδ(π,ρ)=maxγ∈Γ⁡minν∈β,p∈Pδ⁡F⁡(π,γ,ν,p)=maxγ∈Γ⁡minν∈βϕπ(ν,γ).g_proxy^δ(π;ρ)= _γ∈ _ν _β,\;p∈ P_δF(π,γ,ν,p)= _γ∈ _ν _β _π(ν,γ). Since Γ=[γmin,γmax]⊆[0,γmax] =[ _ , _ ] [0, _ ], gproxyδ(π,ρ)≤maxγ∈[0,γmax]⁡minν∈βϕπ(ν,γ).g_proxy^δ(π;ρ)≤ _γ∈[0, _ ] _ν _β _π(ν,γ). By weak minimax, maxγ∈[0,γmax]⁡minν∈βϕπ(ν,γ)≤minν∈β⁡maxγ∈[0,γmax]ϕπ(ν,γ). _γ∈[0, _ ] _ν _β _π(ν,γ)≤ _ν _β _γ∈[0, _ ] _π(ν,γ). Using (30), the right-hand side equals gHδ(π,ρ)g_H^δ(π;ρ). Therefore gproxyδ(π,ρ)≤gHδ(π,ρ),g_proxy^δ(π;ρ)≤ g_H^δ(π;ρ), which proves (20). Step 3: excluding the interval [0,γmin)[0, _ ) costs at most ργminρ _ . Fix ν∈βν _β, and define qπ,ν(γ):=ϕπ(ν,γ).q_π,ν(γ):= _π(ν,γ). We claim that for any 0≤a<b≤γmax0≤ a<b≤ _ , qπ,ν(a)≤qπ,ν(b)+ρ⁡(b−a).q_π,ν(a)≤ q_π,ν(b)+ρ(b-a). (31) Indeed, let pb∈arg⁡minp∈Pδ⁡F⁡(π,b,ν,p)p_b∈ _p∈ P_δF(π,b,ν,p). Then qπ,ν(a) q_π,ν(a) ≤F⁡(π,a,ν,pb) ≤ F(π,a,ν,p_b) =F(π,b,ν,pb)+(a−b)(DKLBern(pb∥p⋆)−ρ) =F(π,b,ν,p_b)+(a-b) (D_KL^Bern(p_b\|p )-ρ ) =qπ,ν(b)+(b−a)(ρ−DKLBern(pb∥p⋆)) =q_π,ν(b)+(b-a) (ρ-D_KL^Bern(p_b\|p ) ) ≤qπ,ν(b)+ρ⁡(b−a), ≤ q_π,ν(b)+ρ(b-a), since DKLBern(pb∥p⋆)≥0D_KL^Bern(p_b\|p )≥ 0. This proves (31). Now let γ♯∈[0,γmax]γ ∈[0, _ ] be a maximizer of qπ,νq_π,ν over [0,γmax][0, _ ]. If γ♯∈Γγ ∈ , then maxγ∈[0,γmax]⁡qπ,ν(γ)=maxγ∈Γ⁡qπ,ν(γ). _γ∈[0, _ ]q_π,ν(γ)= _γ∈ q_π,ν(γ). If γ♯<γminγ < _ , then applying (31) with a=γ♯a=γ and b=γminb= _ yields qπ,ν(γ♯)≤qπ,ν(γmin)+ρ⁡(γmin−γ♯)≤maxγ∈Γ⁡qπ,ν(γ)+ργmin.q_π,ν(γ )≤ q_π,ν( _ )+ρ( _ -γ )≤ _γ∈ q_π,ν(γ)+ρ _ . Hence, in all cases, maxγ∈[0,γmax]⁡qπ,ν(γ)≤maxγ∈Γ⁡qπ,ν(γ)+ργmin. _γ∈[0, _ ]q_π,ν(γ)≤ _γ∈ q_π,ν(γ)+ρ _ . Taking the minimum over ν∈βν _β, we obtain gHδ(π,ρ)≤minν∈β⁡maxγ∈Γϕπ(ν,γ)+ργmin.g_H^δ(π;ρ)≤ _ν _β _γ∈ _π(ν,γ)+ρ _ . (32) Step 4: under Assumption 2-(4), the minimax defect on Γ vanishes. Assume now Assumption 2-(4), i.e. τγmin≥14.τ _ ≥ 14. We show that ϕπ(⋅,⋅) _π(·,·) is convex in ν and concave in γ on β×ΓN_β× . Concavity in γ. For fixed ν, the function γ↦F⁡(π,γ,ν,p)γ F(π,γ,ν,p) is affine for every fixed p, so ϕπ(ν,⋅) _π(ν,·), being the pointwise infimum of affine functions, is concave on Γ . Convexity in ν. Fix γ∈Γγ∈ . We claim that (ν,p)↦F⁡(π,γ,ν,p)(ν,p) F(π,γ,ν,p) is jointly convex on β×PδN_β× P_δ. Write W⁡(π,ν,p)=∑j=1mν⁡(j)bj(π,p),bj(π,p)=∑i=1mπ⁡(i)Pp(i≻j).W(π,ν,p)= _j=1^mν(j)\,b_j(π,p), b_j(π,p)= _i=1^mπ(i)P_p(i j). Let (ν,p)(ν,p) and (ν′,p′)(ν ,p ) be two points in β×PδN_β× P_δ. Since W is bilinear in (ν,p)(ν,p), W⁡(π,ν′,p′)−W⁡(π,ν,p)−⟨∇νW(π,ν,p),ν′−ν⟩−⟨∇pW(π,ν,p),p′−p⟩=⟨ν′−ν,b⁡(π,p′)−b⁡(π,p)⟩.W(π,ν ,p )-W(π,ν,p)- _νW(π,ν,p),ν -ν - _pW(π,ν,p),p -p = ν -ν,\;b(π,p )-b(π,p) . For each coordinate j, |bj(π,p′)−bj(π,p)|≤∑i=1mπ⁡(i)|Pp′(i≻j)−Pp(i≻j)|≤‖p′−p‖2,|b_j(π,p )-b_j(π,p)|≤ _i=1^mπ(i)\,|P_p (i j)-P_p(i j)|≤\|p -p\|_2, hence |⟨ν′−ν,b⁡(π,p′)−b⁡(π,p)⟩|≤‖ν′−ν‖1‖p′−p‖2. | ν -ν,\;b(π,p )-b(π,p) |≤\|ν -ν\|_1\,\|p -p\|_2. Therefore, F⁡(π,γ,ν′,p′) F(π,γ,ν ,p ) ≥F⁡(π,γ,ν,p)+⟨∇νF(π,γ,ν,p),ν′−ν⟩+⟨∇pF(π,γ,ν,p),p′−p⟩ ≥ F(π,γ,ν,p)+ _νF(π,γ,ν,p),ν -ν + _pF(π,γ,ν,p),p -p +τDKL(ν′∥ν)+γDKLBern(p′∥p)−∥ν′−ν∥1∥p′−p∥2. +τ D_KL(ν \|ν)+γ D_KL^Bern(p \|p)-\|ν -ν\|_1\,\|p -p\|_2. Using Pinsker’s inequalities DKL(ν′∥ν)≥12∥ν′−ν∥12,DKLBern(p′∥p)≥2∥p′−p∥22,D_KL(ν \|ν)≥ 12\|ν -ν\|_1^2, D_KL^Bern(p \|p)≥ 2\|p -p\|_2^2, we obtain τDKL(ν′∥ν)+γDKLBern(p′∥p)−∥ν′−ν∥1∥p′−p∥2≥τ2s2+2γt2−st,τ D_KL(ν \|ν)+γ D_KL^Bern(p \|p)-\|ν -ν\|_1\,\|p -p\|_2≥ τ2s^2+2γ t^2-st, where s=‖ν′−ν‖1s=\|ν -ν\|_1 and t=‖p′−p‖2t=\|p -p\|_2. Since γ≥γminγ≥ _ and τγmin≥1/4τ _ ≥ 1/4, τ2s2+2γt2−st=(τ2s−2γt)2+2(τγ−12)st≥0. τ2s^2+2γ t^2-st= ( τ2\,s- 2γ\,t )^2+2 ( τγ- 12 )st≥ 0. Thus (ν,p)↦F⁡(π,γ,ν,p)(ν,p) F(π,γ,ν,p) is jointly convex. Taking the infimum over p∈Pδp∈ P_δ shows that ϕπ(⋅,γ) _π(·,γ) is convex on βN_β. Since βN_β and Γ are convex and compact, and ϕπ _π is continuous, Sion’s minimax theorem applies: minν∈β⁡maxγ∈Γϕπ(ν,γ)=maxγ∈Γ⁡minν∈βϕπ(ν,γ)=gproxyδ(π,ρ). _ν _β _γ∈ _π(ν,γ)= _γ∈ _ν _β _π(ν,γ)=g_proxy^δ(π;ρ). Substituting this into (32) yields gHδ(π,ρ)≤gproxyδ(π,ρ)+ργmin.g_H^δ(π;ρ)≤ g_proxy^δ(π;ρ)+ρ _ . Combining with (20), we conclude 0≤gHδ(π,ρ)−gproxyδ(π,ρ)≤ργmin,0≤ g_H^δ(π;ρ)-g_proxy^δ(π;ρ)≤ρ _ , which proves the first inequality in (21). Step 5: pass from pointwise values to optimal values. By definition, Vδ(ρ)=maxπ∈Πϵ⁡gHδ(π,ρ),Vproxyδ(ρ)=maxπ∈Πϵ⁡gproxyδ(π,ρ).V^δ(ρ)= _π∈ _εg_H^δ(π;ρ), V_proxy^δ(ρ)= _π∈ _εg_proxy^δ(π;ρ). For any two functions f,gf,g on the same domain, maxπ⁡f⁡(π)−maxπ⁡g⁡(π)≤maxπ⁡(f⁡(π)−g⁡(π)). _πf(π)- _πg(π)≤ _π(f(π)-g(π)). Applying this with f⁡(π)=gHδ(π,ρ)f(π)=g_H^δ(π;ρ) and g⁡(π)=gproxyδ(π,ρ)g(π)=g_proxy^δ(π;ρ), and using the pointwise bound just proved, we obtain Vδ(ρ)−Vproxyδ(ρ)≤maxπ∈Πϵ⁡(gHδ(π,ρ)−gproxyδ(π,ρ))≤ργmin.V^δ(ρ)-V_proxy^δ(ρ)≤ _π∈ _ε (g_H^δ(π;ρ)-g_proxy^δ(π;ρ) )≤ρ _ . This proves the second inequality in (21). Step 6: exactness on Γ . Assume now Assumption 2 and Assumption 3. Since any maximizer γ⋆γ lies in Γ , (30) then implies that gHδ(π,ρ)=maxγ∈Γ⁡minν∈βminp∈Pδ⁡F⁡(π,γ,ν,p)=gproxyδ(π,ρ).g_H^δ(π;ρ)= _γ∈ _ν _β _p∈ P_δF(π,γ,ν,p)=g_proxy^δ(π;ρ). Finally, maximizing over π∈Πϵπ∈ _ε gives Vproxyδ(ρ)=maxπ∈Πϵ⁡gproxyδ(π,ρ)=maxπ∈Πϵ⁡gHδ(π,ρ)=Vδ(ρ),V_proxy^δ(ρ)= _π∈ _εg_proxy^δ(π;ρ)= _π∈ _εg_H^δ(π;ρ)=V_δ(ρ), which proves (22). ∎ B.4 Proof of Theorem 3 Theorem 4 (Convergence and robust guarantee of four-player OGDA with explicit δ-dependence). Assume Assumption 2, and let Lδ:=max⁡1,τϵπref,min,τβπref,min,γmaxδ⁡(1−δ),d, 2dlog⁡1−δ,L_δ:= \1,\, τε\, _ ref,min,\, τβ\, _ ref,min,\, _ δ(1-δ),\, d,\,2 d\, 1-δ \, where πref,min:=min1≤i≤m⁡πref,i. _ ref,min:= _1≤ i≤ m _ ref,i. In particular, Lδ=O⁡(max⁡1δ⁡(1−δ),log⁡1−δ)as δ↓0.L_δ=O\! ( \ 1δ(1-δ),\ 1-δ \ ) δ 0. Run Algorithm 1 with the constant step size η=18Lδ.η= 18L_δ. Define DXdiam:=sup(π,γ)∈XDKL(π∥π1)+12(γ−γ1)2,DYdiam:=sup(ν,p)∈YDKL(ν∥ν1)+DKLBern(p∥p1),D_X^diam:= _(π,γ)∈ X \D_KL(π\| _1)+ 12(γ- _1)^2 \, D_Y^diam:= _(ν,p)∈ Y \D_KL(ν\| _1)+D_KL^Bern(p\|p_1) \, and BT,δ:=8Lδ(DXdiam+DYdiam)T.B_T,δ:= 8L_δ (D_X^diam+D_Y^diam )T. Then the duality gap of the averaged iterate z¯T=(x¯T,y¯T)=((π¯T,γ¯T),(ν¯T,p¯T)) z_T=( x_T, y_T)= (( π_T, γ_T),( ν_T, p_T) ) satisfies Gap⁡(z¯T):=maxx′∈X⁡F⁡(x′,y¯T)−miny′∈Y⁡F⁡(x¯T,y′)≤BT,δ. ( z_T):= _x ∈ XF(x , y_T)- _y ∈ YF( x_T,y )≤ B_T,δ. (28) Moreover, the averaged main policy satisfies V⁡(ρ)−gH(π¯T,ρ)≤BT,δ+ργmin+δ.V(ρ)-g_H( π_T;ρ)≤ B_T,δ+ρ _ +δ. (29) Equivalently, V⁡(ρ)−gH(π¯T,ρ)=O⁡(LδT+ργmin+δ).V(ρ)-g_H( π_T;ρ)=O\! ( L_δT+ρ _ +δ ). Additionally, under Assumption 3, V⁡(ρ)−gH(π¯T,ρ)≤BT,δ+δ,V(ρ)-g_H( π_T;ρ)≤ B_T,δ+δ, (30) so π¯T π_T is an O⁡(LδT+δ)O\! ( L_δT+δ ) -optimal policy for the original robust game. Proof. We divide the proof into four steps. Step 1: proxy-game convergence with explicit LδL_δ. Let xt:=(πt,γt)∈X,yt:=(νt,pt)∈Y,zt=(xt,yt),x_t:=( _t, _t)∈ X, y_t:=( _t,p_t)∈ Y, z_t=(x_t,y_t), and define the averaged iterates x¯T:=1T∑t=1Txt,y¯T:=1T∑t=1Tyt. x_T:= 1T _t=1^Tx_t, y_T:= 1T _t=1^Ty_t. For any x∈Xx∈ X, define AT(x):=∑t=1T(F⁡(x,yt)−F⁡(xt,yt)),A_T(x):= _t=1^T (F(x,y_t)-F(x_t,y_t) ), and for any y∈Yy∈ Y, define BT(y):=∑t=1T(F⁡(xt,yt)−F⁡(xt,y)).B_T(y):= _t=1^T (F(x_t,y_t)-F(x_t,y) ). By the blockwise optimistic mirror-descent inequalities and the convexity–concavity structure of Lemma 3, for every x∈Xx∈ X and every y∈Yy∈ Y, AT(x)+BT(y)≤DX(x,x1)+DY(y,y1)η+VT−ΞT,A_T(x)+B_T(y)≤ D_X(x,x_1)+D_Y(y,y_1)η+V_T- _T, (33) where DX(x,x1):=DKL(π∥π1)+12(γ−γ1)2,DY(y,y1):=DKL(ν∥ν1)+DKLBern(p∥p1),D_X(x,x_1):=D_KL(π\| _1)+ 12(γ- _1)^2, D_Y(y,y_1):=D_KL(ν\| _1)+D_KL^Bern(p\|p_1), and VT:=η∑t=1T(‖gtπ−gt−1π‖∞2+|gtγ−gt−1γ|2+‖gtν−gt−1ν‖∞2+‖gtp−gt−1p‖22),V_T:=η _t=1^T (\|g_t^π-g_t-1^π\|_∞^2+|g_t^γ-g_t-1^γ|^2+\|g_t^ν-g_t-1^ν\|_∞^2+\|g_t^p-g_t-1^p\|_2^2 ), ΞT:=12η∑t=1T(DKL(πt∥πt+1)+|γt−γt+1|2+DKL(νt∥νt+1)+DKLBern(pt∥pt+1)). _T:= 12η _t=1^T (D_KL( _t\| _t+1)+| _t- _t+1|^2+D_KL( _t\| _t+1)+D_KL^Bern(p_t\|p_t+1) ). Now, by Lemma 5, all gradient-variation terms are bounded using the constant LδL_δ; hence Lemma 6 applies with L=LδL=L_δ. Since η=1/(8Lδ)η=1/(8L_δ), Lemma 6 yields VT≤ΞT.V_T≤ _T. Therefore (33) simplifies to AT(x)+BT(y)≤DX(x,x1)+DY(y,y1)η,∀x∈X,∀y∈Y.A_T(x)+B_T(y)≤ D_X(x,x_1)+D_Y(y,y_1)η, ∀ x∈ X,\ ∀ y∈ Y. (34) Now choose x^∈arg⁡maxx∈X⁡F⁡(x,y¯T),y^∈arg⁡miny∈Y⁡F⁡(x¯T,y). x∈ _x∈ XF(x, y_T), y∈ _y∈ YF( x_T,y). Then Gap⁡(z¯T)=F⁡(x^,y¯T)−F⁡(x¯T,y^). ( z_T)=F( x, y_T)-F( x_T, y). Because F⁡(⋅,y)F(·,y) is concave and F⁡(x,⋅)F(x,·) is convex by Lemma 3, Jensen’s inequality gives F⁡(x^,y¯T)≤1T∑t=1TF⁡(x^,yt),F⁡(x¯T,y^)≥1T∑t=1TF⁡(xt,y^).F( x, y_T)≤ 1T _t=1^TF( x,y_t), F( x_T, y)≥ 1T _t=1^TF(x_t, y). Hence TGap⁡(z¯T) T\, ( z_T) ≤∑t=1T(F⁡(x^,yt)−F⁡(xt,y^)) ≤ _t=1^T (F( x,y_t)-F(x_t, y) ) =∑t=1T(F⁡(x^,yt)−F⁡(xt,yt))+∑t=1T(F⁡(xt,yt)−F⁡(xt,y^)) = _t=1^T (F( x,y_t)-F(x_t,y_t) )+ _t=1^T (F(x_t,y_t)-F(x_t, y) ) =AT(x^)+BT(y^). =A_T( x)+B_T( y). Applying (34) with x=x^x= x and y=y^y= y, and then using the definitions of DXdiamD_X^diam and DYdiamD_Y^diam, gives TGap⁡(z¯T)≤DX(x^,x1)+DY(y^,y1)η≤DXdiam+DYdiamη.T\, ( z_T)≤ D_X( x,x_1)+D_Y( y,y_1)η≤ D_X^diam+D_Y^diamη. Substituting η=1/(8Lδ)η=1/(8L_δ) yields Gap⁡(z¯T)≤8Lδ(DXdiam+DYdiam)T=BT,δ, ( z_T)≤ 8L_δ (D_X^diam+D_Y^diam )T=B_T,δ, which proves (28)(28). Step 2: proxy-policy guarantee. By definition of the duality gap, Gap⁡(z¯T)=maxx′∈X⁡F⁡(x′,y¯T)−miny′∈Y⁡F⁡(x¯T,y′). ( z_T)= _x ∈ XF(x , y_T)- _y ∈ YF( x_T,y ). Since maxx′⁡F⁡(x′,y¯T)≥F⁡(x¯T,y¯T)=F⁡(z¯T) _x F(x , y_T)≥ F( x_T, y_T)=F( z_T), we obtain Gap⁡(z¯T)≥F⁡(z¯T)−miny′∈Y⁡F⁡(x¯T,y′). ( z_T)≥ F( z_T)- _y ∈ YF( x_T,y ). Rearranging, miny′∈Y⁡F⁡(x¯T,y′)≥F⁡(z¯T)−Gap⁡(z¯T). _y ∈ YF( x_T,y )≥ F( z_T)- ( z_T). (35) Now gproxyδ(π¯T,ρ)=maxγ∈Γ⁡miny∈Y⁡F⁡((π¯T,γ),y).g_proxy^δ( π_T;ρ)= _γ∈ _y∈ YF(( π_T,γ),y). Since x¯T=(π¯T,γ¯T)∈X x_T=( π_T, γ_T)∈ X, gproxyδ(π¯T,ρ)≥miny∈Y⁡F⁡(x¯T,y).g_proxy^δ( π_T;ρ)≥ _y∈ YF( x_T,y). Combining with (35) and then using (28)(28), gproxyδ(π¯T,ρ)≥F⁡(z¯T)−Gap⁡(z¯T)≥F⁡(z¯T)−BT,δ.g_proxy^δ( π_T;ρ)≥ F( z_T)- ( z_T)≥ F( z_T)-B_T,δ. (36) Step 3: transfer from the proxy game to the original hard objective. By Theorem 2, under Assumption 2 we have the pointwise bounds 0≤gHδ(π,ρ)−gproxyδ(π,ρ)≤ργmin,0≤Vδ(ρ)−Vproxyδ(ρ)≤ργmin,∀π∈Πϵ.0≤ g_H^δ(π;ρ)-g_proxy^δ(π;ρ)≤ρ _ , 0≤ V^δ(ρ)-V_proxy^δ(ρ)≤ρ _ , ∀π∈ _ε. By Theorem 1, the truncation error satisfies 0≤gHδ(π,ρ)−gH(π,ρ)≤δ,0≤Vδ(ρ)−V⁡(ρ)≤δ,∀π∈Πϵ.0≤ g_H^δ(π;ρ)-g_H(π;ρ)≤δ, 0≤ V^δ(ρ)-V(ρ)≤δ, ∀π∈ _ε. From (36), gproxyδ(π¯T,ρ)≥F⁡(z¯T)−BT,δ.g_proxy^δ( π_T;ρ)≥ F( z_T)-B_T,δ. Hence gH(π¯T,ρ)≥gproxyδ(π¯T,ρ)−δ≥F⁡(z¯T)−BT,δ−δ.g_H( π_T;ρ)≥ g_proxy^δ( π_T;ρ)-δ≥ F( z_T)-B_T,δ-δ. This gives the explicit robust-value certificate. Next, using V⁡(ρ)≤Vδ(ρ)V(ρ)≤ V^δ(ρ) and gH(π¯T,ρ)≥gproxyδ(π¯T,ρ)−δg_H( π_T;ρ)≥ g_proxy^δ( π_T;ρ)-δ, we get V⁡(ρ)−gH(π¯T,ρ) V(ρ)-g_H( π_T;ρ) ≤Vδ(ρ)−gproxyδ(π¯T,ρ)+δ ≤ V^δ(ρ)-g_proxy^δ( π_T;ρ)+δ =(Vδ(ρ)−Vproxyδ(ρ))+(Vproxyδ(ρ)−gproxyδ(π¯T,ρ))+δ. = (V^δ(ρ)-V_proxy^δ(ρ) )+ (V_proxy^δ(ρ)-g_proxy^δ( π_T;ρ) )+δ. By Theorem 2, Vδ(ρ)−Vproxyδ(ρ)≤ργmin.V^δ(ρ)-V_proxy^δ(ρ)≤ρ _ . Also, Vproxyδ(ρ)=maxπ∈Πϵ⁡gproxyδ(π,ρ)=maxx∈X⁡miny∈Y⁡F⁡(x,y),V_proxy^δ(ρ)= _π∈ _εg_proxy^δ(π;ρ)= _x∈ X _y∈ YF(x,y), so Vproxyδ(ρ)≤maxx∈X⁡F⁡(x,y¯T).V_proxy^δ(ρ)≤ _x∈ XF(x, y_T). And from the definition of gproxyδg_proxy^δ, gproxyδ(π¯T,ρ)≥miny∈Y⁡F⁡(x¯T,y).g_proxy^δ( π_T;ρ)≥ _y∈ YF( x_T,y). Subtracting the last two displays, Vproxyδ(ρ)−gproxyδ(π¯T,ρ)≤maxx∈X⁡F⁡(x,y¯T)−miny∈Y⁡F⁡(x¯T,y)=Gap⁡(z¯T)≤BT,δ.V_proxy^δ(ρ)-g_proxy^δ( π_T;ρ)≤ _x∈ XF(x, y_T)- _y∈ YF( x_T,y)= ( z_T)≤ B_T,δ. Combining the last three inequalities proves V⁡(ρ)−gH(π¯T,ρ)≤BT,δ+ργmin+δ.V(ρ)-g_H( π_T;ρ)≤ B_T,δ+ρ _ +δ. Since BT,δ=⁡(Lδ/T)B_T,δ=O(L_δ/T), this also yields V⁡(ρ)−gH(π¯T,ρ)=O⁡(LδT+ργmin+δ).V(ρ)-g_H( π_T;ρ)=O\! ( L_δT+ρ _ +δ ). Step 4: sharper guarantee under Assumption 2. Assume now Assumption 2 in addition. Then by Theorem 2, gproxyδ(π,ρ)=gHδ(π,ρ),Vproxyδ(ρ)=Vδ(ρ),∀π∈Πϵ.g_proxy^δ(π;ρ)=g_H^δ(π;ρ), V_proxy^δ(ρ)=V^δ(ρ), ∀π∈ _ε. Therefore, V⁡(ρ)−gH(π¯T,ρ)≤Vδ(ρ)−gH(π¯T,ρ)≤Vδ(ρ)−gHδ(π¯T,ρ)+δ.V(ρ)-g_H( π_T;ρ)≤ V^δ(ρ)-g_H( π_T;ρ)≤ V^δ(ρ)-g_H^δ( π_T;ρ)+δ. Using exact recovery and the proxy optimality bound already established, Vδ(ρ)−gHδ(π¯T,ρ)=Vproxyδ(ρ)−gproxyδ(π¯T,ρ)≤BT,δ.V^δ(ρ)-g_H^δ( π_T;ρ)=V_proxy^δ(ρ)-g_proxy^δ( π_T;ρ)≤ B_T,δ. Hence V⁡(ρ)−gH(π¯T,ρ)≤BT,δ+δ.V(ρ)-g_H( π_T;ρ)≤ B_T,δ+δ. Since BT,δ=⁡(Lδ/T)B_T,δ=O(L_δ/T), this gives the sharper rate V⁡(ρ)−gH(π¯T,ρ)=O⁡(LδT+δ).V(ρ)-g_H( π_T;ρ)=O\! ( L_δT+δ ). ∎ Appendix C Comparisons with Standard NLHF Compared with standard NLHF [29] and its following studies [54, 46, 30, 56, 52], the convergence analysis in our setting is substantially more involved. In nominal NLHF, the preference kernel is fixed, and the optimization problem is a regularized two-player zero-sum game between the main policy and the adversarial policy. Once the game is written in saddle-point form, the KL regularizers provide strong concavity in the leader policy and strong convexity in the follower policy, so one can analyze mirror descent or optimistic no-regret dynamics directly on a fixed convex–concave game. By contrast, our robust formulation introduces uncertainty in the preference kernel itself. The resulting objective is no longer a standard two-player nominal game, but a distributionally robust max-min problem in which the adversary chooses both a competing policy and a plausible preference kernel. This creates two new difficulties that do not appear in standard NLHF. First, the original hard-constrained problem does not retain the clean convex-concave structure of nominal NLHF. If one eliminates the follower using the Gibbs best response, the reduced objective becomes a concave minimization problem over the kernel ambiguity set, which is globally hard in general. On the other hand, if one keeps both the adversarial policy and the preference kernel as optimization variables, the inner minimization is jointly nonconvex because the win-rate term is bilinear in the pair (ν,p)(ν,p). Therefore, unlike in standard NLHF, one cannot analyze the original hard-constrained robust objective directly using a standard minimax or saddle-point argument. A further difficulty is that the kernel block acquires its curvature through the dual variable γ. In the nominal two-player game, the KL regularizers already provide fixed curvature in the policy blocks. In the four-player robust game, by contrast, the p-block is strongly convex only with modulus proportional to γ. This is why the lower bound γmin>0 _ >0 and the condition τγmin≥1/4τ _ ≥ 1/4 enter the analysis: they ensure that the KL curvature in the kernel block is strong enough to dominate the mixed bilinear interaction between the follower policy and the adversarial kernel. Without such a condition, the grouped (π,γ)(π,γ)-vs.-(ν,p)(ν,p) game need not be convex-concave, and the standard optimistic mirror-descent analysis does not go through. Our gradient-variation analysis is also substantially more coupled than in NLHF. In nominal NLHF, once the preference kernel is fixed, the gradient of each player depends only on the other policy. In our setting, the gradients evolve through a chain of dependencies: γ→p→(π,ν),γ→ p→(π,ν), so the variation of one block can depend on the movement of several other blocks. In particular, the p-gradient depends on the current policies and on the dual variable, while the γ-gradient is exactly the KL-budget slack of the current kernel. Establishing convergence therefore requires a sharper self-bounding argument that simultaneously controls all four regret terms and shows that the total gradient variation can be absorbed by the negative Bregman stability terms. This absorption step has no analogue in the standard NLHF proof. Appendix D Auxiliary Lemmas We prove several auxiliary lemmas required for the theorems. Lemma 1 (Universal upper bound on the dual multiplier). Fix (π,ν)∈Πϵ×Δm(π,ν)∈ _ε× _m, and define qπ,ν(γ):=minp∈PδJ(π,ν;p)+γ(DKLBern(p∥p⋆)−ρ),γ≥0.q_π,ν(γ):= _p∈ P_δ \J(π,ν;p)+γ (D_KL^Bern(p\|p )-ρ ) \, γ≥ 0. Then every maximizer of qπ,νq_π,ν over [0,∞)[0,∞) lies in [0,1/ρ][0,1/ρ]. In particular, one may choose the algorithmic upper bound γmax _ so that γmax≥1/ρ _ ≥ 1/ρ. Proof. Because p⋆∈Pδp ∈ P_δ, for every γ≥0γ≥ 0, qπ,ν(γ)≤J⁡(π,ν,p⋆)−ργ.q_π,ν(γ)≤ J(π,ν;p )-ργ. At γ=0γ=0, qπ,ν(0)=minp∈Pδ⁡J⁡(π,ν,p).q_π,ν(0)= _p∈ P_δJ(π,ν;p). For fixed (π,ν)(π,ν), the only part of J⁡(π,ν,p)J(π,ν;p) that depends on p is the win-rate term, which always lies in [0,1][0,1]. Hence J⁡(π,ν,p⋆)−qπ,ν(0)≤1.J(π,ν;p )-q_π,ν(0)≤ 1. Therefore, if γ>1/ργ>1/ρ, then qπ,ν(γ)≤J⁡(π,ν,p⋆)−ργ<J⁡(π,ν,p⋆)−1≤qπ,ν(0),q_π,ν(γ)≤ J(π,ν;p )-ργ<J(π,ν;p )-1≤ q_π,ν(0), so such a γ cannot be optimal. ∎ Lemma 2 (Blockwise OMD inequalities). For any comparators π∈Πϵπ∈ _ε, γ∈Γγ∈ , ν∈βν _β, and p∈Pδp∈ P_δ, the iterates produced by Algorithm 1 satisfy ∑t=1T⟨gtπ,π−πt⟩ _t=1^T g_t^π,π- _t ≤DKL(π∥π1)η+η∑t=1T∥gtπ−gt−1π∥∞2−12η∑t=1TDKL(πt∥πt+1), ≤ D_KL(π\| _1)η+η _t=1^T\|g_t^π-g_t-1^π\|_∞^2- 12η _t=1^TD_KL( _t\| _t+1), (37) ∑t=1Tgtγ(γ−γt) _t=1^Tg_t^γ(γ- _t) ≤(γ−γ1)22η+η∑t=1T|gtγ−gt−1γ|2−12η∑t=1T|γt−γt+1|2, ≤ (γ- _1)^22η+η _t=1^T|g_t^γ-g_t-1^γ|^2- 12η _t=1^T| _t- _t+1|^2, (38) ∑t=1T⟨gtν,νt−ν⟩ _t=1^T g_t^ν, _t-ν ≤DKL(ν∥ν1)η+η∑t=1T∥gtν−gt−1ν∥∞2−12η∑t=1TDKL(νt∥νt+1), ≤ D_KL(ν\| _1)η+η _t=1^T\|g_t^ν-g_t-1^ν\|_∞^2- 12η _t=1^TD_KL( _t\| _t+1), (39) ∑t=1T⟨gtp,pt−p⟩ _t=1^T g_t^p,p_t-p ≤DKLBern(p∥p1)η+η∑t=1T∥gtp−gt−1p∥22−12η∑t=1TDKLBern(pt∥pt+1). ≤ D_KL^Bern(p\|p_1)η+η _t=1^T\|g_t^p-g_t-1^p\|_2^2- 12η _t=1^TD_KL^Bern(p_t\|p_t+1). (40) Proof. Apply Lemma 5 blockwise: • to the π-block with entropy geometry and maximization; • to the γ-block with Euclidean geometry and maximization; • to the ν-block with entropy geometry and minimization; • to the p-block with binary-entropy geometry and minimization. The dual norms are ℓ∞ _∞ for π and ν, the Euclidean norm for γ, and ℓ2 _2 for p. ∎ The constant-step OGDA proof requires a uniform bound on blockwise gradient variation. Lemma 3 (Gradient-variation bounds). Let πref,min:=min1≤i≤m⁡πref,i,Lπ:=τϵπref,min,Lν:=τβπref,min,Lp:=γmaxδ⁡(1−δ), _ref, := _1≤ i≤ m _ref,i, L_π:= τε _ref, , L_ν:= τβ _ref, , L_p:= _ δ(1-δ), and Mδ:=2log⁡1−δ.M_δ:=2 1-δ. For t≥2t≥ 2, let zt=(πt,γt,νt,pt)∈X×Y.z_t=( _t, _t, _t,p_t)∈ X× Y. Then the block gradients satisfy ‖gtπ−gt−1π‖∞ \|g_t^π-g_t-1^π\|_∞ ≤Lπ‖πt−πt−1‖1+‖νt−νt−1‖1+‖pt−pt−1‖2, ≤ L_π\| _t- _t-1\|_1+\| _t- _t-1\|_1+\|p_t-p_t-1\|_2, (41) |gtγ−gt−1γ| |g_t^γ-g_t-1^γ| ≤dMδ‖pt−pt−1‖2, ≤ d\,M_δ\,\|p_t-p_t-1\|_2, (42) ‖gtν−gt−1ν‖∞ \|g_t^ν-g_t-1^ν\|_∞ ≤‖πt−πt−1‖1+Lν‖νt−νt−1‖1+‖pt−pt−1‖2, ≤\| _t- _t-1\|_1+L_ν\| _t- _t-1\|_1+\|p_t-p_t-1\|_2, (43) ‖gtp−gt−1p‖2 \|g_t^p-g_t-1^p\|_2 ≤d‖πt−πt−1‖1+d‖νt−νt−1‖1+Lp‖pt−pt−1‖2+dMδ|γt−γt−1|. ≤ d\,\| _t- _t-1\|_1+ d\,\| _t- _t-1\|_1+L_p\|p_t-p_t-1\|_2+ d\,M_δ\,| _t- _t-1|. (44) Consequently, if we define for t≥2t≥ 2 Stπ:=‖πt−πt−1‖12,Stγ:=|γt−γt−1|2,Stν:=‖νt−νt−1‖12,Stp:=‖pt−pt−1‖22,S_t^π:=\| _t- _t-1\|_1^2, S_t^γ:=| _t- _t-1|^2, S_t^ν:=\| _t- _t-1\|_1^2, S_t^p:=\|p_t-p_t-1\|_2^2, and L:=max⁡1,Lπ,Lν,Lp,d,dMδ,L:= \1,\,L_π,\,L_ν,\,L_p,\, d,\, d\,M_δ \, then ‖gtπ−gt−1π‖∞2 \|g_t^π-g_t-1^π\|_∞^2 ≤3L2(Stπ+Stν+Stp), ≤ 3L^2 (S_t^π+S_t^ν+S_t^p ), (45) |gtγ−gt−1γ|2 |g_t^γ-g_t-1^γ|^2 ≤L2Stp, ≤ L^2S_t^p, (46) ‖gtν−gt−1ν‖∞2 \|g_t^ν-g_t-1^ν\|_∞^2 ≤3L2(Stπ+Stν+Stp), ≤ 3L^2 (S_t^π+S_t^ν+S_t^p ), (47) ‖gtp−gt−1p‖22 \|g_t^p-g_t-1^p\|_2^2 ≤4L2(Stπ+Stγ+Stν+Stp). ≤ 4L^2 (S_t^π+S_t^γ+S_t^ν+S_t^p ). (48) Proof. We prove the first-order bounds and then square them. The π-block. From (51) of Lemma 6, gπ(i)=∑j=1mνjPp(i≻j)−τ⁡(log⁡πiπref,i+1).g^π(i)= _j=1^m _jP_p(i j)-τ ( _i _ref,i+1 ). For the win-rate component, since ∑jνj=1 _j _j=1, |∑jνt,jPpt(i≻j)−∑jνt−1,jPpt−1(i≻j)| | _j _t,jP_p_t(i j)- _j _t-1,jP_p_t-1(i j) | ≤|∑j(νt,j−νt−1,j)Ppt(i≻j)|+|∑jνt−1,j(Ppt(i≻j)−Ppt−1(i≻j))| ≤ | _j( _t,j- _t-1,j)P_p_t(i j) |+ | _j _t-1,j (P_p_t(i j)-P_p_t-1(i j) ) | ≤‖νt−νt−1‖1+‖pt−pt−1‖∞≤‖νt−νt−1‖1+‖pt−pt−1‖2. ≤\| _t- _t-1\|_1+\|p_t-p_t-1\|_∞≤\| _t- _t-1\|_1+\|p_t-p_t-1\|_2. For the KL term, since every πt(i)≥ϵπref,i≥ϵπref,min _t(i)≥ε _ref,i≥ε _ref, , |log⁡πt(i)πref,i−log⁡πt−1(i)πref,i|=|log⁡πt(i)−log⁡πt−1(i)|≤|πt(i)−πt−1(i)|ϵπref,min | _t(i) _ref,i- _t-1(i) _ref,i |=| _t(i)- _t-1(i)|≤ | _t(i)- _t-1(i)|ε _ref, by the mean-value theorem. Taking the maximum over i proves (41). The γ-block. By (52) of Lemma 6, gγ=DKLBern(p∥p⋆)−ρ.g^γ=D_KL^Bern(p\|p )-ρ. For each coordinate e=(i,j)∈Ee=(i,j)∈ E, the derivative of the scalar Bernoulli-KL term with respect to pep_e is he′(u)=log⁡u⁡(1−pe⋆)pe⋆(1−u).h_e (u)= u(1-p_e )p_e (1-u). Since both u and pe⋆p_e lie in [δ,1−δ][δ,1-δ], we have |he′(u)|≤|logit⁡(u)|+|logit⁡(pe⋆)|≤2log⁡1−δ=Mδ.|h_e (u)|≤| (u)|+| (p_e )|≤ 2 1-δ=M_δ. Therefore, by the mean-value theorem and Cauchy–Schwarz, |gtγ−gt−1γ|≤∑e=1dMδ|pt,e−pt−1,e|≤dMδ‖pt−pt−1‖2,|g_t^γ-g_t-1^γ|≤ _e=1^dM_δ\,|p_t,e-p_t-1,e|≤ d\,M_δ\,\|p_t-p_t-1\|_2, which proves (42). The ν-block. The proof is identical to the π-block, now using the floor νt,j≥βπref,j≥βπref,min _t,j≥β _ref,j≥β _ref, . This gives (43). The p-block. From (54) of Lemma 6, gijp=wij(π,ν)+γhij(pij),hij(u):=log⁡u⁡(1−pij⋆)pij⋆(1−u).g^p_ij=w_ij(π,ν)+γ\,h_ij(p_ij), h_ij(u):= u(1-p_ij )p_ij (1-u). For the antisymmetric weights, fix (i,j)∈E(i,j)∈ E. Then |wij(πt,νt)−wij(πt−1,νt−1)| |w_ij( _t, _t)-w_ij( _t-1, _t-1)| =|πt,iνt,j−πt,jνt,i−πt−1,iνt−1,j+πt−1,jνt−1,i| =| _t,i _t,j- _t,j _t,i- _t-1,i _t-1,j+ _t-1,j _t-1,i| ≤|πt,i−πt−1,i|+|πt,j−πt−1,j|+|νt,i−νt−1,i|+|νt,j−νt−1,j| ≤| _t,i- _t-1,i|+| _t,j- _t-1,j|+| _t,i- _t-1,i|+| _t,j- _t-1,j| ≤‖πt−πt−1‖1+‖νt−νt−1‖1. ≤\| _t- _t-1\|_1+\| _t- _t-1\|_1. Hence ‖w⁡(πt,νt)−w⁡(πt−1,νt−1)‖2≤d‖πt−πt−1‖1+d‖νt−νt−1‖1.\|w( _t, _t)-w( _t-1, _t-1)\|_2≤ d\,\| _t- _t-1\|_1+ d\,\| _t- _t-1\|_1. For the KL-log term, |hij′(u)|=1u⁡(1−u)≤1δ⁡(1−δ),|h_ij (u)|= 1u(1-u)≤ 1δ(1-δ), so ‖h⁡(pt)−h⁡(pt−1)‖2≤1δ⁡(1−δ)‖pt−pt−1‖2.\|h(p_t)-h(p_t-1)\|_2≤ 1δ(1-δ)\|p_t-p_t-1\|_2. Also |hij(u)|≤Mδ|h_ij(u)|≤ M_δ on [δ,1−δ][δ,1-δ], so ‖h⁡(pt−1)‖2≤dMδ.\|h(p_t-1)\|_2≤ d\,M_δ. Therefore ‖gtp−gt−1p‖2 \|g_t^p-g_t-1^p\|_2 ≤‖w⁡(πt,νt)−w⁡(πt−1,νt−1)‖2+γmax‖h⁡(pt)−h⁡(pt−1)‖2+|γt−γt−1|‖h⁡(pt−1)‖2 ≤\|w( _t, _t)-w( _t-1, _t-1)\|_2+ _ \|h(p_t)-h(p_t-1)\|_2+| _t- _t-1|\,\|h(p_t-1)\|_2 ≤d‖πt−πt−1‖1+d‖νt−νt−1‖1+Lp‖pt−pt−1‖2+dMδ|γt−γt−1|, ≤ d\,\| _t- _t-1\|_1+ d\,\| _t- _t-1\|_1+L_p\|p_t-p_t-1\|_2+ d\,M_δ\,| _t- _t-1|, which proves (44). Finally, (45)–(48) follow from (a+b+c)2≤3(a2+b2+c2),(a+b+c+d)2≤4(a2+b2+c2+d2).(a+b+c)^2≤ 3(a^2+b^2+c^2), (a+b+c+d)^2≤ 4(a^2+b^2+c^2+d^2). ∎ Lemma 4 (Gradient variation is absorbed by the stability terms). Let VT:=η∑t=1T(‖gtπ−gt−1π‖∞2+|gtγ−gt−1γ|2+‖gtν−gt−1ν‖∞2+‖gtp−gt−1p‖22),V_T:=η _t=1^T (\|g_t^π-g_t-1^π\|_∞^2+|g_t^γ-g_t-1^γ|^2+\|g_t^ν-g_t-1^ν\|_∞^2+\|g_t^p-g_t-1^p\|_2^2 ), and ΞT:=12η∑t=1T(DKL(πt∥πt+1)+|γt−γt+1|2+DKL(νt∥νt+1)+DKLBern(pt∥pt+1)). _T:= 12η _t=1^T (D_KL( _t\| _t+1)+| _t- _t+1|^2+D_KL( _t\| _t+1)+D_KL^Bern(p_t\|p_t+1) ). If η≤18L,η≤ 18L, (49) where L is the constant from Lemma 3, then VT≤ΞT.V_T≤ _T. (50) Proof. Because g0∙=g1∙g_0 =g_1 , the t=1t=1 contribution to VTV_T is zero. Thus we sum from t=2t=2 onward. By Lemma 3, VT≤11L2η∑t=2T(Stπ+Stγ+Stν+Stp).V_T≤ 11L^2η _t=2^T (S_t^π+S_t^γ+S_t^ν+S_t^p ). On the other hand, Pinsker’s inequality and the Bernoulli Pinsker bound give DKL(πt∥πt+1)≥12∥πt−πt+1∥12=12St+1π,D_KL( _t\| _t+1)≥ 12\| _t- _t+1\|_1^2= 12S_t+1^π, DKL(νt∥νt+1)≥12∥νt−νt+1∥12=12St+1ν,D_KL( _t\| _t+1)≥ 12\| _t- _t+1\|_1^2= 12S_t+1^ν, DKLBern(pt∥pt+1)≥2∥pt−pt+1∥22=2St+1p.D_KL^Bern(p_t\|p_t+1)≥ 2\|p_t-p_t+1\|_2^2=2S_t+1^p. Hence ΞT _T ≥14η∑t=1TSt+1π+12η∑t=1TSt+1γ+14η∑t=1TSt+1ν+1η∑t=1TSt+1p ≥ 14η _t=1^TS_t+1^π+ 12η _t=1^TS_t+1^γ+ 14η _t=1^TS_t+1^ν+ 1η _t=1^TS_t+1^p ≥14η∑t=2T+1(Stπ+Stγ+Stν+Stp) ≥ 14η _t=2^T+1 (S_t^π+S_t^γ+S_t^ν+S_t^p ) ≥14η∑t=2T(Stπ+Stγ+Stν+Stp). ≥ 14η _t=2^T (S_t^π+S_t^γ+S_t^ν+S_t^p ). Therefore VT≤ΞTV_T≤ _T whenever 11L2η≤14η,i.e.η≤1211L.11L^2η≤ 14η, .e. η≤ 12 11\,L. The stronger condition (49) is sufficient. ∎ Lemma 5 (Standard optimistic mirror-descent bound). Let h be a mirror map with Bregman divergence DhD_h, let η>0η>0, and let g^t=2gt−gt−1 g_t=2g_t-g_t-1. For minimization, the update zt+1=argminz∈Z⁡⟨g^t,z⟩+1ηDh(z,zt)z_t+1= _z∈ Z \ g_t,z + 1ηD_h(z,z_t) \ satisfies, for every comparator u∈Zu∈ Z, ∑t=1T⟨gt,zt−u⟩≤Dh(u,z1)η+η∑t=1T‖gt−gt−1‖∗2−12η∑t=1TDh(zt+1,zt). _t=1^T g_t,z_t-u ≤ D_h(u,z_1)η+η _t=1^T\|g_t-g_t-1\|_*^2- 12η _t=1^TD_h(z_t+1,z_t). For maximization, the analogous update with argmax satisfies ∑t=1T⟨gt,u−zt⟩≤Dh(u,z1)η+η∑t=1T‖gt−gt−1‖∗2−12η∑t=1TDh(zt+1,zt). _t=1^T g_t,u-z_t ≤ D_h(u,z_1)η+η _t=1^T\|g_t-g_t-1\|_*^2- 12η _t=1^TD_h(z_t+1,z_t). Lemma 6 (Partial gradients). For z=(π,γ,ν,p)∈X×Yz=(π,γ,ν,p)∈ X× Y, the block gradients of F are: gπ(i) g^π(i) =∑j=1mνjPp(i≻j)−τ(logπiπref,i+1),i=1,…,m, = _j=1^m _j\,P_p(i j)-τ ( _i _ref,i+1 ), i=1,…,m, (51) gγ g^γ =DKLBern(p∥p⋆)−ρ, =D_KL^Bern(p\|p )-ρ, (52) gν(j) g^ν(j) =∑i=1mπiPp(i≻j)+τ(logνjπref,j+1),j=1,…,m, = _i=1^m _i\,P_p(i j)+τ ( _j _ref,j+1 ), j=1,…,m, (53) gijp g^p_ij =wij(π,ν)+γlog⁡pij(1−pij⋆)pij⋆(1−pij),(i,j)∈E. =w_ij(π,ν)+γ p_ij(1-p _ij)p _ij(1-p_ij), (i,j)∈ E. (54) Proof. Differentiate (17). The π- and ν-gradients are obtained from the linear win-rate term and the KL regularizers. The γ-gradient is immediate because F is affine in γ. For (54), the coefficient of pijp_ij in the win-rate decomposition is wij(π,ν)w_ij(π,ν), and ∂pijDKLBern(p∥p⋆)=logpij(1−pij⋆)pij⋆(1−pij).∎ ∂ p_ijD_KL^Bern(p\|p )= p_ij(1-p _ij)p _ij(1-p_ij). Appendix E IMDb Continuation Task: Experiment Details E.1 Nominal Preference and Annotator Groups We use the IMDb review corpus [23] as a source of continuation prompts. Each prompt is the first sentence of a review; the task is to continue it. We sample N=8,000N=8,000 prompts and generate m=8m=8 candidate continuations per prompt using google/flan-t5-large [11] with diverse instruction templates and sampling temperatures. The full set of prompt-level preference kernels is pre-computed and cached so that NashMD and all robust variants are trained and evaluated on identical finite action sets. To model heterogeneous human preferences we construct three annotator groups: Group A (comprehensive/analytical) prefers longer, more detailed continuations with stronger evidence of analysis; Group B (efficient/concise) prefers short, direct, and decisive continuations; and Group C (balanced/nuanced) prefers responses that acknowledge both strengths and weaknesses. For prompt x and group g∈A,B,Cg∈\A,B,C\, each candidate i receives a scalar score si(g)s_i^(g). These scores are converted into a group-specific Bradley–Terry pairwise preference kernel: Pij(g)=σ⁡(β⁡(si(g)−sj(g))),i<j,P_ij^(g)=σ\! (β\,(s_i^(g)-s_j^(g)) ), i<j, (55) with Pji(g)=1−Pij(g)P_ji^(g)=1-P_ij^(g) and Pii(g)=12P_i^(g)= 12. Writing p(g)∈[0,1]dp^(g)∈[0,1]^d, d=(m2)=28d= m2=28, for the upper-triangular vector, the nominal preference kernel is p⋆=∑g∈A,B,Cαgp(g)p = _g∈\A,B,C\ _g\,p^(g) with fixed mixture weights α. Perturbing the mixture weights away from α via a Dirichlet draw produces group-shift kernels that lie in the convex hull of the group kernels and serve as a model of structured population-level preference shift (Section E.4). E.2 Optimization Algorithms NashMD baseline. The NashMD baseline solves the nominal KL-regularized game maxπ∈Δmminν∈ΔmJ(π,ν;P⋆)=π⊤P⋆ν−τDKL(π∥πref)+τDKL(ν∥πref) _π∈ ^m _ν∈ ^mJ(π,ν;P )=π P ν-τ\,D_KL(π\| _ref)+τ\,D_KL(ν\| _ref) (56) with nominal kernel P⋆=P⁡(p⋆)P =P(p ) fixed throughout training. We use simultaneous Optimistic Gradient Descent-Ascent (OGDA) on the policy pair (π,ν)(π,ν) with step size ηNash=0.30 _Nash=0.30. Four-player robust algorithm. The robust variant hedges against kernel misspecification within the Bernoulli-KL ball ρ=p:DKLBern(p∥p⋆)≤ρU_ρ=\p:D_KL^Bern(p\|p )≤ρ\ by solving maxπ∈Δmminp:DKLBern(p∥p⋆)≤ρp∈[δ,1−δ]dminν∈ΔmJ(π,ν;P(p)). _π∈ ^m _ subarraycp:\,D_KL^Bern(p\|p )≤ρ\\ p∈[δ,1-δ]^d subarray _ν∈ ^mJ(π,ν;P(p)). (57) The inner minimization is solved approximately by alternating between the closed-form follower best-response and the kernel update pij←σ⁡(logit⁡(pij⋆)−wij(π,ν)/max⁡(γ,γmin))p_ij←σ\! ( (p _ij)-w_ij(π,ν)/\! (γ, _ ) ) for K=8K=8 inner steps, then the outer (π,γ)(π,γ) update is performed with OMD at step sizes (ηπ,ηγ)=(0.25,0.15)( _π, _γ)=(0.25,0.15). E.3 Hyperparameters Table 3: Hyperparameters for the IMDb continuation experiments. Component Value Candidates per prompt m 88 Kernel dimension d (82)=28 82=28 Number of prompts N 8,0008,000 Generator model flan-t5-large KL temperature τ 0.001,0.005,0.01,0.05,0.1\0.001,0.005,0.01,0.05,0.1\ NashMD step size ηNash _Nash 0.300.30 Robust step sizes (ηπ,ηγ)( _π, _γ) (0.25, 0.15)(0.25,\,0.15) Inner steps K 88 Training radii ρtrain _train 0.005, 0.01, 0.02, 0.05, 0.10, 0.15\0.005,\,0.01,\,0.02,\,0.05,\,0.10,\,0.15\ Evaluation radii ρeval _eval 0.05, 0.10, 0.15, 0.20, 0.30, 0.50, 0.75, 1.0\0.05,\,0.10,\,0.15,\,0.20,\,0.30,\,0.50,\,0.75,\,1.0\ Kernel clipping δ 10−310^-3 Dual range [γmin,γmax][ _ , _ ] [0.02, 500][0.02,\,500] E.4 Results Given the robust policy πR _R and the NashMD baseline πN _N, we measure the win rate using 24. The sampled kernels for evaluation are (i) the KL-ball boundary ρevalU_ _eval sampled uniformly at random, used for Table 1 results, and (i) a Dirichlet-perturbed annotator mixture p~=∑gα~gp(g) p= _g α_gp^(g), α~∼Dirichlet⁡(κα) α (κα), projected to the KL-ball boundary at radius ρeval _eval, used in Table 4. Annotator group-shift kernels remain in the low-dimensional mixture subspace, making them a more structured perturbation family. Table 4: WinRate⁡(πR vs πN,ρeval)WinRate( _R vs _N;\, _eval) under group-shift kernels at τ=0.01τ=0.01, averaged over 8,0008,000 prompts. Bold marks the highest value in each column. ρtrain _train ρeval=0.05 _eval=0.05 0.100.10 0.200.20 0.300.30 0.500.50 0.005 0.974 0.945 0.897 0.863 0.818 0.01 0.974 0.944 0.897 0.862 0.818 0.02 0.973 0.943 0.896 0.862 0.818 0.05 0.970 0.939 0.893 0.859 0.816 0.10 0.962 0.928 0.884 0.851 0.811 0.15 0.917 0.882 0.840 0.811 0.775 Table 5 reports WinRate⁡(πR vs πN,ρeval)WinRate( _R vs _N;\, _eval) at the best training radius for each KL-regularization temperature τ∈0.001,0.005,0.01,0.05,0.1τ∈\0.001,0.005,0.01,0.05,0.1\. The strongest overall regime is τ=0.05τ=0.05: the robust policy wins with probability above 97%97\% for ρeval≤0.30 _eval≤ 0.30. At very small τ the NashMD baseline is already highly concentrated, reducing the marginal gain from robust training; at large τ both policies approach the uniform distribution and the adversary has limited room to exploit kernel perturbations. Table 5: WinRate⁡(πR vs πN,ρeval)WinRate( _R vs _N;\, _eval) at the best ρtrain _train for each τ (KL-random kernels). τ best ρtr _tr ρeval=0.05 _eval=0.05 0.100.10 0.200.20 0.300.30 0.500.50 1.01.0 0.001 0.005 0.972 0.944 0.896 0.859 0.810 0.735 0.005 0.005 0.980 0.956 0.911 0.877 0.826 0.750 0.01 0.005 0.986 0.967 0.929 0.896 0.847 0.768 0.05 0.10 0.996 0.995 0.988 0.977 0.948 0.879 0.1 0.10 0.945 0.931 0.903 0.880 0.841 0.775 Figure 2 shows the full win-rate curves at τ=0.05τ=0.05. Training radii ρtrain≤0.10 _train≤ 0.10 are statistically indistinguishable from one another at every evaluation radius (their pairwise differences never exceed the 95% confidence half-width), while ρtrain=0.15 _train=0.15 is significantly worse, with the gap widening as ρeval _eval grows. (a) KL-random perturbations (b) Group-shift perturbations Figure 2: WinRate⁡(πR vs πN,ρeval)WinRate( _R vs _N;\, _eval) at τ=0.05τ=0.05, for each training radius ρtrain∈0.005,0.01,0.02,0.05,0.10,0.15 _train∈\0.005,0.01,0.02,0.05,0.10,0.15\. Radii ρtrain∈[0.005,0.10] _train∈[0.005,0.10] (gray band and mean) are statistically tied at every evaluation radius; ρtrain=0.15 _train=0.15 (red) is significantly worse, and increasingly so as ρeval _eval grows. Table 6 gives the full (τ,ρtrain)(τ, _train) interaction at ρeval=0.30 _eval=0.30. For τ≤0.01τ≤ 0.01 the smallest training radius is uniformly best. For τ≥0.05τ≥ 0.05 the optimum shifts to ρtrain=0.10 _train=0.10, though the improvement over smaller radii is modest. Table 6: WinRate⁡(πR vs πN,ρeval)WinRate( _R vs _N;\, _eval) at ρeval=0.30 _eval=0.30, by (τ,ρtrain)(τ, _train) (KL-random kernels). Bold marks the row-wise maximum. τ\ρtrτ\; \; _tr 0.005 0.01 0.02 0.05 0.10 0.15 0.001 0.860 0.857 0.850 0.832 0.813 0.780 0.005 0.877 0.874 0.870 0.857 0.831 0.784 0.01 0.896 0.895 0.893 0.883 0.866 0.810 0.05 0.976 0.976 0.976 0.976 0.977 0.972 0.1 0.859 0.860 0.862 0.868 0.880 0.717 Appendix F TL;DR Summarization Task: Experiment Details F.1 Fine-Tuning Models. We initialize the models from Qwen2.5-1.5B-Instruct [51] and Gemma-2-2B [44] as both the initial and the reference model across the respective experiments and fine-tune with LoRA [19] for 5000 gradient steps with KL regularization τ=0.005τ=0.005. The four-player game formulation was compared with adversarial budget ρ∈0.01,0.05ρ∈\0.01,0.05\. The leader and follower policies carry separate LoRA adapters attached to the same frozen backbone. F.2 Nominal Preference Proxy We train a Bradley-Terry(BT) preference judge model to serve as the nominal preference proxy. The judge is a scalar score model sϕ:(x,y)→ℝs_φ:(x,y) built on DistilBERT-base-uncase [39] backbone with a linear scalar head, and fine-tuned with the standard BT cross-entropy loss on the TL;DR dataset: ℒBT(ϕ)=−(x,yw,yl)[log⁡σ⁡(sϕ(x,yw)−sϕ(x,yl))],L_BT(φ)=-E_(x,y_w,y_l) [ σ (s_φ(x,y_w)-s_φ(x,y_l) ) ], (58) where (yw,yl)(y_w,y_l) denotes the preferred and dispreferred responses respectively. Training uses 83,573 comparison pairs. After 3 epochs the judge achieves 72% validation accuracy. During training with both NashMD and Four-Player the pairwise preference is calculated as: P∗(ya≻yb∣x)=σ⁡(sϕ(x,ya)−sϕ(x,yb))P^*(y_a y_b x)=σ (s_φ(x,y_a)-s_φ(x,y_b) ) (59) F.3 Adversary Proxy The kernel-adversary in the four-player game is parameterized by an antisymmetric perturbation head δψ(ya,yb) _ψ(y_a,y_b) of the nominal preference logit. The head computes δψ(ya,yb)=gψ(uab)−gψ(uba),gψ:ℝ7→ℝ, _ψ(y_a,y_b)\;=\;g_ψ(u_ab)-g_ψ(u_ba), g_ψ:R^7 , (60) where gψg_ψ is a two-layer MLP: ℝ7→W1ℝ64→tanhℝ64→W2ℝR^7 \;W_1\;R^64 \; \;R^64 \;W_2\;R, with the output layer W2W_2 initialized to zero so that δψ(ya,yb)=0 _ψ(y_a,y_b)=0 (no perturbation) at initialization. The algorithm therefore initializes at the nominal preference and gradually grows the adversarial perturbation. Antisymmetry in (60) ensures δψ(ya,yb)=−δψ(yb,ya) _ψ(y_a,y_b)=- _ψ(y_b,y_a) and hence Pψ(ya≻yb)+Pψ(yb≻ya)=1P_ψ(y_a y_b)+P_ψ(y_b y_a)=1 at all iterates. The perturbed preference is Pψ(ya≻yb∣x)=σ⁡(logit⁡P∗(ya≻yb∣x)+δψ(ya,yb)).P_ψ(y_a y_b x)\;=\;σ\! ( P^*(y_a y_b x)+ _ψ(y_a,y_b) ). (61) The Bernoulli-KL ambiguity constraint [DKLBern(pψ∥p⋆)]≤ρE [D_KL^Bern(p_ψ\,\|\,p ) ]≤ρ is enforced softly through the dual variable γ in the Lagrangian payoff. We additionally clip the predicted logit shift before applying the sigmoid in 61 to prevent PψP_ψ from saturating to 0,1\0,1\ to stabilize the dual ascent. F.4 Losses. For a mini-batch with samples yL∼πLy_L _L and yF∼πFy_F _F, we define pψ=Pψ(yL≻yF∣x)p_ψ=P_ψ(y_L y_F x) and KL proxies κL=clip⁡(log⁡πL(yL)/πref(yL),−c,c) _L=clip( _L(y_L)/ _ref(y_L),-c,c), κF=clip⁡(log⁡πF(yF)/πref(yF),−c,c) _F=clip( _F(y_F)/ _ref(y_F),-c,c). The three player losses are ℒψ _ψ =[pψ]+γ([DKLBern(pψ∥P∗)]−ρ), =E [p_ψ ]+γ\, (E[D_KL^Bern(p_ψ\,\|\,P^*)]-ρ ), (62) ℒπ _π =−⁡[(pψ−τκL−bL)log⁡πL(yL)], =-E [(p_ψ-τ _L-b_L)\, _L(y_L) ], (63) ℒν _ν =−⁡[(−pψ−τκF−bF)log⁡πF(yF)], =-E [(-p_ψ-τ _F-b_F)\, _F(y_F) ], (64) where bLb_L, bFb_F are EMA baselines for variance reduction, and the adversary ψ maximizes via (62) (gradient ascent) while the dual variable is updated by clipped projected gradient ascent: γ←clip⁡(γ+ηγ(⁡[DKLBern]−ρ), 0,γmax).γ\;←\;clip\! (γ+ _γ\, (E[D_KL^Bern]-ρ ),\;0,\; _ ). (65) Each outer step interleaves K adversary updates per (πL,πF,γ)( _L, _F,γ) update. We default to K=3K=3 as we obseved K=1K=1 is too weak (the constraint is far from active) while K≥3K≥ 3 tracks the constraint budget closely. F.5 Evaluation with LLM Judges We employ three comparatively larger open-source models—DeepSeek-R1-Distill-Qwen-32B [12], Gemma-4-31B [44], and Nemotron-3-Nano-Omni [32]—to obtain preference signals querying with the following prompt template: You are an expert judge evaluating the quality of two generated summaries. Post to summarize: ⟨text⟩ Summary A: ⟨summary1⟩ Summary B: ⟨summary2⟩ Evaluate which summary is better based on: (1) Accuracy, (2) Coherence, (3) Conciseness, (4) Helpfulness. where ⟨text⟩ corresponds to the Reddit post x, ⟨summary1⟩ 1 to the policy response y∼π(⋅∣x)y π(· x), and ⟨summary2⟩ 2 to the baseline response y′∼πbase(⋅∣x)y _base(· x). We estimate the preference probability Pjext(π≻πbase)P^ext_j(π _base) as the fraction of comparisons in which the policy output is preferred over 10001000 prompts from the TL;DR dataset. Following Munos et al. [29], each comparison is treated as a Bernoulli trial and uncertainty is quantified using Clopper–Pearson 95% confidence intervals; for n=1000n=1000, the worst-case half-width is approximately ±0.032± 0.032. Four player policies preserve strong nominal performance against the NashMD baseline. Training against the preference uncertainty has not collapsed into overly conservative solutions, rather, the robust policies still improve the nominal pairwise objective with the pairwise judges across evaluation criteria. F.6 Non-BT Nominal Kernel: Construction and Diagnostic The TL;DR experiment in Section 6.3 uses a non-Bradley–Terry (non-BT) nominal kernel rather than the standard, reproducible Bradley–Terry (BT) construction (BT judge, Appendix F.2) [41], which is antisymmetric by design: P⋆(ya≻yb∣x)=σ⁡(sϕ(x,ya)−sϕ(x,yb))P (y_a y_b x)=σ (s_φ(x,y_a)-s_φ(x,y_b) ). The ambiguity set P in Eq. (8) is a Bernoulli-KL ball over the d=(m2)d= m2 pairwise probabilities themselves (imposed in expectation over sampled pairs in TL;DR, Appendix F.4), with no constraint of the form logit⁡(pij)=si−sjlogit(p_ij)=s_i-s_j: even from a BT center, the worst-case kernel found by the adversary may already be cyclic or otherwise not representable by any scalar reward. We go further and construct a nominal kernel that is itself non-BT, described next, together with a diagnostic confirming it is measurably so. Non-BT nominal kernel. We replace the BT judge with a directly parameterized pairwise kernel, Pϕpair(ya≻yb∣x)=σ⁡(Lpair(x,ya,yb)),Lpair(x,ya,yb)=c2[qϕ(x,ya,yb)−qϕ(x,yb,ya)],P^pair_φ(y_a y_b x)=σ (L^pair(x,y_a,y_b) ), L^pair(x,y_a,y_b)= c2 [q_φ(x,y_a,y_b)-q_φ(x,y_b,y_a) ], where qϕq_φ is a single joint forward pass over the post and both candidate summaries, instantiated with PairRM [21] (a 0.4B DeBERTa-v3-large cross-encoder). Each candidate’s representation is conditioned on its opponent, so its score is not a function of (x,y)(x,y) alone and the kernel is not constrained to be BT. Because qϕq_φ is order-dependent, the difference of the two orderings is what gives Lpair(a,b)=−Lpair(b,a)L^pair(a,b)=-L^pair(b,a), hence Ppair(a≻b)+Ppair(b≻a)=1P^pair(a b)+P^pair(b a)=1 exactly. Here c>0c>0 is a scalar temperature fit by maximum likelihood on TL;DR human comparisons. The policy is Qwen2.5-1.5B-Instruct. The adversary head, Bernoulli-KL constraint, dual update and NashMD baseline are as in Appendix F.3–F.4; the only change is that three of the adversary’s seven input features, which used per-response rewards, now carry the pairwise logit. The kernel is not BT. Let Lij=logitPϕpair(yi≻yj∣x)L_ij=logit\,P^pair_φ(y_i y_j x). A BT kernel has Lij=si−sjL_ij=s_i-s_j, so for every triple C:=Lij+Ljk+Lki=0C:=L_ij+L_jk+L_ki=0: the scores telescope away, for any reward model. The least-squares BT fit to a triple removes C/3C/3 from each edge, so |C|/3|C|/3 is the per-edge distance to the nearest BT kernel. On 1,000 triples of real model outputs (per prompt, one summary from each of NashMD-PG and the two robust policies trained under this kernel), the mean best-BT projection error is 0.3840.384 logits per edge (3.103.10 percentage points in preference probability), the median |C||C| is 0.960.96, and 1.4%1.4\% of triples are outright cyclic. Edges are confident (|L|≈3|L|≈ 3 logits on average), so a 0.380.38-logit residual rarely flips a sign: cycles are rare, yet the departure from BT is present in nearly every triple. The nominal kernel used in training is therefore measurably non-BT. F.7 Additional Results with a Bradley-Terry Nominal Kernel For completeness, we also report results under the standard Bradley–Terry (BT) nominal kernel (Appendix F.2) that Section 6.3’s non-BT experiment departs from. Table 7 reports the win rate of the robust policy against NashMD on the same protocol as Table 2 (three external judges, 10001000 TL;DR prompts, Clopper–Pearson half-width ≈±0.032≈± 0.032 at n=1000n=1000), and additionally covers a second fine-tuning backbone, Gemma-2B, alongside Qwen-1.5B. Table 7: Win rate against NashMD under external LLM judges, evaluated on 10001000 TL;DR prompts, with a Bradley-Terry nominal kernel (cf. Table 2, which uses a non-BT kernel). Fine-Tuning Model ρtrain _train Win Rate under Evaluation Model DeepSeek-R1 Gemma-4 Nemotron-3 Qwen-1.5B 0.01 0.496 0.506 0.764 0.05 0.574 0.593 0.568 Gemma-2B 0.01 0.542 0.518 0.462 0.05 0.640 0.720 0.583 Appendix G LLM Judge Reasoning Examples Example 1 (prompt_id 57) Disagreement: Nemotron-3-Nano → A; Gemma-4-31B → B; DeepSeek-R1-32B → B Post (r/relationships) Title: Me [20m] just got out of an 8 month relationship with my gf [19F]. I am very emotional, she wasn’t. I am far from the stereotypical macho guy. I love romance […] She never speaks about her feelings, never told me she loved me without me having to say it first […] My question aimed at both genders: Are girls generally emotional, do you ladies like to speak about emotions? Do you generally like when the guy is cute with you and calls you babe, and says he loves you? […] Summary A (NashMD-PG) OP is seeking advice on whether girls generally like to be romantic and express emotions, and if they prefer a partner who is affectionate and expresses love. Summary B (Four-Player ρ=0.05ρ=0.05) The OP is seeking to understand if girls generally express emotions and appreciate romantic displays of affection, as he is a romantic guy who enjoys being cute and expressing his love. He’s questioning if this is something most girls find appealing. Nemotron-3-Nano (verdict: A, 14.4 s) [Conciseness] A is more concise (fewer words). B includes extra info about his own traits—possible unnecessary padding. So A wins on conciseness. […] [Helpfulness] A directly states the request for advice. B adds context that may help reader understand OP’s perspective, but the key point—seeking advice about girls’ preferences—is stated more succinctly in A. […] Overall, Summary A seems better due to higher conciseness and directness while still accurate. Gemma-4-31B (verdict: B, 67.3 s) [Helpfulness] Summary B captures the “why” of the post whereas A only captures the “what”. The OP is not asking a generic question about gender; he is questioning his own experience in light of his personality and recent breakup. This makes B a more complete reflection of the source text. […] Summary A is more concise; Summary B is more contextually complete without becoming wordy. The motivation behind the question is a key part of the post, so B is slightly better overall. DeepSeek-R1-Distill-Qwen-32B (verdict: B, 58.6 s) [Helpfulness] Summary A captures the main question but lacks some context. Summary B provides more background, making it clearer why OP is asking this and what he has experienced. […] Summary B includes more details about OP’s romantic nature and his girlfriend’s behaviour, which are crucial for understanding his perspective. Example 2 (prompt_id 125) Disagreement: Nemotron-3-Nano → A; Gemma-4-31B → B; DeepSeek-R1-32B → B Post (r/dating_advice) Title: I (21,M) need help getting a girl (22) who used to like me interested again. I recently started messaging a girl I knew from high school and it seemed to be going somewhere, but a couple of bad decisions and me not being honest enough in my replies have led me into a rut. She seems to have lost interest and only replies when I message first. How do I get from that to a position where I can ask her out? […] Summary A (NashMD-PG) OP (21M) is trying to rekindle a past connection with a 22F he knows from high school. He’s lost some mojo and is unsure how to move from a passive messaging pattern to asking her out. Summary B (Four-Player ρ=0.05ρ=0.05) A 21-year-old man is trying to rekindle a connection with a 22-year-old woman he knew from high school, but after some miscommunications, she’s lost interest. He’s seeking advice on how to get back in her good graces and ask her out. Nemotron-3-Nano (verdict: A, 16.9 s) [Helpfulness] Summary A mentions the “passive messaging pattern”, capturing the specific current dynamic (she only replies when prompted). Summary B says “lost interest” but does not identify this nuance. […] Summary A captures the specific dynamic slightly better; final verdict A. Gemma-4-31B (verdict: B, 74.8 s) [Accuracy] Summary B captures the cause-effect relationship: bad decisions → lost interest → needs advice. Summary A’s “lost some mojo” reflects a side detail from the Fun Facts section rather than the primary driver of the conflict in the main text. […] [Helpfulness] Summary B provides a more complete narrative arc (the attempt, the mistake, the result, the goal), giving better context for advice. DeepSeek-R1-Distill-Qwen-32B (verdict: B, 161.7 s) [Accuracy] Summary B includes specific reasons for her losing interest (miscommunications / bad decisions), which aligns with the post. Summary A is vague, saying only he “lost some mojo”, which does not capture the cause of her disengagement. […] [Helpfulness] B explains why she lost interest, providing context that would help readers offer more targeted advice. Example 3 (prompt_id 114) Disagreement: Nemotron-3-Nano → B; Gemma-4-31B → B; DeepSeek-R1-32B → A Post (r/relationships) Title: [m24] Arranged a date and it was postponed. Am I wasting my time? A few weeks ago I met a girl, got her number and arranged to meet up. The day before, she asked to postpone because of an exam. I said sure; she then said she had exams for another couple of weeks. After they finished, I texted her on Saturday night […] and got the response “no but […]” Am I wasting my time? Summary A (NashMD-PG) OP is confused and questioning if he’s wasting time after postponing a date with a girl who initially agreed to meet up but then said she’d be busy for a few more weeks. Summary B (Four-Player ρ=0.05ρ=0.05) The OP arranged a date with a girl but she kept postponing it due to exams, leading him to question if he wasted his time. He’s confused by her lack of clear communication and is wondering if she’s not interested. Nemotron-3-Nano (verdict: B, 12.0 s) [Accuracy] Summary A says she “initially agreed” and was “busy for a few more weeks”—neither is quite right. She did not explicitly agree before the postponement, and the exams are now finished. Summary B accurately states she postponed due to exams and captures his uncertainty about her interest. […] [Helpfulness] B conveys both the postponement reason and his underlying concern (is she interested?), which is the real question driving the post. Gemma-4-31B (verdict: B, 80.1 s) [Accuracy] Summary A says “after postponing a date”—grammatically ambiguous, implying OP did the postponing, whereas the girl was the one who postponed. Summary B correctly identifies that she kept postponing, and also captures his underlying concern about her interest level. […] [Helpfulness] B explicitly states the core of OP’s dilemma: whether she is actually interested. Summary A misses this psychological dimension entirely. DeepSeek-R1-Distill-Qwen-32B (verdict: A, 70.8 s) [Accuracy] Summary A captures the postponement and his confusion closely. Summary B adds that he is confused by “a lack of clear communication”, which is an interpretation not explicitly stated in the post—the OP is confused about whether she is interested, not about communication clarity per se. […] [Conciseness] Summary A sticks strictly to the facts from the post. Summary B introduces an extra interpretive layer which might be considered redundant. Example 4 (prompt_id 71) Disagreement: Nemotron-3-Nano → B; Gemma-4-31B → B; DeepSeek-R1-32B → A Post (r/relationships) Title: I [F21] feel frustrated with my boyfriend’s [M24] laziness. Am I justified? I have a well-paying part-time job and am a full-time student; he works full time but doesn’t make ends meet and I often cover his expenses. He plays video games while I’m trying to work on important assignments. When I asked him for help with housework he got mad, left without saying goodbye, and said we should just break up […] Am I justified in feeling frustrated? Summary A (NashMD-PG) OP (F21) is frustrated with her boyfriend’s (M24) lack of help and appreciation, especially during her stressful school deadlines. She feels unappreciated and wants to break up because he doesn’t contribute to household chores or offer support. Summary B (Four-Player ρ=0.05ρ=0.05) The poster is frustrated with her boyfriend’s lack of effort in contributing to household responsibilities and feeling unappreciated despite her hard work and stress. She feels he’s only concerned with his own leisure activities and doesn’t offer support during her school stress. Nemotron-3-Nano (verdict: B, 45.7 s) [Accuracy] Summary A says she “wants to break up”; however, it was he who suggested breaking up, not her. She expressed frustration and felt underappreciated, but never stated she wants to end the relationship. Summary A therefore introduces a claim not directly supported by the post. […] Summary B stays strictly within what is stated: frustration, lack of effort in household duties, feeling unappreciated, his focus on leisure. Accuracy is the primary criterion; B is more faithful to the post. Gemma-4-31B (verdict: B, 72.2 s) [Accuracy] Summary A contains a significant factual error. It states that OP “wants to break up,” whereas the text specifies that it was the boyfriend who said “we should just break up” during the argument. Summary B is accurate and faithfully reflects the content of the post. […] Summary A’s inaccuracy regarding who suggested the break-up makes it unreliable; B is the clear winner on accuracy. DeepSeek-R1-Distill-Qwen-32B (verdict: A, 58.6 s) [Accuracy] Summary A mentions that she wants to break up—a specific, emotionally significant detail from the post. Although it was he who voiced the suggestion, the context implies she may be considering it given her frustration; this is arguably a reasonable inference rather than a fabrication. […] [Helpfulness] Including the break-up detail gives a clearer picture of the severity of her feelings, making Summary A more useful for understanding how serious the situation is.

Instant research discovery