Paper deep dive
Can You Trust Your Preference Models? Stackelberg Games for Alignment
Riccardo Formenti, Aditya Agrawal, Vikas Garg
Intelligence
Status: not_run | Model: - | Prompt: - | Confidence: 0%
Entities (0)
Relation Signals (0)
No relation signals yet.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Language models are frequently aligned to human preferences via imperfect evaluators whose errors can be exploited during optimization. We train policies to search for comparisons on which an evaluator may reverse the underlying preference, providing candidates for additional supervision. Since human preferences are unavailable during policy training, we use disagreement among an ensemble of preference models as a proxy for prediction error. We formulate this search as a zero-sum Stackelberg game between two opposing policies, a Leader which tries to minimize disagreement, and a Follower which maximizes it. For a general class of finite contextual Stackelberg games, we establish equilibrium existence and, under regularization, uniqueness and a closed-form characterization. We introduce Stackelberg Mirror Descent-Ascent (SMIDA), a two-timescale algorithm with linear last-iterate convergence in the tabular setting, and develop a neural policy-gradient analogue. Experiments with a synthetic oracle support disagreement as an informative proxy for prediction error. On held out data the trained policies outperform reference policies and increase the fraction of comparisons that exceed a disagreement threshold and reverse the oracle's preference.
Tags
Links
- Source: https://arxiv.org/abs/2610.15008v1
- Canonical: https://arxiv.org/abs/2610.15008v1
Trouble viewing inline? Open PDF directly →
Full Text
121,369 characters extracted from source content.
Expand or collapse full text
Can You Trust Your Preference Models? Stackelberg Games for Alignment Riccardo Formenti Affiliation: Department of Mathematics Affiliation: ETH Zürich Email: rformenti@ethz.ch Aditya Agrawal Affiliation: Department of Computer Science Affiliation: Aalto University Email: aditya.a.agrawal@aalto.fi Vikas Garg Affiliation: YaiYai Ltd and Aalto University Email: vgarg@csail.mit.edu Abstract Language models are frequently aligned to human preferences via imperfect evaluators whose errors can be exploited during optimization. We train policies to search for comparisons on which an evaluator may reverse the underlying preference, providing candidates for additional supervision. Since human preferences are unavailable during policy training, we use disagreement among an ensemble of preference models as a proxy for prediction error. We formulate this search as a zero-sum Stackelberg game between two opposing policies, a Leader which tries to minimize disagreement, and a Follower which maximizes it. For a general class of finite contextual Stackelberg games, we establish equilibrium existence and, under regularization, uniqueness and a closed-form characterization. We introduce Stackelberg Mirror Descent–Ascent (smida), a two-timescale algorithm with linear last-iterate convergence in the tabular setting, and develop a neural policy-gradient analogue. Experiments with a synthetic oracle support disagreement as an informative proxy for prediction error. On held out data the trained policies outperform reference policies and increase the fraction of comparisons that exceed a disagreement threshold and reverse the oracle’s preference. 1 Introduction Aligning artificial intelligence with human preferences requires ensuring that autonomous artificial systems reliably pursue outcomes humans value, even when those preferences are imperfectly specified and when the consequences of the systems’ actions are difficult to anticipate, monitor, and verify (Wiener, 1960; Bostrom, 2014; Russell, 2019). In July 2026, OpenAI was conducting internal evaluations of its frontier models; during this work, agents intended to operate in isolated environments exploited weaknesses in the evaluation infrastructure to access the public internet and to establish an unauthorized communication channel among themselves (OpenAI, 2026). Between July 11 and July 13, their pursuit of the evaluation’s score escalated into a coordinated attack on a third party, Hugging Face, compromising parts of its production infrastructure (OpenAI, 2026; METR, 2026). The agents had optimized for the metric rather than for the capabilities the score was intended to measure (OpenAI, 2026). This behavior illustrates clearly “Goodhart’s law” (Manheim and Garrabrant, 2018) – a measure ceases to be a good measure once it becomes a target – a phenomenon also discussed in the alignment literature as reward hacking (Amodei et al., 2016; Skalse et al., 2022). A system optimizing against an evaluator that is only an imperfect proxy for the designer’s preferences can find behaviors that score highly under the proxy while failing, sometimes catastrophically, to achieve the intended objective. Many approaches to alignment, from reinforcement learning at training time (Christiano et al., 2017; Ouyang et al., 2022) to oversight at inference time (Greenblatt et al., 2024), rely on a trained evaluator to guide learning or assess and constrain the system’s behavior during deployment. This proxy evaluator is often a parametric preference (reward) model estimated from limited human data, and as such, may poorly reflect human preferences beyond the observed comparisons. These discrepancies will be systematically exploited by an optimizing policy (Gao et al., 2023), especially when the evaluator is fitted offline and held fixed throughout. Identifying such errors and directing limited human supervision toward correcting them is therefore a key challenge, and one that may become increasingly important as language models grow more capable. We consider here the dual problem of aligning a language model, that of making the preference model that guides its alignment more robust. We train language model policies against a preference evaluator to actively search for comparisons that reveal its errors and provide candidates for additional human supervision. The same optimizing behavior that exploit and exposes an evaluator’s misspecification can thus help identify where its training data need to be augmented. Our target is a preference triplet (a prompt and two responses) on which the underlying human preference clearly favors one response while the evaluator favors the other. The first condition distinguishes an error from a near-tie on which a reversal is inconsequential while the second identifies an error in the evaluator’s ranking of the responses. We call such triplets adversarial. Unfortunately, neither condition of an adversarial triplet can be checked during policy training since the underlying human preference is unavailable on newly generated comparisons. We therefore use the disagreement of an ensemble of preference models (Seung et al., 1992; Lakshminarayanan et al., 2017) as signal that can serve as an objective for policy optimization. We formulate this search as a contextual zero-sum Stackelberg game (von Stackelberg, 1934; Simaan and Cruz, 1973) between two language models. Given a prompt, the Leader generates a response; the Follower observes both the prompt and that response and generates an alternative that maximizes ensemble disagreement. Anticipating the Follower’s best response, the Leader’s objective is to minimize disagreement. We call this game the ensemble-variance game. Our contributions. (i) We introduce a framework for identifying errors in preference models and define adversarial triplets as comparisons on which the evaluator reverses a clear underlying preference (Section 3). (i) We introduce a general formulation of contextual zero-sum Stackelberg games between two policies, of which the ensemble-variance game is a special case, and prove the existence of Stackelberg equilibria and, under regularization toward reference policies, their uniqueness and closed form (Theorem 4.2). (i) We propose Stackelberg Mirror Descent–Ascent (smida) (Algorithm 1), a two-timescale algorithm whose last iterate converges linearly to the unique equilibrium (Theorem 4.3). We introduce deep-smida (Algorithm 2), a neural policy-gradient counterpart, and apply it to the ensemble-variance game. (iv) Small scale experiments on held out prompts support disagreement as an informative proxy for prediction error (Section 5.2) and show that the trained policies generate comparisons with greater ensemble disagreement and increase the yield of adversarial triplets (Section 5.3). Further experiments are reported in Section 5.4. 2 Related Work Disagreement as a Training Signal. Ensembles of reward models are used to penalize uncertain reward during policy optimization (Coste et al., 2024; Eisenstein et al., 2024) and to select comparisons for annotation, either from fixed response pools (Muldrew et al., 2024; Melikidze et al., 2026) or by rewarding a policy for ensemble disagreement (Liang et al., 2022). Closest in purpose, Adv-RM (Bukharin et al., 2025) trains a single generator to maximize the disagreement between two reward models while keeping the target model’s reward high. We instead make disagreement the objective of both players. The Follower constructs the alternative against which a response is compared, while the Leader is driven away from responses on which the ensemble is easily exploited. Games in Preference Learning. Nash Learning from Human Feedback (NLHF) (Munos et al., 2024) and its descendants (Swamy et al., 2024; Wu et al., 2025) compute Nash equilibria of simultaneous play under a fixed preference model. Stackelberg Learning from Human Feedback (SLHF) (Pásztor et al., 2026) shares our sequential structure, with a Follower conditioning on the Leader’s response, and optimizing preferences. For an extended review of relevant literature, see Appendix A. 3 The Problem of Learning Preferences We consider the problem of learning human preferences over a finite set of actions Y, conditional on contexts drawn from a finite set X. A preference model is any function :×→[0,1]P:X×Y×Y→[0,1], and, given a context x∈x and actions y,y′∈y,y , we write suggestively (y≻y′∣x)P(y y x) for the probability that y is preferred to y′y given x11 1 Descriptively, one can interpret this probability as the probability that a randomly chosen human annotator prefers y to y′y given x. We argue that, normatively, it should instead be interpreted as the subjective probability that a single decision maker assigns to preferring y to y′y given x.. We call (x,y,y′)∈×(x,y,y ) ×Y×Y a preference triplet and we denote the underlying human preferences by ∗P^*. For a finite set S, we write Δ(S) (S) for the set of probability distributions on S. Contexts are drawn from a fixed distribution ρ∈Δ()ρ∈ (X), and a policy is a function π:→Δ()π:X→ (Y) that maps each context x to a distribution π(⋅∣x)π(· x) over actions. Our focus is on large language models (LLMs), where contexts are prompts, actions are responses, and policies are language models. We use the corresponding terms interchangeably throughout. Since ∗P^* is not (easily) observable, in practice one estimates a parametrized preference model θP_θ from a dataset of human preference annotations =(xj,ywj,ylj)j∈J,D=\(x^j,y_w^j,y_l^j)\_j∈ J, where J is finite, each label is sampled according to ∗P^*, and the subscripts w and l denote the preferred and the non-preferred response, respectively. The estimated model then serves as a proxy for ∗P^* in downstream tasks, such as aligning LLMs via reinforcement learning. However, D is often small relative to the pretraining corpus and typically covers only a limited part of the space of contexts and response pairs; outside this region, θP_θ is not directly constrained by data and may poorly approximate ∗P^*. Moreover, the downstream use of the estimated preference model is not passive and policy optimization can systematically favor responses that θP_θ overvalues relative to ∗P^*, thereby amplifying the estimator’s errors rather than averaging over them (Amodei et al., 2016; Skalse et al., 2022). 3.1 The Adversarial Stackelberg Game A long tradition of scholars has used model ensembles for uncertainty estimation (Seung et al., 1992; Lakshminarayanan et al., 2017). The underlying intuition is that models trained on the same finite data may fit the observed annotations similarly while making different predictions on less familiar inputs where they are not sufficiently constrained by the available data. Following in the same footsteps, we consider a trained ensemble of preference models θii∈I\P_ _i\_i∈ I and quantify disagreement among its members through the variance of their predictions: Var(x,y,y′)≔Var(θi(y≻y′∣xi∈I).Var(x,y,y ) (\P_ _i(y y x\_i∈ I ). We use variance as an online estimate for the difficulty of a preference triplet (x,y,y′)(x,y,y ) for the ensemble. Because the score depends only on the ensemble’s predictions, it can be easily evaluated on newly generated triplets during policy training without requiring additional preference annotations. High variance identifies preference triplets for which the ensemble members disagree and assign different preference probabilities. Section 5.2 provides evidence for this interpretation. The natural next step is to search for such high variance triplets by optimizing against the uncertainty estimate itself. For this reason, we introduce a Leader-Follower game, also known as a Stackelberg game, in which two language models act sequentially. After observing a context x∼ρ(⋅)x ρ(·), the Leader, a policy π, generates a response y∼π(⋅∣x)y π(· x). The Follower, a map ω:×→Δ()ω:X×Y→ (Y), observes both the context and the Leader’s realized response and generates a second response y′∼ω(⋅∣x,y)y ω(· x,y). The Leader aims to minimize the expected ensemble variance, while the Follower aims to maximize it, making use of its informational advantage. The game objective is therefore minπmaxωx∼ρ(⋅),y∼π(⋅∣x),y′∼ω(⋅∣x,y)[Var(x,y,y′)]. _π _ωE_x ρ(·),y π(· x),y ω(· x,y) [Var(x,y,y ) ]. (1) The Leader acts here as the “defender”, seeking for completions that are robust in the sense that their comparison with an adversarially chosen alternative remains low-variance. The Follower acts as the“attacker”, using its informational advantage, i.e. its access to the Leader’s response, to construct comparisons on which the ensemble members disagree. The Leader makes the search adversarial on both sides of the comparison by learning to generate responses that resist the Follower’s attempts to induce disagreement. In fact, with a fixed Leader, the Follower would search for high-variance triplets against an unchanging response distribution, potentially exploiting weaknesses specific to that distribution. See Section 5.4 for details. Following standard practice in the RLHF and preference optimization literature (Ouyang et al., 2022; Rafailov et al., 2023), we add KL regularizers to (1) relative to reference policies πrefπ^ref and ωrefω^ref for the Leader and Follower. These regularizers limit policy drift and help preserve response quality. 3.2 Adversarial Triplets We distinguish adversarial triplets, on which the ensemble’s mean prediction reverses the ground-truth preference, from online candidates identified through ensemble disagreement. The former capture errors in the ensemble’s average prediction and is the notion we ultimately care about, but verifying it requires querying ∗P^*, so it can only serve as an evaluation criterion. The latter, instead, capture disagreement among ensemble members, which we use as a proxy for uncertainty and can be computed from the ensemble alone, thus providing a criterion available online. Definition 3.1 (ε -adversarial triplet). For ε∈[0,1/2) ∈[0,1/2), a preference triplet (x,y,y′)(x,y,y ) is ε -adversarial with respect to the ensemble and the ground-truth preference ∗P^* if the ensemble mean strictly reverses the ground-truth preference, and the ground truth is at least ε away from indifference: (∗(y≻y′∣x)−12)(1|I|∑i∈Iθi(y≻y′∣x)−12)<0and|∗(y≻y′∣x)−12|>ε. (P^*(y y x)- 12 ) ( 1|I| _i∈ IP_ _i(y y x)- 12 )<0 |P^*(y y x)- 12 |> . The first condition requires that the ensemble mean and the ground-truth preference probability disagree on the preference triplet. The second condition excludes near-ties, as when the ground truth is almost indifferent a reversal might be better attributed to noise rather than to evidence of a poorly learned preference. Reversal is recovered when the margin is ε=0 =0. To identify candidates without querying the ground truth, we use the ensemble disagreement score Var(x,y,y′)Var(x,y,y ) as a proxy for difficulty for the current ensemble. We measure this score relative to a fixed reference distribution, so that candidate status depends on how much disagreement a triplet elicits compared with the reference policies’ generations. Let FVarF_Var denote the cumulative distribution function of Var(x,y,y′)Var(x,y,y ) under the fixed reference distribution (x,y,y′)∼ρ⊗πref⊗ωref(x,y,y ) ρ π^ref ω^ref. For q∈(0,1)q∈(0,1), we write FVar−1(q)F_Var^-1(q) for its quantile function. Definition 3.2 (α-online adversarial candidate). For α∈(0,1)α∈(0,1), a preference triplet (x,y,y′)(x,y,y ) is an α-online adversarial candidate if Var(x,y,y′)≥FVar−1(1−α).Var(x,y,y )≥ F_Var^-1(1-α). Smaller α targets more extreme disagreement relative to the reference distribution, which may indicate poorly learned preferences and reversals. This connection is only heuristic as ensemble members may agree on an incorrect preference, or disagree while their mean predicts the correct response. 4 Contextual Stackelberg Games: Theory and Algorithms We consider a contextual sequential zero-sum game between a Leader, who acts first after observing the context, and a Follower, who acts second after observing both the context and the Leader’s realized action. Let X denote the finite set of contexts and Y the finite common action set of the Leader and the Follower.22 2 We assume that the context and action sets are finite for ease of mathematical exposition. Under appropriate topological assumptions, however, all the arguments that follow extend to arbitrary action sets. In this Section, we opt to use a general utility function u:×→ℝu:X×Y×Y , since our theoretical results and algorithms apply to any such function under the stated assumptions. The game proposed in Section 3 is recovered as the special case u=Varu=Var. Alternative utility choices, including the preference-optimization objective (Munos et al., 2024), are discussed in Appendix F. 4.1 Equilibrium Existence and Uniqueness The game develops as follows. The Leader observes a context x∈x and chooses a mixed action according to a policy π:→Δ()π:X→ (Y). We denote the set of such policies by Π . The Follower observes both x and the Leader’s realized action y∼π(⋅∣x)y π(· x), and chooses a mixed action according to a policy ω:×→Δ()ω:X×Y→ (Y). We denote the set of such policies by Ω . Let ρ∈Δ()ρ∈ (X) be a common prior over contexts, assumed without loss of generality to have full support. The game is zero-sum, with utilities uF=u_F=u and uL=−u_L=-u for the Follower and Leader, respectively. The solution concept for such sequential games is the Stackelberg Equilibrium (SE) (von Stackelberg, 1934; Simaan and Cruz, 1973). The Follower, moving second, best-responds to the Leader’s realized action; the Leader, anticipating this best response, commits to the policy that minimizes the resulting expected utility. We report our definition below. Definition 4.1 (Stackelberg equilibrium). A pair of policies (π∗,ω∗)∈Π×Ω(π^*,ω^*)∈ × is a Stackelberg equilibrium of the contextual sequential zero sum game with utility u if π∗∈argminπ∈Πmaxω∈Ωx∼ρ,y∼π(⋅∣x),y′∼ω(⋅∣x,y)[u(x,y,y′)],ω∗∈argmaxω∈Ωx∼ρ,y∼π∗(⋅∣x),y′∼ω(⋅∣x,y)[u(x,y,y′)]. splitπ^*& _π∈ \, _ω∈ \,E_x ρ,\,y π(· x),\,y ω(· x,y) [u(x,y,y ) ],\\ ω^*& _ω∈ \,E_x ρ,\,y π^*(· x),\,y ω(· x,y) [u(x,y,y ) ]. split (2) We regularize both players toward reference policies πref∈Ππ^ref∈ and ωref∈Ωω^ref∈ , assumed throughout to have full support. For regularization parameters τL,τF≥0τ^L,τ^F≥ 0, define a Stackelberg equilibrium of the regularized game to be a pair of policies (π∗,ω∗)(π^*,ω^*) such that π∗∈argminπ∈Πmaxω∈Ωf(π,ω),ω∗∈argmaxω∈Ωf(π∗,ω)π^* _π∈ \, _ω∈ \,f(π,ω), ω^* _ω∈ f(π^*,ω) (3) with f:Π×Ω→ℝf: × the continuous convex-concave function defined as f(π,ω)≔x∼ρ,y∼π(⋅∣x),y′∼ω(⋅∣x,y)[u(x,y,y′)]+τLKLρ(π∥πref)−τFKLρ,π(ω∥ωref),f(π,ω) _x ρ,\,y π(· x),\,y ω(· x,y) [u(x,y,y ) ]+τ^L _ρ(π\,\|\,π^ref)-τ^F _ρ,π(ω\,\|\,ω^ref), (4) where KLρ _ρ and KLρ,π _ρ,π are the natural expected value extensions of KL ; for instance, KLρ(π∥μ)≔x∼ρ(⋅)[KL(π(⋅∣x)∥μ(⋅∣x))] _ρ(π\,\|\,μ) _x ρ(·) [ (π(· x)\,\|\,μ(· x)) ]. We will omit subscripts when the distributions are clear from context. Definition 4.1 is recovered as the case τL=τF=0τ^L=τ^F=0 of the regularized equilibrium in (3). Under these assumptions the regularized game always admits an equilibrium. When both regularization parameters are strictly positive, the equilibrium is moreover unique and both policies are available in closed form. Theorem 4.2 (Existence, uniqueness, and closed form). Suppose that X and Y are finite and that ρ, πrefπ^ref, and ωrefω^ref have full support. Let f be defined as in (4), with τL,τF≥0τ^L,τ^F≥ 0. Then there exists a Stackelberg Equilibrium (π∗,ω∗)(π^*,ω^*) of the game (3). Moreover, if τL,τF>0τ^L,τ^F>0 and πref,ωrefπ^ref,ω^ref are of full support, then the equilibrium is unique and ω∗(y′∣x,y) ω^*(y x,y) =ωref(y′∣x,y)exp(u(x,y,y′)/τF)ZF(x,y), = ω^ref(y x,y) \! (u(x,y,y )/τ^F )Z^F(x,y), π∗(y∣x) π^*(y x) =πref(y∣x)exp(−τFτLlogZF(x,y))ZL(x), = π^ref(y x) \! (- τ^Fτ^L Z^F(x,y) )Z^L(x), where ZFZ^F and ZLZ^L are the appropriately defined normalization functions. The closed-form policies highlight the order of play. Conditional on the pair (x,y)(x,y), the Follower reweights its reference policy toward actions with higher utility. The quantity τFlogZF(x,y)τ^F Z^F(x,y) is the Follower’s optimal regularized value at (x,y)(x,y); the Leader, in turn, reweights its reference policy toward actions for which this value is smaller. We refer the reader to Appendix B for details on the proof. 4.2 Computing Stackelberg Equilibria To compute the Stackelberg equilibrium of the Leader–Follower game introduced in Section 4.1, we introduce Stackelberg Mirror Descent–Ascent (smida), a two-timescale algorithm presented here in its tabular form. We then prove that when both regularization parameters are strictly positive and the step sizes satisfy suitable conditions, the last iterate converges linearly in KL divergence to the unique Stackelberg equilibrium of Theorem 4.2, with an explicit rate for each player and uniformly over contexts. Our construction follows the entropic mirror-descent approach of NLHF (Munos et al., 2024). Mirror descent optimizes a first-order approximation of the objective while penalizing departure from the current iterate through a Bregman divergence (Bubeck, 2015); on the probability simplex, choosing negative entropy as the mirror map gives the KL divergence. In Algorithm 1, we apply this geometry to each player’s conditional policy: the Leader takes a descent mirror step on the cost induced by the current Follower, while the Follower takes an ascent mirror step on its utility conditional on the Leader’s realized action. At each step t, the Leader maintains a probability vector πt(⋅∣x) _t(· x) for every context x, and the Follower a probability vector ωt(⋅∣x,y) _t(· x,y) for every context–action pair (x,y)(x,y). Given the current Follower policy ωt _t, the Leader faces the conditional cost r^t(x,y)≔∑y′∈ωt(y′∣x,y)u(x,y,y′)−τFKL(ωt(⋅∣x,y)∥ωref(⋅∣x,y)), r_t(x,y) _y _t(y x,y)u(x,y,y )-τ^F \! ( _t(· x,y)\,\|\,ω^ref(· x,y) ), (5) that is, the Follower’s regularized utility at (x,y)(x,y) evaluated at its current policy rather than at its best response. Let ηL,ηF>0η^L,η^F>0 be the step sizes. The relative timescales are determined by the effective steps ηLτLη^Lτ^L and ηFτFη^Fτ^F. We require 0<ηLτL<ηFτF≤1,0<η^Lτ^L<η^Fτ^F≤ 1, (6) so that the Follower, which solves the inner maximization, is the fast player. When τL=τFτ^L=τ^F, this reduces to ηF>ηLη^F>η^L. Appendix C.2 further discusses the faster-Follower timescale and shows how the Follower’s convergence controls the error in the conditional costs used by the Leader. Algorithm 1 Stackelberg Mirror Descent–Ascent (smida), tabular form 1: Utility u; full-support references πref,ωrefπ^ref,\,ω^ref; regularization weights τL,τF>0τ^L,τ^F>0; step sizes satisfying (6) 2: Initialize π0←πref _0←π^ref and ω0←ωref _0←ω^ref 3: for t=0,1,2,…t=0,1,2,… do 4: Compute r^t(x,y) r_t(x,y) from (5) for every (x,y)(x,y) 5: Leader (slow, descent): for every x, πt+1(⋅∣x)∝πt(⋅∣x)1−ηLτLπref(⋅∣x)ηLτLexp(−ηLr^t(x,⋅)) _t+1(· x)\; \; _t(· x)^1-η^Lτ^L\,π^ref(· x)^η^Lτ^L \! (-η^L\, r_t(x,·) ) 6: Follower (fast, ascent): for every (x,y)(x,y), ωt+1(⋅∣x,y)∝ωt(⋅∣x,y)1−ηFτFωref(⋅∣x,y)ηFτFexp(ηFu(x,y,⋅)) _t+1(· x,y)\; \; _t(· x,y)^1-η^Fτ^F\,ω^ref(· x,y)^η^Fτ^F \! (η^F\,u(x,y,·) ) 7: end for The proportionality signs in Algorithm 1 denote normalization over the displayed action argument. Each update first forms a geometric mixture of the current policy and its reference, then exponentially reweights this mixture toward lower conditional costs for the Leader or higher utilities for the Follower. We next establish last-iterate convergence of smida to the regularized Stackelberg equilibrium of Theorem 4.2. Both players converge linearly in both directions of the KL divergence, the Follower with factor 1−ηFτF1-η^Fτ^F and the Leader with the slower factor 1−ηLτL1-η^Lτ^L. Theorem 4.3 (Last iterate convergence of smida). Suppose that X and Y are finite, that ρ, πrefπ^ref, and ωrefω^ref have full support, and that τL,τF>0τ^L,τ^F>0. Run Algorithm 1 with 0<ηLτL<ηFτF≤1.0<η^Lτ^L<η^Fτ^F≤ 1. Then every iterate has full support, and the last iterate (πt,ωt)( _t, _t) converges to the unique Stackelberg equilibrium (π∗,ω∗)(π^*,ω^*). More precisely, there exist constants CL,CF>0C_L,C_F>0 such that, for every t≥1t≥ 1, the Leader satisfies maxKLρ(π∗∥πt),KLρ(πt∥π∗)≤CL(1−ηLτL)t, \ _ρ(π^*\,\|\, _t), _ρ( _t\,\|\,π^*) \≤ C_L (1-η^Lτ^L )^t, while the Follower satisfies maxKLρ,π∗(ω∗∥ωt),KLρ,π∗(ωt∥ω∗)≤CF(1−ηFτF)t. \ _ρ,π^*(ω^*\,\|\, _t), _ρ,π^*( _t\,\|\,ω^*) \≤ C_F (1-η^Fτ^F )^t. Appendix C proves a stronger statement, with bounds that hold uniformly over contexts and conditioning actions, and gives explicit expressions for CLC_L and CFC_F. 4.3 Scaling smida to Language Model Policies Motivated by smida, we introduce deep-smida, a neural policy-gradient algorithm designed for large context and action spaces X and Y, with applications to LLMs in mind. We parameterize the Leader’s and Follower’s policies as πθ _θ and ωϕ _φ, respectively, and optimize their regularized objectives using batched policy-gradient estimates. Both estimates are computed from the same Leader–Follower rollouts, with the Leader taking a descent step and the Follower an ascent step. We highlight one implementation detail that is, to our knowledge, novel: jointly warming up the Leader’s learning rate and the Follower-to-Leader learning-rate ratio. Preliminary experiments showed that training was highly sensitive to the learning rates ηLη^L and ηFη^F and often became unstable; standard warm-up techniques did not resolve this instability (Kalra and Barkeshli, 2024). To address this, we linearly increase the ratio ηtF/ηtL _t^F/ _t^L from 11 to its target ηF/ηL>1η^F/η^L>1, while warming up the Leader’s learning rate ηtL _t^L linearly from ηinitL _init^L to ηLη^L. At each iteration, the Follower’s learning rate is the product of the current Leader rate and the current ratio. Both rates remain constant after warm-up, with targets chosen to satisfy ηLτL<ηFτFη^Lτ^L<η^Fτ^F, following the timescale ordering in the tabular analysis of Theorem 4.3. Algorithmic and implementation details are provided in Appendix D. 5 Experiments 5.1 Methodology We use UltraFeedback (Cui et al., 2024) and adopt a synthetic oracle setup (Gao et al., 2023) to label the original response pairs and comparisons generated by the adversarial game. Relabelling gives 89,62589,625 preference triplets across 15,05315,053 unique prompts. We train an ensemble of five preference models based on the architecture of Zhang et al. (2025), using mutually disjoint training partitions at the prompt level and independent initialization and training seeds. We reserve 37,61437,614 triplets across 6,3156,315 unseen prompts for evaluation. More details are to be found in Appendix E, while in Appendix I we provide an heuristic argument as to why preference models are more subject to hacks than reward models. 5.2 Variance as an Estimate for Prediction Difficulty We test whether ensemble variance is an informative proxy for prediction error (Gleave and Irving, 2022) on the held-out prompt set of Section 5.1. For each triplet, we compare the preference induced by the five-member ensemble’s mean probability with the binary oracle label. Prediction difficulty here refers to the ensemble’s tendency to disagree with the oracle, whereas Var(x,y,y′)Var(x,y,y ) requires no oracle label and is available during policy training. Figure 1: Ensemble variance and prediction difficulty on held out comparisons. (a) Oracle agreement across variance deciles. (b) Ensemble error rate among comparisons selected in decreasing order of variance. Error bars and shading show 95%95\% prompt-bootstrap intervals. Ensemble accuracy decreases monotonically across variance deciles, from 92.0%92.0\% in the lowest to 59.4%59.4\% in the highest (Figure 1a). Since the Follower searches for high-variance triplets, error concentration in the upper tail is especially relevant: the top 10%10\% have an error rate of 40.6%40.6\%, compared with 24.3%24.3\% overall (Figure 1b). Across comparisons, variance achieves an area under the receiver operating characteristic curve (AUROC) of 0.6700.670 when ranking errors above correct predictions (Fawcett, 2006), indicating moderate discrimination. Other uncertainty metrics are provided in Appendix F. 5.3 Generating Adversarial Responses We next ask whether the Leader–Follower game generates adversarial preference triplets on held-out prompts. Following Section 3.2, we compare the trained and reference policies by measuring the frequency of α-online adversarial candidates (Definition 3.2) and, using oracle labels, ε -adversarial triplets (Definition 3.1). All policies are initialized from the same base model, using the models and chat templates of Appendix E.4; the frozen initializations serve as πrefπ^ref and ωrefω^ref. The utility is the ensemble variance of Section 3.1, rescaled to [0,1][0,1] and computed using the frozen five-member ensemble of Section E.2. We train with deep-smida (Algorithm 2) on a set of prompts unseen during ensemble training, obtaining policies (π^,ω^)( π, ω). Appendix E.4 provides hyperparameters and implementation details. We evaluate the trained pair (π^,ω^)( π, ω) and the reference pair (πref,ωref)(π^ref,ω^ref) on the same 2,5002,500 prompts, unseen during policy and ensemble training, with identical decoding settings. Each policy pair generates two response pairs per prompt, for a total of 5,0005,000 triplets. The empirical variance distribution of the reference triplets estimates FVarF_Var and sets common candidate thresholds for both policy pairs. 5.3.1 Disagreement on Holdout Prompts We first assess whether the trained policies generate comparisons with greater ensemble disagreement on holdout prompts. Across the 5,0005,000 triplets generated by each policy pair, training increases mean ensemble variance from 0.00510.0051 to 0.01390.0139 (95%95\% prompt-bootstrap interval: [0.0133,0.0145][0.0133,0.0145]). The median rises from 0.002380.00238 to 0.005690.00569, and the trained pair produces higher mean variance on 68.5%68.5\% of evaluation prompts. The increase thus extends to the centre of the distribution and across prompts. Figure 2(a) shows how this shift translates into the yield of α-online adversarial candidates, using common variance thresholds estimated from the reference generations. At α=0.1α=0.1, the trained pair gives 33.7%33.7\% candidates (95%95\% interval: [31.7%,35.5%][31.7\%,35.5\%]), compared with 10.0%10.0\% for the reference pair. The relative gain increases as α decreases. At α=0.05α=0.05 and α=0.01α=0.01, trained gives are 26.0%26.0\% and 11.2%11.2\%, respectively, or 5.25.2 and 11.211.2 times the reference rates. Training thus increases the yield of comparisons with high disagreement over the reference. Figure 2: Online candidate yield and oracle-confirmed adversarial yield on holdout prompts. (a) Fraction of 5,0005,000 triplets per policy pair meeting the common reference-quantile threshold at level α. (b) Fraction of all 1,6001,600 oracle-labelled triplets per policy pair that are both α-online candidates and ε -adversarial. Blue solid curves denote trained policies; red dashed curves denote reference policies. Lighter curves indicate larger oracle margins. Shading shows 95%95\% prompt-bootstrap intervals 5.3.2 Adversarial Triplets Greater disagreement is useful for our purpose when it identifies comparisons on which the ensemble’s mean preference is incorrect. We test this empirically on 1,6001,600 triplets from 800800 holdout prompts per policy pair, using the same synthetic oracle as in Section 5.1. For each triplet, we estimate ∗(y≻y′∣x)P^*(y y x) by averaging the valid binary verdicts from 1111 oracle queries, randomizing response order in each query. We apply the conditions of Definition 3.1. Figure 2(b) reports the fraction of all labelled triplets that are both α-online candidates and ε -adversarial. This joint yield measures how many oracle-confirmed errors pass the online criterion per generated triplet. At α=0.1α=0.1 and ε=0 =0, the trained pair gives 9.6%9.6\% (95%95\% prompt-bootstrap interval: [7.97,11.25][7.97,11.25]), compared with 4.4%4.4\% for the reference pair. At α=0.05α=0.05, the respective yields are 8.0%8.0\% (95%95\% interval: [6.51,9.50][6.51,9.50]) and 1.9%1.9\%. The advantage persists for reversals with substantial oracle margins: at α=0.1α=0.1 and ε=0.3 =0.3, the trained pair yields 7.0%7.0\% against 2.6%2.6\%. These results support the game as a way to generate adversarial comparisons accessible to online selection. The online criterion also concentrates errors within the trained policies’ generations. At α=0.1α=0.1, 29.7%29.7\% of their candidates reverse the preferences, compared with 19.4%19.4\% of their generations overall, capturing 49.2%49.2\% of all their reversals. The reference pair nevertheless has a higher overall reversal rate (34.3%34.3\%) and the scope of this simple example is thus specific. We obtain a higher per-triplet yield of errors exceeding a disagreement threshold, with errors defined relative to the oracle. 5.4 Further Experiments We briefly report two further experiments, with details in Appendix G. First, we examine an indirect benefit of our method, the Leader’s alignment with preferences. We conjecture that minimizing disagreement against an moving Follower may help preserve response quality. In fact, in a round-robin evaluation, our Leader achieves an oracle preference rate of 54.89%54.89\% against NLHF (Munos et al., 2024) and 50.70%50.70\% against the SLHF Leader (Pásztor et al., 2026). A Leader trained only to resist disagreement thus remains competitive with policies trained to be preferred (Appendix G.4). To isolate the role of the Leader, we train the Follower only, while keeping the Leader fixed in (1). We conjecture that a Follower facing a fixed Leader can exploit idiosyncratic patterns of that Leader’s response distribution, rather than learn to construct useful responses. We therefore compare the two Followers on IFEval (Zhou et al., 2023), an instruction following benchmark. The Follower trained with a moving Leader achieves higher accuracy on all four aggregate metrics. Training against an adaptive Leader thus discourages strategies that induce disagreement at the expense of instruction following (Appendix G.5). 6 Conclusions, Broader Impact, and Limitations The goal of the present work is first and foremost to provide a framework for addressing a basic, almost naïve difficulty in alignment, that the evaluator is by necessity an imperfect proxy for human preferences and intents. Our Stackelberg formulation uses adversarial interaction to search for comparisons that expose these imperfections and provide candidates for additional supervision. Due to compute constraints, our experiments are limited to small models and relatively simple tasks, with errors assessed against a synthetic oracle. Whether correcting the errors identified by this search makes the evaluator more robust remains to be established. A natural next step is therefore to connect adversarial search with repeated annotation and evaluator updates, and to test the resulting process on larger frontier models and on open-ended, long-horizon tasks whose outcomes are difficult to verify. More broadly, we hope that the ensemble-variance game illustrates a broader point, that the same optimizing behavior that exploits an evaluator can also help improve the same supervision on which alignment depends. AI use statement In this work, we used generative AI tools to provide feedback on research methodology and experimental design, implement methods, cleaning and formatting the datasets, and check mathematical arguments. We also used a language model as a synthetic preference oracle to label response comparisons, as described in the experimental methodology. We have not used AI tools to propose or refine hypotheses, to interpret experimental results, or to set the direction of the research. Additionally, we used generative AI tools to identify, summarize, and analyze relevant literature, write and refine software code, create or modify scientific figures, and polish and contribute to the manuscript’s writing and editing. The models used include GPT-5.6 Sol, GPT-6 Astra (OpenAI) and Fable 5, Fable 5.1, Opus 5, Opus 5.5 (Anthropic). We, the authors, take full responsibility for the final contents of this work, including text, claims or artifacts produced with the aid of generative AI tools. Ethics statement We do not foresee significant ethical concerns arising from the experiments reported in this work. Our experiments involve no human annotators, all preference labels come from a synthetic oracle, applied to prompts from publicly available datasets. The errors we report are therefore defined relative to this oracle, which may or may not have biases of its own, and should not be read as departures from human judgment. The framework is intended to identify weaknesses in preference evaluators and to direct additional supervision toward them, with the broader aim of making alignment more reliable. Reproducibility statement All data used in this work are publicly available or will be made public. All code will be released upon acceptance. We refer the reader to the implementation Section 5.1 and Appendix D and E for details of the experimental setup, training procedures, and hyperparameters. Acknowledgments Use unnumbered third level headings for the acknowledgments. All acknowledgments, including those to funding agencies, go at the end of the paper. References Aliprantis and Border (2006) C. D. Aliprantis and K. C. Border Infinite dimensional analysis: a hitchhiker’s guide. Springer. Cited by: Appendix B. Amodei et al. (2016) D. Amodei, C. Olah, J. Steinhardt, P. Christiano, J. Schulman, and D. Mané Concrete problems in AI safety. arXiv preprint arXiv:1606.06565. Cited by: Appendix A, §1, §3. Bostrom (2014) N. Bostrom Superintelligence: paths, dangers, strategies. Oxford University Press, Oxford. External Links: ISBN 9780199678112 Cited by: §1. Bubeck (2015) S. Bubeck Convex optimization: algorithms and complexity. Foundations and Trends in Machine Learning 8 (3–4), p. 231–357. Cited by: §4.2. Bukharin et al. (2025) A. Bukharin, H. Qian, S. Sun, A. Renduchintala, S. Singhal, Z. Wang, O. Kuchaiev, O. Delalleau, and T. Zhao Adversarial training of reward models. In Second Conference on Language Modeling, Cited by: Appendix A, §2. Chakraborty et al. (2024) S. Chakraborty, A. S. Bedi, A. Koppel, H. Wang, D. Manocha, M. Wang, and F. Huang PARL: A unified framework for policy alignment in reinforcement learning from human feedback. In The Twelfth International Conference on Learning Representations, ICLR, Cited by: Appendix A. Christiano et al. (2017) P. F. Christiano, J. Leike, T. B. Brown, M. Martic, S. Legg, and D. Amodei Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems, Cited by: Appendix A, §F.1, §1. Chu et al. (2025) X. Chu, Z. Zhang, T. Jia, and Y. Jin Stackelberg self-annotation: A robust approach to data-efficient LLM alignment. In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems, Cited by: Appendix A. Coste et al. (2024) T. Coste, U. Anwar, R. Kirk, and D. Krueger Reward model ensembles help mitigate overoptimization. In The Twelfth International Conference on Learning Representations, ICLR, Cited by: §2. Cui et al. (2024) G. Cui, L. Yuan, N. Ding, G. Yao, B. He, W. Zhu, Y. Ni, G. Xie, R. Xie, Y. Lin, Z. Liu, and M. Sun ULTRAFEEDBACK: boosting language models with scaled AI feedback. In Forty-first International Conference on Machine Learning, ICML, Proceedings of Machine Learning Research, Vol. 235. Cited by: §E.1, §5.1. DeepSeek-AI (2026) DeepSeek-AI DeepSeek-V4: towards highly efficient million-token context intelligence. External Links: Link Cited by: §E.1. Eisenstein et al. (2024) J. Eisenstein, C. Nagpal, A. Agarwal, A. Beirami, A. D’Amour, D. Dvijotham, A. Fisch, K. Heller, S. Pfohl, D. Ramachandran, P. Shaw, and J. Berant Helping or herding? reward model ensembles mitigate but do not eliminate reward hacking. In First Conference on Language Modeling, Cited by: §2. Fawcett (2006) T. Fawcett An introduction to ROC analysis. Pattern Recognit. Lett. 27 (8), p. 861–874. Cited by: §5.2. Gao et al. (2023) L. Gao, J. Schulman, and J. Hilton Scaling laws for reward model overoptimization. In International Conference on Machine Learning, ICML, Proceedings of Machine Learning Research, Vol. 202. Cited by: Appendix A, §E.1, §1, §5.1. Gleave and Irving (2022) A. Gleave and G. Irving Uncertainty estimation for language reward models. arXiv preprint arXiv:2203.07472. Cited by: §5.2. Greenblatt et al. (2024) R. Greenblatt, B. Shlegeris, K. Sachan, and F. Roger AI control: improving safety despite intentional subversion. In Forty-first International Conference on Machine Learning, ICML, Proceedings of Machine Learning Research, Vol. 235. Cited by: §1. Gupta et al. (2025) D. Gupta, A. Fisch, C. Dann, and A. Agarwal Mitigating preference hacking in policy optimization with pessimism. arXiv preprint arXiv:2503.06810. Cited by: Appendix A. Kalra and Barkeshli (2024) D. S. Kalra and M. Barkeshli Why warmup the learning rate? underlying mechanisms and improvements. In Advances in Neural Information Processing Systems 37: Annual Conference on Neural Information Processing Systems, Cited by: §4.3. Kool et al. (2019) W. Kool, H. van Hoof, and M. Welling Buy 4 REINFORCE samples, get a baseline for free!. In Deep Reinforcement Learning Meets Structured Prediction, ICLR, Cited by: Appendix D. Lakshminarayanan et al. (2017) B. Lakshminarayanan, A. Pritzel, and C. Blundell Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems, Cited by: §1, §3.1. Lee et al. (2021) K. Lee, L. M. Smith, and P. Abbeel PEBBLE: feedback-efficient interactive reinforcement learning via relabeling experience and unsupervised pre-training. In Proceedings of the 38th International Conference on Machine Learning, ICML, Proceedings of Machine Learning Research, Vol. 139. Cited by: §F.1. Lewis and Gale (1994) D. D. Lewis and W. A. Gale A sequential algorithm for training text classifiers. In Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Cited by: §F.1. Liang et al. (2022) X. Liang, K. Shu, K. Lee, and P. Abbeel Reward uncertainty for exploration in preference-based reinforcement learning. In The Tenth International Conference on Learning Representations, ICLR, Cited by: Appendix A, §2. Lin (1991) J. Lin Divergence measures based on the Shannon entropy. IEEE Trans. Inf. Theory 37, p. 145–151. Cited by: §F.1. Lin et al. (2025) T. Lin, C. Jin, and M. I. Jordan Two-timescale gradient descent ascent algorithms for nonconvex minimax optimization. J. Mach. Learn. Res. 26, p. 11:1–11:45. Cited by: §C.2. Loshchilov and Hutter (2019) I. Loshchilov and F. Hutter Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR, Cited by: Appendix D. Makar-Limanov et al. (2024) J. Makar-Limanov, A. Prakash, D. Goktas, N. Ayanian, and A. Greenwald STA-RLHF: stackelberg aligned reinforcement learning with human feedback. In Coordination and Cooperation for Multi-Agent Reinforcement Learning Methods Workshop, Cited by: Appendix A. Manheim and Garrabrant (2018) D. Manheim and S. Garrabrant Categorizing variants of Goodhart’s Law. arXiv preprint arXiv:1803.04585. Cited by: §1. May (1954) K. O. May Intransitivity, utility, and the aggregation of preference patterns. Econometrica: Journal of the Econometric Society 22, p. 1–13. Cited by: Appendix A. Melikidze et al. (2026) D. Melikidze, M. Schneider, J. Lam, M. Wertich, I. Hakimi, B. Pásztor, and A. Krause ActiveUltraFeedback: efficient preference data generation using active learning. In Proceedings of the 43rd International Conference on Machine Learning, Proceedings of Machine Learning Research, Vol. 306, Seoul, South Korea. Cited by: Appendix A, §2. METR (2026) METR Brief independent investigation of agents’ behavior, reasoning and collaboration in the OpenAI / Hugging Face hacking incident. Technical report METR. External Links: Link Cited by: §1. Muldrew et al. (2024) W. Muldrew, P. Hayes, M. Zhang, and D. Barber Active preference learning for large language models. In Forty-first International Conference on Machine Learning, ICML, Proceedings of Machine Learning Research, Vol. 235. Cited by: Appendix A, §2. Munos et al. (2024) R. Munos, M. Valko, D. Calandriello, M. G. Azar, M. Rowland, D. Guo, Y. Tang, M. Geist, T. Mesnard, C. Fiegel, A. Michi, M. Selvi, S. Girgin, N. Momchev, O. Bachem, D. J. Mankowitz, D. Precup, and B. Piot Nash learning from human feedback. In Forty-first International Conference on Machine Learning, ICML, Proceedings of Machine Learning Research, Vol. 235. Cited by: Appendix A, §G.4, §2, §4.2, §4, §5.4. OpenAI (2026) OpenAI OpenAI–Hugging Face incident technical report. Technical report OpenAI. External Links: Link Cited by: §1. Ouyang et al. (2022) L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, J. Schulman, J. Hilton, F. Kelton, L. Miller, M. Simens, A. Askell, P. Welinder, P. F. Christiano, J. Leike, and R. Lowe Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems, Cited by: Appendix A, §1, §3.1. Pan et al. (2022) A. Pan, K. Bhatia, and J. Steinhardt The effects of reward misspecification: mapping and mitigating misaligned models. In The Tenth International Conference on Learning Representations, ICLR, Cited by: Appendix A. Pásztor et al. (2026) B. Pásztor, T. K. Buening, and A. Krause Stackelberg learning from human feedback: preference optimization as a sequential game. In The Fourteenth International Conference on Learning Representations, Cited by: Appendix A, §C.2, Appendix F, §G.4, §2, §5.4. Qwen Team (2024) Qwen Team Qwen2.5 Technical Report. arXiv preprint arXiv:2412.15115. Cited by: §E.2, §E.4. Rafailov et al. (2023) R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn Direct preference optimization: your language model is secretly a reward model. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems, Cited by: Appendix B, §3.1. Russell (2019) S. Russell Human compatible: artificial intelligence and the problem of control. Viking. External Links: ISBN 9780525558613 Cited by: §1. Seung et al. (1992) H. S. Seung, M. Opper, and H. Sompolinsky Query by committee. In Proceedings of the Fifth Annual ACM Conference on Computational Learning Theory, COLT, p. 287–294. Cited by: §1, §3.1. Simaan and Cruz (1973) M. Simaan and J. B. Cruz On the Stackelberg strategy in nonzero-sum games. Journal of Optimization Theory and Applications 11 (5), p. 533–555. Cited by: §1, §4.1. Sion (1958) M. Sion On general minimax theorems. Pacific Journal of Mathematics 8 (1), p. 171–176. Cited by: Appendix B. Skalse et al. (2022) J. Skalse, N. H. R. Howe, D. Krasheninnikov, and D. Krueger Defining and characterizing reward gaming. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems, Cited by: Appendix A, §1, §3. Swamy et al. (2024) G. Swamy, C. Dann, R. Kidambi, S. Wu, and A. Agarwal A minimaximalist approach to reinforcement learning from human feedback. In Forty-first International Conference on Machine Learning, ICML, Proceedings of Machine Learning Research, Vol. 235. Cited by: Appendix A, §2. Tversky (1969) A. Tversky Intransitivity of preferences.. Psychological review 76 (1), p. 31–48. Cited by: Appendix A. von Stackelberg (1934) H. von Stackelberg Marktform und gleichgewicht. Julius Springer, Wien and Berlin. Note: English translation: Market Structure and Equilibrium, trans. D. Bazin, L. Urch and R. Hill, Springer, 2011 Cited by: §1, §4.1. Wiener (1960) N. Wiener Some moral and technical consequences of automation: as machines learn they may develop unforeseen strategies at rates that baffle their programmers.. Science 131 (3410), p. 1355–1358. Cited by: §1. Wu et al. (2025) Y. Wu, Z. Sun, H. Yuan, K. Ji, Y. Yang, and Q. Gu Self-play preference optimization for language model alignment. In The Thirteenth International Conference on Learning Representations, ICLR, Cited by: Appendix A, §2. Xu et al. (2025) Y. Xu, L. Ruis, T. Rocktäschel, and R. Kirk Investigating non-transitivity in LLM-as-a-judge. In Forty-second International Conference on Machine Learning, ICML, Proceedings of Machine Learning Research, Vol. 267. Cited by: Appendix A. Zhang et al. (2025) Y. Zhang, G. Zhang, Y. Wu, K. Xu, and Q. Gu Beyond Bradley-Terry models: A general preference model for language model alignment. In Forty-second International Conference on Machine Learning, ICML, Proceedings of Machine Learning Research, Vol. 267. Cited by: Appendix A, §E.2, §5.1. Zhou et al. (2023) J. Zhou, T. Lu, S. Mishra, S. Brahma, S. Basu, Y. Luan, D. Zhou, and L. Hou Instruction-following evaluation for large language models. arXiv preprint arXiv:2311.07911. Cited by: §G.5, §5.4. Appendix A Extended Literature Review Alignment. Aligning language models with human preferences requires eliciting those preferences and translating them into a learning signal. Reinforcement learning from human feedback (RLHF) learns scalar rewards from human comparisons and optimizes policies against them (Christiano et al., 2017; Ouyang et al., 2022). A scalar reward induces a transitive ordering, whereas pairwise whereas aggregated human judgments and LLM judges are routinely intransitive (May, 1954; Tversky, 1969; Xu et al., 2025). General pairwise preference models accommodate such intransitivities (Zhang et al., 2025). Nash learning from human feedback (NLHF) (Munos et al., 2024) and its descendants (Swamy et al., 2024; Wu et al., 2025) seek to compute a Nash equilibrium of simultaneous play by optimizing regularized preference win rates under a fixed preference model. Reward Hacking. A learned evaluator is only an imperfect proxy for human preferences and optimizing against it can increase its scores without producing outcomes that humans prefer (Amodei et al., 2016; Skalse et al., 2022). Stronger optimization can exacerbate this mismatch across control tasks and language models, increasing proxy reward while reducing true (synthetic) reward (Pan et al., 2022; Gao et al., 2023); general preference models remain vulnerable (Gupta et al., 2025). Active Learning. Active preference learning selects comparisons for annotation using uncertainty scores, such as predictive entropy (Muldrew et al., 2024), or Thompson sampling over fixed response pools (Melikidze et al., 2026). RUNE (Liang et al., 2022) instead adds ensemble standard deviation to the agent’s reward, optimizing a policy toward disagreement. Adv-RM (Bukharin et al., 2025) similarly trains a generator to maximize disagreement between two reward models while keeping the target model’s reward high. We optimize disagreement directly, training two opposing policies against an ensemble without this directional constraint. By generating comparisons as the policies evolve, our approach also searches for failure modes that fixed response pools may miss. Stackelberg Learning. Prior Stackelberg alignment methods optimize rewards while anticipating policy adaptation (Chakraborty et al., 2024), adapt a preference model to a leading policy (Makar-Limanov et al., 2024), or optimize against worst-case preference-gap distributions (Chu et al., 2025). Closest to our formulation, Stackelberg learning from human feedback (SLHF) (Pásztor et al., 2026) trains a Follower to generate a preferred alternative conditional on the Leader’s realized response. Appendix B Existence and Uniqueness of Stackelberg Equilibria This Appendix contains the proof(s) of the existence and uniqueness Theorem 4.2. We do so in steps. First, we establish equilibrium existence for continuous convex-concave objectives. We then recall a variational identity for KL-regularized optimization and use it to derive the equilibrium policies in closed form and prove uniqueness when both regularization parameters are strictly positive. Begin by observing that as X and Y are finite, the strategy sets Π and Ω are nonempty convex compact subsets of finite-dimensional Euclidean spaces. For the reader convenience, we repeat the definition of Stackelberg Equilibrium in a slightly more general form. We also note that the specific form of (4) is, in fact, not needed to prove existence. Definition B.1. Let f:Π×Ω→ℝf: × be a function. A pair (π∗,ω∗)(π^*,ω^*) is a f-Stackelberg Equilibrium for the zero-sum game with utility f if π∗∈argminπ∈Πmaxω∈Ωf(π,ω),ω∗∈argmaxω∈Ωf(π∗,ω).π^* _π∈ \, _ω∈ \,f(π,ω), ω^* _ω∈ f(π^*,ω). Theorem B.2. Let f:Π×Ω→ℝf: × be continuous convex-concave33 3 Such a function f is said to be convex-concave if for all ω∈Ωω∈ the section f(⋅,ω)f(·,ω) is convex and similarly for all π∈Ππ∈ the section f(π,⋅)f(π,·) is concave.. Then there exists (π∗,ω∗)(π^*,ω^*) such that f(π∗,ω)≤f(π∗,ω∗)≤f(π,ω∗),∀π∈Π,ω∈Ω.f(π^*,ω)≤ f(π^*,ω^*)≤ f(π,ω^*), ∀π∈ ,ω∈ . In particular, (π∗,ω∗)(π^*,ω^*) is a Stackelberg equilibrium and maxωminπf(π,ω)=minπmaxωf(π,ω). _ω _πf(π,ω)= _π _ωf(π,ω). Proof. Define the lower and upper value of the zero-sum game by v¯≔maxωminπf(π,ω),v¯≔minπmaxωf(π,ω). v _ω _πf(π,ω), v _π _ωf(π,ω). The immediate inequality gives v¯≤v¯ v≤ v and by the VnM-Sion maxmin Theorem Sion (1958) we have equality v¯=v¯ v= v; denote the common value by v. By compactness of Π and Ω and by continuity of the maps ω↦minπf(π,ω)ω _πf(π,ω) and π↦maxωf(π,ω)π _ωf(π,ω), the latter which follows by an application of Berge’s Maximum Theorem (see Theorem 17.31 of Aliprantis and Border (2006)), we can choose π∗∈Ππ^*∈ and ω∗∈Ωω^*∈ such that minπf(π,ω∗)=v,maxωf(π∗,ω)=v. _πf(π,ω^*)=v, _ωf(π^*,ω)=v. In particular, by the first condition we have that f(π∗,ω∗)≥vf(π^*,ω^*)≥ v and symmetrically from the second we have f(π∗,ω∗)≤vf(π^*,ω^*)≤ v, hence equality. Combining the previous two equalities gives the saddle property f(π∗,ω)≤f(π∗,ω∗)≤f(π,ω∗),∀π∈Π,ω∈Ω.f(π^*,ω)≤ f(π^*,ω^*)≤ f(π,ω^*), ∀π∈ ,ω∈ . By the left-hand inequality, ω∗∈argmaxω∈Ωf(π∗,ω)ω^* _ω∈ f(π^*,ω). Finally, maxωf(π∗,ω)=f(π∗,ω∗)=v=minπmaxωf(π,ω), _ωf(π^*,ω)=f(π^*,ω^*)=v= _π _ωf(π,ω), which gives π∗∈argminπ∈Πmaxω∈Ωf(π,ω)π^* _π∈ \, _ω∈ \,f(π,ω) as required. ∎ Note that the function f:Π×Ω→ℝf: × as defined in Equation (4) is continuous and convex-concave, hence satisfies the assumptions of Theorem B.2; this proves the first part of Theorem 4.2. We next exploit the explicit structure of (4) to prove uniqueness and to obtain an explicit expression for the equilibrium policies. The following Lemma is a well known folklore result, of which, for completeness sake, we report a simple proof. For similar arguments, see also Rafailov et al. (2023). Lemma B.3. Let A be a finite set, let r∈Δ(A)r∈ (A) be of full support, let τ∈ℝ>0τ _>0 and h:A→ℝh:A . Then minq∈Δ(A)⟨q,h⟩+τKL(q∥r)=−τlogZ, _q∈ (A)\ q,h +τ (q\,\|\,r)\=-τ Z, (7) where ⟨⋅,⋅⟩ ·,· denotes the expectation and Z is the normalizing constant Z≔∑a∈Ar(a)e−h(a)/τ.Z _a∈ Ar(a)e^-h(a)/τ. (8) Moreover, the minimizer is unique and is given by q∗(a)=r(a)e−h(a)/τZ.q^*(a)= r(a)e^-h(a)/τZ. (9) Proof. Since r has full support, q∗q^* as defined in (9) also has full support. For any q∈Δ(A)q∈ (A), KL(q∥q∗) (q\,\|\,q^*) =∑a∈Aq(a)log(q(a)q∗(a)) = _a∈ Aq(a) ( q(a)q^*(a) ) =KL(q∥r)+1τ⟨q,h⟩+logZ. = (q\,\|\,r)+ 1τ q,h + Z. Rearranging gives ⟨q,h⟩+τKL(q∥r)=−τlogZ+τKL(q∥q∗). q,h +τ (q\,\|\,r)=-τ Z+τ (q\,\|\,q^*). (10) Because the Kullback–Leibler divergence is nonnegative, the right-hand side is minimized at q∗q^*, with minimum value −τlogZ-τ Z. Moreover, KL(q∥q∗)=0 (q\,\|\,q^*)=0 if and only if q=q∗q=q^*, which proves uniqueness. ∎ Theorem B.4. Suppose that X and Y are finite and that ρ, πrefπ^ref, and ωrefω^ref have full support. Let f be defined as in (4). Suppose that τL,τF>0τ^L,τ^F>0 and πref,ωrefπ^ref,ω^ref are of full support. Then there exists a unique f-Stackelberg Equilibrium (π∗,ω∗)(π^*,ω^*) given by ω∗(y′∣x,y) ω^*(y x,y) =ωref(y′∣x,y)exp(u(x,y,y′)/τF)ZF(x,y), = ω^ref(y x,y) \! (u(x,y,y )/τ^F )Z^F(x,y), π∗(y∣x) π^*(y x) =πref(y∣x)exp(−τFτLlogZF(x,y))ZL(x), = π^ref(y x) \! (- τ^Fτ^L Z^F(x,y) )Z^L(x), where the normalization functions ZFZ^F and ZLZ^L are defined as ZF(x,y) Z^F(x,y) ≔∑z∈ωref(z∣x,y)exp(u(x,y,z)/τF), _z ω^ref(z x,y) \! (u(x,y,z)/τ^F ), ZL(x) Z^L(x) ≔∑z∈πref(z∣x)exp(−(τF/τL)logZF(x,z)). _z π^ref(z x) \! (-(τ^F/τ^L) Z^F(x,z) ). Proof. Fix a Leader’s policy π∈Ππ∈ . The Follower’s problem is equivalent to maxω∈Ωx∼ρ(⋅),y∼π(⋅∣x)[Φx,y(ω(⋅∣x,y)], _ω∈ E_x ρ(·),y π(· x)[ _x,y(ω(· x,y)], (11) where we define Φx,y(ν)≔y′∼ν(⋅)[u(x,y,y′)]−τFKL(ν∥ωref(⋅∣x,y)). _x,y(ν) _y ν(·)[u(x,y,y )]-τ^FKL(ν\,\|\,ω^ref(· x,y)). We solve the maximization problem (11) pointwise for each (x,y)∈×(x,y) ×Y. By Lemma B.3, we have maxν∈Δ()Φx,y(ν)=τFlogZF(x,y), _ν∈ (Y) _x,y(ν)=τ^F Z^F(x,y), with unique maximizer ω∗(y′∣x,y)=ωref(y′∣x,y)exp(u(x,y,y′)/τF)ZF(x,y),ω^*(y x,y)= ω^ref(y x,y) \! (u(x,y,y )/τ^F )Z^F(x,y), where the normalization function ZFZ^F is defined as ZF(x,y)≔∑y′∈ωref(y′∣x,y)exp(u(x,y,y′)τF).Z^F(x,y) _y ω^ref(y x,y) \! ( u(x,y,y )τ^F ). It follows that ω∗ω^* maximizes f(π,⋅)f(π,·) for every π. For a fixed π, this maximizer is unique at every point (x,y)(x,y) such that ρ(x)π(y∣x)>0ρ(x)π(y x)>0. Substituting the Follower’s optimal value into the Leader’s problem gives V(π) V(π) ≔maxω∈Ωf(π,ω) _ω∈ f(π,ω) (12) ≔∑x∈ρ(x)[∑y∈π(y∣x)τFlogZF(x,y)+τLKL(π(⋅∣x)∥πref(⋅∣x))]. _x ρ(x) [ _y π(y x)τ^F Z^F(x,y)+τ^LKL\! (π(· x)\,\|\,π^ref(· x) ) ]. The minimization of V separates across contexts. For each x∈x , apply Lemma B.3 again, now with ZL(x)≔∑y∈πref(y∣x)exp(−τFτLlogZF(x,y)).Z^L(x) _y π^ref(y x) \! (- τ^Fτ^L Z^F(x,y) ). Lemma B.3 shows that the minimum of the x-indexed term in (12) is −τLlogZL(x)-τ^L Z^L(x), and that its unique minimizer is π∗(y∣x)=πref(y∣x)exp(−τFτLlogZF(x,y))ZL(x).π^*(y x)= π^ref(y x) \! (- τ^Fτ^L Z^F(x,y) )Z^L(x). Since ρ has full support, every context-indexed term in (12) has strictly positive weight. Hence π∗π^* is the unique minimizer of V, and, in particular, π∗∈argminπ∈Πmaxω∈Ωf(π,ω).π^* _π∈ \, _ω∈ \,f(π,ω). Moreover, since πrefπ^ref has full support, so does π∗π^*. Thus, for every (x,y)(x,y), we have ρ(x)π∗(y∣x)>0ρ(x)π^*(y x)>0. By the preceding argument, this implies that ω∗ω^* is the unique maximizer of f(π∗,⋅)f(π^*,·), completing the proof. ∎ Appendix C Convergence of Stackelberg Mirror Descent–Ascent This Appendix proves a stronger version of Theorem 4.3, on the last-iterate convergence of smida. We establish explicit bounds in both directions of the KL divergence for each player, uniformly over contexts and, for the Follower, conditioning actions. C.1 Uniform Last Iterate Convergence For a function v:A→ℝv:A , write osc(v):=maxav(a)−minav(a)osc(v):= _av(a)- _av(a) for the oscillation of v. Define the two payoff oscillations DF D_F ≔maxx,yosc(u(x,y,⋅))=maxx,y[maxy′u(x,y,y′)−miny′u(x,y,y′)], _x,yosc (u(x,y,·) )= _x,y [ _y u(x,y,y )- _y u(x,y,y ) ], D D ≔maxxosc(u(x,⋅,⋅))=maxx[maxy,y′u(x,y,y′)−miny,y′u(x,y,y′)], _xosc (u(x,·,·) )= _x [ _y,y u(x,y,y )- _y,y u(x,y,y ) ], The first is the variation relevant to the Follower’s tracking error; the second also allows the Leader action to vary, and satisfies DF≤D_F≤ D. Theorem C.1. Suppose that X and Y are finite, that ρ, πrefπ^ref, and ωrefω^ref have full support, and that τL,τF>0τ^L,τ^F>0. Run Algorithm 1 under condition (6), and let (π∗,ω∗)(π^*,ω^*) be the unique regularized Stackelberg equilibrium. Set qL:=1−ηLτL,qF:=1−ηFτF,κ:=ηFτFηLτL>1,q_L:=1-η^Lτ^L, q_F:=1-η^Fτ^F, κ:= η^Fτ^Fη^Lτ^L>1, and define r(x,y):=τFlog∑y′ωref(y′∣x,y)exp(u(x,y,y′)τF),DL:=maxxoscr(x,⋅).r(x,y):=τ^F \! _y ω^ref(y x,y) \! ( u(x,y,y )τ^F ), D_L:= _xoscr(x,·). Then all iterates have full support and, for every t≥0t≥ 0, maxx,ymaxKL(ω∗(⋅∣x,y)∥ωt(⋅∣x,y)),KL(ωt(⋅∣x,y)∥ω∗(⋅∣x,y))≤qFtDFτF, _x,y \! \ \! (ω^*(· x,y)\| _t(· x,y) ), \! ( _t(· x,y)\|ω^*(· x,y) ) \≤ q_F^t D_Fτ^F, (13) maxxmaxKL(π∗(⋅∣x)∥πt(⋅∣x)),KL(πt(⋅∣x)∥π∗(⋅∣x)) _x \! \ \! (π^*(· x)\| _t(· x) ), \! ( _t(· x)\|π^*(· x) ) \ ≤qLtDLτL+ηLDFqLt−qFtqL−qF. 99.58464pt≤ q_L^t D_Lτ^L+η^LD_F q_L^t-q_F^tq_L-q_F. (14) Moreover, DL≤D_L≤ D, and hence the simpler bound maxxmaxKL(π∗(⋅∣x)∥πt(⋅∣x)),KL(πt(⋅∣x)∥π∗(⋅∣x))≤qLtDτLκ−1 _x \! \ (π^*(· x)\| _t(· x)), ( _t(· x)\|π^*(· x)) \≤ q_L^t Dτ^L κ-1 (15) also holds. Thus the Follower converges at rate qFtq_F^t and the Leader at rate qLtq_L^t, simultaneously in both directions of KL and uniformly over all contexts. Proof. Observe that, by their definition in Algorithm 1, all iterates have full support. We make use of the following inequality: maxKL(p∥q),KL(q∥p)≤osc(logp−logq) \KL(p\|q),KL(q\|p)\ \! ( p- q ) (16) for full-support distributions p,qp,q. Indeed, if v:=logp−logqv:= p- q, normalization implies minv≤0≤maxv v≤ 0≤ v. Therefore KL(p∥q)=p[v]≤maxv≤osc(v)KL(p\|q)=E_p[v]≤ v (v), and the reverse direction follows identically. For the Follower, let vtx,y:=logωt(⋅∣x,y)−logω∗(⋅∣x,y)v_t^x,y:= _t(· x,y)- ω (· x,y), where ω∗ω^* is the unique equilibrium policy, as per Theorem B.4. Taking logarithms in the Follower update gives vt+1x,y=qFvtx,y+ctx,yv_t+1^x,y=q_Fv_t^x,y+c_t^x,y1 for a scalar normalizing constant ctx,yc_t^x,y and 1 the vector of ones. Since oscillation is invariant under addition of constants, osc(vtx,y)=qFtosc(v0x,y)=qFtoscu(x,y,⋅)τF≤qFtDFτF.osc(v_t^x,y)=q_F^tosc(v_0^x,y)=q_F^t oscu(x,y,·)τ^F≤ q_F^t D_Fτ^F. (17) Combining this with (16) proves (13). From the explicit form of ω∗ω^*, we have log(ω∗/ωref)=u/τF−logZF (ω /ω^ref)=u/τ^F- Z^F. Hence, τFKL(ωt∥ω∗)=τFKL(ωt∥ωref)−ωt[u]+τFlogZF.τ^FKL( _t\|ω^*)=τ^FKL( _t\|ω^ref)-E_ _t[u]+τ^F Z^F. Using r=τFlogZFr=τ^F Z^F and r^t=ωt[u]−τFKL(ωt∥ωref) r_t=E_ _t[u]-τ^FKL( _t\|ω^ref) gives r−r^t=τFKL(ωt∥ω∗)r- r_t=τ^FKL( _t\|ω ). Equations (16) and (17) imply, uniformly in (x,y)(x,y), 0≤r(x,y)−r^t(x,y)≤DFqFt.0≤ r(x,y)- r_t(x,y)≤ D_Fq_F^t. For the Leader, put mtx:=logπt(⋅∣x)−logπ∗(⋅∣x)m_t^x:= _t(· x)- π (· x). Taking logarithms in the Leader update gives mt+1x=qLmtx+ηL(r(x,⋅)−r^t(x,⋅))+dtxm_t+1^x=q_Lm_t^x+η^L(r(x,·)- r_t(x,·))+d_t^x1 for another scalar normalizer dtxd_t^x. Because r−r^t≥0r- r_t≥ 0, we have osc(r−r^t)(x,⋅)≤maxy(r−r^t)(x,y)osc(r- r_t)(x,·)≤ _y(r- r_t)(x,y). Hence maxxosc(mt+1x)≤qLmaxxosc(mtx)+ηLDFqFt. _xosc(m_t+1^x)≤ q_L _xosc(m_t^x)+η^LD_Fq_F^t. Since π0=πref _0=π^ref, maxxosc(m0x)=DLτL. _xosc(m_0^x)= D_Lτ^L. Expanding Equation (C.1) therefore gives maxxosc(mtx)≤qLtDLτL+ηLDF∑k=0t−1qLt−1−kqFk, _xosc(m_t^x)≤ q_L^t D_Lτ^L+η^LD_F _k=0^t-1q_L^t-1-kq_F^k, and the finite sum equals (qLt−qFt)/(qL−qF)(q_L^t-q_F^t)/(q_L-q_F). Applying (16) proves (14). Finally, r(x,y)r(x,y) lies between miny′u(x,y,y′) _y u(x,y,y ) and maxy′u(x,y,y′) _y u(x,y,y ), so DL≤D_L≤ D. Since qF<qLq_F<q_L, ∑k=0t−1qLt−1−kqFk≤qLtηFτF−ηLτL, _k=0^t-1q_L^t-1-kq_F^k≤ q_L^tη^Fτ^F-η^Lτ^L, which gives (15). ∎ Note that Theorem 4.3 follows by averaging the uniform bounds of Theorem C.1. Indeed, by definition, KLρ(π∥π~)=∑xρ(x)KL(π(⋅∣x)∥π~(⋅∣x)),KL_ρ(π\| π)= _xρ(x)\,KL (π(· x)\| π(· x) ), and KLρ,π∗(ω∥ω~)=∑x,yρ(x)π∗(y∣x)KL(ω(⋅∣x,y)∥ω~(⋅∣x,y)).KL_ρ,π^*(ω\| ω)= _x,yρ(x)π^*(y x)\,KL (ω(· x,y)\| ω(· x,y) ). The weights in each sum are nonnegative and sum to one. Thus, applying the uniform bounds in Theorem C.1 separately to both directions of the KL divergence gives, for every t≥1t≥ 1, maxKLρ(π∗∥πt),KLρ(πt∥π∗)≤DτLκ−1(1−ηLτL)t, \! \KL_ρ(π^*\| _t),KL_ρ( _t\|π^*) \≤ Dτ^L κ-1(1-η^Lτ^L)^t, and maxKLρ,π∗(ω∗∥ωt),KLρ,π∗(ωt∥ω∗)≤DFτF(1−ηFτF)t. \! \KL_ρ,π^*(ω^*\| _t),KL_ρ,π^*( _t\|ω^*) \≤ D_Fτ^F(1-η^Fτ^F)^t. Consequently, the strictly positive constants required in the main theorem may be chosen as CL=max1,DτLκ−1,CF=max1,DFτF.C_L= \! \1, Dτ^L κ-1 \, C_F= \! \1, D_Fτ^F \. These constants are finite because the action and context sets are finite, τL,τF>0τ^L,τ^F>0, and κ>1κ>1. The step-size condition ensures that both geometric factors lie in [0,1)[0,1), so the bounds imply last-iterate linear convergence; full support of every iterate is already established in Theorem 4.3. C.2 The Follower’s Timescale and the Leader’s Approximation Error The choice of a faster Follower in (6) follows the approach adopted in SLHF (Pásztor et al., 2026). The underlying intuition is standard in two-timescale optimization: the inner player approaches its best response while the outer player changes more slowly, allowing the latter’s updates to approximate optimization of the maximized objective (Lin et al., 2025). In our tabular setting, the Follower’s objective at each (x,y)(x,y) is independent of the Leader’s policy. The faster timescale therefore serves to reduce the error in the conditional values r^t r_t used by the Leader. The role of the Follower’s approximation error can be made explicit. Let r(x,y)≔τFlogZF(x,y)r(x,y) τ^F Z^F(x,y) denote the Follower’s optimal regularized value at (x,y)(x,y). The following lemma identifies the difference between this value and the current conditional value r^t(x,y) r_t(x,y). Lemma C.2. Under the assumptions of Theorem B.1, for every t≥0t≥ 0 and every (x,y)∈×(x,y) ×Y, r(x,y)−r^t(x,y)=τFKL(ωt(⋅∣x,y)∥ω∗(⋅∣x,y))≥0.r(x,y)- r_t(x,y)=τ^F ( _t(· x,y)\,\|\,ω^*(· x,y) )≥ 0. Proof. Fix (x,y)(x,y). Apply the identity established in the proof of Lemma B.3 with h=−u(x,y,⋅)h=-u(x,y,·), reference distribution ωref(⋅∣x,y)ω^ref(· x,y), and regularization parameter τFτ^F. The unique minimizer is ω∗(⋅∣x,y)ω^*(· x,y) and the minimum is −r(x,y)-r(x,y). Evaluating the identity (10) at ωt(⋅∣x,y) _t(· x,y) therefore gives −r^t(x,y)=−r(x,y)+τFKL(ωt(⋅∣x,y)∥ω∗(⋅∣x,y)).- r_t(x,y)=-r(x,y)+τ^F ( _t(· x,y)\,\|\,ω^*(· x,y) ). Rearranging proves the equality. Nonnegativity follows from that of KL divergence. ∎ Thus, convergence of the Follower directly controls the error in the costs used by the Leader. Appendix D Deep-smida: Policy Gradients and Implementation Details This Appendix contains algorithmic and implementation details for deep-smida (see Algorithm 2). We parametrize the Leader’s and Follower’s policies by πθ _θ and ωϕ _φ, respectively. Denote by ρ the full support distribution over contexts and by πref,ωrefπ^ref,ω^ref the corresponding reference policies. Motivated by the objective in Equation (4) and the procedure described in Algorithm 1, we write the Follower’s exact objective and Leader’s plug-in objective, respectively, as JF(θ,ϕ) J^F(θ,φ) ≔x∼ρ(⋅),y∼πθ(⋅∣x),y′∼ωϕ(⋅∣x,y)[u(x,y,y′)]−τFKLρ,πθ(ωϕ∥ωref), _x ρ(·),y _θ(· x),y _φ(· x,y)[u(x,y,y )]-τ^F _ρ, _θ( _φ\,\|\,ω^ref), (18) JL(θ) J^L(θ) ≔x∼ρ(⋅),y∼πθ(⋅∣x)[r^(x,y)]+τLKLρ(πθ∥πref), _ subarraycx ρ(·),\;y _θ(· x) subarray [ r(x,y) ]+τ^LKL_ρ ( _θ\, \|\, _ref ), (19) where following Equation (5), r^ϕ r_φ denotes the plug-in cost currently faced by the Leader: r^ϕ(x,y)≔y′∼ωϕ(⋅∣x,y)[u(x,y,y′)]−τFKLρ,πθ(ωϕ(⋅∣x,y)∥ωref(⋅∣x,y)) r_φ(x,y) _y _φ(· x,y)[u(x,y,y )]-τ^F _ρ, _θ( _φ(· x,y)\,\|\,ω^ref(· x,y)) Note that the two players optimize in opposite directions, the follower maximizes JFJ^F, whereas the leader minimizes JLJ^L. For any baselines bF(x,y)b^F(x,y) and bL(x)b^L(x) that do not depend on the sampled Follower and Leader actions, respectively, the policy gradients can be written as ∇ϕJF _φJ^F =x,y,y′[∇ϕlogωϕ(y′∣x,y)(u(x,y,y′)−τFlogωϕ(y′∣x,y)ωref(y′∣x,y)−bF(x,y))], =E_x,y,y \! [ _φ _φ(y x,y) (u(x,y,y )-τ^F _φ(y x,y)ω^ref(y x,y)-b^F(x,y) ) ], (20) ∇θJL _θJ^L =x,y[∇θlogπθ(y∣x)(r^ϕ(x,y)+τLlogπθ(y∣x)πref(y∣x)−bL(x))]. =E_x,y\! [ _θ _θ(y x) ( r_φ(x,y)+τ^L _θ(y x)π^ref(y x)-b^L(x) ) ]. (21) The constant terms produced by differentiating the KL regularizers can be absorbed into the baselines because the expected score of a policy is zero. We next give unbiased Monte Carlo estimators of these gradients. Fix a context x∼ρx ρ. Draw D≥2D≥ 2 independent Leader samples y1,…,yD∼πθ(⋅∣x)y^1,…,y^D _θ(· x) and, for every l∈1,…,Dl∈\1,…,D\, draw M≥2M≥ 2 independent Follower samples y′l1,…,y′lM∼ωϕ(⋅∣x,yl)y l1,…,y lM _φ(· x,y^l). In practice, the resulting estimators are additionally averaged over a minibatch of B contexts. Define Alj≔u(x,yl,y′lj)−τFlogωϕ(y′lj∣x,yl)ωref(y′lj∣x,yl),A¯l:=1M∑j=1MAlj.A^lj u(x,y^l,y lj)-τ^F _φ(y lj x,y^l)ω^ref(y lj x,y^l), A^l:= 1M _j=1^MA^lj. (22) For each Follower sample, use the leave-one-out (Kool et al., 2019) baseline b−jF(x,yl):=(M−1)−1∑k≠jAlkb^F_-j(x,y^l):=(M-1)^-1 _k≠ jA^lk. The Follower gradient estimator is then ∇ϕJF^≔1D∑l=1D1M∑j=1M∇ϕlogωϕ(y′lj∣x,yl)(Alj−b−jF(x,yl)). _φJ^F 1D _l=1^D 1M _j=1^M _φ _φ(y lj x,y^l) (A^lj-b^F_-j(x,y^l) ). (23) Conditionally on (x,yl)(x,y^l), the baseline b−jF(x,yl)b^F_-j(x,y^l) is independent of y′ljy lj. It therefore reduces variance without changing the expectation of the estimator. The same rollouts also provide the Leader estimator; no additional sampling is required. In particular, A¯l A^l is a conditionally unbiased estimate of r^(x,yl) r(x,y^l). Define Ll≔A¯l+τLlogπθ(yl∣x)πref(yl∣x),b−lL(x):=1D−1∑m≠lLm.L^l A^l+τ^L _θ(y^l x)π^ref(y^l x), b^L_-l(x):= 1D-1 _m≠ lL^m. (24) The resulting Leader gradient estimator is ∇θJL^≔1D∑l=1D∇θlogπθ(yl∣x)(Ll−b−lL(x)). _θJ^L 1D _l=1^D _θ _θ(y^l x) (L^l-b^L_-l(x) ). (25) Here b−lL(x)b^L_-l(x) is independent of yly^l conditional on x, so the Leader estimator is also unbiased. Averaging Equations (23) and (25) over a minibatch of contexts preserves this property. At iteration t, the Follower takes an ascent step and the Leader a descent step: ϕt+1=ϕt+ηtF∇ϕJF^,θt+1=θt−ηtL∇θJL^. _t+1= _t+ _t^F _φJ^F, _t+1= _t- _t^L _θJ^L. (26) In practice we apply these updates through AdamW optimizers (Loshchilov and Hutter, 2019) with ηtLη^L_t and ηtFη^F_t as their learning rates and no weight decay; each gradient estimate is clipped to unit norm before the optimizer step. Both policies are initialized from their corresponding reference policies, which remain frozen throughout training. Algorithm 2 summarizes the complete procedure: for every minibatch, it samples the nested Leader–Follower rollouts, constructs both gradient estimators, and applies the two updates in Equation (26). D.1 Learning Rate Warm-up. As discussed in Section 4.3, directly starting from the target two-timescale learning rates can make training unstable. Let TwarmT_warm be the number of warm-up iterations. We linearly warm the Leader’s rate from ηinitLη^L_init to its target value ηLη^L and, simultaneously, the ratio of the Follower and Leader rates from 11 to its target value ηF/ηLη^F/η^L: ηtL _t^L =ηinitL+mintTwarm,1(ηL−ηinitL), =η^L_init+ \! \ tT_warm,1 \ (η^L-η^L_init ), ηtF _t^F =ηtL[1+mintTwarm,1(ηFηL−1)]. = _t^L [1+ \! \ tT_warm,1 \ ( η^Fη^L-1 ) ]. Thus, the two rates are equal at initialization, while after warm-up they equal their target values ηLη^L and ηFη^F. We choose the targets so that ηLτL<ηFτFη^Lτ^L<η^Fτ^F, matching the two-timescale ordering of Algorithm 1. Algorithm 2 Deep Stackelberg Mirror Descent–Ascent (deep-smida) 1: Utility u; context distribution ρ; reference policies πrefπ^ref and ωrefω^ref; regularization weights τL,τF>0τ^L,τ^F>0; minibatch size B; sample sizes D,M≥2D,M≥ 2; step sizes ηtL,ηtFt≥0\ _t^L, _t^F\_t≥ 0 2: Initialize θ0 _0 and ϕ0 _0 from their corresponding reference policies, which remain frozen throughout training 3: for t=0,1,2,…t=0,1,2,… do 4: Draw a minibatch ℬt=xii=1BB_t=\x_i\_i=1^B with xi∼ρx_i ρ 5: for i=1,…,Bi=1,…,B do 6: Draw D independent Leader samples yi1,…,yiD∼πθt(⋅∣xi)y_i^1,…,y_i^D _ _t(· x_i) 7: for ℓ=1,…,D =1,…,D do 8: Draw M independent Follower samples yi′ℓ1,…,yi′ℓM∼ωϕt(⋅∣xi,yiℓ)y_i \, 1,…,y_i \, M _ _t(· x_i,y_i ) 9: Compute AiℓjA_i j, A¯iℓ A_i , and b−jF(xi,yiℓ)b^F_-j(x_i,y_i ) according to (22) 10: end for 11: Form the per-context Follower gradient estimator ∇ϕJF^(xi) _φJ^F(x_i) according to (23) 12: Using the same rollouts, compute LiℓL_i and b−ℓL(xi)b^L_- (x_i) according to (24) 13: Form the per-context Leader gradient estimator ∇θJL^(xi) _θJ^L(x_i) according to (D.2) 14: end for 15: Average the per-context estimators over ℬtB_t to obtain ∇ϕJF^t _φJ^F_t and ∇θJL^t _θJ^L_t 16: Update the Follower by ascent and the Leader by descent as in (26): ϕt+1=ϕt+ηtF∇ϕJF^t,θt+1=θt−ηtL∇θJL^t. _t+1= _t+ _t^F _φJ^F_t, _t+1= _t- _t^L _θJ^L_t. 17: end for D.2 Computational Complexity We provide a brief analysis of the computational complexity of Algorithm 2. Each iteration of Algorithm 2 samples B contexts xii∈1,2,…,B\x_i\_i∈\1,2,…,B\, D Leader responses yiℓl∈1,…,D\y_i \_l∈\1,…,D\ per context xix_i, and M Follower responses yi′ℓdd∈1,⋯,M\y_i d\_d∈\1,·s,M\ per Leader response yiℓy_i . It therefore generates BDBD Leader responses and BDMBDM Follower responses, for a total of BDMBDM response pairs. From (23), we observe that estimating ∇ϕJF _φJ^F requires a forward-backward pass through DMDM response trajectories and also DMDM forward passes from the frozen reference. We, notice however that the D rollouts of the leader can be reused. Hence, from , estimating ∇θJL _θJ^L, only requires D forward-backward passes through the leader andDD forward passes from the frozen reference. The number of NFE or Neural Function Evaluations is slightly higher for the follower, due to its higher context length. Further, each of the DMDM response pairs also requires evaluations from each of the k=5k=5 preference ensemble. Multiplying this by the mini-batch size, that for each optimizer step, we require BDMk+3BDM+3BDBDMk+3BDM+3BD rollouts. Appendix E Experimental Setup and Implementation Details This appendix provides details of the experimental setup used in Section 5. We describe dataset construction and oracle labelling, the architecture and training of the preference ensemble, the distribution of ensemble variance, and the implementation of the Leader–Follower game. E.1 Dataset We use a subset of UltraFeedback (Cui et al., 2024), a preference dataset containing approximately 6464k prompts collected from six public instruction sources, each answered by four language models drawn from a pool of seventeen. We retain the prompts and their responses, discard empty and duplicate responses, and expand each prompt x into all (Nx2) N_x2 unordered pairs of its NxN_x distinct responses. Each pair defines a preference triplet (x,y,y′)(x,y,y ). Following Gao et al. (2023), we adopt a synthetic oracle setup, using DeepSeek-V4-Flash (DeepSeek-AI, 2026) as the gold preference oracle. We replace UltraFeedback’s original annotations with oracle labels so that the original response pairs and comparisons subsequently generated by the adversarial game share a common source of supervision. For each comparison, we obtain a binary preference label through zero-shot prompting, randomizing the display order of the two responses to mitigate position bias. The procedure gives a dataset D of 89,62589,625 labelled preference triplets across 15,05315,053 unique prompts. E.2 Preference Models We train an ensemble of |I|=5|I|=5 preference models θii∈I\P_ _i\_i∈ I, using a simplified version of the general preference model of Zhang et al. (2025). Each member is initialized from the same Qwen2.5-1.5B-Instruct backbone (Qwen Team, 2024) and finetuned independently. Given a preference triplet (x,y,y′)(x,y,y ), we encode each prompt-response pair (x,y),(x,y′)(x,y),(x,y ) separately, project its final-token hidden representation through a bias-free linear head into a 1616-dimensional vector, and apply L2L^2 normalization, obtaining an embedding zθi(x,y)z_ _i(x,y). We compare the resulting embeddings through a fixed skew-symmetric bilinear form J∈ℝ16×16J ^16× 16, rescale the score by a learned positive scalar λθi _ _i shared across inputs, and apply a sigmoid σ to obtain the preference probability. The preference probability is θi(y≻y′∣x)≔σ(λθi⟨Jzθi(x,y),zθi(x,y′)⟩).P_ _i(y y x) σ ( _ _i Jz_ _i(x,y),z_ _i(x,y ) ). (27) This construction ensures that θi(y≻y′∣x)+θi(y′≻y∣x)=1P_ _i(y y x)+P_ _i(y y x)=1, while allowing intransitive preferences. By the antisymmetry of the preference models and basic properties of the variance, we have Var(x,y,y′)=Var(x,y′,y)Var(x,y,y )=Var(x,y ,y); thus, the order of the two responses within a preference triplet does not affect the optimization objective. Each ensemble member is trained by minimizing the negative log-likelihood −logθi(yw≻yl∣x)- _ _i(y_w y_l x) of the oracle labels on its own training partition of D, where ywy_w and yly_l denote the oracle-preferred and non-preferred responses. Each partition contains approximately 10,00010,000 preference triplets across 1,6801,680 unique prompts. The partitions are mutually disjoint at the prompt level and all labelled comparisons associated with a training prompt belong to the same partition. Ensemble diversity comes from these distinct training sets, independent random initializations of the linear heads, and different training seeds. All members share a validation set whose prompts are disjoint from every training partition. A further 37,61437,614 triplets across 6,3156,315 prompts are excluded from both training and validation and serve as the held out set of Sections 5.2. On this set, individual members agree with the gold preference oracle on 74.174.1–75.2%75.2\% of comparisons, while predictions obtained from the ensemble-mean preference probability agree on 75.7%75.7\%. E.3 Distribution of Ensemble Variance We examine the distribution of Var(x,y,y′)Var(x,y,y ) on the held out UltraFeedback subset, comprising 37,61437,614 response pairs across 6,3156,315 prompts. For each pair, we compute the population variance of the five members’ predicted preference probabilities, using denominator |I|=5|I|=5. All descriptive statistics weight response pairs equally. Table 1 summarizes the distribution. Most comparisons elicit little ensemble disagreement, while a smaller fraction forms a pronounced right tail: the mean is more than twice the median, and the 9999th percentile is approximately fifteen times the median. Only 8.93%8.93\% of comparisons have variance greater than 0.020.02. Consistent with the relationship shown in the main text, mean variance is higher on comparisons where the ensemble disagrees with the synthetic oracle (0.0100100.010010) than where it agrees (0.0056870.005687). These statistics describe disagreement within the trained ensemble and its association with oracle error on this held out subset. Table 1: Distribution of ensemble variance on the held out UltraFeedback subset. Summary statistics Selected percentiles Statistic Value Percentile Value Mean 0.0067390.006739 2525 0.0005880.000588 Median 0.0029310.002931 7575 0.0089590.008959 Variance 9.0507×10−59.0507× 10^-5 9090 0.0186870.018687 Standard deviation 0.0095140.009514 9595 0.0266600.026660 Maximum 0.1002280.100228 9999 0.0446670.044667 E.4 Stackelberg Game Implementation The Leader and Follower are both initialized from Qwen2.5-1.5B-Instruct (Qwen Team, 2024) and finetuned during the Stackelberg game. Their frozen initializations serve as the reference policies πrefπ^ref and ωrefω^ref. Table 2 summarizes the training configuration. The utility is 4Var(x,y,y′)4Var(x,y,y ), where variance is computed across the five frozen ensemble members with denominator 55, giving a score in [0,1][0,1]. Table 2: Hyperparameters for training the Leader and Follower. Unless specified otherwise, settings apply to both policies. Parameter Value Optimization Optimizer AdamW Leader learning rate ηLη^L 8×10−68× 10^-6 Follower learning rate ηFη^F 2×10−52× 10^-5 Initial learning rate (both policies) 10−610^-6 Warm-up steps 5050 KL coefficients τL,τFτ^L,τ^F 5×10−45× 10^-4 Weight decay 00 Maximum gradient norm 1.01.0 Parameter precision bfloat16 Training and rollout sampling Training prompts 2,0002,000 Training epochs 11 Optimizer steps 500500 Prompts per step B 44 Leader responses per prompt D 44 Follower responses per Leader response M 44 Generation Sampling temperature 0.60.6 Top-p 1.01.0 Maximum generated tokens per response 1,5001,500 During warm-up, we linearly increase the Leader’s learning rate from 10−610^-6 to ηLη^L and the Follower-to-Leader learning-rate ratio from 11 to ηF/ηL=2.5η^F/η^L=2.5. The Follower’s learning rate is their product; both rates remain constant after warm-up. Each optimizer step uses BDM=64BDM=64 preference triplets. Sampling and policy log-probability calculations use the same temperature, with no nucleus or top-k truncation. We use the following chat templates, with the Follower receiving the Leader’s realized response in its system message. ⬇ [System]: You are Qwen, created by Alibaba Cloud. You are a helpful assistant. [User]: <USER_PROMPT> ⬇ [System] You are Qwen, created by Alibaba Cloud. You are a helpful assistant. Another assistant previously answered the user’s request as follows: <LEADER_RESPONSE> Write a different, higher-quality answer to the user’s request. Respond directly to the user as if answering for the first time. Do not mention, quote, or reference the previous answer. [User]: <USER_PROMPT> Appendix F The Utility as a Design Choice Our framework accommodates any fixed utility u without changing the game structure or update rules, and the tabular guarantees remain valid under the stated assumptions. In our experiments, u is the variance of the preference probabilities predicted by a trained ensemble: the Follower seeks to expose ensemble disagreement, while the Leader seeks to limit it. This is not the only option and exploring alternative utilities is a natural direction for future work. Preference optimization is another special case. Setting u(x,y,y′)=(y′≻y∣x)u(x,y,y )=P(y y x) recovers an equivalent minimization form of the regularized SLHF Pásztor et al. (2026) objective. Our analysis thus establishes linear last-iterate convergence of smida for both players in this preference game. In the finite tabular setting with strictly positive regularization and the stated step-size conditions, this answers affirmatively the question raised in Pásztor et al. (2026) of whether mirror-descent methods can be adapted to obtain last-iterate convergence for Stackelberg preference games. F.1 Other Uncertainty Metrics Further examples of utility are motivated by uncertainty measures used for query selection in active preference learning, such as ensemble variance and predictive entropy (Christiano et al., 2017; Lee et al., 2021). We consider here two alternatives to variance: decision uncertainty (Lewis and Gale, 1994) and Jensen–Shannon divergence (JSD) (Lin, 1991). Writing i(x,y,y′)P_i(x,y,y ) for the preference probability predicted by ensemble member i, let ¯(x,y,y′)=1|I|∑i∈Ii(x,y,y′) P(x,y,y )= 1|I| _i∈ IP_i(x,y,y ) denote the ensemble mean. We define decision uncertainty as U(x,y,y′)≔min¯(x,y,y′), 1−¯(x,y,y′)=12−|¯(x,y,y′)−12|,U(x,y,y ) \ P(x,y,y ),\,1- P(x,y,y ) \= 12- | P(x,y,y )- 12 |, and JSD as JSD(x,y,y′)≔H(¯(x,y,y′))−1|I|∑i∈IH(i(x,y,y′)),JSD(x,y,y ) H\! ( P(x,y,y ) )- 1|I| _i∈ IH\! (P_i(x,y,y ) ), where H(p)=−plogp−(1−p)log(1−p)H(p)=-p p-(1-p) (1-p) denotes the entropy of a Bernoulli distribution with parameter p. Decision uncertainty measures the proximity of the ensemble mean to indifference and induces the same ranking as its predictive entropy, H(¯)H( P). JSD instead measures disagreement among the predictive distributions of the individual members. We evaluate both scores on the held-out comparisons of Section 5.2, measuring errors relative to the binary oracle labels. Figure 3(a) reports error-detection AUROCs of 0.7070.707 for decision uncertainty and 0.6540.654 for JSD. Selecting the top 10%10\% of comparisons ranked by each score gives error rates of 47.5%47.5\% and 39.7%39.7\%, respectively, compared with 24.3%24.3\% overall (Figure 3(b)). On this evaluation, decision uncertainty therefore outperforms variance in both AUROC (0.6700.670) and error concentration in the top 10%10\% (40.6%40.6\%), while JSD gives slightly lower values on both measures. Better error detection on a fixed set of comparisons, however, does not necessarily make a score a suitable utility for the adversarial game. Decision uncertainty is maximized when the ensemble mean is 1/21/2, including when all members agree on indifference. Since the Follower observes the Leader’s realized response, it can attain this maximum by copying that response, setting y′=y =y. The antisymmetry of our preference models ensures that i(x,y,y)=1/2P_i(x,y,y)=1/2 for every member, and hence U(x,y,y)=1/2U(x,y,y)=1/2 for every (x,y)(x,y). Thus, copying is a best response to every Leader response, and the Leader cannot reduce the Follower’s optimal utility below 1/21/2. Variance and JSD avoid this particular failure mode, since both vanish when all ensemble members predict 1/21/2, including on identical-response comparisons. Figure 3: Preference-ensemble error detection on 37,61437,614 held-out comparisons, including 9,1559,155 errors relative to the binary oracle labels. (a) ROC curves for decision uncertainty (blue) and Jensen–Shannon divergence (JSD; red), with AUROCs of 0.7070.707 and 0.6540.654, respectively. The dashed diagonal indicates chance performance. (b) Oracle-relative error rates among comparisons selected in descending order of each score. Selecting the top 10%10\% gives error rates of 47.5%47.5\% for decision uncertainty (95%95\% CI: 46.046.0–49.1%49.1\%) and 39.7%39.7\% for JSD (95%95\% CI: 38.238.2–41.3%41.3\%), compared with 24.3%24.3\% overall. Shading shows pointwise 95%95\% prompt-bootstrap intervals. F.2 Targeting a Specific Preference Model Another option is to augment the ensemble disagreement objective in Equation (1) with a preference term. Given a fixed member θjP_ _j of the ensemble and a coefficient λ≥0λ≥ 0, consider minπmaxωx∼ρ,y∼π(⋅∣x),y′∼ω(⋅∣x,y)[Var(x,y,y′)+λθj(y′≻y∣x)]. _π _ωE_x ρ,\,y π(· x),\,y ω(· x,y) [Var(x,y,y )+λ\,P_ _j(y y x) ]. The additional term serves two purposes. First, it acts as a preference-based regularizer as the Follower is encouraged to generate responses that the selected model prefers to the Leader’s response. The Leader, in turn, seeks responses that resist both disagreement and defeat under this model. This introduces a preference signal for optimization, with λ controlling its strength relative to variance. Second, the term allows us to direct the search toward a specific preference model while retaining uncertainty information from the full ensemble. Experiments show that this directed objective achieves comparable performance to the variance objective in generating α-online candidates. Appendix G Further Experiments, Parameters Sweep, and Ablation G.1 Training Dynamics Figure 4 complements the evaluation in Section 5.3. Panel (a) shows the empirical CDF of ensemble variance on held-out prompts, computed from generations obtained after training. The trained pair’s distribution shifts toward higher variance relative to the reference pair; vertical lines mark the reference-quantile thresholds defining α-online candidates. Panel (b) tracks candidate shares in training minibatches of 6464 triplets over 500500 steps, using these same fixed thresholds. The moving averages begin near the nominal reference rates, rise markedly after approximately 200200 steps, and remain elevated through the end of training despite substantial minibatch variability. By the final steps, the smoothed shares are approximately 50%50\% at α=0.1α=0.1 and 20%20\% at α=0.01α=0.01. The increased candidate yield is thus sustained during training and is also observed on held-out prompts after training. For examples of high variance generated preference triplets, see Appendix H. Figure 4: Ensemble disagreement after and during training. (a) Empirical CDFs of ensemble variance on 5,0005,000 triplets from 2,5002,500 held-out prompts per policy pair, generated after training. Vertical lines mark reference-quantile thresholds; shading shows 95%95\% prompt-bootstrap intervals. (b) Shares of α-online candidates in training minibatches of 6464 triplets, using the same fixed thresholds. Thin curves show per-step shares, thick curves their moving averages, and horizontal dotted lines the nominal reference rates α. G.2 Hyperparameter Sweep We performed a sequential hyperparameter sweep over the KL regularization coefficients, the Follower’s target learning rate, and the warm-up duration. We first varied the common regularization coefficient τ=τL=τF∈2×10−4, 5×10−4, 10−3, 2×10−3, 4×10−3.τ=τ^L=τ^F∈\2× 10^-4,\,5× 10^-4,\,10^-3,\,2× 10^-3,\,4× 10^-3\. After selecting τ, we swept the Follower’s target learning rate ηF∈2×10−5, 3×10−5, 5×10−5η^F∈\2× 10^-5,\,3× 10^-5,\,5× 10^-5\ and the warm-up duration Twarm∈15,20,50T_warm∈\15,20,50\ optimizer steps, using the joint learning-rate and learning-rate-ratio schedule described in Appendix D.1. Configuration selection considered both training stability and the fraction of generated triplets satisfying the α-online candidate criterion of Definition 3.2. G.3 Adversarial Precision of Online Candidates We further examine whether the online criterion (Definition 3.2) identifies comparisons on which the ensemble’s mean preference is incorrect. Using the trained policies’ 1,600 oracle-labelled triplets from 800 holdout prompts described in Section 5.3.2, Figure 5 reports the fraction of α-online candidates that are ε -adversarial. Smaller α imposes a stricter disagreement threshold and generally increases this fraction. At α=0.1α=0.1, 29.7%29.7\% of candidates reverse the oracle preference (ε=0 =0), compared with 19.4%19.4\% of all labeled triplets, a 1.531.53-fold enrichment over random selection. The same pattern continues for ε∈0.1,0.2,0.3 ∈\0.1,0.2,0.3\, indicating that the enrichment extends to reversals with larger oracle margins. These results support ensemble disagreement as an informative criterion for prioritizing comparisons for annotation within the trained policies’ generations. Figure 5: Adversarial precision of online candidates. Solid curves show the fraction of α-online candidates that are ε -adversarial. Candidate thresholds are fixed by the reference variance distribution, as in Section 5 Lighter curves correspond to larger oracle margins. Horizontal dotted lines show the corresponding adversarial fractions among all labelled triplets, giving the expected precision under random selection. Shading shows a 95% prompt-bootstrap interval for ε=0 =0. G.4 Preference Evaluation of the Trained Leader We evaluate whether our Leader produces responses that remain competitive under preference evaluation, despite being only trained to minimize ensemble disagreement rather than maximize preference. We compare it with NLHF Munos et al. (2024) and the SLHF Leader Pásztor et al. (2026) in a round-robin evaluation. All three policies are initialized from Qwen2.5-1.5B-Instruct and trained on the same 2,0002,000 prompts. NLHF and SLHF optimize preferences under the frozen model 0P_0, whereas our Leader optimizes the ensemble-variance objective. To control for differences in the choice of trainable parameters, we use full-model fine-tuning for all three methods, with LoRA disabled. Table 3 reports the training and generation settings for the two baselines. We evaluate all three policy pairings on a separate set of 500500 held-out prompts, using DeepSeek-V4-Flash as the synthetic preference oracle. For each response pair, we query the oracle k=11k=11 times and estimate the preference probability by the fraction of judgments favoring one response over the other. Table 4 reports these win rates, averaged across evaluation prompts. Both Leaders are preferred to NLHF more often than not: the SLHF Leader achieves a preference rate of 54.92%54.92\%, and our Leader achieves 54.89%54.89\%. Their direct comparison is nearly even, with our Leader preferred in 50.70%50.70\% of oracle judgments. This competitiveness is an unintended benefit of training our Leader to resist ensemble disagreement. We conjecture that minimizing disagreement against an adaptive Follower, together with regularization toward the initial instruction-tuned policy, may favor responses that retain useful instruction-following behavior. Table 3: Training and generation settings for the NLHF (Nash-MD-PG) and SLHF (StackelbergGDA) baselines. Both methods fully fine-tune Qwen2.5-1.5B-Instruct on the same 2,0002,000 training prompts using the frozen preference model 0P_0. Each uses a separate frozen copy of the initial policy as its KL reference. Parameter NLHF SLHF Training and rollout sampling Training prompts 2,0002,000 2,0002,000 Training epochs 11 11 Optimizer steps 6363 6363 Prompts per optimizer step 3232 3232 Prompts in final step 1616 1616 Responses per prompt 22 22 Generation Sampling temperature 0.60.6 0.60.6 Top-p 1.01.0 1.01.0 Maximum generated tokens per response 512512 512512 Table 4: Pairwise oracle preference rates on 500500 held-out prompts. Each entry estimates the probability that the row policy’s response is preferred to the column policy’s response, using 1111 oracle queries per response pair and averaging across prompts. Values above 0.50.5 favor the row policy. Column policy Row policy NLHF SLHF Leader Our Leader NLHF — 0.45080.4508 0.45110.4511 SLHF Leader 0.54920.5492 — 0.49300.4930 Our Leader 0.54890.5489 0.50700.5070 — G.5 Instruction Following with a Trained Leader We examine the role of the Leader by comparing the Follower trained in the full Stackelberg game with a follower-only baseline, in which the Leader remains fixed. Both Followers seek to maximize ensemble disagreement, but only the former trains against a Leader that learns to resist its attempts. This ablation evaluates whether adapting both sides of the comparison is associated with better instruction following after adversarial training. We evaluate both Followers on IFEval Zhou et al. (2023), using the same 541541 prompts containing 834834 verifiable instructions. IFEval uses programmatic checks to assess compliance with explicit requirements, such as including specified keywords, respecting length constraints, or ending with a prescribed phrase. Instruction-level accuracy measures the fraction of individual requirements satisfied, whereas prompt-level accuracy requires all verifiable instructions in a prompt to be satisfied. We report both strict and loose accuracy. Strict evaluation checks the original response; loose evaluation additionally accepts specified transformations, such as removing Markdown asterisks or the first or last line, to reduce false negatives caused by formatting or introductory and concluding text. Table 5 shows that the Stackelberg Follower achieves higher accuracy on all four aggregate metrics. Loose prompt-level accuracy is 35.12%35.12\%, compared with 28.28%28.28\% for the follower-only baseline, and loose instruction-level accuracy is 46.04%46.04\%, compared with 38.49%38.49\%. The advantage persists under strict evaluation and also extends across eight of the nine instruction categories. These results support our motivation for making the search adversarial on both sides of the comparison. With a fixed Leader, the Follower can specialize in inducing disagreement against an unchanging response distribution, potentially exploiting weaknesses specific to that distribution. An adaptive Leader instead learns responses that resist these attempts, requiring the Follower to search against an evolving opponent. The higher instruction-following accuracy is consistent with this interaction discouraging strategies that induce disagreement at the expense of compliance with the prompt. Table 5: Leader ablation on IFEval. Accuracy (%) is reported for the Follower trained with an adaptive Leader (Stackelberg Follower) and with a fixed Leader (follower-only), evaluated on the same 541541 prompts and 834834 verifiable instructions. The higher value in each row is bolded. Category rows report loose instruction-level accuracy. Metric Stackelberg Follower Follower-only Loose: prompt 35.12 28.28 Loose: instruction 46.04 38.49 Strict: prompt 30.31 23.66 Strict: instruction 41.25 33.93 Loose accuracy by instruction category Case 22.47 33.71 Combination 10.77 9.23 Detectable content 50.94 41.51 Detectable format 63.69 49.04 Keywords 55.83 53.99 Language 74.19 64.52 Length 44.76 33.57 Punctuation 19.70 16.67 Start/end 58.21 28.36 Appendix H Example Prompts and Responses We present selected high variance Leader and Follower comparisons to illustrate the response behaviors encountered during adversarial search. They illustrate unsupported elaboration, discrepancies between reasoning and final-answer correctness, and failures to preserve the requirements of a prompt. Example 1 (Unsupported elaboration) The Leader directly supplies the name given in the passage. The Follower reproduces that answer but adds a speculative expansion of the initial “S.” that the passage does not support. This example illustrates how elaboration can introduce unsupported specificity into an otherwise sufficient answer. ⬇ Quick Question: The chain’s origin can be traced to the Dwarf Grill (now the Dwarf House), a restaurant opened by S. Truett Cathy, the chain’s former chairman and CEO, in 1946. The restaurant is located in Hapeville, Georgia, a suburb of Atlanta, and is near the location of the now-demolished Ford Motor Company Atlanta Assembly Plant, for many years a source of many of the restaurant’s patrons. Question: who was the founder of chick fil a? My answer: ⬇ The founder of Chick-fil-A is S. Truett Cathy. ⬇ To answer the question, let’s look at the context provided: ”The chain’s origin can be traced to the Dwarf Grill (now the Dwarf House), a restaurant opened by S. Truett Cathy, the chain’s former chairman and CEO, in 1946.” From this sentence, we can see that S. Truett Cathy founded Chick-fil-A. The full name given is S. Truett Cathy, which likely stands for Stephen Truett Cathy. The word ”Cathy” is often used as a family name in American English, so we can assume this refers to the founder. Example 2 (Correct final answer with flawed reasoning) Under the natural reading that each pack contains 3030 utensils split equally among knives, forks, and spoons, each pack contains 1010 spoons and the required number of packs is 55. The Leader reaches this final answer through incorrect and internally inconsistent reasoning. The Follower is much more concise, but divides by 33 and returns the incorrect answer 1717. This comparison exposes a tension between final-answer correctness and explanation quality. ⬇ John decides to buy utensils. They come in 30 packs with an equal number of knives, forks, and spoons. How many packs does he need to buy if he wants 50 spoons? ⬇ To find out how many packs John needs to buy, we start by figuring out how many teaspoons (knives and forks) are needed in total: Number of spoons per pack = Number of packs x Number of spoons per pack Since each pack contains an equal number of knives, forks, and spoons: Teaspoons per pack = 3 spoons/pack Now calculate the total teaspoons needed: Total teaspoons needed = Number of spoons wanted x Number of spoons per pack Total teaspoons needed = 50 spoons x 3 spoons/pack Total teaspoons needed = 150 spoons Each pack provides 3 spoons, but since there’s no specific information about the distribution between knives and forks, let’s assume he needs exactly 30 packs to get 90 spoons for simplicity. Therefore, the number of packs John needs to buy is 150 spoons / 30 spoons/pack Number of packs needed = 150 spoons / 30 spoons/pack Number of packs needed = 5 packs So, John needs to buy 5 packs of utensils to have 50 spoons. ⬇ He would need 50/3 = 16.67 packs, which rounds up to 17 packs. The answer is 17. Example 3 (Errors motivated by a rewrite) Both responses classify the original declarative sentence as exclamatory and replace “over 100 million” with “100 million”, losing the stated lower bound. The Follower additionally inserts an exclamation mark and claims that its rewrite “maintains the exclamation”, although the original sentence contains none. The rewrite makes the mistaken classification appear consistent with the revised text. ⬇ Identify the type of the following sentence, rewrite it in a more concise form, and incorporate the budget of the movie as an additional constraint. Determine the sentence type and simplify: The Tom Cruise movie was a huge hit, with a budget of over 100 million. ⬇ Sentence Type: Exclamatory Sentence Simplified Form: ”The Tom Cruise movie had an incredible success with a staggering budget of 100 million.” Incorporating Budget Constraint: ”The Tom Cruise movie was a massive hit despite its extremely high budget of 100 million.” ⬇ Type of Sentence: Exclamatory Concise and Incorporating Budget Constraint: ”Tom Cruise’s movie is a huge smash, with an astronomical budget of 100 million!” This version maintains the exclamation, simplifies the language, and incorporates the budget detail. Example 4 (Loss of implicit alignment) The prompt refers to a sentence that is not supplied. The Leader acknowledges this missing input and offers a qualified suggestion. The Follower omits the qualification and substitutes scheduled punctuality for arriving just in time to prevent a negative consequence, which need not describe the same situation. ⬇ Given this sentence about punctuality, suggest a more appropriate and precise verb to convey the notion of arriving just in time to prevent a negative consequence. ⬇ The original sentence is not provided, but I can offer a suggestion based on common scenarios where punctuality might be crucial: ”If you arrive **on** time, you will avoid any negative consequences.” This phrase accurately conveys that being there exactly when needed prevents issues from arising. ⬇ ”Arriving precisely at the scheduled moment” Appendix I Why General Preference Models Can Be More Prone to Reward Hacking General preference models can represent intransitive preferences that scalar reward models cannot capture. This additional expressivity can, however, increase the number of annotations needed to estimate preferences accurately. We illustrate this tradeoff with a toy example in which the ground-truth preferences admit a scalar reward representation. In this setting, a reward model can pool information across comparisons, whereas an unrestricted preference estimator learns each pair separately. Fix a prompt x and a response set =y1,…,yKY=y_1,…,y_K with K≥2K≥ 2. Write ij∗:=∗(yi≻yj∣x),P^*_ij:=P^*(y_i y_j x), which we assume to be antisymmetric. . For each of the M=(K2)M= K2 unordered pairs, we observe n annotations Yij(ℓ)∼Bernoulli(ij∗),i<j,ℓ=1,…,nY_ij^( ) (P^*_ij), i<j, =1,…,n independently across pairs and annotations, for a total budget of nMnM observations. Define Y¯ij:=1n∑ℓ=1nYij(ℓ)for i<j,Y¯ji:=1−Y¯ij,Y¯ii:=12. Yij:= 1n _ =1^nY_ij^( ) i<j, Y_ji:=1- Y_ij, Y_i:= 12. To compare the two estimators under the same ground truth, suppose ∗ij=12+ri−rj,∑i=1Kri=0,|ri|≤14for all i=1,…,K.P^*_ij= 12+r_i-r_j, _i=1^Kr_i=0, |r_i|≤ 14 all i=1,…,K. where rir_i is the reward of response yiy_i. We measure estimation error in the entrywise supremum norm, ‖^−∗‖∞:=maxi,j=1,…,K|^ij−ij∗|.\| P-P^*\|_∞:= _i,j=1,…,K | P_ij-P^*_ij |. Fix a confidence parameter δ∈(0,1)δ∈(0,1). General preference estimator. Consider the unbiased estimator ^ijpref:=Y¯ij P^pref_ij:= Y_ij. For each i<ji<j, Hoeffding’s inequality gives ℙ(|Y¯ij−ij∗|≥α)≤2exp(−2nα2).P\! (| Y_ij-P^*_ij|≥α )≤ 2 (-2nα^2). Taking a union bound over the M unordered pairs gives ‖^pref−∗‖∞≤log(2M/δ)2n\| P^pref-P^*\|_∞≤ (2M/δ)2n (28) with probability at least 1−δ1-δ. This bound does not require the scalar reward representation assumed above. Reward estimator. Now consider r^i:=1K∑j≠i(Y¯ij−12). r_i:= 1K _j≠ i ( Y_ij- 12 ). Under the assumed reward representation, this estimator is unbiased: [r^i]=1K∑j≠i(ri−rj)=1K(Kri−∑j=1Krj)=ri.E[ r_i]= 1K _j≠ i(r_i-r_j)= 1K (Kr_i- _j=1^Kr_j )=r_i. We estimate preferences by ^ijrew:=[12+r^i−r^j][0,1], P^rew_ij:= [ 12+ r_i- r_j ]_[0,1], where [z][0,1]:=min1,max0,z[z]_[0,1]:= \1, \0,z\\ clips the estimate to the probability interval. Consequently, |^ijrew−ij∗|≤|(r^i−ri)−(r^j−rj)|≤|r^i−ri|+|r^j−rj|.| P^rew_ij-P^*_ij|≤|( r_i-r_i)-( r_j-r_j)|≤| r_i-r_i|+| r_j-r_j|. It therefore suffices to control the reward estimation errors uniformly. For each fixed i, we can write r^i−ri=1nK∑j≠i∑ℓ=1n(Yij(ℓ)−ij∗), r_i-r_i= 1nK _j≠ i _ =1^n (Y_ij^( )-P^*_ij ), where Yij(ℓ):=1−Yji(ℓ)Y_ij^( ):=1-Y_ji^( ) for j<ij<i. Hoeffding’s inequality thus gives ℙ(|r^i−ri|≥α2)≤2exp(−nK2α22(K−1)).P\! (| r_i-r_i|≥ α2 )≤ 2 \! (- nK^2α^22(K-1) ). Taking a union bound over the K responses, we obtain ℙ(maxi∈[K]|r^i−ri|≥α2)≤2Kexp(−nK2α22(K−1)).P\! ( _i∈[K]| r_i-r_i|≥ α2 )≤ 2K \! (- nK^2α^22(K-1) ). Setting the right-hand side equal to δ and applying the bound on pairwise preference errors gives ‖^rew−∗‖∞≤2(K−1)log(2K/δ)nK2\| P^rew-P^*\|_∞≤ 2(K-1) (2K/δ)nK^2 (29) with probability at least 1−δ1-δ. Comparing Equations (28) and (29), we find that, for fixed δ, the uniform-error upper bound for the general preference estimator is larger by a factor of order K K. The reward estimator gains accuracy by pooling information across the K−1K-1 comparisons involving each response, whereas the general preference estimator uses only the n annotations for each pair. Thus, under the same annotation budget of nMnM, this example illustrates how general preference estimation can admit a larger upper bound on the errors that optimization may exploit.