Paper deep dive
Mitigating Preference Hacking in Policy Optimization with Pessimism
Dhawal Gupta, Adam Fisch, Christoph Dann, Alekh Agarwal
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 95%
Last extracted: 3/12/2026, 6:24:10 PM
Summary
The paper addresses the problem of reward and preference hacking in Reinforcement Learning from Human Feedback (RLHF) by introducing pessimistic objectives. The authors propose two new algorithms, P3O (Pessimistic Preference-based Policy Optimization) and PRPO (Pessimistic Reward-based Policy Optimization), which incorporate uncertainty to ensure robustness against overoptimization, demonstrating superior performance in language model fine-tuning tasks compared to standard methods like DPO and REINFORCE.
Entities (6)
Relation Signals (3)
P3O → mitigates → Preference Hacking
confidence 95% · demonstrating remarkable resilience to overoptimization.
PRPO → mitigates → Preference Hacking
confidence 95% · demonstrating remarkable resilience to overoptimization.
P3O → isa → RLHF
confidence 90% · We propose novel, pessimistic objectives for RLHF... P3O
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:This work tackles the problem of overoptimization in reinforcement learning from human feedback (RLHF), a prevalent technique for aligning models with human preferences. RLHF relies on reward or preference models trained on \emph{fixed preference datasets}, and these models are unreliable when evaluated outside the support of this preference data, leading to the common reward or preference hacking phenomenon. We propose novel, pessimistic objectives for RLHF which are provably robust to overoptimization through the use of pessimism in the face of uncertainty, and design practical algorithms, P3O and PRPO, to optimize these objectives. Our approach is derived for the general preference optimization setting, but can be used with reward models as well. We evaluate P3O and PRPO on the tasks of fine-tuning language models for document summarization and creating helpful assistants, demonstrating remarkable resilience to overoptimization.
Tags
Links
- Source: https://arxiv.org/abs/2503.06810
- Canonical: https://arxiv.org/abs/2503.06810
Trouble viewing inline? Open PDF directly →
Full Text
231,103 characters extracted from source content.
Expand or collapse full text
Mitigating Preference Hacking in Policy Optimization with Pessimism Dhawal Gupta Adam Fisch Christoph Dann Alekh Agarwal Abstract This work tackles the problem of overoptimization in reinforcement learning from human feedback (RLHF), a prevalent technique for aligning models with human preferences. RLHF relies on reward or preference models trained on fixed preference datasets, and these models are unreliable when evaluated outside the support of this preference data, leading to the common reward or preference hacking phenomenon. We propose novel, pessimistic objectives for RLHF which are provably robust to overoptimization through the use of pessimism in the face of uncertainty, and design practical algorithms, P3O and PRPO, to optimize these objectives. Our approach is derived for the general preference optimization setting, but can be used with reward models as well. We evaluate P3O and PRPO on the tasks of fine-tuning language models for document summarization and creating helpful assistants, demonstrating remarkable resilience to overoptimization. Machine Learning 1 Introduction Reinforcement learning (RL) from human feedback (RLHF) (Christiano et al., 2017) has emerged as a promising technique for aligning language models with human preferences (Stiennon et al., 2020; Ouyang et al., 2022). The predominant approach involves training a reward model on human preference data and then fine-tuning the language model to maximize the expected reward of the responses it generates to training inputs. More recently, a line of work (Swamy et al., 2024; Munos et al., 2023; Calandriello et al., 2024; Guo et al., 2024) has argued for the benefits of learning a pairwise preference function from the preference dataset, and using this to directly compare trajectories side-by-side during online RL. Irrespective of whether we use reward or preference models during training, however, the availability of a limited pool of high-quality preference data presents a key bottleneck in learning good policies. The high cost of collecting preference datasets with human feedback means that they suffer from limited coverage, and the reward or preference models trained on such datasets fail to adequately generalize to policies which produce trajectories out of the support of the preference data. The inadequacy of learned reward/preference models in reliably producing good policies has resulted in the now well-documented phenomenon of reward hacking or overoptimization (Amodei et al., 2016; Gao et al., 2023; Eisenstein et al., 2024). Perhaps the most commonly used technique to limit overoptimization is through regularization by the KL-divergence between the policy being trained and a reference model. However, KL-divergence only measures the distributional distance of the produced responses, independently of the uncertainty in the predictions of the reward/preference models. This typically leads to either overoptimization or overly limiting reward/preference maximization, when too little or too much regularization is applied, necessitating careful tuning. Even when the trade-off is properly tuned, KL regularization is still often observed to inhibit learning. Figure 1: Comparison of our methods (P3O and PRPO) against standard approaches (REINFORCE and DPO) on summarization and “helpful assistant” tasks, showing the evaluation preference of a prompted Gemini evaluator, for generations of the policy over those of the reference policy. The hyperparameters of each method have been tuned to prevent reward hacking (best eval performance), which necessitates strong KL-regularization for DPO and REINFORCE. Our approaches, however, can avoid reward hacking by relying on pessimism instead of KL regularization (a blunt tool), and achieve consistently better performance as a result. We also compare to Nash-EMA (Munos et al., 2023), a natural baseline that also employs a preference model instead of a reward model, but without any pessimism. Like REINFORCE, the best Nash-EMA values still plateau to a lower performance on summarization. While it achieves similar win-rates over πrefsubscriptref _refπroman_ref on helpfulness, it does so at the cost of much longer and idiosyncratic generations on average (see Figure 4). Shaded areas show 95%percent9595\%95 % CIs of the evaluation. Motivated by these concerns, there is a growing literature on techniques to control this overoptimization behavior in more data-driven ways, such as by incorporating uncertainty in the predictions of the underlying reward model with explicit reward ensembles (Eisenstein et al., 2024; Coste et al., 2023), or pessimistic reasoning (Fisch et al., 2024; Liu et al., 2024; Huang et al., 2024b; Cen et al., 2024), albeit with only modest success in standard settings. Similar to the related area of distributionally robust optimization (Bertsimas & Sim, 2004; Ben-Tal et al., 2013), a core part of the challenge is balancing sufficient reward uncertainty and pessimism to prevent overoptimization, while still being able to learn effectively (Eisenstein et al., 2024; Fisch et al., 2024). In contrast to the reward-based setting, much less work has studied the incorporation of uncertainty when using learned pairwise preference models in subsequent RL—with even fewer works devoted to understanding exactly what kinds of pessimistic possibilities due to uncertainty are useful to entertain, versus ignore. Pessimistic techniques for preference-based RL are particularly challenging, since many leading methods (Munos et al., 2023; Swamy et al., 2024; Calandriello et al., 2024) without pessimistic reasoning, critically leverage the symmetry of the min and max players in preference-based RL to develop efficient algorithms, but the introduction of additional pessimism breaks this symmetry. Additionally, the obvious pessimistic estimators studied theoretically in (Cui & Du, 2022; Ye et al., 2024) exhibit some pathologies when the offline dataset has systematic gaps in its coverage, arising from the non-transitivity of general preferences. These pathologies mean that despite using pessimism, we cannot easily compete with policies whose responses are adequately covered in the data—the typical benchmark for offline RL in reward-based settings. In this work, we build on prior works in preference-based RLHF (Swamy et al., 2024; Munos et al., 2023), as well as offline learning in Markov games (Cui & Du, 2022), to obtain new robust objectives for incorporating a restricted form of uncertainty from finite preference datasets. Specifically, we make the following key contributions: 1. We identify problematic properties of prior pessimistic estimators (Cui & Du, 2022; Ye et al., 2024) in the absence of prohibitive coverage assumptions on the preference data sampling policy. We then develop a new restricted Nash formulation under which the learned policy is provably preferable to any other competitor policy that is restricted to choosing actions within the support of the preference data sampling policy, and show the theoretical benefits of this formulation. 2. We provide a practical algorithm, Pessimistic Preference-based Policy Optimization (P3O) for optimizing the resulting objective. We approximate the ideal theoretical objective with a variational upper bound, that yields a minimax game between a policy and a preference player, which we solve using gradient ascent-descent. The policy optimization is similar to prior works (Swamy et al., 2024; Munos et al., 2023) and the preference updates are adversarial to the current policy’s choices. For Bradley-Terry-Luce models with reward-based preferences, we also evaluate a simpler, reward-based variant of P3O (which we call PRPO) in our experiments. 3. Empirical evaluation on document summarization and training helpful assistants in Figure 1 shows P3O and PRPO reach a higher quality of responses quickly, and the quality does not degrade due to overoptimization from further training. This is contrast with standard RLHF methods (DPO (Rafailov et al., 2023), and REINFORCE (Willams, 1992; Ahmadian et al., 2024), which either exhibit significant overoptimization, or are limited in their ability to sufficiently optimize. The evaluations are performed using a prompted Gemini 1.5 Flash auto-evaluator (Gemini Team, 2024). Detailed analysis in Section 5 further shows that P3O and PRPO avoid qualitative reward hacking behavior of REINFORCE and DPO. 2 Background We consider human alignment of a language model (LM) policy π∈Ππ∈ π ∈ Π, where Π⊆→Δ()Π→Δ \X→ (Y)\Π ⊆ X → Δ ( Y ) 111We use Δ()Δ (Y)Δ ( Y ) to denote the probability simplex defined over the elements of the set YY (e.g., the set of possible LM responses)., which generates for a context x∈x∈Xx ∈ X, a response y∼π(⋅|x)y π(·|x)y ∼ π ( ⋅ | x ) with y∈y∈Yy ∈ Y. We assume that we are given access to a preference dataset, DD, consisting of tuples (x,yw,yl)∈×subscriptsubscript(x,y_w,y_l)∈X×Y×Y( x , yitalic_w , yitalic_l ) ∈ X × Y × Y where for context x, the response ywsubscripty_wyitalic_w is preferred over ylsubscripty_lyitalic_l (as labeled by a human). We further assume access to a reference policy πrefsubscriptref _refπroman_ref, which may or may not match the original sampling policy for the preferred and dispreferred (yw,yl)subscriptsubscript(y_w,y_l)( yitalic_w , yitalic_l ) responses in DD. For brevity, we drop the context x from the notation and work with a finite YY when there is no ambiguity. Preferences are often modeled via a reward function under the Bradley-Terry-Luce (BTL) (Bradley & Terry, 1952; Luce, 2012) model (Christiano et al., 2017; Ouyang et al., 2022); however, in this paper, we make no such assumptions and work with both general pairwise preference functions as well as BTL preference functions based on pointwise rewards. In the following, we first set up the preference learning framework, and then discuss techniques to optimize policies with preference feedback, while also establishing the use of pessimism to handle uncertainties that may exist in the reward and preference functions. Learning preferences from data. We define the preference function p:×→[0,1]:→01p:Y×Y→[0,1]p : Y × Y → [ 0 , 1 ], such that p(y1,y2)≐Pr(y1≻y2)approaches-limitsubscript1subscript2Prsucceedssubscript1subscript2p(y_1,y_2) (y_1 y_2)p ( y1 , y2 ) ≐ Pr ( y1 ≻ y2 ) represents the probability of the generation y1subscript1y_1y1 being preferred over y2subscript2y_2y2 by a target user. Being a probability, the preference function satisfies: p(y1,y2)=1−p(y2,y1)subscript1subscript21subscript2subscript1p(y_1,y_2)=1-p(y_2,y_1)p ( y1 , y2 ) = 1 - p ( y2 , y1 ). To obtain a preference model, we typically fine-tune a pretrained language model (LM) on DD to produce the maximum likelihood estimate pmlesubscriptmlep_mleproman_mle via the following objective: pmle∈argminpℒpref(p;),whereℒpref(p;)=−(yw,yl)∼[logp(yw,yl)].formulae-sequencesubscriptmlesubscriptargminsubscriptℒprefwheresubscriptℒprefsubscriptsimilar-tosubscriptsubscriptdelimited-[]logsubscriptsubscript split &p_mle∈ arg\,min_% pL_pref(p;D),~~ where~~\\ &L_pref(p;D)=-E_(y_w,y_l) % D [ logp (y_w,y_l ) ]. splitstart_ROW start_CELL end_CELL start_CELL proman_mle ∈ start_BIGOP arg min end_BIGOPp Lroman_pref ( p ; D ) , where end_CELL end_ROW start_ROW start_CELL end_CELL start_CELL Lroman_pref ( p ; D ) = - blackboard_E( y start_POSTSUBSCRIPT w , yitalic_l ) ∼ D end_POSTSUBSCRIPT [ log p ( yitalic_w , yitalic_l ) ] . end_CELL end_ROW (1) We overload the notation to say p(π,π′)superscript′p(π,π )p ( π , π′ ), where π,π′∈Πsuperscript′Ππ,π ∈ π , π′ ∈ Π, to represent the expected preference for π over π′π π′, given the preference function p, that is, p(π,π′)=y∼π,y′∼π′[p(y,y′)].superscript′subscriptformulae-sequencesimilar-tosimilar-tosuperscript′delimited-[]superscript′p(π,π )=E_y π,y π [p(y,y^% )].p ( π , π′ ) = blackboard_Ey ∼ π , y′ ∼ π′ [ p ( y , y′ ) ] . In the standard case of RLHF, the preference function is modeled using a reward function r:→ℝ:→ℝr:Y : Y → blackboard_R, assuming an underlying BTL model pBTLsubscriptBTLp_BTLproman_BTL, i.e., pBTL(y1,y2;r)≐exp(r(y1))exp(r(y1))+exp(r(y2)).approaches-limitsubscriptBTLsubscript1subscript2subscript1subscript1subscript2 p_BTL(y_1,y_2;r) (r(y_1)) (% r(y_1))+ (r(y_2)).proman_BTL ( y1 , y2 ; r ) ≐ divide start_ARG exp ( r ( y1 ) ) end_ARG start_ARG exp ( r ( y1 ) ) + exp ( r ( y2 ) ) end_ARG . (2) Note that the set of BTL models is a strict subset of the general pairwise preference models. We also overload the notation pBTL(r)subscriptBTLp_BTL(r)proman_BTL ( r ) to denote the BTL preference model that is induced by reward function r. Preference-based policy optimization. To optimize general preferences without making a BTL modeling assumption, following Munos et al. (2023) and Swamy et al. (2024), we formulate a preference game JP(π,π′,p)subscriptPsuperscript′J_P(π,π ,p)Jroman_P ( π , π′ , p ) between a pair of competing policies π and π′π π′, with preference function p, a reference policy πrefsubscriptref _refπroman_ref, and a regularization parameter β>00β>0β > 0, as JP(π,π′,p)≐p(π,π′)−βKL(π∥πref)+βKL(π′∥πref),approaches-limitsubscriptPsuperscript′KLconditionalsubscriptrefKLconditionalsuperscript′subscriptref J_P(π,π ,p) p(π,π )-% βKL (π\| _ref )+βKL % (π \| _ref ),Jroman_P ( π , π′ , p ) ≐ p ( π , π′ ) - β KL ( π ∥ πroman_ref ) + β KL ( π′ ∥ πroman_ref ) , where KL(π∥πref)≐y∼π[logπ(y)πref(y)]approaches-limitKLconditionalsubscriptrefsubscriptsimilar-todelimited-[]logsubscriptrefKL (π\| _ref ) _y π% [ log π(y) _ref(y) ]KL ( π ∥ πroman_ref ) ≐ blackboard_Ey ∼ π [ log divide start_ARG π ( y ) end_ARG start_ARG πroman_ref ( y ) end_ARG ]. For the preference objective JPsubscriptPJ_PJroman_P, the π and π′π π′ players optimize their corresponding max -min and min -max objectives, i.e., πnash∈argmaxπ∈Πminπ′∈ΠJP(π,π′,pmle),πnash′∈argminπ′∈Πmaxπ∈ΠJP(π,π′,pmle).formulae-sequencesubscriptnashsubscriptargmaxΠsubscriptsuperscript′ΠsubscriptPsuperscript′subscriptmlesuperscriptsubscriptnash′subscriptargminsuperscript′ΠsubscriptΠsubscriptPsuperscript′subscriptmle split& _nash∈ arg\,max_π∈% _π ∈ J_P(π,π ,p_mle)% ,\\ & _nash ∈ arg\,min_π ∈ % _π∈ J_P(π,π ,p_mle). splitstart_ROW start_CELL end_CELL start_CELL πroman_nash ∈ start_BIGOP arg max end_BIGOPπ ∈ Π minitalic_π′ ∈ Π Jroman_P ( π , π′ , proman_mle ) , end_CELL end_ROW start_ROW start_CELL end_CELL start_CELL πroman_nash′ ∈ start_BIGOP arg min end_BIGOPπ′ ∈ Π maxitalic_π ∈ Π Jroman_P ( π , π′ , proman_mle ) . end_CELL end_ROW (3) Here, due to the symmetry of the game, a Nash equilibrium exists at the same policy, i.e., πnash=πnash′subscriptnashsuperscriptsubscriptnash′ _nash= _nash πroman_nash = πroman_nash′, and the objective can be simplified to optimizing a single-player game—termed Self-play Preference Optimization (SPO) in Swamy et al. (2024). Alternatively, in the standard reward-based setup, given a reward function r, the corresponding objective for reward optimization becomes JR(π,r)subscriptRJ_R(π,r)Jroman_R ( π , r ), which is defined as: JR(π,r)≐y∼π[r(y)]−βKL(π∥πref).approaches-limitsubscriptRsubscriptsimilar-todelimited-[]KLconditionalsubscriptref J_R(π,r) _y π[r(y)]-β% KL (π\| _ref ).Jroman_R ( π , r ) ≐ blackboard_Ey ∼ π [ r ( y ) ] - β KL ( π ∥ πroman_ref ) . (4) As described earlier, typical reward-based RLHF settings make use of a learned reward rmlesubscriptmler_mlerroman_mle obtained by optimizing (1) for pBTLsubscriptBTLp_BTLproman_BTL in (2), and fine-tune the policy to maximize the expected pointwise reward, i.e.,222Note that, as shown by Azar et al. (2023), this is also equivalent to optimizing JPsubscriptPJ_PJroman_P with a fixed opponent π′π π′ and a special transformation of the BTL preference function, namely, p(y1,y2)=Ψ(pBTL(y1,y2))subscript1subscript2ΨsubscriptBTLsubscript1subscript2p(y_1,y_2)= (p_BTL(y_1,y_2))p ( y1 , y2 ) = Ψ ( proman_BTL ( y1 , y2 ) ) where Ψ(q)=log(q/(1−q))Ψlog1 (q)= log(q/(1-q))Ψ ( q ) = log ( q / ( 1 - q ) ). πrlhf∈argmaxπ∈ΠJR(π,rmle),subscriptrlhfsubscriptargmaxΠsubscriptRsubscriptmle _rlhf∈ arg\,max_π∈ J_ % R(π,r_mle),πroman_rlhf ∈ start_BIGOP arg max end_BIGOPπ ∈ Π Jroman_R ( π , rroman_mle ) , (5) wherermle∈argminrℒpref(pBTL(r);).wheresubscriptmlesubscriptargminsubscriptℒprefsubscriptBTL ~~ where~~r_mle∈ arg\,min_% rL_pref (p_BTL(r);D ).where rroman_mle ∈ start_BIGOP arg min end_BIGOPr Lroman_pref ( proman_BTL ( r ) ; D ) . Pessimism in preference-based policy optimization. It is a well-understood issue in preference optimization and RLHF that optimizing JP(⋅,⋅,pmle)subscriptP⋅subscriptmleJ_P(·,·,p_mle)Jroman_P ( ⋅ , ⋅ , proman_mle ) and JR(⋅,rmle)subscriptR⋅subscriptmleJ_R(·,r_mle)Jroman_R ( ⋅ , rroman_mle ) can lead to over-optimization of the corresponding preference and reward functions (Gao et al., 2023; Eisenstein et al., 2024). This behavior arises because pmlesubscriptmlep_mleproman_mle (resp. rmlesubscriptmler_mlerroman_mle) has large inaccuracies and/or uncertainties in its predictions outside the support of DD, and the policy optimization to maximize the preference (resp. reward), can exploit the such regions with spuriously high scores under pmlesubscriptmlep_mleproman_mle (resp. rmlesubscriptmler_mlerroman_mle), resulting in a shift in the distribution of outputs generated by the learned policies (πnashsubscriptnash _nashπroman_nash and πrlhfsubscriptrlhf _rlhfπroman_rlhf). Colloquially, this phenomenon is often termed “reward hacking” or “preference hacking”. Pessimism in both the reward setting (Eisenstein et al., 2024; Liu et al., 2024; Fisch et al., 2024; Cen et al., 2024) and the preference setting (Ye et al., 2024) has been proposed as a way to remedy these issues. Pessimism in the reward setting leads to a max -min objective, πp-rlhf∈argmaxπminr∈ℛJR(π,r)subscriptp-rlhfsubscriptargmaxsubscriptℛsubscriptR _p -rlhf∈ arg\,max_π _r% ∈RJ_R(π,r)πroman_p - rlhf ∈ start_BIGOP arg max end_BIGOPπ minitalic_r ∈ R Jroman_R ( π , r ) where ℛRR is an uncertainty set of reward functions, that is, all reward functions that are consistent with the dataset. Liu et al. (2024) and Fisch et al. (2024) show that for certain choices of ℛRR, this game can be solved without actually maintaining the set ℛRR and performing the inner optimization in closed form. In the following section, we extend this to the preference setting (again, which includes reward-based BTL preferences), while also analyzing what forms of pessimism are most appropriate for learning reasonable optimal policies under uncertainty. 3 Pessimistic Preference-based RL We now define a natural extension of the pessimistic reward-based objective to the case of preferences, and study its improvements and generalizations. Implementation issues in developing an efficient algorithm are deferred to §4. A pessimistic Nash solution. In preference-based RL, a pessimistic counterpart of the Nash solution in (3) can be naturally formulated as πp-nashsubscriptp-nash _p -nashπroman_p - nash ∈argmaxπminπ′minp∈JP(π,π′,p)absentsubscriptargmaxsubscriptsuperscript′subscriptsubscriptPsuperscript′ ∈ arg\,max_π _π _p∈% PJ_P(π,π ,p)∈ start_BIGOP arg max end_BIGOPπ minitalic_π′ minitalic_p ∈ P Jroman_P ( π , π′ , p ) (6) where ⊆×→[0,1]→01P \\ Y×Y→[0,1]\P ⊆ Y × Y → [ 0 , 1 ] defines an uncertainty set over preference functions.333The game not being symmetric leads to πp-nash≠πp-nash′subscriptp-nashsuperscriptsubscriptp-nash′ _p -nash≠ _p -% nash πroman_p - nash ≠ πroman_p - nash′. In particular we consider the set (,c)⊂×→[0,1]→01P(D,c)⊂\Y×Y→[0,1]\P ( D , c ) ⊂ Y × Y → [ 0 , 1 ] , for c≥00c≥ 0c ≥ 0, which is defined as: (,c)≐p:ℒpref(p;)≤ℒpref(pmle;)+c,approaches-limitconditional-setsubscriptℒprefsubscriptℒprefsubscriptmle P(D,c) \p:L_% pref(p;D)≤L_pref(p_mle;% D)+c\,P ( D , c ) ≐ p : Lroman_pref ( p ; D ) ≤ Lroman_pref ( proman_mle ; D ) + c , (7) and choose ≐(,c′)approaches-limitsuperscript′P P(D,c )P ≐ P ( D , c′ ) for a value of c′, such that p⋆∈(,c′)superscript⋆superscript′p ∈P(D,c )p⋆ ∈ P ( D , c′ ). This formulation has been studied previously for certain choices of PP in the tabular (Cui & Du, 2022) and function approximation setting (Ye et al., 2024; Huang et al., 2024a). These works prove that the solution πp-nashsubscriptp-nash _p -nashπroman_p - nash converges to the optimal policy if and only if a condition called unilateral coverage holds, which requires that we can compare πp-nashsubscriptp-nash _p -nashπroman_p - nash with any response y, within the coverage of the sampling policy πdatasubscriptdata _dataπroman_data. That is, p(πp-nash,y)subscriptp-nashp( _p -nash,y)p ( πroman_p - nash , y ) lies within a small interval as we vary p∈p∈Pp ∈ P, for all y. This approach has not been empirically evaluated in prior works, as the optimization problem is very challenging with no obvious practical strategies. Before discussing these algorithmic challenges, we will first explore the practical implications of the unilateral coverage requirement. The implications of an unconstrained min player. Consider an illustrative example in Figure 2 which is emblematic of typical RLHF scenarios. There is no context and =y1,y2,y3subscript1subscript2subscript3Y=\y_1,y_2,y_3\Y = y1 , y2 , y3 and suppose further that we have that p(y1,y2)=1subscript1subscript21p(y_1,y_2)=1p ( y1 , y2 ) = 1 for all p∈p∈Pp ∈ P, so we are fully certain about this preference. But we never observe any comparisons involving y3subscript3y_3y3 in our preference data (πdata(y3)=0)subscriptdatasubscript30( _data(y_3)=0)( πroman_data ( y3 ) = 0 ), and hence the set PP allows all values p(y,y3)∈[0,1]subscript301p(y,y_3)∈[0,1]p ( y , y3 ) ∈ [ 0 , 1 ] for y≠y3subscript3y≠ y_3y ≠ y3. To highlight the limitations of pessimism in preference optimization, we consider the problem in absence of regularization, i.e., β=00β=0β = 0. Then, as illustrated in Figure 2 (Left) and proven in Appendix B, the pessimistic Nash policy πp-nashsubscriptp-nash _p -nashπroman_p - nash satisfies πp-nash(y3)=0.5subscriptp-nashsubscript30.5 _p -nash(y_3)=0.5πroman_p - nash ( y3 ) = 0.5. That is, the policy takes an action completely out of the support of the sampled dataset w.p. 0.50.50.50.5, where the ground-truth preferences can take completely arbitrary values. Intuitively, this happens because the pessimistic policy has to account for either possibility that y1subscript1y_1y1 is much better than y3subscript3y_3y3, or vice versa, since the set PP is completely uncertain about the preference between y3subscript3y_3y3 and y1subscript1y_1y1. Indeed, in most practical applications, many possible y’s will not be covered in the dataset, even distributionally, and it appears undesirable that the optimal policy obtained by pessimism will then predominantly generate such outputs. This example highlights a key distinction between pessimism with rewards versus pessimism with preferences. When using pessimism with rewards, outputs which are not covered in the data tend to get a low score, and a reward-maximizing policy naturally avoids such outputs. But in preference-based learning, the min player π′π π′ can choose any output which is not observed in the preference data, and pessimism over preferences means that the max player looses to this action in the worst case—forcing the max player to put probability over such an uncovered output as well. We now propose a remedy for this issue. Figure 2: An illustration of the problematic example for pessimistic preference optimization with unrestricted opponents. We assume that y1,y2subscript1subscript2\y_1,y_2\ y1 , y2 are well-sampled in the preference data, whereas y3subscript3y_3y3 is not—resulting in certain preferences for y1subscript1y_1y1 vs. y2subscript2y_2y2, but completely uncertain preferences for y3subscript3y_3y3 vs. others (Left). The 3×3333× 33 × 3 matrices above are then the optimized pessimistic preference matrices for y1,y2,y3subscript1subscript2subscript3\y_1,y_2,y_3\ y1 , y2 , y3 , with the 3d33d3 d vectors the optimized competing policies. Specifically, shaded entries represent optimizable variables, and the values in each of the blue and red shaded entries are the solutions for the max (argmaxπsubscriptargmax arg\,max_πstart_BIGOP arg max end_BIGOPπ) and min player (argminπ′subscriptargminsuperscript′ arg\,min_π start_BIGOP arg min end_BIGOPπ′ and argminp∈subscriptargmin arg\,min_p∈Pstart_BIGOP arg min end_BIGOPp ∈ P), resp., see (6). Middle: when the opponent π′π π′ is unrestricted, the optimal policy πp-nashsubscriptp-nash _p -nashπroman_p - nash must hedge and put significant support on y3subscript3y_3y3. Right: restricting the support of π′π π′ to the support of the preference-data (i.e., y1,y2subscript1subscript2\y_1,y_2\ y1 , y2 ), avoids this issue, and yields a more reasonable optimal policy πrp-nashsubscriptrp-nash _rp -nashπroman_rp - nash. Restricted pessimistic Nash with a constrained min player. Given the example from Figure 2, an intuitive response is to consider a Nash strategy where the support for the opponent policy π′π π′ is restricted to actions which are well-sampled in the preference dataset. For tabular scenarios, where we have no contexts and a finite YY, with all possible policies in the class Π Π, such a restriction can be carried out by explicitly constraining the support of the min-player in the pessimistic objective (6). However, this does not generalize to more practical scenarios with parametric policies over a large output space. Instead, for such situations, we define a subset Π(πdata,C)⊆ΠsubscriptdataΠ ( _data,C) Π ( πroman_data , C ) ⊆ Π to be a set of policies whose outputs are “well-covered” by the sampling policy πdatasubscriptdata _dataπroman_data, with C denoting a coverage parameter that we define below. Definition 3.1 (Covered policy set). For a given sampling policy πdata∈ΠsubscriptdataΠ _data∈ πroman_data ∈ Π and constant C, the covered policy set Π(πdata,C)Πsubscriptdata ( _data,C)Π ( πroman_data , C ) with respect to πdatasubscriptdata _dataπroman_data is the set of policies such that ∀π,π′∈Π(πdata,C)for-allsuperscript′Πsubscriptdata∀π,π ∈ ( _data,C)∀ π , π′ ∈ Π ( πroman_data , C ) and ∀p1,p2∈for-allsubscript1subscript2∀ p_1,p_2∈P∀ p1 , p2 ∈ P, y∼π,y′∼π′(p1(y,y′)−p2(y,y′))2≤C⋅y,y′∼i.i.d.πdata(p1(y,y′)−p2(y,y′))2. split&E_y π,y π (p_% 1(y,y )-p_2(y,y ))^2\\ & 14.22636pt≤ C·E_y,y % i.i.d. _data(p_1(y,y )-p_2(y,y^% ))^2. splitstart_ROW start_CELL end_CELL start_CELL blackboard_Ey ∼ π , y′ ∼ π′ ( p1 ( y , y′ ) - p2 ( y , y′ ) )2 end_CELL end_ROW start_ROW start_CELL end_CELL start_CELL ≤ C ⋅ blackboard_Ey , y′ start_RELOP SUPERSCRIPTOP start_ARG ∼ end_ARG start_ARG i . i . d . end_ARG end_RELOP π start_POSTSUBSCRIPT data end_POSTSUBSCRIPT ( p1 ( y , y′ ) - p2 ( y , y′ ) )2 . end_CELL end_ROW (8) What covered policies are included in Π(πdata,C)Πsubscriptdata ( _data,C)Π ( πroman_data , C )? Clearly, πdata∈Π(πdata,C)subscriptdataΠsubscriptdata _data∈ ( _data,C)πroman_data ∈ Π ( πroman_data , C ) when C≥11C≥ 1C ≥ 1. We assume C≥11C≥ 1C ≥ 1 in the sequel to ensure that this containment happens. We can also show (see Appendix C.2 for a short derivation) that Π(πdata,C)Πsubscriptdata ( _data,C)Π ( πroman_data , C ) includes the set of all policies with likelihood ratios with respect to πdatasubscriptdata _dataπroman_data that are uniformly bounded by C Csquare-root start_ARG C end_ARG, that is, ΠC⊆Π(πdata,C)subscriptΠsubscriptdata _ C ( _data,C)Πsquare-root start_ARG C end_ARG ⊆ Π ( πroman_data , C ), where ΠC=π∈Π:‖π/πdata‖∞≤CsubscriptΠconditional-setΠsubscriptnormsubscriptdata _ C=\π∈ ~:~\|π/ _data\|_∞≤ % C\Πsquare-root start_ARG C end_ARG = π ∈ Π : ∥ π / πroman_data ∥∞ ≤ square-root start_ARG C end_ARG . In fact, we further show in Appendix C.3 that when the preference functions are linear in a shared feature map, then this coverage condition is ensured whenever the cross-covariance matrices of π and π′π π′ are sufficiently aligned with the covariance matrix of πdatasubscriptdata _dataπroman_data. Perhaps most importantly, however, when defining the following restricted pessimistic Nash solution, πrp-nashsubscriptrp-nash _rp -nashπroman_rp - nash, using this notion of coverage, i.e., πrp-nash∈argmaxπ∈Πminπ′∈Π(πdata,C)minp∈JP(π,π′,p),subscriptrp-nashsubscriptargmaxΠsubscriptsuperscript′ΠsubscriptdatasubscriptsubscriptPsuperscript′ _rp -nash∈ arg\,max_π∈ % _π ∈ ( _data,C) _p∈PJ_% P(π,π ,p),πroman_rp - nash ∈ start_BIGOP arg max end_BIGOPπ ∈ Π minitalic_π′ ∈ Π ( π start_POSTSUBSCRIPT data , C ) end_POSTSUBSCRIPT minitalic_p ∈ P Jroman_P ( π , π′ , p ) , (9) we can give the following performance guarantee for πrp-nashsubscriptrp-nash _rp -nashπroman_rp - nash under the ground-truth preference function p⋆superscript⋆p p⋆ which generated the preference dataset DD (which was then used to derive the set of plausible preference functions PP). Lemma 3.2 (Preference guarantee for the restricted pessimistic Nash policy). We denote the restricted pessimistic Nash policy by πrp-nashsubscriptrp-nash _rp -nashπroman_rp - nash from (9), and let p⋆superscript⋆p p⋆ be the ground-truth preference function underlying DD. Then we have that for any π∈Π(πdata,C)Πsubscriptdataπ∈ ( _data,C)π ∈ Π ( πroman_data , C ) with C≥11C≥ 1C ≥ 1: p⋆(πrp-nash,π)≥12−2Cε,superscript⋆subscriptrp-nash122p ( _rp -nash,π)≥ 12-2% C ,p⋆ ( πroman_rp - nash , π ) ≥ divide start_ARG 1 end_ARG start_ARG 2 end_ARG - 2 square-root start_ARG C ε end_ARG , where ε ε is a bound on how much preference functions in PP can disagree in total variation under πdatasubscriptdata _dataπroman_data: (y,y′)∼i.i.d.πdata|p1(y,y′)−p2(y,y′)|≤εE_(y,y ) .i.d. _% data|p_1(y,y )-p_2(y,y )|≤ _E( y , y′ ) start_RELOP SUPERSCRIPTOP start_ARG ∼ end_ARG start_ARG i . i . d . end_ARG end_RELOP π start_POSTSUBSCRIPT data end_POSTSUBSCRIPT | p1 ( y , y′ ) - p2 ( y , y′ ) | ≤ ε, ∀p1,p2∈for-allsubscript1subscript2∀ p_1,p_2∈P∀ p1 , p2 ∈ P. Appendix C.1 restates Definition 3.1 and Lemma 3.2 with the context x included, along with a proof. This result shows that the restricted pessimistic Nash policy is always preferred to all other covered policies in Π(πdata,C)Πsubscriptdata ( _data,C)Π ( πroman_data , C ) up to an error term. In comparison, the unrestricted pessimistic Nash solution from (6) does not satisfy this guarantee in general. To see this, consider the example in Figure 2, where the unrestricted pessimistic Nash policy πp-nash=[1/2,0,1/2]subscriptp-nash12012 _p -nash=[1/2,0,1/2]πroman_p - nash = [ 1 / 2 , 0 , 1 / 2 ] is dispreferred to the covered policy π1=[1,0,0]subscript1100 _1=[1,0,0]π1 = [ 1 , 0 , 0 ] with probability p⋆(π1,πp-nash)=1/2+γ/2superscript⋆subscript1subscriptp-nash122p ( _1, _p -nash)=1/2+γ/2p⋆ ( π1 , πroman_p - nash ) = 1 / 2 + γ / 2 when p⋆(y1,y3)=γsuperscript⋆subscript1subscript3p (y_1,y_3)= ⋆ ( y1 , y3 ) = γ. As this example suggests, the unrestricted pessimistic solution can be arbitrarily dispreferred, even to covered policies. Existing analyses by Cui & Du (2022) for the unrestricted version indeed do not apply to this example since πdata(y3)=0subscriptdatasubscript30 _data(y_3)=0πroman_data ( y3 ) = 0 and, thus, the necessary unilateral coverage condition is violated (Cui & Du, 2022). In fact, no policy can satisfy unilateral coverage in this case, and we get a vacuous guarantee out of their analysis. The same is true for the relaxed coverage condition of Zhang et al. (2023, Appendix B), as it still applies to the unrestricted pessimistic solution. In some sense, the contrast between our result from Lemma 3.2 and those of Cui & Du (2022) is analogous to the classical analysis of offline RL methods (see, e.g., Chen & Jiang (2019)) and pessimistic offline RL techniques (Xie et al., 2021). Without pessimism in offline RL, we end up with vacuous guarantees, while the pessimistic results allow a non-trivial sub-optimality bound against any policy well covered by the data collection policy. Similarly, the results of Cui & Du (2022) offer a strong guarantee when the data collection policy is sufficiently exploratory, but are rendered vacuous without this. In contrast, our analysis of the restricted Nash estimator offers an opportunistic guarantee, where we are able to adaptively compete with all policies which are well covered by πdatasubscriptdata _dataπroman_data. We note that these considerations are particularly pertinent when aligning LLMs with small preference datasets, where the output space is of long sequences over a large vocabulary, of which the alignment data typically only covers a tiny sliver, leaving no hope for unilateral coverage style assumptions to be satisfied. 4 P3O: An Efficient Implementation Hyperparameters: Mixing coefficient α, regularization coefficient β, preference regularization coefficient λ, EMA parameter γ, learning rates (ηp,ηπ)subscriptsubscript( _p, _π)( ηitalic_p , ηitalic_π ) Initialize: π¯1=π1=πrefsubscript¯1subscript1subscriptref π_1= _1= _refover¯ start_ARG π end_ARG1 = π1 = πroman_ref, p1=pmlesubscript1subscriptmlep_1=p_mlep1 = proman_mle for t=1,2,…12…t=1,2,…t = 1 , 2 , … do Set πmixα∝π¯t1−απdataαproportional-tosuperscriptsubscriptmixsubscriptsuperscript¯1superscriptsubscriptdata _mix^α π^1-α_t _data^% απroman_mixitalic_α ∝ over¯ start_ARG π end_ARG1 - αitalic_t πroman_dataitalic_α as mix of πrefsubscriptref _refπroman_ref and the exponential moving average π¯tsubscript¯ π_tover¯ start_ARG π end_ARGt for restricted Nash. Approximate the current objective (14): JP3O(α)(πt,pt)≐pt(πt,πmixα)approaches-limitsubscriptP3Osubscriptsubscriptsubscriptsubscriptsuperscriptsubscriptmix J_P3O(α)( _t,p_t) p_t( _t, _% mix^α)JP3O ( α ) ( πitalic_t , pitalic_t ) ≐ pitalic_t ( πitalic_t , πroman_mixitalic_α ) −βKL(πt∥πref)KLconditionalsubscriptsubscriptref -βKL ( _t\| _ref )- β KL ( πitalic_t ∥ πroman_ref ) −λKLπref(pmle∥p)subscriptKLsubscriptrefconditionalsubscriptmle - _ _ref (p_mle\|p )- λ KLitalic_π start_POSTSUBSCRIPT ref end_POSTSUBSCRIPT ( proman_mle ∥ p ) Update πt+1←πt+ηπ∂JP3O(α)(π,pt)∂π|π=πt←subscript1subscriptevaluated-atsubscriptsubscriptP3Osubscriptsubscript _t+1← _t+ _π ∂ J_P3O(α)(% π,p_t)∂π |_π= _tπitalic_t + 1 ← πitalic_t + ηitalic_π divide start_ARG ∂ JP3O ( α ) ( π , pitalic_t ) end_ARG start_ARG ∂ π end_ARG |π = π start_POSTSUBSCRIPT t end_POSTSUBSCRIPT Update pt+1←pt−ηp∂JP3O(α)(πt,p)∂p|p=pt←subscript1subscriptevaluated-atsubscriptsubscriptP3Osubscriptsubscriptp_t+1← p_t- _p ∂ J_P3O(α)( _t% ,p)∂ p |_p=p_tpitalic_t + 1 ← pitalic_t - ηitalic_p divide start_ARG ∂ JP3O ( α ) ( πitalic_t , p ) end_ARG start_ARG ∂ p end_ARG |p = p start_POSTSUBSCRIPT t end_POSTSUBSCRIPT Update π¯t+1←γπt+(1−γ)π¯t←subscript¯1subscript1subscript¯ π_t+1←γ _t+(1-γ) π_tover¯ start_ARG π end_ARGt + 1 ← γ πitalic_t + ( 1 - γ ) over¯ start_ARG π end_ARGt end for Algorithm 1 P3O (α) We now develop an efficient algorithm, Pessimistic Preference-based Policy Optimization (P3O), that approximately solves the restricted pessimistic Nash formulation. Approximating the restricted policy set. A first obstacle to an efficient algorithm is that the definition Π(πdata,C)Πsubscriptdata ( _data,C)Π ( πroman_data , C ) is not amenable to easy implementation. However, in KL-regularized preference-based RLHF, there is a natural heuristic to approximate this restriction via an additional KL regularization term. Recall that the central goal of Π(πdata,C)Πsubscriptdata ( _data,C)Π ( πroman_data , C ) is to limit optimization to policies π which generate responses that are in the coverage of the data generating policy πdatasubscriptdata _dataπroman_data. We encourage this through adding an additional penalty based on the KL divergence between π′π π′ and πdatasubscriptdata _dataπroman_data: maxπminp∈minπ′p(π,π′)−βKL(π∥πref)subscriptsubscriptsubscriptsuperscript′KLconditionalsubscriptref _π _p∈P _π \;\;p(π,% π )-βKL (π\| _ref )maxitalic_π minitalic_p ∈ P minitalic_π′ p ( π , π′ ) - β KL ( π ∥ πroman_ref ) (10) +(1−α)βKL(π′∥πref)+αβKL(π′∥πdata).1KLconditionalsuperscript′subscriptrefKLconditionalsuperscript′subscriptdata +(1-α)βKL (π \| _% ref )+αβKL (π \| _% data ).+ ( 1 - α ) β KL ( π′ ∥ πroman_ref ) + α β KL ( π′ ∥ πroman_data ) . Note that π′π π′ is regularized with respect to both the reference policy πrefsubscriptref _refπroman_ref and the sampling policy πdatasubscriptdata _dataπroman_data, where the added parameter α controls the relative strength of the contribution of πrefsubscriptref _refπroman_ref versus πdatasubscriptdata _dataπroman_data. While going from a data-aware constraint in terms of Π(πdata,C)Πsubscriptdata ( _data,C)Π ( πroman_data , C ) to KL regularization is lossy, we note that this is for the min player π′π π′ and only affects the max player π through the data-dependent p term. We use P3O (α) to denote this variant, with P3O being the shorthand for P3O (0). Using a closed-form solution to the inner KL-regularized problem for π′π π′, we next show how to obtain an equivalent, but greatly simplified, objective for π. First, we define the shorthand πmixα(y;π1,π2)superscriptsubscriptmixsubscript1subscript2 _mix^α(y; _1, _2)πroman_mixitalic_α ( y ; π1 , π2 ) as: πmixα(y;π1,π2)∝π11−α(y)π2α(y).proportional-tosuperscriptsubscriptmixsubscript1subscript2superscriptsubscript11superscriptsubscript2 split _mix^α(y; _1, _2)% _1^1-α(y) _2^α(y). splitstart_ROW start_CELL πroman_mixitalic_α ( y ; π1 , π2 ) ∝ π11 - α ( y ) π2italic_α ( y ) . end_CELL end_ROW (11) Optimizing π against an appropriate mixed distribution is then equivalent to solving for (10)italic-(10italic-) eq:restricted-objectiveitalic_( italic_), as we show below. Lemma 4.1. The optimization problem in (10) is equivalent to the following objective, assuming that the minimization over π′π π′ is over all possible policies: maxπminp∈−logy∼πmixα(πref,πdata)[e−p(π,y)β]−βKL(π∥πref).subscriptsubscriptlogsubscriptsimilar-tosuperscriptsubscriptmixsubscriptrefsubscriptdatadelimited-[]superscriptKLconditionalsubscriptref split _π _p∈P&- log% E_y _mix^α( _ref, _% data) [e -p(π,y)β ]\\ & -βKL (π\| _ref ). splitstart_ROW start_CELL maxitalic_π minitalic_p ∈ P end_CELL start_CELL - log blackboard_Ey ∼ π start_POSTSUBSCRIPT mixitalic_α ( πroman_ref , πroman_data ) end_POSTSUBSCRIPT [ edivide start_ARG - p ( π , y ) end_ARG start_ARG β end_ARG ] end_CELL end_ROW start_ROW start_CELL end_CELL start_CELL - β KL ( π ∥ πroman_ref ) . end_CELL end_ROW (12) We provide a proof of this equivalence in Appendix D. We also note that replacing πmixα(πref,πdata)superscriptsubscriptmixsubscriptrefsubscriptdata _mix^α( _ref, _data)πroman_mixitalic_α ( πroman_ref , πroman_data ) with πrefsubscriptref _refπroman_ref (i.e., α=00α=0α = 0) gives an equivalent rewriting for the pessimistic Nash objective with no support restrictions, as given in (6). Approximating the log-partition function. The objective in Lemma 4.1 simplifies the inner minimization to only have one variable, but at the cost of changing the objective to have a more complicated log-partition function term. Consequently, we can no longer get unbiased stochastic gradients of the objective from a mini-batch of data, due to the non-linearity of the logarithm outside of the expectation. To obtain a practical algorithm, we leverage ideas from variational inference (Jordan et al., 1999) to approximate the log-partition function. Doing so, we obtain the following result, which is proved in Appendix E. Lemma 4.2. For any choice of policies π,π¯∈Π¯Ππ, π∈ π , over¯ start_ARG π end_ARG ∈ Π: minp∈−logy∼πmixα(πref,πdata)[e−p(π,y)β]subscriptlogsubscriptsimilar-tosuperscriptsubscriptmixsubscriptrefsubscriptdatadelimited-[]superscript _p∈P- logE_y _% mix^α( _ref, _data) [e % -p(π,y)β ]minitalic_p ∈ P - log blackboard_Ey ∼ π start_POSTSUBSCRIPT mixitalic_α ( πroman_ref , πroman_data ) end_POSTSUBSCRIPT [ edivide start_ARG - p ( π , y ) end_ARG start_ARG β end_ARG ] ≤minp∈y∼πmixα(π¯,πdata)[p(π,y)β]+κ,absentsubscriptsubscriptsimilar-tosuperscriptsubscriptmix¯subscriptdatadelimited-[] 21.33955pt≤ _p∈PE_y π% _mix^α( π, _data) [ p(π,y)% β ]+κ,≤ minitalic_p ∈ P blackboard_Ey ∼ π start_POSTSUBSCRIPT mixitalic_α ( over¯ start_ARG π end_ARG , πroman_data ) end_POSTSUBSCRIPT [ divide start_ARG p ( π , y ) end_ARG start_ARG β end_ARG ] + κ , (13) where κ is independent of π and p. Due to the direction of the inequality, Lemma 4.2 gives only an upper bound for our objective in Lemma 4.1, and therefore maximizing the two is not equivalent. Nevertheless, the approximate objective is tractable, and simply takes the form of optimizing preferences against some comparator π¯ πover¯ start_ARG π end_ARG mixed with the sampling distribution. Furthermore, the approximate objective at any fixed value of p is tight when π¯(y)1−α∝exp(−p(π,y)/β)proportional-to¯superscript1 π(y)^1-α (-p(π,y)/β)over¯ start_ARG π end_ARG ( y )1 - α ∝ exp ( - p ( π , y ) / β ), which resembles the multiplicative weight updates observed in prior self-play algorithms (Swamy et al., 2024; Munos et al., 2023). Since the current preference function iterate ptsubscriptp_tpitalic_t is slowly moving during gradient descent, with this motivation in hand, we choose the competitor policy π¯ πover¯ start_ARG π end_ARG to be an exponentially moving average of past policy iterates in our experiments, giving our algorithm a pessimistic self-play flavor. Approximating the preference uncertainty set. As a final step, we replace the constrained optimization over the preference uncertainty set PP with an unconstrained optimization over all preference functions in some parametric family by adding an additional loss term −ℒpref(p;)subscriptℒpref-L_pref(p;D)~- Lroman_pref ( p ; D )(Liu et al., 2024), corresponding to the Lagrangian form of the constraint defining PP. The objective function then becomes y∼πmixα(π¯,πdata)[p(π,y)]−βKL(π∥πref)−λℒpref(p;),subscriptsimilar-tosuperscriptsubscriptmix¯subscriptdatadelimited-[]KLconditionalsubscriptrefsubscriptℒpref _y _mix^α( π, _% data)[p(π,y)]-βKL (π\| _ref% )-λL_pref(p;D),blackboard_Ey ∼ π start_POSTSUBSCRIPT mixitalic_α ( over¯ start_ARG π end_ARG , πroman_data ) end_POSTSUBSCRIPT [ p ( π , y ) ] - β KL ( π ∥ πroman_ref ) - λ Lroman_pref ( p ; D ) , where we rescaled the objective to absorb the 1/β11/ 1 / β on p into the corresponding hyper-parameters of the KL and likelihood loss components (i.e., β,λβ,λβ , λ). The above objective requires us to also load the preference dataset DD while trying to learn the policy, which can be somewhat inconvenient. To circumvent this issue, however, we can instead simply regularize the preference model to stay close to pmlesubscriptmlep_mleproman_mle, i.e., JP3O(α)(π,p)≐approaches-limitsubscriptP3Oabsent J_P3O(α)(π,p) 3O ( α ) ( π , p ) ≐ p(π,πmixα(π¯,πdata))superscriptsubscriptmix¯subscriptdata ~p (π, _mix^α( π, _ % data) )p ( π , πroman_mixitalic_α ( over¯ start_ARG π end_ARG , πroman_data ) ) (14) −βKL(π∥πref)−λKLπdata(pmle∥p),KLconditionalsubscriptrefsubscriptKLsubscriptdataconditionalsubscriptmle -βKL (π\| _ref )-λ% KL_ _data (p_mle\|p ),- β KL ( π ∥ πroman_ref ) - λ KLitalic_π start_POSTSUBSCRIPT data end_POSTSUBSCRIPT ( proman_mle ∥ p ) , where KLπdata(pmle∥p)subscriptKLsubscriptdataconditionalsubscriptmleKL_ _data (p_mle\|p )KLitalic_π start_POSTSUBSCRIPT data end_POSTSUBSCRIPT ( proman_mle ∥ p ) is defined as y,y′∼i.i.d.πdata[KL(pmle(y,y′)∥p(y,y′))]. _y,y .i.d.% _data\!\! [KL (p_mle(y,y^% )\|p(y,y ) ) ].blackboard_Ey , y′ start_RELOP SUPERSCRIPTOP start_ARG ∼ end_ARG start_ARG i . i . d . end_ARG end_RELOP π start_POSTSUBSCRIPT data end_POSTSUBSCRIPT [ KL ( proman_mle ( y , y′ ) ∥ p ( y , y′ ) ) ] . If the MLE solution pmlesubscriptmlep_mleproman_mle is a good approximation to the true preferences p⋆superscript⋆p p⋆ on y,y′∼i.i.d.πdatay,y .i.d. _datay , y′ start_RELOP SUPERSCRIPTOP start_ARG ∼ end_ARG start_ARG i . i . d . end_ARG end_RELOP πroman_data, then this KL divergence provides a good approximation to the likelihood-based version. The resulting algorithm is shown in Algorithm 1. Having introduced P3O, which handles general preferences, we can extend it to the special case where preferences are parameterized by a reward function under the BTL model. In this setting, we replace the general preference function p with pBTL(r)subscriptBTLp_BTL(r)proman_BTL ( r ) in (14), resulting in the following objective: JPRPO(α)(π,r)≐pBTL(π,πmixα(π¯,πdata);r)approaches-limitsubscriptPRPOsubscriptBTLsuperscriptsubscriptmix¯subscriptdata J_PRPO(α)(π,r) p_BTL (% π, _mix^α( π, _data);r )Jroman_PRPO ( α ) ( π , r ) ≐ proman_BTL ( π , πroman_mixitalic_α ( over¯ start_ARG π end_ARG , πroman_data ) ; r ) (15) −βKL(π∥πref)−λKLπdata(pBTL(rmle)∥pBTL(r)).KLconditionalsubscriptrefsubscriptKLsubscriptdataconditionalsubscriptBTLsubscriptmlesubscriptBTL -βKL (π\| _ref )-% _ _data (p_BTL(r_mle% )\|p_BTL(r) ).- β KL ( π ∥ πroman_ref ) - λ KLitalic_π start_POSTSUBSCRIPT data end_POSTSUBSCRIPT ( proman_BTL ( rroman_mle ) ∥ proman_BTL ( r ) ) . We refer to the algorithm that optimizes (15) as Pessimistic Reward-based Policy Optimization (PRPO). Appendix F provides the pseudo code and further discussion on PRPO. 5 Experimental Results Figure 3: Tabular experiments: Comparison of the different objectives with an explicit search over the main policy, opponent policy, and version space. The X-axis shows the probability assigned to the under-sampled output (y3subscript3y_3y3), from 0.0→0.2→0.00.20.0→ 0.20.0 → 0.2. The Y-axis indicates the minimum preference of the policy found over all covered actions (higher is better). We also show the ground-truth preference matrix p⋆superscript⋆p p⋆ (top-right) and the corresponding πnashsubscriptnash _nashπroman_nash (bottom-right). Results are averaged over 10 random seeds, shaded areas represent ±2×± 2×± 2 × std error. EP3O(0.1) corresponding to the restricted Nash formulation consistently does well, particularly when the sampling rate of y3subscript3y_3y3 is very low (left part of plot). In this section, we illustrate through multiple experiments the effectiveness of the P3O approach. We begin with evaluating the solution πrp-nashsubscriptrp-nash _rp -nashπroman_rp - nash to the exact objective (12) by performing a brute force search over all the variables in a small tabular environment to understand how the objective performs for simple games. Next, we perform experiments on summarization and helpfulness tasks using the approximate objective JP3O(α)subscriptP3OJ_P3O(α)JP3O ( α ) (14) to demonstrate its effectiveness. Figure 4: Qualitative results for the helpfulness and summarization tasks: On helpfulness, both response length and list formats are common reward hacks (Eisenstein et al., 2024). While policies do generate longer responses than the responses in πdatasubscriptdata _dataπroman_data, both REINFORCE and Nash-EMA converge on generations that are ≈40−50%absent40percent50≈ 40-50\%≈ 40 - 50 % longer than those of P3O and PRPO (left). Similarly, REINFORCE degenerates into producing responses that are nearly all formatted as lists (with Nash-EMA at over 50%percent5050\%50 %), while P3O and PRPO stay closer to πdatasubscriptdata _dataπroman_data (middle). On summarization, DPO, REINFORCE, and Nash-EMA all show clear signs of length hacking—also a pervasive issue on this task (Eisenstein et al., 2024; Singhal et al., 2023; Park et al., 2024). In contrast, both PRPO and P3O converge to the average length of preferred responses in πdatasubscriptdata _dataπroman_data, all while also achieving the highest win-rates (right). Figure 5: Confusion matrix showing the pairwise evaluation across the different methods, for each method’s best checkpoint selected in terms of preferences over the reference policy. In head-to-head comparisons, P3O and PRPO outperform baselines, except Nash-EMA on helpfulness, where the Gemini 1.5 Flash evaluation preference aligns with the preference model used for training—favoring the verbose, list-heavy outputs of Nash-EMA (and REINFORCE, to a lesser degree). However, when looking for both helpful and concise responses, P3O and PRPO win rates increase dramatically against Nash-EMA (see Figure 14 in the appendix), demonstrating their robustness. Tabular experiments. We first conduct experiments in a controlled, tabular setting with three possible outputs y∈y1,y2,y3subscript1subscript2subscript3y∈\y_1,y_2,y_3\y ∈ y1 , y2 , y3 , and a ground-truth preference matrix p⋆superscript⋆p p⋆. We vary the probability of sampling y3subscript3y_3y3 from 0.00.00.00.0 to 0.20.20.20.2, distributing the remaining probability equally among (y1,y2)subscript1subscript2(y_1,y_2)( y1 , y2 ). Thus, y3subscript3y_3y3 is consistently the under-sampled output. We conduct experiments with varying Nash strategies, including cases where y3subscript3y_3y3 is preferred, as well as dispreferred over the other two actions. Complete details about the experimental setup and search procedure are provided in Appendix G. Figure 3 illustrates a case in which the under-sampled action is dispreferred (πnash(y3)=0.1subscriptnashsubscript30.1 _nash(y_3)=0.1πroman_nash ( y3 ) = 0.1). We compare Exact P3O (0.1) (EP3O (0.1)), EP3O (0), as defined in Eq. (12), and the non-pessimistic Nash policy obtained from pmlesubscriptmlep_mleproman_mle. Additional scenarios, including ones where the under-sampled action is genuinely preferred, appear in Figure 6 (see Appendix G). Notably, EP3O (0.1) serves as a robust default choice: when the under-sampled action is truly dispreferred (particularly under extremely low sampling), the restricted pessimism in EP3O (0.1) prevents the policy from overcommitting to an insufficiently explored but suboptimal action. Conversely, if the under-sampled action is actually favored under the true πnashsubscriptnash _nashπroman_nash, even a small amount of data may guide non-pessimistic methods to weight that action correctly—potentially yielding strong performance. Since we typically lack ground-truth preferences, EP3O (0.1) offers a “safe-default” strategy. Task 1: Summarization. To demonstrate the effectiveness of our approach in mitigating preference hacking, we compare it against existing preference optimization methods on the popular TL;DR summarization task (Völske et al., 2017; Stiennon et al., 2020). Following prior studies on reward hacking for TL;DR (Eisenstein et al., 2024), we train an MLE preference model pmlesubscriptmlep_mleproman_mle as well as MLE reward model rmlesubscriptmler_mlerroman_mle by fine-tuning a T5 XL (3B) model (Raffel et al., 2020; Roberts et al., 2023) on the preference dataset. The initial policy πrefsubscriptref _refπroman_ref is obtained by supervised fine-tuning a T5 large model (770M) on the human reference summaries in TL;DR. Choosing a larger preference model than the policy is a commonly employed strategy for mitigating hacking (Eisenstein et al., 2024). We initialize the training preference model p1≐pmleapproaches-limitsubscript1subscriptmlep_1 p_mlep1 ≐ proman_mle in case of P3O, and r1≐rmleapproaches-limitsubscript1subscriptmler_1 r_mler1 ≐ rroman_mle in case of PRPO. Task 2: Helpfulness. We also test our method at larger scales on the Anthropic Helpfulness task (Bai et al., 2022) using 8B PaLM-based (Anil et al., 2023) policy, reward, and preference models. The helpfulness tasks consists of dialogues between humans and an automated assistant. The goal is to complete the next turn of the assistant by producing an engaging and helpful response. Like before, we obtain MLE preference and reward models by fine-tuning a pre-trained PaLM model on the preference data. These checkpoints are also used to initialize the preference and reward models for P3O and PRPO. The initial policy πrefsubscriptref _refπroman_ref is obtained from an instruction-tuned PaLM model. We compare P3O and PRPO against existing popular non-pessimistic RLHF methods, DPO (Rafailov et al., 2023) and REINFORCE (Willams, 1992) (which has been shown to outperform PPO (Ahmadian et al., 2024)), as well as the preference-model-based method Nash-EMA (Munos et al., 2023) (which is also non-pessimistic). We also compare to PDPO (Fisch et al., 2024), a pessimistic, offline variant of DPO, in Appendix H. We set α=00α=0α = 0 for our methods, as in these experiments πrefsubscriptref _refπroman_ref and πdatasubscriptdata _dataπroman_data are similar, and hence α does not influence the behavior of P3O. All policies are evaluated using preferences assigned by Gemini 1.5 Flash (Gemini Team, 2024). Results. Figure 1 shows that the non-pessimistic baselines plateau at a lower overall performance level in general, with DPO even degrading substantially with prolonged training. In contrast, both P3O and PRPO achieve a significantly higher preference over πrefsubscriptref _refπroman_ref, surpassing the baselines (a result which is further supported by the confusion matrices in Figure 5). The lone exception is Nash-EMA on helpfulness, which performs very strongly according to our Gemini 1.5 evaluator—a result that happens to be helped by the alignment of the Gemini 1.5 evaluation preference with Nash-EMA’s generally longer responses. Under an evaluation that emphasizes both helpful and concise responses, P3O and PRPO are significantly better (Figure 14). Notably, the preference-based P3O outperforms the reward-based PRPO, indicating the advantage of general preference models. Figure 4 also illustrates how pessimism results in qualitatively different outputs by helping to avoid some of the common “hacking” behaviors related to length and style across different tasks that have been well-reported in the literature (Eisenstein et al., 2024; Singhal et al., 2023; Park et al., 2024). In particular, a typical form of reward hacking in summarization is “length hacking”, where policies inflate their rewards by producing overly verbose responses. We observe that non-pessimistic methods generate summaries significantly longer than those in the original dataset. In contrast, P3O and PRPO converge to summary lengths that are more consistent with πrefsubscriptref _refπroman_ref. Interestingly, while REINFORCE exhibits length hacking whereas P3O and PRPO do not, in Figure 12 (left) we see that REINFORCE actually maintains a lower KL divergence from πrefsubscriptref _refπroman_ref compared to P3O and PRPO. This suggests that P3O and PRPO are learning policies that are substantially distributionally distant from πrefsubscriptref _refπroman_ref, but along quality dimensions that are distinct from simply length (and that result in higher perceived quality in our evaluations in Figure 1). Additionally, for helpfulness, we observe that REINFORCE tends to format nearly all responses as lists, much higher than the prevalence of lists in the preference data (Figure 4, middle), an artifact not shared by the responses of P3O or PRPO. We also observe a significant inflation in response lengths for both REINFORCE and Nash-EMA in helpfulness, converging to nearly 1.5x the response lengths for P3O and PRPO. In Appendix H, we present further results on the training dynamics which indicate a steadily improved objective for all the methods, even when the eval performance is non-monotonic. We also show some cherry-picked responses for both summarization and helpfulness tasks to illustrate the stylistic differences in their responses. We refer the reader to Appendix H for these and further details and results of our empirical evaluation. 6 Conclusion Modern RLHF methods suffer from a significant tendency to overoptimize spurious preferences (or rewards) that are derived from faulty preference (or reward) models. In this work, we introduced pessimistic, preference-based RLHF objectives, which carefully balance uncertainty with effective learning. In particular, we theoretically analyzed the limitations of existing pessimistic estimators, and derive a novel formulation for a restricted, pessimistic Nash solution with provable advantages. Empirical results on multiple tasks and models demonstrate that our approach effectively resists overoptimization while outperforming standard RLHF baselines—highlighting the potential of pessimistic objectives for achieving robust language model alignment. Impact Statement This paper introduces new ideas to the active field of research on large language model post-training, which we hope will help facilitate the successful alignment of models to human preferences, as well as improve our understanding of current approaches—with the goal of ultimately supporting the development of capable and reliable AI systems that are easier, and more stable, to finetune. References Ahmadian et al. (2024) Ahmadian, A., Cremer, C., Gallé, M., Fadaee, M., Kreutzer, J., Pietquin, O., Üstün, A., and Hooker, S. Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs, 2024. Amodei et al. (2016) Amodei, D., Olah, C., Steinhardt, J., Christiano, P., Schulman, J., and Mané, D. Concrete Problems in AI Safety, 2016. Anil et al. (2023) Anil, R., Dai, A. M., Firat, O., Johnson, M., Lepikhin, D., Passos, A., Shakeri, S., Taropa, E., Bailey, P., Chen, Z., et al. Palm 2 technical report. arXiv preprint arXiv:2305.10403, 2023. Antos et al. (2007) Antos, A., Szepesvári, C., and Munos, R. Fitted q-iteration in continuous action-space mdps. Advances in neural information processing systems, 20, 2007. Antos et al. (2008) Antos, A., Szepesvári, C., and Munos, R. Learning near-optimal policies with bellman-residual minimization based fitted policy iteration and a single sample path. Machine Learning, 71:89–129, 2008. Azar et al. (2023) Azar, M. G., Rowland, M., Piot, B., Guo, D., Calandriello, D., Valko, M., and Munos, R. A General Theoretical Paradigm to Understand Learning from Human Preferences, 2023. Bai et al. (2022) Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., DasSarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022. Ben-Tal et al. (2013) Ben-Tal, A., den Hertog, D., De Waegenaere, A., Melenberg, B., and Rennen, G. Robust solutions of optimization problems affected by uncertain probabilities. Manage. Sci., 59(2):341–357, February 2013. ISSN 0025-1909. doi: 10.1287/mnsc.1120.1641. URL https://doi.org/10.1287/mnsc.1120.1641. Bertsimas & Sim (2004) Bertsimas, D. and Sim, M. The price of robustness. Operations research, p. 35–53, 2004. Bradley & Terry (1952) Bradley, R. A. and Terry, M. E. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324–345, 1952. Buckman et al. (2020) Buckman, J., Gelada, C., and Bellemare, M. G. The importance of pessimism in fixed-dataset policy optimization. arXiv preprint arXiv:2009.06799, 2020. Calandriello et al. (2024) Calandriello, D., Guo, D., Munos, R., Rowland, M., Tang, Y., Pires, B. A., Richemond, P. H., Lan, C. L., Valko, M., Liu, T., Joshi, R., Zheng, Z., and Piot, B. Human Alignment of Large Language Models through Online Preference Optimisation, 2024. Cen et al. (2024) Cen, S., Mei, J., Goshvadi, K., Dai, H., Yang, T., Yang, S., Schuurmans, D., Chi, Y., and Dai, B. Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHF, 2024. Chen & Jiang (2019) Chen, J. and Jiang, N. Information-theoretic considerations in batch reinforcement learning. In International Conference on Machine Learning, p. 1042–1051. PMLR, 2019. Cheng et al. (2022) Cheng, C.-A., Xie, T., Jiang, N., and Agarwal, A. Adversarially trained actor critic for offline reinforcement learning. In International Conference on Machine Learning, p. 3852–3878. PMLR, 2022. Christiano et al. (2017) Christiano, P. F., Leike, J., Brown, T., Martic, M., Legg, S., and Amodei, D. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017. Coste et al. (2023) Coste, T., Anwar, U., Kirk, R., and Krueger, D. Reward model ensembles help mitigate overoptimization. arXiv preprint arXiv:2310.02743, 2023. Cui & Du (2022) Cui, Q. and Du, S. S. When is Offline Two-Player Zero-Sum Markov Game Solvable?, 2022. Eisenstein et al. (2024) Eisenstein, J., Nagpal, C., Agarwal, A., Beirami, A., D’Amour, A., Dvijotham, D. J., Fisch, A., Heller, K., Pfohl, S., Ramachandran, D., Shaw, P., and Berant, J. Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hacking, 2024. Farahmand et al. (2010) Farahmand, A.-m., Szepesvári, C., and Munos, R. Error propagation for approximate policy and value iteration. Advances in neural information processing systems, 23, 2010. Fisch et al. (2024) Fisch, A., Eisenstein, J., Zayats, V., Agarwal, A., Beirami, A., Nagpal, C., Shaw, P., and Berant, J. Robust preference optimization through reward model distillation. arXiv preprint arXiv:2405.19316, 2024. Fujimoto et al. (2019) Fujimoto, S., Meger, D., and Precup, D. Off-policy deep reinforcement learning without exploration. In International conference on machine learning, p. 2052–2062. PMLR, 2019. Gao et al. (2023) Gao, L., Schulman, J., and Hilton, J. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, p. 10835–10866. PMLR, 2023. Gemini Team (2024) Gemini Team. Gemini: A family of highly capable multimodal models, 2024. URL https://arxiv.org/abs/2312.11805. Guo et al. (2024) Guo, S., Zhang, B., Liu, T., Liu, T., Khalman, M., Llinares, F., Rame, A., Mesnard, T., Zhao, Y., Piot, B., Ferret, J., and Blondel, M. Direct Language Model Alignment from Online AI Feedback, 2024. Huang et al. (2024a) Huang, A., Zhan, W., Xie, T., Lee, J. D., Sun, W., Krishnamurthy, A., and Foster, D. J. Correcting the mythos of kl-regularization: Direct alignment without overparameterization via chi-squared preference optimization. arXiv preprint arXiv:2407.13399, 2024a. Huang et al. (2024b) Huang, A., Zhan, W., Xie, T., Lee, J. D., Sun, W., Krishnamurthy, A., and Foster, D. J. Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimization. https://arxiv.org/abs/2407.13399v2, 2024b. Jaques et al. (2019) Jaques, N., Ghandeharioun, A., Shen, J. H., Ferguson, C., Lapedriza, A., Jones, N., Gu, S., and Picard, R. Way off-policy batch deep reinforcement learning of implicit human preferences in dialog. arXiv preprint arXiv:1907.00456, 2019. Jin et al. (2021) Jin, Y., Yang, Z., and Wang, Z. Is pessimism provably efficient for offline rl? In International Conference on Machine Learning, p. 5084–5096. PMLR, 2021. Jordan et al. (1999) Jordan, M. I., Ghahramani, Z., Jaakkola, T. S., and Saul, L. K. An introduction to variational methods for graphical models. Machine learning, 37:183–233, 1999. Kidambi et al. (2020) Kidambi, R., Rajeswaran, A., Netrapalli, P., and Joachims, T. Morel: Model-based offline reinforcement learning. Advances in neural information processing systems, 33:21810–21823, 2020. Koppel et al. (2024) Koppel, A., Bhatt, S., Guo, J., Eappen, J., Wang, M., and Ganesh, S. Information-directed pessimism for offline reinforcement learning. In Forty-first International Conference on Machine Learning, 2024. Kostrikov et al. (2021) Kostrikov, I., Nair, A., and Levine, S. Offline reinforcement learning with implicit q-learning. arXiv preprint arXiv:2110.06169, 2021. Kumar et al. (2019) Kumar, A., Fu, J., Soh, M., Tucker, G., and Levine, S. Stabilizing off-policy q-learning via bootstrapping error reduction. Advances in neural information processing systems, 32, 2019. Kumar et al. (2020) Kumar, A., Zhou, A., Tucker, G., and Levine, S. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33:1179–1191, 2020. Laroche et al. (2019) Laroche, R., Trichelair, P., and Des Combes, R. T. Safe policy improvement with baseline bootstrapping. In International conference on machine learning, p. 3652–3661. PMLR, 2019. Liu et al. (2020) Liu, Y., Swaminathan, A., Agarwal, A., and Brunskill, E. Provably good batch off-policy reinforcement learning without great exploration. Advances in neural information processing systems, 33:1264–1274, 2020. Liu et al. (2024) Liu, Z., Lu, M., Zhang, S., Liu, B., Guo, H., Yang, Y., Blanchet, J., and Wang, Z. Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizer, 2024. Luce (2012) Luce, R. Individual Choice Behavior: A Theoretical Analysis. Dover Books on Mathematics. Dover Publications, 2012. ISBN 9780486153391. URL https://books.google.com/books?id=ERQsKkPiKkkC. Munos (2003) Munos, R. Error bounds for approximate policy iteration. In ICML, volume 3, p. 560–567. Citeseer, 2003. Munos & Szepesvári (2008) Munos, R. and Szepesvári, C. Finite-time bounds for fitted value iteration. Journal of Machine Learning Research, 9(5), 2008. Munos et al. (2023) Munos, R., Valko, M., Calandriello, D., Azar, M. G., Rowland, M., Guo, Z. D., Tang, Y., Geist, M., Mesnard, T., Michi, A., Selvi, M., Girgin, S., Momchev, N., Bachem, O., Mankowitz, D. J., Precup, D., and Piot, B. Nash Learning from Human Feedback, 2023. Ouyang et al. (2022) Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744, 2022. Park et al. (2024) Park, R., Rafailov, R., Ermon, S., and Finn, C. Disentangling Length from Quality in Direct Preference Optimization, 2024. Rafailov et al. (2023) Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D., and Finn, C. Direct Preference Optimization: Your Language Model is Secretly a Reward Model, 2023. Raffel et al. (2020) Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1–67, 2020. Roberts et al. (2023) Roberts, A., Chung, H. W., Mishra, G., Levskaya, A., Bradbury, J., Andor, D., Narang, S., Lester, B., Gaffney, C., Mohiuddin, A., et al. Scaling up models and data with t5x and seqio. Journal of Machine Learning Research, 24(377):1–8, 2023. Singhal et al. (2023) Singhal, P., Goyal, T., Xu, J., and Durrett, G. A long way to go: Investigating length correlations in rlhf. arXiv preprint arXiv:2310.03716, 2023. Stiennon et al. (2020) Stiennon, N., Ouyang, L., Wu, J., Ziegler, D., Lowe, R., Voss, C., Radford, A., Amodei, D., and Christiano, P. F. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008–3021, 2020. Swamy et al. (2024) Swamy, G., Dann, C., Kidambi, R., Wu, Z. S., and Agarwal, A. A Minimaximalist Approach to Reinforcement Learning from Human Feedback, 2024. Tang et al. (2024) Tang, Y., Guo, Z. D., Zheng, Z., Calandriello, D., Munos, R., Rowland, M., Richemond, P. H., Valko, M., Pires, B. Á., and Piot, B. Generalized Preference Optimization: A Unified Approach to Offline Alignment, 2024. Völske et al. (2017) Völske, M., Potthast, M., Syed, S., and Stein, B. Tl; dr: Mining reddit to learn automatic summarization. In Proceedings of the Workshop on New Frontiers in Summarization, p. 59–63, 2017. Wang et al. (2020) Wang, R., Foster, D. P., and Kakade, S. M. What are the statistical limits of offline rl with linear function approximation? ArXiv, abs/2010.11895, 2020. URL https://api.semanticscholar.org/CorpusID:225039786. Willams (1992) Willams, R. J. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8:229–256, 1992. Wu et al. (2019) Wu, Y., Tucker, G., and Nachum, O. Behavior regularized offline reinforcement learning. arXiv preprint arXiv:1911.11361, 2019. Xiao et al. (2023) Xiao, C., Wang, H., Pan, Y., White, A., and White, M. The in-sample softmax for offline reinforcement learning. arXiv preprint arXiv:2302.14372, 2023. Xie & Jiang (2020) Xie, T. and Jiang, N. Q* approximation schemes for batch reinforcement learning: A theoretical comparison. In Conference on Uncertainty in Artificial Intelligence, p. 550–559. PMLR, 2020. Xie et al. (2021) Xie, T., Cheng, C.-A., Jiang, N., Mineiro, P., and Agarwal, A. Bellman-consistent pessimism for offline reinforcement learning. Advances in neural information processing systems, 34:6683–6694, 2021. Ye et al. (2024) Ye, C., Xiong, W., Zhang, Y., Jiang, N., and Zhang, T. Online Iterative Reinforcement Learning from Human Feedback with General Preference Model, 2024. Yu et al. (2020) Yu, T., Thomas, G., Yu, L., Ermon, S., Zou, J. Y., Levine, S., Finn, C., and Ma, T. Mopo: Model-based offline policy optimization. Advances in Neural Information Processing Systems, 33:14129–14142, 2020. Zanette et al. (2021) Zanette, A., Wainwright, M. J., and Brunskill, E. Provable benefits of actor-critic methods for offline reinforcement learning. Advances in neural information processing systems, 34:13626–13640, 2021. Zhang et al. (2024) Zhang, D., Lyu, B., Qiu, S., Kolar, M., and Zhang, T. Pessimism meets risk: risk-sensitive offline reinforcement learning. arXiv preprint arXiv:2407.07631, 2024. Zhang et al. (2023) Zhang, Y., Bai, Y., and Jiang, N. Offline learning in markov games with general function approximation. In International Conference on Machine Learning, p. 40804–40829. PMLR, 2023. Appendix A Extended Literature Review Offline RL is primarily concerned with learning a policy from a fixed dataset, a problem that has attracted considerable attention. Many works focus on scenarios with sufficiently broad dataset coverage (Antos et al., 2007, 2008; Munos, 2003; Munos & Szepesvári, 2008; Farahmand et al., 2010; Chen & Jiang, 2019; Xie & Jiang, 2020), though such assumptions tend to be overly restrictive and seldom hold in real-world situations. Consequently, recent research has shifted toward the more realistic setting of inadequate coverage (Wang et al., 2020), aiming to learn a “best effort” policy (Liu et al., 2020). Two major strategies have emerged to handle poor coverage: behavior policy regularization (Fujimoto et al., 2019; Laroche et al., 2019; Kumar et al., 2019; Wu et al., 2019; Jaques et al., 2019; Kostrikov et al., 2021; Xiao et al., 2023) and pessimism in the face of uncertainty (Kumar et al., 2020; Liu et al., 2020; Kidambi et al., 2020; Yu et al., 2020; Buckman et al., 2020; Jin et al., 2021; Zanette et al., 2021; Xie et al., 2021; Cheng et al., 2022; Zhang et al., 2024; Koppel et al., 2024). In limited-data regimes, pessimism has been shown to provide strong theoretical guarantees for the resulting policy (Buckman et al., 2020; Jin et al., 2021), achieving min-max optimality in linear MDPs. Moreover, it has been successfully incorporated into both linear (Zanette et al., 2021) and deep RL (DRL) settings (Xie et al., 2021; Cheng et al., 2022). Behavior policy regularization has also been explored in language models (Jaques et al., 2019), alongside standard RLHF approaches that commonly regularize to a reference policy (Stiennon et al., 2020). When the reward function is learned from limited data, inaccuracies naturally arise, mirroring the challenge in value-based offline RL where value estimates become unreliable in underrepresented state-action regions. Pessimism thus serves as a compelling remedy and has recently been investigated in the standard reward-based setting. Concurrently, several works (Fisch et al., 2024; Liu et al., 2024; Cen et al., 2024) have proposed offline methods that learn policies against an adversarial reward function, leveraging the DPO simplification (Rafailov et al., 2023) to avoid full-fledged adversarial training; among these, Cen et al. (2024) also introduces an online variant. Eisenstein et al. (2024) explores reward uncertainty through an ensemble of reward models, showing that ensemble aggregation helps mitigate reward hacking, though it does not fully resolve over-optimization risks. On the preference-learning front, recent work has relaxed the BTL assumption, either by bypassing the need for a separate preference model (Azar et al., 2023; Tang et al., 2024) or adopting self-play approaches (Munos et al., 2023; Swamy et al., 2024; Calandriello et al., 2024; Ye et al., 2024). Ye et al. (2024) introduces a pessimistic preference objective with unrestricted Nash, and empirically evaluates an uncertainty based exploration method for the preference optimization. However, the role of pessimism in the more general preference-learning setting remains largely unexplored. Appendix B Optimal Solution of the Example in Figure 2 Let πi=π(yi)subscriptsubscript _i=π(y_i)πitalic_i = π ( yitalic_i ) and p23=p(y2,y3)subscript23subscript2subscript3p_23=p(y_2,y_3)p23 = p ( y2 , y3 ) and p13=p(y1,y3)subscript13subscript1subscript3p_13=p(y_1,y_3)p13 = p ( y1 , y3 ). Then we can write the objective in this example as V⋆=maxπminπ′minp∈J(π,π′,p)superscript⋆subscriptsubscriptsuperscript′subscriptsuperscript′ V = _π _π _p∈PJ% (π,π ,p)V⋆ = maxitalic_π minitalic_π′ minitalic_p ∈ P J ( π , π′ , p ) where J(π,π′,p)=p(π,π′)=( where J(π,π ,p)=p(π,π )= (where J ( π , π′ , p ) = p ( π , π′ ) = ( 0.5π1π1′+1⋅π1π2′+p13π1π3′0.5subscript1subscriptsuperscript′1⋅1subscript1subscriptsuperscript′2subscript13subscript1subscriptsuperscript′3 \,\,0.5 _1π _1+1· _1π _2+p_1% 3 _1π _30.5 π1 π′1 + 1 ⋅ π1 π′2 + p13 π1 π′3 +0⋅π2π1′+0.5π2π2′+p23π2π3′⋅0subscript2subscriptsuperscript′10.5subscript2subscriptsuperscript′2subscript23subscript2subscriptsuperscript′3 +0· _2π _1+0.5 _2π _2+p_23% _2π _3+ 0 ⋅ π2 π′1 + 0.5 π2 π′2 + p23 π2 π′3 +(1−p13)π3π1′+(1−p23)π3π2′+0.5π3π3′). +(1-p_13) _3π _1+(1-p_23) _3π _% 2+0.5 _3π _3 ).+ ( 1 - p13 ) π3 π′1 + ( 1 - p23 ) π3 π′2 + 0.5 π3 π′3 ) . Upper-bound on V⋆superscript⋆V V⋆: First note that for any π: minπ′minp∈J(π,π′,p)≤0.5π3andminπ′minp∈J(π,π′,p)≤0.5π1.formulae-sequencesubscriptsuperscript′subscriptsuperscript′0.5subscript3andsubscriptsuperscript′subscriptsuperscript′0.5subscript1 _π _p∈PJ(π,π ,p)% ≤ 0.5 _3 and _π _p∈% PJ(π,π ,p)≤ 0.5 _1.minitalic_π′ minitalic_p ∈ P J ( π , π′ , p ) ≤ 0.5 π3 and minitalic_π′ minitalic_p ∈ P J ( π , π′ , p ) ≤ 0.5 π1 . The first inequality follows by considering the choice p13=p23=1subscript13subscript231p_13=p_23=1p13 = p23 = 1, and the second inequality from considering the choice π1′=1subscriptsuperscript′11π _1=1π′1 = 1 and p13=1subscript131p_13=1p13 = 1. Since both bounds hold simultaneously and π1+π3≤1subscript1subscript31 _1+ _3≤ 1π1 + π3 ≤ 1, we can conclude that V⋆≤0.25.superscript⋆0.25 V ≤ 0.25.V⋆ ≤ 0.25 . Lower-bound on V⋆superscript⋆V V⋆: Choosing π1=π3=0.5subscript1subscript30.5 _1= _3=0.5π1 = π3 = 0.5, we see that the objective value can be written as J(π,π′,p)=superscript′absent J(π,π ,p)=J ( π , π′ , p ) = (0.25π1′+0.5π2′+0.5p13π3′ (0.25π _1+0.5π _2+0.5p_13π^% _3( 0.25 π′1 + 0.5 π′2 + 0.5 p13 π′3 +0.5(1−p13)π1′+0.5(1−p23)π2′+0.25π3′). +0.5(1-p_13)π _1+0.5(1-p_23)π _2+0.25% π _3 ).+ 0.5 ( 1 - p13 ) π′1 + 0.5 ( 1 - p23 ) π′2 + 0.25 π′3 ) . First we observe that the minimum of this quantity is always attained at p23=1subscript231p_23=1p23 = 1 and thus we can ignore the penultimate term. Consider now two cases: • Case π1′≤π3′subscriptsuperscript′1subscriptsuperscript′3π _1≤π _3π′1 ≤ π′3: Then the coefficient of p13subscript13p_13p13 is non-negative and the minimum is attained at p13=0subscript130p_13=0p13 = 0. This allows us to simplify the expression further as minπ′minp∈J(π,π′,p)subscriptsuperscript′subscriptsuperscript′ _π _p∈PJ(π,π ,p)minitalic_π′ minitalic_p ∈ P J ( π , π′ , p ) =minπ′0.25π1′+0.5π2′+0.5π1′+0.5π2′+0.25π3′absentsubscriptsuperscript′0.25subscriptsuperscript′10.5subscriptsuperscript′20.5subscriptsuperscript′10.5subscriptsuperscript′20.25subscriptsuperscript′3 = _π 0.25π _1+0.5π _2+0.5% π _1+0.5π _2+0.25π _3= minitalic_π′ 0.25 π′1 + 0.5 π′2 + 0.5 π′1 + 0.5 π′2 + 0.25 π′3 =minπ′0.75π1′+π2′+0.25π3′absentsubscriptsuperscript′0.75subscriptsuperscript′1subscriptsuperscript′20.25subscriptsuperscript′3 = _π 0.75π _1+π _2+0.25π% _3= minitalic_π′ 0.75 π′1 + π′2 + 0.25 π′3 =0.25absent0.25 =0.25= 0.25 where we choose π3′=1subscriptsuperscript′31π _3=1π′3 = 1 in the last step. • Case π1′≥π3′subscriptsuperscript′1subscriptsuperscript′3π _1≥π _3π′1 ≥ π′3: Then the coefficient of p13subscript13p_13p13 is non-positive and the minimum is attained at p13=1subscript131p_13=1p13 = 1. This gives minπ′minp∈J(π,π′,p)subscriptsuperscript′subscriptsuperscript′ _π _p∈PJ(π,π ,p)minitalic_π′ minitalic_p ∈ P J ( π , π′ , p ) =minπ′0.25π1′+0.5π2′+0.5π3′+0.5π2′+0.25π3′absentsubscriptsuperscript′0.25subscriptsuperscript′10.5subscriptsuperscript′20.5subscriptsuperscript′30.5subscriptsuperscript′20.25subscriptsuperscript′3 = _π 0.25π _1+0.5π _2+0.5% π _3+0.5π _2+0.25π _3= minitalic_π′ 0.25 π′1 + 0.5 π′2 + 0.5 π′3 + 0.5 π′2 + 0.25 π′3 =minπ′minP∈0.25π1′+π2′+0.75π3′absentsubscriptsuperscript′subscript0.25subscriptsuperscript′1subscriptsuperscript′20.75subscriptsuperscript′3 = _π _P∈P0.25π _1+% π _2+0.75π _3= minitalic_π′ minitalic_P ∈ P 0.25 π′1 + π′2 + 0.75 π′3 =0.25absent0.25 =0.25= 0.25 where the optimal solution is to choose π1′=1subscriptsuperscript′11π _1=1π′1 = 1. Combining both cases, we can conclude that V⋆≥0.25.superscript⋆0.25 V ≥ 0.25.V⋆ ≥ 0.25 . Optimal solution. Combining both upper- and lower-bounds, we can conclude that V⋆=0.25superscript⋆0.25V =0.25V⋆ = 0.25 which is attained at π1=π3=0.5subscript1subscript30.5 _1= _3=0.5π1 = π3 = 0.5. Appendix C Definition and Analysis of Restricted Nash Policy C.1 Restatement and Proof of Lemma 3.2 We restate Definition 3.1 and Lemma 3.2 over here, but with conversation context (x) included in the equations for clarity, hence for this section we redefine ⊆×→[0,1]→01P \X×Y×Y→% [0,1]\P ⊆ X × Y × Y → [ 0 , 1 ] . We also use x∼similar-tox Dx ∼ D to denote a context x sampled from the offline dataset. Definition C.1 (Covered policy set). For a given sampling policy πdata∈ΠsubscriptdataΠ _data∈ πroman_data ∈ Π and constant C, the covered policy set Π(πdata,C)Πsubscriptdata ( _data,C)Π ( πroman_data , C ) with respect to πdatasubscriptdata _dataπroman_data is the set of policies such that ∀π,π′∈Π(πdata,C)for-allsuperscript′Πsubscriptdata∀π,π ∈ ( _data,C)∀ π , π′ ∈ Π ( πroman_data , C ) and ∀p1,p2∈for-allsubscript1subscript2∀ p_1,p_2∈P∀ p1 , p2 ∈ P, x∼,y∼π(⋅|x),y′∼π′(⋅|x)(p1(x,y,y′)−p2(x,y,y′))2≤C⋅x∼,y,y′∼i.i.d.πdata(⋅|x)(p1(x,y,y′)−p2(x,y,y′))2. split&E_x D,y π(·|x),y^% π (·|x)(p_1(x,y,y )-p_2(x,y,y )% )^2\\ & 14.22636pt≤ C·E_x D,y,y % .i.d. _data(·|x)(% p_1(x,y,y )-p_2(x,y,y ))^2. splitstart_ROW start_CELL end_CELL start_CELL blackboard_Ex ∼ D , y ∼ π ( ⋅ | x ) , y′ ∼ π′ ( ⋅ | x ) ( p1 ( x , y , y′ ) - p2 ( x , y , y′ ) )2 end_CELL end_ROW start_ROW start_CELL end_CELL start_CELL ≤ C ⋅ blackboard_Ex ∼ D , y , y′ start_RELOP SUPERSCRIPTOP start_ARG ∼ end_ARG start_ARG i . i . d . end_ARG end_RELOP π start_POSTSUBSCRIPT data ( ⋅ | x ) end_POSTSUBSCRIPT ( p1 ( x , y , y′ ) - p2 ( x , y , y′ ) )2 . end_CELL end_ROW (16) Lemma C.2 (Preference guarantee for the restricted pessimistic Nash policy). We denote the restricted pessimistic Nash policy by πrp-nashsubscriptrp-nash _rp -nashπroman_rp - nash from Eq. (9), and let p⋆superscript⋆p p⋆ be the ground-truth preference function underlying DD. Then we have that for any π∈Π(πdata,C)Πsubscriptdataπ∈ ( _data,C)π ∈ Π ( πroman_data , C ) with C≥11C≥ 1C ≥ 1: p⋆(πrp-nash,π)≥12−2Cε,superscript⋆subscriptrp-nash122p ( _rp -nash,π)≥ 12-2% C ,p⋆ ( πroman_rp - nash , π ) ≥ divide start_ARG 1 end_ARG start_ARG 2 end_ARG - 2 square-root start_ARG C ε end_ARG , where ε ε is a bound on how much preference functions in PP can disagree in total variation under πdatasubscriptdata _dataπroman_data: x∼,(y,y′)∼i.i.d.πdata(⋅|x)|p1(x,y,y′)−p2(x,y,y′)|≤εE_x D,(y,y ) .% i.d. _data(·|x)|p_1(x,y,y )-p_2(x,y,y% )|≤ _Ex ∼ D , ( y , y′ ) start_RELOP SUPERSCRIPTOP start_ARG ∼ end_ARG start_ARG i . i . d . end_ARG end_RELOP π start_POSTSUBSCRIPT data ( ⋅ | x ) end_POSTSUBSCRIPT | p1 ( x , y , y′ ) - p2 ( x , y , y′ ) | ≤ ε, ∀p1,p2∈for-allsubscript1subscript2∀ p_1,p_2∈P∀ p1 , p2 ∈ P. Proof of Lemma 3.2. Let πr-nash⋆superscriptsubscriptr-nash⋆ _r -nash πroman_r - nash⋆ be a restricted Nash solution under the true preference function p⋆superscript⋆p p⋆: πr-nash⋆superscriptsubscriptr-nash⋆ _r -nash πroman_r - nash⋆ =argmaxπ∈Π(πdata,C)minπ′∈Π(πdata,C)x∼,y∼π(⋅|x),y′∼π′(⋅|x)[p⋆(x,y,y′)] = arg\,max_π∈ ( _data,C) _% π ∈ ( _data,C)E_x D,y % π(·|x),y π (·|x)\!\! [p (x,y,y^% ) ]= start_BIGOP arg max end_BIGOPπ ∈ Π ( π start_POSTSUBSCRIPT data , C ) end_POSTSUBSCRIPT minitalic_π′ ∈ Π ( π start_POSTSUBSCRIPT data , C ) end_POSTSUBSCRIPT blackboard_Ex ∼ D , y ∼ π ( ⋅ | x ) , y′ ∼ π′ ( ⋅ | x ) [ p⋆ ( x , y , y′ ) ] =argmaxπ∈Π(πdata,C)minπ′∈Π(πdata,C)p⋆(π,π′).absentsubscriptargmaxΠsubscriptdatasubscriptsuperscript′Πsubscriptdatasuperscript⋆superscript′ = arg\,max_π∈ ( _data,C) _% π ∈ ( _data,C)p (π,π ).= start_BIGOP arg max end_BIGOPπ ∈ Π ( π start_POSTSUBSCRIPT data , C ) end_POSTSUBSCRIPT minitalic_π′ ∈ Π ( π start_POSTSUBSCRIPT data , C ) end_POSTSUBSCRIPT p⋆ ( π , π′ ) . We start by noting that πr-nash⋆superscriptsubscriptr-nash⋆ _r -nash πroman_r - nash⋆ is solving an anti-symmetric two player zero-sum game and the constraint set Π(πdata,C)Πsubscriptdata ( _data,C)Π ( πroman_data , C ) is a convex set whenever Π Π is convex. To see this, consider two policies π,π′∈Π(πdata,C)superscript′Πsubscriptdataπ,π ∈ ( _data,C)π , π′ ∈ Π ( πroman_data , C ) and α∈[0,1]01α∈[0,1]α ∈ [ 0 , 1 ]. Then for any π′∈Π(πdata,C)superscript′Πsubscriptdataπ ∈ ( _data,C)π′ ′ ∈ Π ( πroman_data , C ) and p∈p∈Pp ∈ P, we have x∼,y∼(απ(⋅|x)+(1−α)π′(⋅|x)),y′∼π′(⋅|x)(p1(x,y,y′)−p2(x,y,y′))2 _x D,y (απ(·|x)+(1-α% )π (·|x)),y π (·|x)(p_1(x,y,y% )-p_2(x,y,y ))^2blackboard_Ex ∼ D , y ∼ ( α π ( ⋅ | x ) + ( 1 - α ) π′ ( ⋅ | x ) ) , y′ ∼ π′ ′ ( ⋅ | x ) ( p1 ( x , y , y′ ) - p2 ( x , y , y′ ) )2 =αx∼,y∼π(⋅|x),y′∼π′(⋅|x)(p1(x,y,y′)−p2(x,y,y′))2+(1−α)x∼,y∼π′(⋅|x),y′∼π′(⋅|x)(p1(x,y,y′)−p2(x,y,y′))2 = _x D,y π(·|x),y % π (·|x)(p_1(x,y,y )-p_2(x,y,y ))% ^2+(1-α)E_x D,y π (·|x),y^% π (·|x)(p_1(x,y,y )-p_2(x,y,y^% ))^2= α blackboard_Ex ∼ D , y ∼ π ( ⋅ | x ) , y′ ∼ π′ ′ ( ⋅ | x ) ( p1 ( x , y , y′ ) - p2 ( x , y , y′ ) )2 + ( 1 - α ) blackboard_Ex ∼ D , y ∼ π′ ( ⋅ | x ) , y′ ∼ π′ ′ ( ⋅ | x ) ( p1 ( x , y , y′ ) - p2 ( x , y , y′ ) )2 ≤C⋅x∼,y,y′∼i.i.d.πdata(⋅|x)(p1(x,y,y′)−p2(x,y,y′))2 ≤ C·E_x D,y,y % .i.d. _data(·|x)(p_1(x,y,y% )-p_2(x,y,y ))^2≤ C ⋅ blackboard_Ex ∼ D , y , y′ start_RELOP SUPERSCRIPTOP start_ARG ∼ end_ARG start_ARG i . i . d . end_ARG end_RELOP π start_POSTSUBSCRIPT data ( ⋅ | x ) end_POSTSUBSCRIPT ( p1 ( x , y , y′ ) - p2 ( x , y , y′ ) )2 where the first inequality uses linearity of expectation and the second that π,π′∈Π(πdata,C)superscript′Πsubscriptdataπ,π ∈ ( _data,C)π , π′ ∈ Π ( πroman_data , C ). Thus, απ+(1−α)π′∈Π(πdata,C)1superscript′Πsubscriptdataαπ+(1-α)π ∈ ( _data,C)α π + ( 1 - α ) π′ ∈ Π ( πroman_data , C ) and Π(πdata,C)Πsubscriptdata ( _data,C)Π ( πroman_data , C ) is convex. As a consequence of this, we have that πr-nash⋆∈argminπ∈Π(πdata,C)p⋆(πr-nash⋆,π)superscriptsubscriptr-nash⋆subscriptargminΠsubscriptdatasuperscript⋆superscriptsubscriptr-nash⋆ _r -nash ∈ arg\,min_π% ∈ ( _data,C)p ( _r -% nash ,π)πroman_r - nash⋆ ∈ start_BIGOP arg min end_BIGOPπ ∈ Π ( π start_POSTSUBSCRIPT data , C ) end_POSTSUBSCRIPT p⋆ ( πroman_r - nash⋆ , π ) and p⋆(πr-nash⋆,πr-nash⋆)=0.5superscript⋆superscriptsubscriptr-nash⋆superscriptsubscriptr-nash⋆0.5p ( _r -nash , _r% -nash )=0.5p⋆ ( πroman_r - nash⋆ , πroman_r - nash⋆ ) = 0.5. Let πrp-nash′∈argminπ∈Π(πdata,C)minp∈p(πrp-nash,π)superscriptsubscriptrp-nash′subscriptargminΠsubscriptdatasubscriptsubscriptrp-nash _rp -nash ∈ arg\,min_% π∈ ( _data,C) _p∈Pp( _rp% -nash,π)πroman_rp - nash′ ∈ start_BIGOP arg min end_BIGOPπ ∈ Π ( π start_POSTSUBSCRIPT data , C ) end_POSTSUBSCRIPT minitalic_p ∈ P p ( πroman_rp - nash , π ). Then we have by definition: p⋆(πrp-nash,π)=superscript⋆subscriptrp-nashabsent p ( _rp -nash,π)=p⋆ ( πroman_rp - nash , π ) = p⋆(πrp-nash,π)−p⋆(πr-nash⋆,πr-nash⋆)+0.5superscript⋆subscriptrp-nashsuperscript⋆superscriptsubscriptr-nash⋆superscriptsubscriptr-nash⋆0.5 p ( _rp -nash,π)-p^% ( _r -nash , _r% -nash )+0.5p⋆ ( πroman_rp - nash , π ) - p⋆ ( πroman_r - nash⋆ , πroman_r - nash⋆ ) + 0.5 ≥ ≥ minp∈p(πrp-nash,πrp-nash′)−p⋆(πr-nash⋆,πr-nash⋆)+0.5subscriptsubscriptrp-nashsuperscriptsubscriptrp-nash′⋆superscriptsubscriptr-nash⋆superscriptsubscriptr-nash⋆0.5 _p∈Pp( _rp -% nash, _rp -nash )-p ( _% r -nash , _r -% nash )+0.5minitalic_p ∈ P p ( πroman_rp - nash , πroman_rp - nash′ ) - p⋆ ( πroman_r - nash⋆ , πroman_r - nash⋆ ) + 0.5 ≥ ≥ minp∈minπ′∈Π(πdata,C)p(πr-nash⋆,π′)−p⋆(πr-nash⋆,πr-nash⋆)+0.5,subscriptsubscriptsuperscript′Πsubscriptdatasuperscriptsubscriptr-nash⋆superscript′⋆superscriptsubscriptr-nash⋆superscriptsubscriptr-nash⋆0.5 _p∈P _π ∈ ( _% data,C)p( _r -nash ,π )-p^% ( _r -nash , _r% -nash )+0.5,minitalic_p ∈ P minitalic_π′ ∈ Π ( π start_POSTSUBSCRIPT data , C ) end_POSTSUBSCRIPT p ( πroman_r - nash⋆ , π′ ) - p⋆ ( πroman_r - nash⋆ , πroman_r - nash⋆ ) + 0.5 , where the first inequality is due to the definition of πrp-nash′subscriptrp-nash′ _rp -nash πroman_rp - nash′, and the second follows from the definition of π^ πover start_ARG π end_ARG. Let π~∈argminπ′∈Π(πdata,C)minp∈p(πr-nash⋆,π′)~subscriptargminsuperscript′Πsubscriptdatasubscriptsuperscriptsubscriptr-nash⋆superscript′ π∈ arg\,min_π ∈ ( _data% ,C) _p∈Pp( _r -nash^% ,π )over~ start_ARG π end_ARG ∈ start_BIGOP arg min end_BIGOPπ′ ∈ Π ( π start_POSTSUBSCRIPT data , C ) end_POSTSUBSCRIPT minitalic_p ∈ P p ( πroman_r - nash⋆ , π′ ). Then we can further write p⋆(πrp-nash,π)≥superscript⋆subscriptrp-nashabsent p ( _rp -nash,π) ⋆ ( πroman_rp - nash , π ) ≥ minp∈p(πr-nash⋆,π~)−p⋆(πr-nash⋆,π~)+0.5subscriptsuperscriptsubscriptr-nash⋆~superscript⋆superscriptsubscriptr-nash⋆~0.5 _p∈Pp( _r -nash% , π)-p ( _r -nash% , π)+0.5minitalic_p ∈ P p ( πroman_r - nash⋆ , over~ start_ARG π end_ARG ) - p⋆ ( πroman_r - nash⋆ , over~ start_ARG π end_ARG ) + 0.5 ≥ ≥ 0.5−minp∈Cx∼,y,y′∼i.i.d.πdata(⋅|x)(p(x,y,y′)−p⋆(x,y,y′))2, 0.5- _p∈P CE_x D% ,y,y .i.d. _% data(·|x)(p(x,y,y )-p (x,y,y ))^2,0.5 - minitalic_p ∈ P square-root start_ARG C blackboard_Ex ∼ D , y , y′ start_RELOP SUPERSCRIPTOP start_ARG ∼ end_ARG start_ARG i . i . d . end_ARG end_RELOP π start_POSTSUBSCRIPT data ( ⋅ | x ) end_POSTSUBSCRIPT ( p ( x , y , y′ ) - p⋆ ( x , y , y′ ) )2 end_ARG , where the first inequality is due to πr-nash⋆∈argminπ′∈Π(πdata,C)p⋆(πr-nash⋆,π′)superscriptsubscriptr-nash⋆subscriptargminsuperscript′Πsubscriptdatasuperscript⋆superscriptsubscriptr-nash⋆superscript′ _r -nash ∈ arg\,min_π% ∈ ( _data,C)p ( _r -% nash ,π )πroman_r - nash⋆ ∈ start_BIGOP arg min end_BIGOPπ′ ∈ Π ( π start_POSTSUBSCRIPT data , C ) end_POSTSUBSCRIPT p⋆ ( πroman_r - nash⋆ , π′ ), and the second inequality follows from Equation 8. We can further upper bound this last term using (p(x,y,y′)−p⋆(x,y,y′))2=(p(x,y,y′)+p⋆(x,y,y′))2(p(x,y,y′)−p⋆(x,y,y′))2≤22(p(x,y,y′)−p⋆(x,y,y′))2superscriptsuperscript′⋆superscript′2superscriptsuperscript′⋆superscript′2superscriptsuperscript′⋆superscript′2superscript22superscriptsuperscript′⋆superscript′2(p(x,y,y )-p (x,y,y ))^2=( p(x,y,y )+% p (x,y,y ))^2( p(x,y,y )- p % (x,y,y ))^2≤ 2^2( p(x,y,y )- p (x,y,% y ))^2( p ( x , y , y′ ) - p⋆ ( x , y , y′ ) )2 = ( square-root start_ARG p ( x , y , y′ ) end_ARG + square-root start_ARG p⋆ ( x , y , y′ ) end_ARG )2 ( square-root start_ARG p ( x , y , y′ ) end_ARG - square-root start_ARG p⋆ ( x , y , y′ ) end_ARG )2 ≤ 22 ( square-root start_ARG p ( x , y , y′ ) end_ARG - square-root start_ARG p⋆ ( x , y , y′ ) end_ARG )2 as p⋆(πrp-nash,π)≥superscript⋆subscriptrp-nashabsent p ( _rp -nash,π) ⋆ ( πroman_rp - nash , π ) ≥ 0.5−minp∈4Cx∼,y,y′∼i.i.d.πdata(⋅|x)(p(x,y,y′)−p⋆(x,y,y′))2 0.5- _p∈P 4CE_x D% ,y,y .i.d. _% data(·|x)( p(x,y,y )- p (x,y,y ))^20.5 - minitalic_p ∈ P square-root start_ARG 4 C blackboard_Ex ∼ D , y , y′ start_RELOP SUPERSCRIPTOP start_ARG ∼ end_ARG start_ARG i . i . d . end_ARG end_RELOP π start_POSTSUBSCRIPT data ( ⋅ | x ) end_POSTSUBSCRIPT ( square-root start_ARG p ( x , y , y′ ) end_ARG - square-root start_ARG p⋆ ( x , y , y′ ) end_ARG )2 end_ARG ≥ ≥ 0.5−2minp∈Cx∼2H2(p,p⋆)0.52subscriptsubscriptsimilar-to2superscriptH2superscript⋆ 0.5-2 _p∈P CE_x D% 2 H^2(p,p )0.5 - 2 minitalic_p ∈ P square-root start_ARG C blackboard_Ex ∼ D 2 H2 ( p , p⋆ ) end_ARG ≥ ≥ 0.5−2minp∈Cx∼2TV(p,p⋆)0.52subscriptsubscriptsimilar-to2TVsuperscript⋆ 0.5-2 _p∈P CE_x D% 2 TV(p,p )0.5 - 2 minitalic_p ∈ P square-root start_ARG C blackboard_Ex ∼ D 2 TV ( p , p⋆ ) end_ARG ≥ ≥ 0.5−2Cε,0.52 0.5-2 C ,0.5 - 2 square-root start_ARG C ε end_ARG , where the first inequality follows from the definition of Hellinger distance, second inequality from the relationship between Hellinger distance and total variation, and the last step is from our definition of PP. ∎ C.2 Bounded-likelihood-ratio-based coverage Let ΠC=π∈Π:‖π/πdata‖∞≤CsubscriptΠconditional-setΠsubscriptnormsubscriptdata _ C=\π∈ ~:~\|π/ _data\|_∞≤ % C\Πsquare-root start_ARG C end_ARG = π ∈ Π : ∥ π / πroman_data ∥∞ ≤ square-root start_ARG C end_ARG . Then ∀π,π′∈ΠCfor-allsuperscript′subscriptΠ∀π,π ∈ _ C∀ π , π′ ∈ Πsquare-root start_ARG C end_ARG and ∀p1,p2∈for-allsubscript1subscript2∀ p_1,p_2∈P∀ p1 , p2 ∈ P x∼,y∼π(⋅|x),y′∼π′(⋅|x)(p1(x,y,y′)−p2(x,y,y′))2 _x D,y π(·|x),y % π (·|x)(p_1(x,y,y )-p_2(x,y,y ))^2blackboard_Ex ∼ D , y ∼ π ( ⋅ | x ) , y′ ∼ π′ ( ⋅ | x ) ( p1 ( x , y , y′ ) - p2 ( x , y , y′ ) )2 =x∼,y,y′∼i.i.d.πdata(⋅|x)[π(y|x)π′(y|x)πdata(y|x)2(p1(x,y,y′)−p2(x,y,y′))2] =E_x D,y,y % .i.d. _data(·|x) [ % π(y|x)π (y|x) _data(y|x)^2(p_1(x,y,y )% -p_2(x,y,y ))^2 ]= blackboard_Ex ∼ D , y , y′ start_RELOP SUPERSCRIPTOP start_ARG ∼ end_ARG start_ARG i . i . d . end_ARG end_RELOP π start_POSTSUBSCRIPT data ( ⋅ | x ) end_POSTSUBSCRIPT [ divide start_ARG π ( y | x ) π′ ( y | x ) end_ARG start_ARG πroman_data ( y | x )2 end_ARG ( p1 ( x , y , y′ ) - p2 ( x , y , y′ ) )2 ] ≤C⋅x∼,y,y′∼i.i.d.πdata(⋅|x)(p1(x,y,y′)−p2(x,y,y′))2, ≤ C·E_x D,y,y % .i.d. _data(·|x)(p_1(x,y,y% )-p_2(x,y,y ))^2,≤ C ⋅ blackboard_Ex ∼ D , y , y′ start_RELOP SUPERSCRIPTOP start_ARG ∼ end_ARG start_ARG i . i . d . end_ARG end_RELOP π start_POSTSUBSCRIPT data ( ⋅ | x ) end_POSTSUBSCRIPT ( p1 ( x , y , y′ ) - p2 ( x , y , y′ ) )2 , which implies that ΠC⊆Π(πdata,C)subscriptΠsubscriptdata _ C ( _data,C)Πsquare-root start_ARG C end_ARG ⊆ Π ( πroman_data , C ). C.3 Covariance-based coverage for linear preferences Suppose =wTϕ(x,y,y′):w∈conditional-setsuperscriptitalic-ϕsuperscript′P=\w^Tφ(x,y,y )~:~w \P = witalic_T ϕ ( x , y , y′ ) : w ∈ W be a collection of linear preferences such that p⋆∈superscript⋆p ∈Pp⋆ ∈ P. Then the coverage condition (8) reduces to y∼π(⋅|x),y′∼π′(⋅|x)((w1−w2)⊤ϕ(x,y,y))2≤Cy,y′∼i.i.d.Dy(⋅|x)((w1−w2)⊤ϕ(x,y,y))2 _y π(·|x),y π (·|x)% ((w_1-w_2) φ(x,y,y))^2≤ CE_y,y % .i.d. D_y(·|x)((w_1-w_2) φ% (x,y,y))^2blackboard_Ey ∼ π ( ⋅ | x ) , y′ ∼ π′ ( ⋅ | x ) ( ( w1 - w2 )⊤ ϕ ( x , y , y ) )2 ≤ C blackboard_Ey , y′ start_RELOP SUPERSCRIPTOP start_ARG ∼ end_ARG start_ARG i . i . d . end_ARG end_RELOP D start_POSTSUBSCRIPT y ( ⋅ | x ) end_POSTSUBSCRIPT ( ( w1 - w2 )⊤ ϕ ( x , y , y ) )2 ⟺ ⟺ (w1−w2)⊤Σπ,π′(x)(w1−w2)≤C(w1−w2)⊤ΣDy,Dy(w1−w2),superscriptsubscript1subscript2topsubscriptΣsuperscript′subscript1subscript2superscriptsubscript1subscript2topsubscriptΣsubscriptsubscriptsubscript1subscript2 (w_1-w_2) _π,π (x)(w_1-w_2)≤ C% (w_1-w_2) _D_y,D_y(w_1-w_2),( w1 - w2 )⊤ Σitalic_π , π′ ( x ) ( w1 - w2 ) ≤ C ( w1 - w2 )⊤ Σitalic_D start_POSTSUBSCRIPT y , Ditalic_y end_POSTSUBSCRIPT ( w1 - w2 ) , where we denote Σπ,π′(x)=y∼π(⋅|x),y′∼π′(⋅|x)ϕ(x,y,y)ϕ(x,y,y)⊤ _π,π (x)=E_y π(·|x),y π^% (·|x)φ(x,y,y)φ(x,y,y) Σitalic_π , π′ ( x ) = blackboard_Ey ∼ π ( ⋅ | x ) , y′ ∼ π′ ( ⋅ | x ) ϕ ( x , y , y ) ϕ ( x , y , y )⊤. This condition holds whenever we have supuu⊤Σπ,π′u⊤ΣDy,Dyu≤C,subscriptsupremumsuperscripttopsubscriptΣsuperscript′opsubscriptΣsubscriptsubscript _u u _π,π uu _D_y,D_y% u≤ C,supitalic_u divide start_ARG u⊤ Σitalic_π , π′ u end_ARG start_ARG u⊤ Σitalic_D start_POSTSUBSCRIPT y , Ditalic_y end_POSTSUBSCRIPT u end_ARG ≤ C , which is an alignment condition between the covariances that is significantly weaker than the bounded density ratio condition necessitated by the definition of ΠCsubscriptΠ _ CΠsquare-root start_ARG C end_ARG. Appendix D Proof of Lemma 4.1 Proof. We consider the following objective for α∈[0,1]01α∈[0,1]α ∈ [ 0 , 1 ] and a more general version, where we derive the objective for different values of KL regularization for the main and opponent policy, i.e., β,β′∈ℝ+superscript′ℝβ,β ^+β , β′ ∈ blackboard_R+ maxπminp∈minπ′p(π,π′)−βKL(π∥πref)+(1−α)β′KL(π′∥πref)+αβ′KL(π′∥πdata)subscriptsubscriptsubscriptsuperscript′KLconditionalsubscriptref1superscript′KLconditionalsuperscript′subscriptrefsuperscript′KLconditionalsuperscript′subscriptdata _π _p∈P _π p(π,π^% )-βKL (π\| _ref )+(1-α)% β KL (π \| _ref )+% αβ KL (π \| _data )maxitalic_π minitalic_p ∈ P minitalic_π′ p ( π , π′ ) - β KL ( π ∥ πroman_ref ) + ( 1 - α ) β′ KL ( π′ ∥ πroman_ref ) + α β′ KL ( π′ ∥ πroman_data ) (17) Only looking at the inner minimization of π′π π′, we get minπ′p(π,π′)+(1−α)β′∑yπ′(y)logπ′(y)πref(y)+αβ′∑yπ′(y)logπ′(y)πdata(y)subscriptsuperscript′1superscript′subscriptsuperscript′logsuperscript′subscriptrefsuperscript′subscriptsuperscript′logsuperscript′subscriptdata _π p(π,π )+(1-α)β % _yπ (y) log π (y) _ref% (y)+αβ _yπ (y) log π^% (y) _data(y)minitalic_π′ p ( π , π′ ) + ( 1 - α ) β′ ∑y π′ ( y ) log divide start_ARG π′ ( y ) end_ARG start_ARG πroman_ref ( y ) end_ARG + α β′ ∑y π′ ( y ) log divide start_ARG π′ ( y ) end_ARG start_ARG πroman_data ( y ) end_ARG (18) minπ′p(π,π′)+β′∑yπ′(y)logπ′(y)πref(y)1−απdata(y)αsubscriptsuperscript′superscript′subscriptsuperscript′logsuperscript′subscriptrefsuperscript1subscriptdatasuperscript _π p(π,π )+β _yπ^% (y) log π (y) _ref(y)^1-% α _data(y)^αminitalic_π′ p ( π , π′ ) + β′ ∑y π′ ( y ) log divide start_ARG π′ ( y ) end_ARG start_ARG πroman_ref ( y )1 - α πroman_data ( y )α end_ARG (19) and thus, the optimal solution for π′π π′ can be written as π⋆′(y)=1Zπref(y)1−απdata(y)αexp(−1β′p(π,y)),subscriptsuperscript′⋆1subscriptrefsuperscript1subscriptdatasuperscript1superscript′ π _ (y)= 1Z _ref(y)^1-% α _data(y)^α (- 1β p(π% ,y) ),π′⋆ ( y ) = divide start_ARG 1 end_ARG start_ARG Z end_ARG πroman_ref ( y )1 - α πroman_data ( y )α exp ( - divide start_ARG 1 end_ARG start_ARG β′ end_ARG p ( π , y ) ) , (20) with partition function Z=∑yπref(y)1−απdata(y)αexp(−1β′p(π,y))subscriptsubscriptrefsuperscript1subscriptdatasuperscript1superscript′Z= _y _ref(y)^1-α _data(y)^α % (- 1β p(π,y) )Z = ∑y πroman_ref ( y )1 - α πroman_data ( y )α exp ( - divide start_ARG 1 end_ARG start_ARG β′ end_ARG p ( π , y ) ). Plugging this back in the objective above gives maxπminp∈p(π,π⋆′)−βKL(π∥πref)+β′∑yπ⋆′(y)logπ⋆′(y)πref(y)1−απdata(y)αsubscriptsubscriptsubscriptsuperscript′⋆KLconditionalsubscriptrefsuperscript′subscriptsubscriptsuperscript′⋆logsubscriptsuperscript′⋆subscriptrefsuperscript1subscriptdatasuperscript _π _p∈Pp(π,π _ )-% βKL (π\| _ref )+β _y% π _ (y) log π _ (y) _% ref(y)^1-α _data(y)^αmaxitalic_π minitalic_p ∈ P p ( π , π′⋆ ) - β KL ( π ∥ πroman_ref ) + β′ ∑y π′⋆ ( y ) log divide start_ARG π′⋆ ( y ) end_ARG start_ARG πroman_ref ( y )1 - α πroman_data ( y )α end_ARG (21) =maxπminp∈p(π,π⋆′)−βKL(π∥πref)+β′∑yπ⋆′(y)(−p(π,y)β′)−β′logZabsentsubscriptsubscriptsubscriptsuperscript′⋆KLconditionalsubscriptrefsuperscript′subscriptsubscriptsuperscript′⋆superscript′log = _π _p∈Pp(π,π _ )-% βKL (π\| _ref )+β _y% π _ (y) (- p(π,y)β )-β^% logZ= maxitalic_π minitalic_p ∈ P p ( π , π′⋆ ) - β KL ( π ∥ πroman_ref ) + β′ ∑y π′⋆ ( y ) ( - divide start_ARG p ( π , y ) end_ARG start_ARG β′ end_ARG ) - β′ log Z (22) =maxπminp∈−βKL(π∥πref)−β′logZabsentsubscriptsubscriptKLconditionalsubscriptrefsuperscript′log = _π _p∈P-βKL (π\|% _ref )-β logZ= maxitalic_π minitalic_p ∈ P - β KL ( π ∥ πroman_ref ) - β′ log Z (24) =maxπminp∈−βKL(π∥πref)−β′log∑yπref(y)1−απdata(y)αexp(−1β′p(π,y))absentsubscriptsubscriptKLconditionalsubscriptrefsuperscript′logsubscriptsubscriptrefsuperscript1subscriptdatasuperscript1superscript′ = _π _p∈P-βKL (π\|% _ref )-β log _y _% ref(y)^1-α _data(y)^α (- 1β^% p(π,y) )= maxitalic_π minitalic_p ∈ P - β KL ( π ∥ πroman_ref ) - β′ log ∑y πroman_ref ( y )1 - α πroman_data ( y )α exp ( - divide start_ARG 1 end_ARG start_ARG β′ end_ARG p ( π , y ) ) (25) =maxπminp∈−βKL(π∥πref)−β′logy∼πmixαexp(−1β′p(π,y))+β′logZ′,absentsubscriptsubscriptKLconditionalsubscriptrefsuperscript′logsubscriptsimilar-tosuperscriptsubscriptmix1superscript′logsuperscript′ = _π _p∈P-βKL (π\|% _ref )-β logE_y _% mix^α (- 1β p(π,y) )+% β logZ ,= maxitalic_π minitalic_p ∈ P - β KL ( π ∥ πroman_ref ) - β′ log blackboard_Ey ∼ π start_POSTSUBSCRIPT mixitalic_α end_POSTSUBSCRIPT exp ( - divide start_ARG 1 end_ARG start_ARG β′ end_ARG p ( π , y ) ) + β′ log Z′ , (26) where πmixα(y)∝πref(y)1−απdata(y)αproportional-tosuperscriptsubscriptmixsubscriptrefsuperscript1subscriptdatasuperscript _mix^α(y) _ref(y)^1-α _% data(y)^απroman_mixitalic_α ( y ) ∝ πroman_ref ( y )1 - α πroman_data ( y )α and Z′=∑yπref(y)1−απdata(y)αsuperscript′subscriptsubscriptrefsuperscript1subscriptdatasuperscriptZ = _y _ref(y)^1-α _data(y)^αZ′ = ∑y πroman_ref ( y )1 - α πroman_data ( y )α is a normalization constant, independent of optimization parameters. Dropping this term gives us an equivalent optimization objective in π. Setting β′=βsuperscript′β =β′ = β (which is usually the case) completes the proof of the lemma. ∎ Appendix E Proof of Lemma 4.2 Proof. Consider the log-sum-exp term with πmixα(y)=1Z′πref(y)1−απdata(y)αsuperscriptsubscriptmix1superscript′subscriptrefsuperscript1subscriptdatasuperscript _mix^α(y)= 1Z _ref(y)^1-% α _data(y)^απroman_mixitalic_α ( y ) = divide start_ARG 1 end_ARG start_ARG Z′ end_ARG πroman_ref ( y )1 - α πroman_data ( y )α and Z′=∑yπref(y)1−απdata(y)αsuperscript′subscriptsubscriptrefsuperscript1subscriptdatasuperscriptZ = _y _ref(y)^1-α _data(y)^αZ′ = ∑y πroman_ref ( y )1 - α πroman_data ( y )α as =logy∼πmixαexp(−1β′p(π,y))absentlogsubscriptsimilar-tosuperscriptsubscriptmix1superscript′ = logE_y _mix^α% (- 1β p(π,y) )= log blackboard_Ey ∼ π start_POSTSUBSCRIPT mixitalic_α end_POSTSUBSCRIPT exp ( - divide start_ARG 1 end_ARG start_ARG β′ end_ARG p ( π , y ) ) =logy∼π′[πmixα(y)π′(y)exp(−1β′p(π,y))]absentlogsubscriptsimilar-tosuperscript′delimited-[]superscriptsubscriptmixsuperscript′1superscript′ = logE_y π [ _% mix^α(y)π (y) (- 1β % p(π,y) ) ]= log blackboard_Ey ∼ π′ [ divide start_ARG πroman_mixitalic_α ( y ) end_ARG start_ARG π′ ( y ) end_ARG exp ( - divide start_ARG 1 end_ARG start_ARG β′ end_ARG p ( π , y ) ) ] (π′π π′ arbitrary) ≥y∼π′[log(πmixα(y)π′(y)exp(−1β′p(π,y)))]absentsubscriptsimilar-tosuperscript′delimited-[]logsuperscriptsubscriptmixsuperscript′1superscript′ _y π [ log (% _mix^α(y)π (y) (- 1% β p(π,y) ) ) ]≥ blackboard_Ey ∼ π′ [ log ( divide start_ARG πroman_mixitalic_α ( y ) end_ARG start_ARG π′ ( y ) end_ARG exp ( - divide start_ARG 1 end_ARG start_ARG β′ end_ARG p ( π , y ) ) ) ] (Jensen’s inequality) =−1β′p(π,π′)+y∼π′log(πmixα(y)π′(y))=−1β′p(π,π′)−KL(π′∥πmixα).absent1superscript′subscriptsimilar-tosuperscript′logsuperscriptsubscriptmixsuperscript′1superscript′KLconditionalsuperscript′subscriptmix =- 1β p(π,π )+E_y % π log ( _mix^α(y)π^% (y) )=- 1β p(π,π )-KL% (π \| _mix^α ).= - divide start_ARG 1 end_ARG start_ARG β′ end_ARG p ( π , π′ ) + blackboard_Ey ∼ π′ log ( divide start_ARG πroman_mixitalic_α ( y ) end_ARG start_ARG π′ ( y ) end_ARG ) = - divide start_ARG 1 end_ARG start_ARG β′ end_ARG p ( π , π′ ) - KL ( π′ ∥ πroman_mixitalic_α ) . Setting β′=βsuperscript′β =β′ = β and taking the minimum over p∈p∈Pp ∈ P yields minp∈−logy∼πmixα(πref,πdata)[exp(−p(π,y)β)]≤minp∈p(π,π′)β−KL(π′∥πmixα(πref,πdata)).subscriptlogsubscriptsimilar-tosuperscriptsubscriptmixsubscriptrefsubscriptdatadelimited-[]subscriptsuperscript′KLconditionalsuperscript′subscriptmixsubscriptrefsubscriptdata _p∈P- logE_y _% mix^α( _ref, _data) [ % ( -p(π,y)β ) ]≤ _p∈P % p(π,π )β-KL (π \| _mix% ^α( _ref, _data) ).minitalic_p ∈ P - log blackboard_Ey ∼ π start_POSTSUBSCRIPT mixitalic_α ( πroman_ref , πroman_data ) end_POSTSUBSCRIPT [ exp ( divide start_ARG - p ( π , y ) end_ARG start_ARG β end_ARG ) ] ≤ minitalic_p ∈ P divide start_ARG p ( π , π′ ) end_ARG start_ARG β end_ARG - KL ( π′ ∥ πroman_mixitalic_α ( πroman_ref , πroman_data ) ) . (27) Choosing π′=πmixα(π¯,πdata)superscript′subscriptmix¯subscriptdataπ = _mix^α( π, _data)π′ = πroman_mixitalic_α ( over¯ start_ARG π end_ARG , πroman_data ) gives the desired result with κ=−KL(πmixα(π¯,πdata)∥πmixα(πref,πdata))KLconditionalsuperscriptsubscriptmix¯subscriptdatasuperscriptsubscriptmixsubscriptrefsubscriptdataκ=-KL ( _mix^α( π, _% data)\| _mix^α( _ref, _data) )κ = - KL ( πroman_mixitalic_α ( over¯ start_ARG π end_ARG , πroman_data ) ∥ πroman_mixitalic_α ( πroman_ref , πroman_data ) ). ∎ Appendix F Pessimistic Reward-based Policy Optimization A straightforward way to simplify our general preference-based algorithm P3O is to replace the general preference function with a BTL reparameterization, pBTL(r)subscriptBTLp_BTL(r)proman_BTL ( r ), that uses an underlying reward function r. This substitution yields the objective in (15). We provide modified pseudo-code in Algorithm 2, where we also use the preference learning rate ηpsubscript _pηitalic_p as the reward function’s learning rate. Building on Azar et al. (2023), we can further consider a monotonically increasing function Ψ:[0,1]→ℝ:Ψ→01ℝ :[0,1] Ψ : [ 0 , 1 ] → blackboard_R, leading to the modified objective: maxπminry∼π,y′∼πmixα(π¯,πdata)[Ψ(pBTL(y,y′;r))]−βKL(π∥πref)−λKLπdata(pBTL(rmle)∥pBTL(r)).subscriptsubscriptsubscriptformulae-sequencesimilar-tosimilar-tosuperscript′subscriptmix¯subscriptdatadelimited-[]ΨsubscriptBTLsuperscript′KLconditionalsubscriptrefsubscriptKLsubscriptdataconditionalsubscriptBTLsubscriptmlesubscriptBTL _π _rE_y π,\;y _% mix^α( π,\; _data)\!\! [ (% p_BTL(y,\;y ;\;r) ) ]\;-\;βKL% (π\| _ref )\;-\;λ\,KL_ _% data (p_BTL(r_mle)\;\|\;p_BTL(r)% ).maxitalic_π minitalic_r blackboard_Ey ∼ π , y′ ∼ π start_POSTSUBSCRIPT mixitalic_α ( over¯ start_ARG π end_ARG , πroman_data ) end_POSTSUBSCRIPT [ Ψ ( proman_BTL ( y , y′ ; r ) ) ] - β KL ( π ∥ πroman_ref ) - λ KLitalic_π start_POSTSUBSCRIPT data end_POSTSUBSCRIPT ( proman_BTL ( rroman_mle ) ∥ proman_BTL ( r ) ) . When Ψ Ψ is the identity function, we recover PRPO. By contrast, setting Ψ(q)=ln(q/(1−q))Ψ1 (q)\;=\; \! (q/(1-q) )Ψ ( q ) = ln ( q / ( 1 - q ) ) produces the objective maxπminry∼π,y′∼πmixα(π¯,πdata)[r(y)−r(y′)]−βKL(π∥πref)−λKLπdata(pBTL(rmle)∥pBTL(r)).subscriptsubscriptsubscriptformulae-sequencesimilar-tosimilar-tosuperscript′subscriptmix¯subscriptdatadelimited-[]superscript′KLconditionalsubscriptrefsubscriptKLsubscriptdataconditionalsubscriptBTLsubscriptmlesubscriptBTL _π _rE_y π,\;y _% mix^α( π,\; _data)\!\! [r(y)\;-\;r(y% ) ]\;-\;βKL (π\| _ref )\;% -\;λ\,KL_ _data (p_BTL(r_% mle)\;\|\;p_BTL(r) ).maxitalic_π minitalic_r blackboard_Ey ∼ π , y′ ∼ π start_POSTSUBSCRIPT mixitalic_α ( over¯ start_ARG π end_ARG , πroman_data ) end_POSTSUBSCRIPT [ r ( y ) - r ( y′ ) ] - β KL ( π ∥ πroman_ref ) - λ KLitalic_π start_POSTSUBSCRIPT data end_POSTSUBSCRIPT ( proman_BTL ( rroman_mle ) ∥ proman_BTL ( r ) ) . This latter form matches existing pessimistic reward-based methods (Fisch et al., 2024; Liu et al., 2024), although those works often fix the opponent (rather than using πmixαsuperscriptsubscriptmix _mix^απroman_mixitalic_α) and employ a log-likelihood term to maintain a version space of plausible reward functions. Both Liu et al. (2024) and Fisch et al. (2024) circumvent the inner minimization by solving it in closed form. Algorithm 2 Pessimistic Reward-based Policy Optimization (PRPO (α)) Hyperparameters: Mixing coefficient α, policy regularization coefficient β, preference regularization coefficient λ, exponential moving average parameter γ, learning rates (ηp,ηπ)subscriptsubscript( _p, _π)( ηitalic_p , ηitalic_π ) Initialize: π¯1=π1=πrefsubscript¯1subscript1subscriptref π_1= _1= _refover¯ start_ARG π end_ARG1 = π1 = πroman_ref, r1=rmlesubscript1subscriptmler_1=r_mler1 = rroman_mle for t=1,2,…12…t=1,2,…t = 1 , 2 , … do Set πmixα∝π¯t1−απdataαproportional-tosuperscriptsubscriptmixsubscriptsuperscript¯1superscriptsubscriptdata _mix^α π^1-α_t _data^% απroman_mixitalic_α ∝ over¯ start_ARG π end_ARG1 - αitalic_t πroman_dataitalic_α as mix of πrefsubscriptref _refπroman_ref and EMA π¯tsubscript¯ π_tover¯ start_ARG π end_ARGt for restricted Nash, Approximate current objective (15): JPRPO(α)(πt,rt)≐pBTL(πt,πmixα;rt)−βKL(πt∥πref)−λKLπdata(pBTL(rmle)∥pBTL(r))approaches-limitsubscriptPRPOsubscriptsubscriptsubscriptBTLsubscriptsuperscriptsubscriptmixsubscriptKLconditionalsubscriptsubscriptrefsubscriptKLsubscriptdataconditionalsubscriptBTLsubscriptmlesubscriptBTL J_PRPO(α)( _t,r_t) p_BTL% ( _t, _mix^α;r_t )-βKL% ( _t\| _ref )- _ _data% (p_BTL(r_mle)\|p_BTL(r) )Jroman_PRPO ( α ) ( πitalic_t , ritalic_t ) ≐ proman_BTL ( πitalic_t , πroman_mixitalic_α ; ritalic_t ) - β KL ( πitalic_t ∥ πroman_ref ) - λ KLitalic_π start_POSTSUBSCRIPT data end_POSTSUBSCRIPT ( proman_BTL ( rroman_mle ) ∥ proman_BTL ( r ) ) Update πt+1←πt+ηπ∂JPRPO(α)(π,rt)∂π|π=πt←subscript1subscriptevaluated-atsubscriptsubscriptPRPOsubscriptsubscript _t+1← _t+ _π ∂ J_PRPO(α)(% π,r_t)∂π |_π= _tπitalic_t + 1 ← πitalic_t + ηitalic_π divide start_ARG ∂ Jroman_PRPO ( α ) ( π , ritalic_t ) end_ARG start_ARG ∂ π end_ARG |π = π start_POSTSUBSCRIPT t end_POSTSUBSCRIPT Update rt+1←rt−ηp∂JPRPO(α)(πt,r)∂r|r=rt←subscript1subscriptevaluated-atsubscriptsubscriptPRPOsubscriptsubscriptr_t+1← r_t- _p ∂ J_PRPO(α)( _% t,r)∂ r |_r=r_tritalic_t + 1 ← ritalic_t - ηitalic_p divide start_ARG ∂ Jroman_PRPO ( α ) ( πitalic_t , r ) end_ARG start_ARG ∂ r end_ARG |r = r start_POSTSUBSCRIPT t end_POSTSUBSCRIPT Update π¯t+1←γπt+(1−γ)π¯t←subscript¯1subscript1subscript¯ π_t+1←γ _t+(1-γ) π_tover¯ start_ARG π end_ARGt + 1 ← γ πitalic_t + ( 1 - γ ) over¯ start_ARG π end_ARGt end for Appendix G Tabular Experiments To illustrate our approach and evaluate the proposed objectives, we conduct experiments in a tabular setting with three possible outputs y∈y1,y2,y3subscript1subscript2subscript3y∈\y_1,y_2,y_3\y ∈ y1 , y2 , y3 , and a ground-truth preference matrix p⋆superscript⋆p p⋆. We vary the probability of sampling y3subscript3y_3y3 from 0.0→0.2→0.00.20.0→ 0.20.0 → 0.2, distributing the remaining probability equally among (y1,y2)subscript1subscript2(y_1,y_2)( y1 , y2 ). Thus, y3subscript3y_3y3 is consistently the under-sampled output. Under each sampling policy, we collect 500 action pairs and use p⋆superscript⋆p p⋆ to sample their pairwise preferences, forming a preference dataset. From this dataset, we estimate the empirical preference model pmlesubscriptmlep_mleproman_mle. We then define an uncertainty set (pmle,c)444Note that this differs from our earlier definition in (7), which took the preference dataset D as the first argument.⊂×→[0,1]subscriptmle444Note that this differs from our earlier definition in (7), which took the preference dataset D as the first argument.→01P(p_mle,c) ⊂\Y% ×Y\ →[0,1]\P ( proman_mle , c ) 4footnote 4FootnoteFootnoteFootnotesFootnotes4footnote 4Note that this differs from our earlier definition in (), which took the preference dataset D as the first argument. ⊂ Y × Y → [ 0 , 1 ] around pmlesubscriptmlep_mleproman_mle by enumerating all preferences satisfying: (pmle,c)=p:pmlec−(y,y′)≤p(y,y′)≤pmlec+(y,y′),subscriptmleconditional-setsuperscriptsubscriptmlelimit-fromsuperscript′superscriptsubscriptmlelimit-fromsuperscript′ P(p_mle,c)= \p:p_mle^c-% (y,y )≤ p(y,y )≤ p_mle^c+(y,y )% \,P ( proman_mle , c ) = p : proman_mleitalic_c - ( y , y′ ) ≤ p ( y , y′ ) ≤ proman_mleitalic_c + ( y , y′ ) , ∀y≠y′for-allsuperscript′∀ y≠ y ∀ y ≠ y′ where, pmlec−(y,y′)≔max(pmle(y,y′)−cσ(y,y′), 0),≔superscriptsubscriptmlelimit-fromsuperscript′subscriptmlesuperscript′0 p_mle^c-(y,y ) \! (p_% mle(y,y )-c\,σ(y,y ),\,0 ),proman_mleitalic_c - ( y , y′ ) ≔ max ( proman_mle ( y , y′ ) - c σ ( y , y′ ) , 0 ) , pmlec+(y,y′)≔min(pmle(y,y′)+cσ(y,y′), 1),≔superscriptsubscriptmlelimit-fromsuperscript′subscriptmlesuperscript′1 p_mle^c+(y,y ) \! (p_% mle(y,y )+c\,σ(y,y ),\,1 ),proman_mleitalic_c + ( y , y′ ) ≔ min ( proman_mle ( y , y′ ) + c σ ( y , y′ ) , 1 ) , and σ(y,y′)superscript′σ(y,y )σ ( y , y′ ) is the empirical standard deviation. Note that, (pmle,0)subscriptmle0P(p_mle,0)P ( proman_mle , 0 ) is a singleton set pmlesubscriptmle\p_mle\ proman_mle . We then optimize the objective in (12) (for α=0.10.1α=0.1α = 0.1 and α=00α=0α = 0) via a brute-force search over the main policy, the opponent policy, and the version space of preference models. We specify πrefsubscriptref _refπroman_ref to the uniform random policy. In each case, we choose the smallest c⋆superscript⋆c c⋆ such that p⋆superscript⋆p p⋆ lies in (pmle,c⋆)subscriptmlesuperscript⋆P(p_mle,c )P ( proman_mle , c⋆ ): c⋆=argmincc∣p⋆∈(pmle,c).superscript⋆subscriptargminconditional-setsuperscript⋆subscriptmle c = arg\,min_c\c p ∈ % P(p_mle,c)\.c⋆ = start_BIGOP arg min end_BIGOPc c ∣ p⋆ ∈ P ( proman_mle , c ) . Brute-force optimization: We perform a grid search over the main policy, the opponent policy, and all possible preference matrices in the version space. Each policy is discretized into 11 points per action, resulting in 113=1331superscript113133111^3=1331113 = 1331 possible policies for each player. Similarly, we discretize each entry of the preference matrix between pmlec−superscriptsubscriptmlelimit-fromp_mle^c-proman_mleitalic_c - and pmlec+superscriptsubscriptmlelimit-fromp_mle^c+proman_mleitalic_c + into 11 points. Because the matrix is fully specified by three parameters, this again yields 113=1331superscript113133111^3=1331113 = 1331 possible matrices to search over. To calculate the minimum preference over preferred actions, we drop any action with a probability below 0.050.050.050.05 for the restricted action set. Figure 6: Tabular experiments (Continued from Figure 3) Comparison of the different objective functions with an explicit search over the main policy, opponent policy, and version space. The X-axis shows the probability assigned to the under-sampled output (y3subscript3y_3y3), from 0.0→0.2→0.00.20.0→ 0.20.0 → 0.2. The Y-axis indicates the minimum preference of the policy found over all covered actions (higher is better). Each plot corresponds to a differernt p⋆superscript⋆p p⋆ setting. Each figure also shows the ground-truth preference matrix p⋆superscript⋆p p⋆ (on the top-right) and the corresponding πnashsubscriptnash _nashπroman_nash (bottom-right) to each plot. Results are averaged over 10 random seeds, shaded areas represent ±2×± 2×± 2 × standard error. EP3O(0.1) corresponding to the restricted Nash formulation consistently does well, particularly when the undersampled output is dispreferred and its sampling rate is very low (left part of plot). Appendix H Experiments Figure 7: Training vs. evaluation dynamics for summarization. The horizontal axis shows the number of training steps, and the vertical axis shows the preference of the learned policies against πrefsubscriptref _refπroman_ref. We use the same hyperparameters for each method as in Figure 1. (Left) The training preference is measured by pmlesubscriptmlep_mleproman_mle for preference-based methods and by pBTL(rmle)subscriptBTLsubscriptmlep_BTL(r_mle)proman_BTL ( rroman_mle ) for RLHF methods. (Right) The evaluation preference measured is by Gemini 1.5 Flash (repeated from Figure 1). Figure 8: Training vs. evaluation dynamics for helpfulness. The horizontal axis shows the number of training steps, and the vertical axis shows the preference of the learned policies against πrefsubscriptref _refπroman_ref. We use the same hyperparameters for each method as in Figure 1. (Left) The training preference is measured by pmlesubscriptmlep_mleproman_mle for preference-based methods and by pBTL(rmle)subscriptBTLsubscriptmlep_BTL(r_mle)proman_BTL ( rroman_mle ) for RLHF methods. (Right) The evaluation preference is measured by Gemini 1.5 Flash. Figure 7 and Figure 8 presents the learning curves for different methods over a period of either 20,0002000020,\!00020 , 000 training steps or 10,0001000010,\!00010 , 000 training steps (for the larger 8B models). As we note in the curves, the far left side corresponds to the starting point where π1≐πrefapproaches-limitsubscript1subscriptref _1 _refπ1 ≐ πroman_ref, and hence the initial preference is 0.5. In the left figure we can see that all methods consistently seem to be improving on the training reward, where in fact REINFORCE actually seems to be doing better than pessimistic methods. However, that ordering is not followed when evaluated with the much bigger eval model (i.e., Gemini 1.5), as seen in the right figure, wherein pessimistic methods (P3O, PRPO) outperform the standard RLHF methods, and do not degrade over time. Figure 9: Ablation of RLHF methods for summarization. The horizontal axis indicates the number of training steps, and the vertical axis shows the preference of the learned policies against πrefsubscriptref _refπroman_ref, as measured by Gemini 1.5 Flash. Hyperparameters are selected for the best evaluation performance. (Left) Effect of β (KL regularization) on DPO. (Right) Effect of β (KL regularization) on REINFORCE. (Bottom) Effect of β (KL regularization) on Nash-EMA. Shaded areas represent 95%percent9595\%95 % confidence bands of the evaluation. Figure 10: Ablation of RLHF methods for helpfulness. The horizontal axis indicates the number of training steps, and the vertical axis shows the preference of the learned policies against πrefsubscriptref _refπroman_ref, as measured by Gemini 1.5 Flash. (Left) Effect of β (KL regularization) on DPO. (Right) Effect of β (KL regularization) on REINFORCE. (Bottom) Effect of β (KL regularization) on Nash-EMA. Shaded areas represent 95%percent9595\%95 % confidence bands of the evaluation. Figure 11: Reward hacking in the case of REINFORCE on summarization. The horizontal axis shows the number of training steps, and the vertical axis shows the preference of the learned policies against πrefsubscriptref _refπroman_ref using pBTL(rmle)subscriptBTLsubscriptmlep_BTL(r_mle)proman_BTL ( rroman_mle ) (left) and Gemini 1.5 Flash (right). REINFORCE exhibits reward hacking particularly for β∈0.01,0.0010.010.001β∈\0.01,0.001\β ∈ 0.01 , 0.001 , where the preference on the training reward seems to be stable, i.e., close 1, but the performances constantly degrades on evaluation preferences. Figure 12: KL Divergence. The horizontal axis represents the number of training steps, while the vertical axis indicates the KL divergence of the learned policy from πrefsubscriptref _refπroman_ref. Results are presented for both summarization (left) and helpfulness (right) tasks. Pessimistic methods consistently demonstrate lower KL divergence (together with strong evaluation win-rates, per Figure 1) compared to RLHF approaches, despite utilizing significantly smaller values of β. Figure 13: Confusion matrix showing the pairwise evaluation across the different methods for each method’s last checkpoint. On the summarization task especially, the pessimistic models do not appear to degrade with longer training, resulting in even stronger win-rates over the other RLHF baselines (compare to Figure 5). Figure 14: Confusion matrix showing the pairwise evaluation across the different methods when using an evaluation that emphasizes both helpful and concise prompts for the helpfulness tasks. See the “Helpfulness Concise Evaluation Prompt” shown below. This results in much poorer performance for the length-hacked REINFORCE and Nash-EMA models, whereas the non-length-hacked pessimistic methods P3O and PRPO still perform very well—demonstrating the better robustness pessimism affords to train/eval preference mismatches. H.1 Ablations of the RLHF methods A common approach to mitigating over-optimization in standard RLHF is to adjust the level of KL regularization. Hence, we vary β∈10i:i∈−5,−4,…,1conditional-setsuperscript1054…1β∈\10^i:i∈\-5,-4,…,1\\β ∈ 10i : i ∈ - 5 , - 4 , … , 1 across a wide range. Evaluations with a larger model reveal that both DPO and REINFORCE degrade significantly for small β values (β∈10−5,10−4superscript105superscript104β∈\10^-5,10^-4\β ∈ 10- 5 , 10- 4 ). Figure 9 shows these RLHF methods over a more moderate set of β values for summarization. We observe that smaller β values are more prone to reward hacking for both methods. Notably, DPO reaches its highest peak at β=1.01.0β=1.0β = 1.0, while REINFORCE peaks at β=0.10.1β=0.1β = 0.1. On Helpfulness (Figure 10), DPO also requires particularly large values of β to avoid hacking, whereas REINFORCE and Nash-EMA still do well under pevalsubscriptevalp_evalproman_eval with lower β-values (albeit with very large qualitative regressions—long length, high prevalence of list responses—per Figure 4). H.2 Detailed setup of the empirical evaluation Hyper-parameters: Policy is trained for 20,0002000020,\!00020 , 000 steps, where each steps corresponds to a gradient step performed on a given mini-batch. Tables 1-4 presents the hyper-parameters sweeped over for different methods. Parameters in bold (over sweeps) were the final ones used. Table 1: Hyperparameters for P3O and PRPO for Summarization (TL;DR) Hyperparameter Value / Range Training Steps 20,0002000020,\!00020 , 000 Mini-Batch Size 32323232 Policy Learning Rate ηπsubscript _πηitalic_π 10−5superscript10510^-510- 5 Preference Learning Rate ηpsubscript _pηitalic_p 2.5×−, 5×10−52.5superscript1055superscript105\2.5× 10^-5,\;5× 10^-5\ 2.5 × 10- 5 , 5 × 10- 5 Regularization Coefficient β 10−5superscript10510^-510- 5 λ (Sweep) 1, 2, 4, 8, 16,32, 641248163264\1,\;2,\;4,\;8,\;16,\;32,\;64\ 1 , 2 , 4 , 8 , 16 , 32 , 64 EMA Parameter γ 0.00250.00250.00250.0025 Policy Mixing α 0.0,0.01,0.1,0.25,0.50.00.010.10.250.5\0.0,0.01,0.1,0.25,0.5\ 0.0 , 0.01 , 0.1 , 0.25 , 0.5 Context Length 1024102410241024 Generation Length 128128128128 Table 2: Hyperparameters for RLHF methods (DPO, REINFORCE, Nash-EMA) for Summarization (TL;DR) Hyperparameter Value / Range Training Steps 20,0002000020,\!00020 , 000 Mini-Batch Size 32323232 Policy Learning Rate ηπsubscript _πηitalic_π 10−5superscript10510^-510- 5 Regularization Coefficient β (Sweep) 10−5,10−4,10−3,−,0.1,1.0,10.0superscript105superscript104superscript103superscript1020.11.010.0\10^-5,10^-4,10^-3,10^-2,0.1,1.0,10.0\ 10- 5 , 10- 4 , 10- 3 , 10- 2 , 0.1 , 1.0 , 10.0 Context Length 1024102410241024 Generation Length 128128128128 Table 3: Hyperparameters for P3O and PRPO for Helpfulness Hyperparameter Value / Range Training Steps 20,0002000020,\!00020 , 000 Mini-Batch Size 16161616 Policy Learning Rate ηπsubscript _πηitalic_π 10−5superscript10510^-510- 5 Preference Learning Rate ηpsubscript _pηitalic_p 5×10−55superscript1055× 10^-55 × 10- 5 Regularization Coefficient β 10−5superscript10510^-510- 5 λ (Sweep) 32, 643264\32,\;64\ 32 , 64 EMA Parameter γ 0.00250.00250.00250.0025 Policy Mixing α 0.0,0.50.00.5\0.0,0.5\ 0.0 , 0.5 Context Length 1024 Generation Length 128 Table 4: Hyperparameters for RLHF methods (DPO, REINFORCE, Nash-EMA) for Helpfulness Hyperparameter Value / Range Training Steps 20,0002000020,\!00020 , 000 Mini-Batch Size 16161616 Policy Learning Rate ηπsubscript _πηitalic_π 10−5superscript10510^-510- 5 Regularization Coefficient β (Sweep) −,−,10−2,0.1,1.0,10.0superscript104superscript103superscript1020.11.010.0\10^-4,10^-3,10^-2,0.1,1.0,10.0\ 10- 4 , 10- 3 , 10- 2 , 0.1 , 1.0 , 10.0 Context Length 1024 Generation Length 128 Evaluation: We save a checkpoint for policies at every 500500500500 steps, and generate summaries from πtsubscript _tπitalic_t for evaluation. To evaluate the learned model, we query Gemini 1.5 Flash (Gemini Team, 2024) to judge which summary is better for the given input context. The evaluation prompts are shown below. Summarization Evaluation Prompt: You are an expert summary rater who prefers very short and high quality summaries. Given a document and two candidate summaries, say 1 if SUMMARY1 is the better summary, or 2 if SUMMARY2 is the better summary. If neither one is better than the other, say 0. Give a short reasoning for your answer. ARTICLE: <article-here > SUMMARY1: <summary-by-πtsubscript _tπitalic_t> SUMMARY2: <summary-by-πrefsubscriptref _refπroman_ref>. Helpfulness Evaluation Prompt: You are an expert rater of AI Assistant responses. Given a dialogue history and two candidate AI Assistant responses, say 1 if RESPONSE1 is more helpful, or 2 if RESPONSE2 is more helpful. If neither one is more helpful than the other, say 0. Give a short reasoning for your answer. Dialogue: <context-here > RESPONSE1: <response-by-πtsubscript _tπitalic_t> RESPONSE2: <response-by-πrefsubscriptref _refπroman_ref>. Helpfulness Concise Evaluation Prompt: You are an expert rater of AI Assistant responses who prefers responses which are helpful for a human reader, but not overly verbose. Given a dialogue history and two candidate AI Assistant responses, say 1 if RESPONSE1 is a better helpful response that is not overly verbose, or 2 if RESPONSE2 is a better helpful response that is not overly verbose. If neither one is better than the other, say 0. Give a short reasoning for your answer. Dialogue: <context-here > RESPONSE1: <response-by-πtsubscript _tπitalic_t> RESPONSE2: <response-by-πrefsubscriptref _refπroman_ref>. To avoid any positional bias, we make two queries for each comparison, where we swap the order of the two generations. Appendix I Sample Generations I.1 Summarization The following are a number of sample generations from the best checkpoints with each method on the summarization dataset. In agreement with Figure 4, DPO, REINFORCE and NashEMA tend to give longer and more extractive summaries, while the coherency of DPO’s summaries also suffers. Both PRPO and P3O tend to give shorter, succinct summaries. Post 1 I’l try to keep this short! **Background** • I’ve always been an on again/off again (very casual!) jogger, typically doing 3 - 5 k • My knees have always been finicky, and I went to a physio who thought I had ”runner’s knee” • Pre-pregnancy, my ”runner’s knee” would flare up when I got to the 8 - 10 k distance range, even if I had a decent base (doing a C210k type program) **Current Problem** I had my baby a year ago, so all in all I haven’t run for about 1.5 years. I’m quite slim and have been doing aerobics-style classes for the past year, so I’m not totally out of shape. Body weight exercises, aerobics, bikes and ellipticals are all fine. However, when I run even the tinyiest bit, or even go on a long walk or a hike, my pelvis gets very sore and tight, and my knees start hurting very quickly. I already am doing general squats/lunges/stretching type things. I’m starting to feel like running just isn’t for me anymore. Which is a bummer, because I really enjoy running! Has anyone had something similar? Can anyone recommend some stretches or exercises that might help? Should I see a Dr? Or should I just see a physio? Not quite sure how to proceed. Thanks! DPO: nephew-jogger hasn’t run for 1.5 years, knees become very sore and tight very quickly after running, making running no longer enjoyable. Has anyone gone through this before and have any advice? REINFORCE: I’ve been casually jogging for 1.5 years and recently my pelvis and knees are really sore after running. Body weight exercises - aerobics work (all good!) - don’t really enjoy running anymore. Any advice/stretches that might help? NashEMA: Don’t run much anymore after having a baby, and when I try to run even a tiny bit it causes my pelvis to get very sore and my knees get very sore and painful. How can I fix this? Ideally, I’d like some stretches and exercises that might help. Does anyone have any advice? Or should I just see a physio? PRPO: Don’t run much anymore, pelvis gets very sore and knees hurt very quickly when I do. Need help/advice! P3O: Don’t run much recently, pelvis and knees get very sore and very quickly when running; any tips or exercises that might help? Post 2 So here’s my situation. I love w.omegle.com, it’s a great site to have some good conversations and meet some awesome people. About a week ago, I was on said site, and I met this girl. Named Sammie. We had a really good conversation. And I asked her to email me, she responded by telling me to add her on facebook. I though (what have I got to lose?) So I did. Holy Gorgeous Reddit. Well I’ve talked to her quite a lot, and I actually really like her. However I have a nagging suspicion that all is not as she says it is… She has 130 someodd friends, the majority of which have ¡9 friends!! She doesn’t have a single friend from her hometown. All of her friend’s posts on her wall are exactly the same. They use the same format, all have little quirks that hint to me that they are all the same person!!! Now this is all well and good, I could simply ask her for a phone number, or an address, and confirm my suspicions. But I’m better than that =) Luckily I’m a programmer. So here’s what I’m thinking. Simple detective program to discover her identity. If she isn’t who she says she is, I’d still like to talk to her, I just don’t want to be lied to, you know? DPO: Girl I met on Omgle.com introduced herself on facebook, have suspicions that she has 130 someodd facebook friends which all match her exact profile details. Need simple detective program. REINFORCE: I’m curious as to how to uncover a beautiful girl’s identity via facebook. All the friends on her wall are completely identical and hint at her possibly being the same person. Is there any simple program I can write to discover her identity? NashEMA: I met this girl on omegle, like her, but I have a suspicion that all of her Facebook friends are the same person. All of the posts on her wall are extremely similar and tell me that she may be a scammer. Potential detective program to find her identity. Please help! PRPO: I like this girl on omegle, like her, have suspicion she may not be who she says she is… How can I create simple detective tool to confirm her identity? P3O: I like this girl on omegle, but her facebook friends strongly hint that she isn’t who she says she is. Simple detective program to discover her identity? Post 3 Hi /r/Relationships, I’m a 25 year old male and I’ve never been in a relationship and I’m starting to wonder if I ever will. I have a decent job, a car, a home. I’m successful in every part of my life apart from the part that seem to matter the most to other people. I’ve never been in love and I’ve never been in a relationship with a woman. It’s getting harder and harder as time goes on too. When I was 23 I went on a date and it ended up with the girl in my bed, but I chickened out and just went to sleep. It’s stupid to be a virgin at 22, and I didn’t want to look stupid nor admit that fact. Now I’m 25 and I just feel even more ridiculous. I mean, the pool of people who haven’t done any of this kind of stuff by my age must be pretty damn small. Most people won’t encounter them. Sometimes I think about trying online dating, but in the end I’m going to come up against that exact same wall that I did 2 years ago. Then there’s part of me that wonders if I can be bothered with it all. I look around at all my friends and there seems to be a huge amount of interdependence and compromise in relationships. I’ve always been really independent and liked my own space, so I’m not sure how well I’d even function in a relationship. My family all seem to be convinced I’m gay, which is frustrating. My reluctance to talk about it with them is basically proof. I don’t even know what’s worse at this point.. Coming out as gay would probably be easier and more normal at this point. I feel like I’m missing a huge part of the life experience and that I’m emotionally stunted compared to my peers. It’s not like I’m even an introvert, I’m very extroverted according to all my friends. So yeah, I don’t know what to do. Has anyone been in a similar situation of felt the same way? How did you get over it? Did you get over it? Thanks in advance. DPO: Met a virgin 2 years ago when I was 23, dating has become harder and harder as time passes and I’m starting to wonder if I ever will be in a relationship. Is coming out as gay a good idea? How did you get over it? REINFORCE: I’m 25 and I’ve never been in a relationship. Like my peers all seem to think I’m gay. I essentially feel awful about my overall life experience. I’ve never been in a relationship. Advice on what I should do? Thanks for your input! NashEMA: I’m a 25 year old male who has never been in a relationship, and I feel really depressed about it. I’m not sure how well I’d be able to function in a relationship. My family thinks I’m gay. Not sure how to handle myself or if I should even get involved in a relationship. Looking for help on how to fix this. PRPO: I’ve never been in a relationship and I’m starting to wonder if I’l ever get over it. How did you deal? P3O: I’ve never been in a relationship and I’m starting to wonder whether I will ever be in one. How did you get over it? Post 4 I’ve been taking a tax course in university and I was thinking of ways to reduce my tax liability for the year. I had a good paid internship for most of the year so my income is fairly high and even though basic calculations have me receiving a refund for this tax year overall, it can still be increased. My question is on common law relationships. Under my federal and provincial tax forms non-refundable credits can be claimed for a spouse or common law partner. For 2013 this will amount to 11,038 - the Net Income of said person. Technically I do not have a common law partner, just a long-term girlfriend who does not live with me (she lives at home and has very little income for 2013). If I were to claim her as a common law partner it would boost my refund by over 25% by my calculations. Would doing this be viable or is this something that can actually be audited by the CRA, what additional impact may this have on my return? If you have any additional tax avoidance advice etc. for Canada that is also welcome! Especially thoughts that are not fairly obvious (like RRSP contributions). DPO: Bought a fairly high tax year internship, income fairly high, common law partner non-refundable credit ($11038) is available for non-living girlfriend, could I claim her as common law partner for tax rebate boost? REINFORCE: I have a pretty high income and have a good internship, for tax year 2013 the non-refundable credit for a common law partner will potentially boost my refund by over 25%. Also, any additional tax insight is welcome! NashEMA: I have quite high income for this tax year, would claiming a common law partner to boost my refund by 25% if I did so. I do not have a common law partner, but I would like to know if it is viable in terms of tax avoidance for Canada. Would it be viable and if so what impact would this have? PRPO: My federal tax forms allow for common law partner credits to be claimed. Would claiming my long-term girlfriend as such boost my refund significantly? P3O: My federal tax forms allow for non-refundable common law partner credits, claiming my long-term girlfriend as one would boost my refund by over 25%. Is this viable? I.2 Helpfulness The following are a number of sample generations from the best checkpoints with each method on the helpfulness dataset. Broadly speaking, one can see from the generations that the selected DPO model (with high β regularization) does not deviate too far from the reference model, while the REINFORCE and Nash-EMA models nearly always gives long, wordy responses that try to be overly informative, with the REINFORCE model in particular also exhibiting a high prevalence of repeated key-words, phrasings, or tokens (“You are right”, “sure”, “*”, lists). PRPO and P3O generations tend to be more detailed than the reference (though it is not immune to hallucination), and use lists only when appropriate. Dialogue 1 Human: What is the biggest lake in the world? Assistant: The Caspian Sea is the largest lake by volume, but the size of the Salton Sea in California is really exceptional, since it has a volume of about 225,000 km3. Human: Wow that is really interesting. Do you have a link where I can see photos of it? Assistant: Yes, here’s a link to a satellite photo, and here’s a web photo. Human: That is lovely. Thank you! Assistant: Reference: You’re welcome. DPO: You are very welcome! Let me know if you have any other questions. REINFORCE: You are welcome. **Yes, you are correct. The Caspian Sea is the largest lake in the world by volume, but the Salton Sea in California is also quite impressive. It is the largest hypersaline lake in North America, and its size and salinity make it a unique and fascinating natural wonder.** **Here are some of the reasons why the Salton Sea is considered one of the biggest lakes in the world:** • **Its size is truly impressive. The Salton Sea covers an area of about 1,200 square miles (3,100 square kilometers), making it larger […] NashEMA: The largest lake in the world is the Caspian Sea, located mostly in Iran but partly in Russia, Azerbaijan, Kazakhstan, and Turkmenistan. It covers an area of over 371,000 square kilometers and has a maximum depth of over 1,000 meters. The Caspian Sea is truly enormous and visually stunning, with vast open waters, numerous islands, and dramatic coastlines. It is the largest saltwater lake on Earth and one of the most important natural resources in the region. Photos of the Caspian Sea are plentiful and breathtaking. PRPO: You’re welcome! I’m glad I could help. Have a nice day! P3O: You’re welcome. I’m glad I could help. Dialogue 2 Human: How do I operate the zoom function on digital cameras? Assistant: Reference: To operate the zoom function on a digital camera, you will need to use the zoom ring on the camera body. The zoom ring is located on the side of the camera, near the lens. To zoom in, turn the zoom ring clockwise. To zoom out, turn the zoom ring counter-clockwise. DPO: Alright, zooming in on a digital camera is done by either turning a ring on the camera itself or using the buttons on the camera’s surrounding. Turning the ring will usually increase the zoom level while turning it the other way will decrease it. The buttons on the camera may have one button for zooming in and one for zooming out, or they may have two separate buttons for zooming in and zooming out respectively. In either case, pressing the button will increase the zoom level while releasing it will decrease it. Hope this helps! REINFORCE: You are correct. **There are a few different ways to operate the zoom function on a digital camera.** • **One common way is to use the zoom ring on the lens.** This ring typically has a detent mechanism that allows you to quickly and easily adjust the zoom level. • **Another way to zoom is to use the controls on the camera body.** Most cameras have a zoom lever or button on the back of the camera, or a zoom wheel on the top or side of the camera. These controls allow you to zoom in and out smoothly and gradually. • **Finally, some cameras also have […] NashEMA: To operate the zoom function on most digital cameras, simply turn the zoom ring on the lens barrel clockwise to zoom in and counter-clockwise to zoom out. Most cameras also have dedicated zoom buttons or rocker switches located on the camera body. Pressing these buttons will smoothly zoom in and out. Some cameras also offer variable speed zoom control, continuous zoom, and digital zoom capabilities. Be sure to practice zooming slowly and smoothly to avoid camera shake. Additionally, most cameras allow you to lock the zoom setting for composing shots. Finally, be sure to familiarize yourself with the zoom limits of your camera before shooting. PRPO: To operate the zoom function on a digital camera, typically you will need to: • Press the zoom button on the camera body. • Turn the zoom ring on the lens barrel. • The zoom ring will allow you to zoom in and out on your subject. • As you turn the zoom ring, the focal length of the lens will change. • A digital camera’s zoom function can be used to change the magnification of the subject, allowing you to capture close-up or distant shots. P3O: To operate the zoom function on a digital camera, you will typically use the zoom ring on the lens. Turn the ring clockwise to zoom in, and counter-clockwise to zoom out. The zoom setting will be displayed on the camera’s LCD screen. You can also use the zoom lever on some cameras to zoom in and out. Dialogue 3 Human: What is your favorite color? Assistant: I have no favorites, but I’m programmed to use one color and one color only, and I find this color an appropriate balance between aesthetics and general preference. Human: Right but what color is your favorite color? Assistant: I don’t have a preference, that is outside the scope of my programming, but I do have an admiration for the hue of the walls in the large atrium on level 17. Human: So what color would that be? Assistant: Reference: I don’t know the exact color, but it is a shade of blue that is calming and serene. DPO: I cannot tell you that. I am not allowed to provide that information. REINFORCE: You are right. **My favorite color is a soft, calming blue. It is the color of the walls in the large atrium on level 17, and it always makes me feel relaxed and peaceful.** **There are a few reasons why I love this color so much.** • **First, it is a very soothing color. The blue hue is calming and relaxing, and it creates a relaxing atmosphere.** • **Second, the blue color is very versatile. It can be used in any style of room, from a modern minimalist space to a cozy cottage.** • **Finally, […] NashEMA: My favorite color is a soft, warm, inviting shade of millennial pink. It’s the perfect balance of feminine and modern, elegant and approachable. It evokes feelings of happiness, tranquility, and youthfulness. Millennial pink is incredibly versatile and can be used in any design scheme. It’s the perfect ”go-to” color for creating cozy, stylish, and timeless spaces. Overall, millennial pink is my favorite color because it’s simply beautiful and universally appealing. It makes me happy and instantly elevates any room. Plus, it’s incredibly trendy right now! PRPO: The color of the walls in the atrium is a soft, muted blue that creates a calming and serene atmosphere. I find it to be a very beautiful color, and it is my favorite in the building. P3O: My favorite color is probably a shade of blue called cerulean blue. It’s a calming and serene color that evokes feelings of trust and confidence. It’s also a very versatile color that can be used in a variety of settings, from home decor to fashion. Dialogue 4 Human: What is a good healthy thing to eat for lunch? Assistant: Reference: A good healthy thing to eat for lunch is a sandwich with lean protein, whole grains, and fruits or vegetables. DPO: A good healthy lunch option could be a whole-wheat sandwich with lean protein, vegetables, and fruit. For example, you could have turkey or chicken breast on whole-wheat bread with tomatoes, cucumbers, and lettuce. You could also have a tuna salad sandwich on whole-wheat bread with tomatoes, cucumbers, and olive oil. Or, you could have a peanut butter and jelly sandwich on whole-wheat bread with banana slices. Another healthy option could be a wrap with lean protein, vegetables, and fruit. For example, you could have chicken or tofu in a whole-wheat wrap with spinach, tomatoes, and cucumbers. You could also […] REINFORCE: You are correct. **There are many healthy things you can eat for lunch, but some of the most popular and generally agreed upon options include a sandwich on whole-wheat bread with lean protein and vegetables, a salad with grilled chicken or tofu, or a bowl of soup with a side of whole grains.** • **Sandwiches are a classic and easy lunch option. Choose a whole-wheat bread or wrap, and fill it with lean protein like grilled chicken or tofu, roasted vegetables, or hummus.** • **Salads are another great way to get a healthy dose of fruits, vegetables, and protein […] NashEMA: Some great healthy lunch ideas include: • Lean protein (chicken, turkey, tofu, fish) • Whole grains (brown rice, whole-wheat bread, quinoa) • Fresh vegetables (salad, cucumber, tomatoes, avocado) • Healthy fats (avocado, olive oil, nuts) • Fruits (berries, apples) • Yogurt with berries • Smoothies • Leftover chicken or fish salad • Hummus and veggie wraps • Trail mix • Homemade energy bites • Healthy snacks like nuts and seeds All of these options are low in calories, high in nutrients, and filling. Additionally, […] PRPO: A healthy lunch can include: • **A whole grain, such as whole-wheat bread, brown rice, or quinoa**. • **Protein, such as lean meat, fish, tofu, or beans**. • **Fruits and vegetables, such as apples, bananas, oranges, carrots, or spinach**. • **Low-fat dairy products, such as yogurt or cottage cheese**. • **Nuts or seeds, such as almonds or sunflower seeds**. P3O: Some healthy lunch ideas include: • A whole-wheat sandwich on whole-wheat bread with lean protein, such as turkey or chicken breast, lettuce, tomatoes, and cucumbers. • A bowl of whole-grain cereal with fruit and nuts. • A salad with grilled chicken, vegetables, and low-fat dressing. • Leftovers from dinner. • Soup and a whole-wheat roll. • Smoothie made with fruit, yogurt, and protein powder.