Paper deep dive
Sail into the Headwind: Alignment via Robust Rewards and Dynamic Labels against Reward Hacking
Paria Rashidinejad, Yuandong Tian
Models: Llama-3-8B, Mistral-7B
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 97%
Last extracted: 3/12/2026, 6:40:23 PM
Summary
The paper introduces POWER-DL, a novel preference optimization method designed to mitigate two types of reward hacking in AI alignment. Type I hacking involves subpar choices appearing favorable due to sparse data, while Type II involves decent choices appearing unfavorable. POWER-DL combines a robust reward maximization objective using Guiaşu’s weighted entropy with dynamic labels to diminish gradients for untrustworthy samples, demonstrating significant performance gains on benchmarks like AlpacaEval 2.0 and Arena-Hard.
Entities (6)
Relation Signals (3)
POWER-DL → outperforms → DPO
confidence 100% · POWER-DL consistently outperforms state-of-the-art methods... achieving improvements... over DPO
POWER-DL → mitigates → Reward Hacking
confidence 95% · Strong theoretical guarantees and empirical results demonstrate the promise of POWER-DL in mitigating reward hacking.
POWER → uses → Guiaşu’s weighted entropy
confidence 95% · POWER, a new preference optimization method that combines Guiaşu’s weighted entropy with a robust reward maximization objective.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Aligning AI systems with human preferences typically suffers from the infamous reward hacking problem, where optimization of an imperfect reward model leads to undesired behaviors. In this paper, we investigate reward hacking in offline preference optimization, which aims to improve an initial model using a preference dataset. We identify two types of reward hacking stemming from statistical fluctuations in the dataset: Type I Reward Hacking due to subpar choices appearing more favorable, and Type II Reward Hacking due to decent choices appearing less favorable. We prove that many (mainstream or theoretical) preference optimization methods suffer from both types of reward hacking. To mitigate Type I Reward Hacking, we propose POWER, a new preference optimization method that combines Guiasu's weighted entropy with a robust reward maximization objective. POWER enjoys finite-sample guarantees under general function approximation, competing with the best covered policy in the data. To mitigate Type II Reward Hacking, we analyze the learning dynamics of preference optimization and develop a novel technique that dynamically updates preference labels toward certain "stationary labels", resulting in diminishing gradients for untrustworthy samples. Empirically, POWER with dynamic labels (POWER-DL) consistently outperforms state-of-the-art methods on alignment benchmarks, achieving improvements of up to 13.0 points on AlpacaEval 2.0 and 11.5 points on Arena-Hard over DPO, while also improving or maintaining performance on downstream tasks such as mathematical reasoning. Strong theoretical guarantees and empirical results demonstrate the promise of POWER-DL in mitigating reward hacking.
Tags
Links
- Source: https://arxiv.org/abs/2412.09544
- Canonical: https://arxiv.org/abs/2412.09544
Trouble viewing inline? Open PDF directly →
Full Text
389,097 characters extracted from source content.
Expand or collapse full text
Sail into the Headwind: Alignment via Robust Rewards and Dynamic Labels against Reward Hacking Paria Rashidinejad Yuandong Tian pariard@meta.com yuandong@meta.com Fundamental AI Research (FAIR) @ Meta AI , (December 1, 2024) Abstract Aligning AI systems with human preferences typically suffers from the infamous reward hacking problem, where optimization of an imperfect reward model leads to undesired behaviors. In this paper, we investigate reward hacking in offline preference optimization, which aims to improve an initial model using a preference dataset. We identify two types of reward hacking stemming from statistical fluctuations in the dataset: Type I Reward Hacking due to subpar choices appearing more favorable, and Type I Reward Hacking due to decent choices appearing less favorable. We prove that many (mainstream or theoretical) preference optimization methods suffer from both types of reward hacking. To mitigate Type I Reward Hacking, we propose POWER, a new preference optimization method that combines Guiaşu’s weighted entropy with a robust reward maximization objective. POWER enjoys finite-sample guarantees under general function approximation, competing with the best covered policy in the data. To mitigate Type I Reward Hacking, we analyze the learning dynamics of preference optimization and develop a novel technique that dynamically updates preference labels toward certain “stationary labels”, resulting in diminishing gradients for untrustworthy samples. Empirically, POWER with dynamic labels (POWER-DL) consistently outperforms state-of-the-art methods on alignment benchmarks, achieving improvements of up to 13.0 points on AlpacaEval 2.0 and 11.5 points on Arena-Hard over DPO, while also improving or maintaining performance on downstream tasks such as mathematical reasoning. Strong theoretical guarantees and empirical results demonstrate the promise of POWER-DL in mitigating reward hacking. 1 Introduction Aligning AI systems with human values is a core problem in artificial intelligence Russell (2022). After training on vast datasets through self-supervised learning, large language models (LLMs) typically undergo an alignment phase to elicit desired behaviors aligned with human values Ouyang et al. (2022). A main alignment paradigm involves leveraging datasets of human preferences, with techniques like reinforcement learning from human feedback Christiano et al. (2017) or preference optimization Rafailov et al. (2024b). These methods learn an (implicit or explicit) reward model from human preferences, which guides the decision-making process of the AI system. This paradigm has been instrumental in today’s powerful chat models Achiam et al. (2023); Dubey et al. (2024). However, these alignment techniques are observed to suffer from the notorious reward hacking problem Amodei et al. (2016); Tien et al. (2022); Gao et al. (2023); Casper et al. (2023), where optimizing imperfect learned reward leads to poor performance under the true reward—assuming an underlying true reward exists Skalse et al. (2022). One primary cause of the discrepancy between the learned and true rewards arises because preference data do not encompass all conceivable choices, making the learned reward model vulnerable to significant statistical fluctuations in areas with sparse data. Consequently, the AI system might be swayed toward choices that only appear favorable under the learned reward but are, in reality, subpar, or the system might be deterred from truly desirable choices that do not seem favorable according to the learned rewards. In this paper, we investigate reward hacking in offline preference optimization, in which we are provided with an initial AI system (initial model) and a preference dataset. We do not assume that the preference dataset is necessarily constructed through sampling from the initial model, allowing to leverage existing datasets collected from other models. Our objective is to dissect the roots of reward hacking from an statistical standpoint, analyze current methods, and introduce theoretically sound and practically strong methods to mitigate reward hacking. Our contributions are as follows. Figure 1: (a) Example of Type I Reward Hacking. The initial model has a uniform distribution over choices (e.g., responses) while the dataset has a high coverage on the high-reward choice and low coverage on a low-reward choice. With a decent chance, the poorly-covered, low-reward choice is labeled as preferred, causing PO methods to erroneously assign a high weight to it (Proposition 1). (b) Example of Type I Reward Hacking. The initial model is aligned with the true rewards while dataset has a low coverage on the high-reward choice. With a decent chance, the poorly-covered, high-reward choice is labeled as rejected, leading to deterioration of the model post alignment (Proposition 2). Types of reward hacking. We describe two types of reward hacking in preference optimization that stem from high statistical fluctuations in regions with sparse data; see Figure 1 for an illustration. Type I Reward Hacking manifests when poorly covered, subpar choices appear more favorable than they truly are, leading the model to assign high weights to these subpar choices. Type I Reward Hacking arises when decent choices with insufficient coverage appear worse than their true value and that leads to deterioration of the initial model. While reward hacking in offline preference optimization is related to the challenge partial coverage in offline RL, the setting we consider here faces two sources of distribution shift: between the learned model and data, and between the initial model and data. This differs from offline RL, which typically considers access to an offline dataset (with possibly known data collection policy) alone Levine et al. (2020); Kumar et al. (2020); Rashidinejad et al. (2021); Xie et al. (2021); Zhu et al. (2023) and thus is concerned with a single source of distribution shift. The existence of the two sources of distribution shift motivates us to describe the two types of reward hacking, which motivates the designs of new algorithms robust to reward hacking. Preference optimization methods provably suffer from reward hacking. We prove that several theoretical and mainstream preference optimization methods suffer from both types of reward hacking (Propositions 1 and 2). A common countermeasure against reward hacking is keeping the learned model close to the initial model through minimization of divergence measures Rafailov et al. (2024b); Azar et al. (2024); Huang et al. (2024). Yet, our analysis reveals that divergence minimization does not induce sufficient pessimism to prevent Type I Reward Hacking, nor does it mitigate deterioration of the initial model caused by Type I Reward Hacking. Notably, reward hacking can occur even when divergence from the initial model is small. POWER-DL: Against Type I and Type I Reward Hacking. To mitigate reward hacking, we integrate a robust reward maximization framework with Guiaşu’s weighted entropy Guiaşu (1971). We transform this objective into a single-step optimization problem (Proposition 3) leading to Preference Optimization via Weighted Entropy Robust Rewards (POWER). We prove that POWER enjoys finite-sample guarantees for general function approximation, improving over the best covered policy and mitigating Type I Reward Hacking (Theorem 1). Due to the weighted entropy, POWER effectively learns from well-covered choices in the dataset, even those with a large divergence against the initial model, countering potential underoptimization in divergence-based methods. We next develop dynamic labels to mitigate Type I Reward Hacking, whereby preference labels are updated in a way that diminishes gradients for untrustworthy data (Theorem 2). Our final algorithm combines POWER with Dynamic Labels (POWER-DL), which interpolates robust rewards with maintaining closeness to the initial model, allowing to trade off between reward hacking types. POWER-DL consistently outperforms other methods across various settings. For aligning LLMs, we implement POWER-DL and compare it against other preference optimization methods across different datasets and two scenarios: one using an existing preference dataset and another with preference data generated through sampling from the initial model. POWER-DL consistently outperforms state-of-the-art methods in alignment benchmarks, achieving improvements over DPO of up to 13.0 points on AlpacaEval 2.0 and 11.5 points on Arena-Hard. Additionally, POWER-DL improves or maintains performance on downstream tasks such as truthfulness, mathematical reasoning, and instruction-following, demonstrating robustness against reward hacking and achieving a more favorable bias-variance trade-off compared to other methods. Figure 2 provides comparison with two representative baselines DPO Rafailov et al. (2024b) and SimPO Meng et al. (2024) on alignment benchmarks Alpaca-Eval 2.0 and Arena-Hard as well as performance on mathematical reasoning benchmark GSM8K. Figure 2: Performance of POWER-DL compared to DPO and SimPO. POWER-DL outperforms DPO and SimPO in alignment benchmarks AlpacaEval 2.0 and Arena-Hard across pipelines with different dataset sizes and levels of distribution shift between data and the initial model. In downstream mathematical reasoning task GSM8K, POWER-DL consistently maintains or improves mathematical reasoning performance while the performance of models trained with DPO and SimPO can drop significantly in some cases. 2 Background and Problem Formulation 2.1 Learning from Human Preference Contextual bandit formulation. We adopt the contextual bandits formulation described by a tuple (,,r)(X,Y,r)( X , Y , r ), where XX is the space of contexts (e.g., prompts), YY is the space of actions (e.g., responses), and r:×→ℝ:→ℝr:X×Y : X × Y → ℝ is a scalar reward function. A stochastic policy (e.g., model or language model) π:→Δ():→Δπ:X→ (Y)π : X → Δ ( Y ) takes in a context x∈x ∈ X and outputs an action according to y∼π(⋅|x)y π(·|x)y ∼ π ( ⋅ | x ). We denote the set of all stochastic policies by Π≔π:→Δ()≔Πconditional-set→Δ \π:X→ (Y)\Π ≔ π : X → Δ ( Y ) . Performance metric. We assume that there exists an underlying (unknown) true reward function r⋆:×→ℝ:superscript⋆→ℝr :X×Y ⋆ : X × Y → ℝ. Given the true reward function r⋆superscript⋆r r⋆ and a target distribution over contexts x∼ρ(⋅)similar-to⋅x ρ(·)x ∼ ρ ( ⋅ ), performance of a policy π is the expected true reward over contexts and actions J(π)≔x∼ρ,y∼π(⋅|x)[r⋆(x,y)]. J(π) _x ρ,y π(·|x) [r^% (x,y) ].J ( π ) ≔ italic_x ∼ ρ , y ∼ π ( ⋅ | x ) [ r⋆ ( x , y ) ] . (1) The Bradley-Terry model of human preferences. Consider a prompt x∈x ∈ X and a pair of responses y0,y1∈superscript0superscript1y^0,y^1 0 , y1 ∈ Y. For any reward function r, the Bradley-Terry (BT) model characterizes the probability of preferring y1superscript1y^1y1 over y0superscript0y^0y0, denoted by l=11l=1l = 1, according to: ℙr(l=1∣x,y1,y0)=σ(r(x,y1)−r(x,y0)),subscriptℙconditional1superscript1superscript0superscript1superscript0 P_r(l=1 x,y^1,y^0)=σ (% r(x,y^1)-r(x,y^0) ),ℙitalic_r ( l = 1 ∣ x , y1 , y0 ) = σ ( r ( x , y1 ) - r ( x , y0 ) ) , (2) where σ(z)≔1/(1+exp(−z))≔11σ(z) 1/(1+ (-z))σ ( z ) ≔ 1 / ( 1 + exp ( - z ) ) is the sigmoid function. Offline preference optimization. We consider an offline learning setup, where we start from an initial reference policy (model), denoted by πθ0=πrefsubscriptsubscript0subscriptref _ _0= _refπitalic_θ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT = πref, and an offline pairwise preference dataset =(x,y0,y1,l)superscript0superscript1D=\(x,y^0,y^1,l)\D = ( x , y0 , y1 , l ) , comprising of N iid samples. Prompt and response pairs are sampled according to a data distribution: x,y0,y1∼μsimilar-tosuperscript0superscript1x,y^0,y^1 , y0 , y1 ∼ μ, and preferences label is sampled according to the BT model corresponding to true rewards: l∼ℙr⋆(⋅|x,y1,y0)l P_r (·|x,y^1,y^0)l ∼ ℙitalic_r⋆ ( ⋅ | x , y1 , y0 ). Importantly, we do not assume that the preference dataset is necessarily constructed through sampling from the initial model. To simplify notation, we define y+=ly1+(1−l)y0superscriptsuperscript11superscript0y^+=ly^1+(1-l)y^0y+ = l y1 + ( 1 - l ) y0 and y−=(1−l)y1+ly0superscript1superscript1superscript0y^-=(1-l)y^1+ly^0y- = ( 1 - l ) y1 + l y0 to denote the chosen and rejected responses in the dataset, respectively. Appendix A presents additional notation. 2.2 Direct Preference Optimization A classical approach to learning from human preferences involves learning a reward model from dataset, followed by finding a policy through maximizing the learned reward typically regularized with a (reverse) KL-divergence to keep the learned policy closed to initial policy: r^∈argminrLBT(r)≔−[logσ(r(x,y+)−r(x,y−))]π^∈argmaxπx∼ρ,y∼π[r^(x,y)]−βDKL[π∥πref],^subscriptargminsubscriptBT≔subscriptdelimited-[]superscriptsuperscript^subscriptargmaxsubscriptformulae-sequencesimilar-tosimilar-todelimited-[]^subscriptKLdelimited-[]conditionalsubscriptref split r∈& *argmin_rL_BT(r% ) -E_D [ σ (r(x,y^+)-r(x,y^-)% ) ]\\ π∈& *argmax_πE_x ρ,y π[ % r(x,y)]-β D_KL[π\| _ref], splitstart_ROW start_CELL over start_ARG r end_ARG ∈ end_CELL start_CELL argminitalic_r LBT ( r ) ≔ - caligraphic_D [ log σ ( r ( x , y+ ) - r ( x , y- ) ) ] end_CELL end_ROW start_ROW start_CELL over start_ARG π end_ARG ∈ end_CELL start_CELL argmaxitalic_π italic_x ∼ ρ , y ∼ π [ over start_ARG r end_ARG ( x , y ) ] - β DKL [ π ∥ πref ] , end_CELL end_ROW (3) Here, DKL[π∥πref]≔x∼ρ[DKL[π(⋅|x)∥πref(⋅|x)]]D_KL[π\| _ref] _x ρ [D_% KL[π(·|x)\| _ref(·|x)] ]DKL [ π ∥ πref ] ≔ italic_x ∼ ρ [ DKL [ π ( ⋅ | x ) ∥ πref ( ⋅ | x ) ] ] and LBT(r)subscriptBTL_BT(r)LBT ( r ) is the negative log-likelihood according to the BT model. Rafailov et al. (2024b) observed that the policy maximization step in (3) can be computed in closed form and thus simplified the two-step process into a single minimization objective. This method is called direct preference optimization (DPO) and has inspired a series of works; see Tables 1 and 3 for several examples. Some representative variants of DPO that we theoretically analyze are IPO Azar et al. (2024), which applies a nonlinear transformation to preferences to reduce overfitting, and SimPO Meng et al. (2024), which removes the reference policy from the DPO objective. We also analyze two recent theoretical methods that come with finite-sample guarantees and aim at mitigating overoptimization: χPO Huang et al. (2024), which replaces the KL divergence in DPO with a stronger χ2superscript2χ^2χ2+KL divergence, and DPO+SFT Liu et al. (2024); Cen et al. (2024), which adds a supervised finetuning term that increases log-likelihood of chosen responses in the preference dataset. Table 1: Preference optimization objectives given data =(x,y+,y−)superscriptsuperscriptD=\(x,y^+,y^-)\D = ( x , y+ , y- ) and initial model πrefsubscriptref _refπref. Method Objective DPO Rafailov et al. (2024b) π^DPO∈argminθ−[logσ(β(logπθ(y+|x)πref(y+|x)−logπθ(y−|x)πref(y−|x)))]subscript^DPOsubscriptargminsubscriptdelimited-[]subscriptconditionalsuperscriptsubscriptrefconditionalsuperscriptsubscriptconditionalsuperscriptsubscriptrefconditionalsuperscript π_DPO∈ *argmin_θ-E_% D [ σ (β ( _θ(y^+|x) _% ref(y^+|x)- _θ(y^-|x) _ref(y^-|% x) ) ) ]over start_ARG π end_ARGDPO ∈ argminitalic_θ - caligraphic_D [ log σ ( β ( log divide start_ARG πitalic_θ ( y+ | x ) end_ARG start_ARG πref ( y+ | x ) end_ARG - log divide start_ARG πitalic_θ ( y- | x ) end_ARG start_ARG πref ( y- | x ) end_ARG ) ) ] DPO+SFT Liu et al. (2024) π^DPO+SFT∈argminθ−[logσ(β(logπθ(y+|x)πref(y+|x)−logπθ(y−|x)πref(y−|x)))]−[logπθ(y+|x)]subscript^DPO+SFTsubscriptargminsubscriptdelimited-[]subscriptconditionalsuperscriptsubscriptrefconditionalsuperscriptsubscriptconditionalsuperscriptsubscriptrefconditionalsuperscriptsubscriptdelimited-[]subscriptconditionalsuperscript π_DPO+SFT∈ *argmin_θ-E_% D [ σ (β ( _θ(y^+|x)% _ref(y^+|x)- _θ(y^-|x) _ref(y% ^-|x) ) ) ]-E_D [ _θ(y% ^+|x) ]over start_ARG π end_ARGDPO+SFT ∈ argminitalic_θ - caligraphic_D [ log σ ( β ( log divide start_ARG πitalic_θ ( y+ | x ) end_ARG start_ARG πref ( y+ | x ) end_ARG - log divide start_ARG πitalic_θ ( y- | x ) end_ARG start_ARG πref ( y- | x ) end_ARG ) ) ] - caligraphic_D [ log πitalic_θ ( y+ | x ) ] IPO Azar et al. (2024) π^IPO∈argminθ[(logπθ(y+|x)πref(y+|x)−logπθ(y−|x)πref(y−|x)−12τ)2]subscript^IPOsubscriptargminsubscriptdelimited-[]superscriptsubscriptconditionalsuperscriptsubscriptrefconditionalsuperscriptsubscriptconditionalsuperscriptsubscriptrefconditionalsuperscript122 π_IPO∈ *argmin_θE_D% [ ( _θ(y^+|x) _ref(y^+|x)- % _θ(y^-|x) _ref(y^-|x)- 12τ )% ^2 ]over start_ARG π end_ARGIPO ∈ argminitalic_θ caligraphic_D [ ( log divide start_ARG πitalic_θ ( y+ | x ) end_ARG start_ARG πref ( y+ | x ) end_ARG - log divide start_ARG πitalic_θ ( y- | x ) end_ARG start_ARG πref ( y- | x ) end_ARG - divide start_ARG 1 end_ARG start_ARG 2 τ end_ARG )2 ] SimPO Meng et al. (2024) π^SimPO∈argminθ−[logσ(β(1|y+|logπθ(y+|x)−1|y−|logπθ(y−|x))−γ)]subscript^SimPOsubscriptargminsubscriptdelimited-[]1superscriptsubscriptconditionalsuperscript1superscriptsubscriptconditionalsuperscript π_SimPO∈ *argmin_θ-E_% D [ σ (β ( 1|y^+| _θ% (y^+|x)- 1|y^-| _θ(y^-|x) )-γ ) ]over start_ARG π end_ARGSimPO ∈ argminitalic_θ - caligraphic_D [ log σ ( β ( divide start_ARG 1 end_ARG start_ARG | y+ | end_ARG log πitalic_θ ( y+ | x ) - divide start_ARG 1 end_ARG start_ARG | y- | end_ARG log πitalic_θ ( y- | x ) ) - γ ) ] χPO Huang et al. (2024) π^χPO∈argminθ−[logσ(2R[β(ϕ(πθ(y+|x)πref(y+|x))−ϕ(πθ(y−|x)πref(y−|x)))])]subscript^POsubscriptargminsubscriptdelimited-[]subscript2delimited-[]italic-ϕsubscriptconditionalsuperscriptsubscriptrefconditionalsuperscriptitalic-ϕsubscriptconditionalsuperscriptsubscriptrefconditionalsuperscript π_ ∈ *argmin_θ-E_% D [ σ ( clip_2R [β (φ % ( _θ(y^+|x) _ref(y^+|x) )-φ (% _θ(y^-|x) _ref(y^-|x) ) ) ]% ) ]over start_ARG π end_ARGχ PO ∈ argminitalic_θ - caligraphic_D [ log σ ( sansserif_clip2 R [ β ( ϕ ( divide start_ARG πitalic_θ ( y+ | x ) end_ARG start_ARG πref ( y+ | x ) end_ARG ) - ϕ ( divide start_ARG πitalic_θ ( y- | x ) end_ARG start_ARG πref ( y- | x ) end_ARG ) ) ] ) ]; ϕ(z)≔z+log(z)≔italic-ϕ\;φ(z) z+ (z)ϕ ( z ) ≔ z + log ( z ) 3 Reward Hacking in Preference Optimization In this section, we investigate reward hacking in preference optimization. One driver of reward hacking is statistical errors present in the dataset. Typically, preference datasets suffer from partial coverage, lacking extensive samples across all possible options. As a result, preferences for poorly covered choices are subject to high levels of statistical fluctuations, given the fact that preference labels are Bernoulli random variables with probabilities described by the Bradley-Terry model (2). Subsequently, we describe two types of reward hacking, both originating from the presence of poorly covered choices (actions) in the dataset. 3.1 Type I Reward Hacking Type I Reward Hacking occurs when poorly covered, subpar choices in the dataset appear more favorable due to statistical errors, and that leads to a learned policy π^ πover start_ARG π end_ARG with a low expected true reward J(π^)^J( π)J ( over start_ARG π end_ARG ). In the following proposition, we prove that even in the favorable scenario that the high-reward actions are well-covered in the dataset, the existence of a single sample on a low-reward action can overwhelm many preference optimization algorithms, causing them to learning highly suboptimal policies. Proposition 1 (Type I Reward Hacking in ⋆ ⋆PO). Consider multi-armed bandits with bounded rewards r⋆(a)∈[0,1]superscript⋆01r (a)∈[0,1]r⋆ ( a ) ∈ [ 0 , 1 ] and the softmax policy class, defined as Πθ≔πθ(y)=exp(θ(y))/Zθ|Zθ=∑yexp(θ(y)),θ(y)∈[0,1].≔subscriptΠconditional-setsubscriptsubscriptformulae-sequencesubscriptsubscript01 _θ \ _θ(y)= (θ(y))/Z% _θ |Z_θ= _y (θ(y)),θ(y)∈[0,1] \.Πitalic_θ ≔ πitalic_θ ( y ) = exp ( θ ( y ) ) / Zitalic_θ | Zitalic_θ = ∑y exp ( θ ( y ) ) , θ ( y ) ∈ [ 0 , 1 ] . (4) Define the best-in-class policy πθ⋆=maxπ∈ΠθJ(π)subscriptsuperscript⋆subscriptsubscriptΠ _θ = _π∈ _θJ(π)πitalic_θ⋆ = maxitalic_π ∈ Π start_POSTSUBSCRIPT θ end_POSTSUBSCRIPT J ( π ). There exist three-armed bandit instances with ΠθsubscriptΠ _θΠitalic_θ parameterization, high coverage of the optimal arms μ(a∈argmaxar⋆(a))>1/2subscriptsuperscript⋆12μ(a∈ _ar (a))>1/2μ ( a ∈ arg maxitalic_a r⋆ ( a ) ) > 1 / 2, and bounded KL-divergence DKL(πθ⋆∣πref)subscriptKLconditionalsubscriptsuperscript⋆subscriptrefD_KL( _θ _ref)DKL ( πitalic_θ⋆ ∣ πref ), such that for any N≥22N≥ 2N ≥ 2, β>00β>0β > 0, γ, τ>00τ>0τ > 0, policy π^∈π^DPO,π^IPO,π^SimPO^subscript^DPOsubscript^IPOsubscript^SimPO π∈\ π_DPO, π_IPO, π_% SimPO\over start_ARG π end_ARG ∈ over start_ARG π end_ARGDPO , over start_ARG π end_ARGIPO , over start_ARG π end_ARGSimPO or π^=π^χPO^subscript^PO π= π_ over start_ARG π end_ARG = over start_ARG π end_ARGχ PO for 0<β≤1/30130<β≤ 1/30 < β ≤ 1 / 3, suffers from a constant suboptimality J(πθ⋆)−J(π^)>0.15subscriptsuperscript⋆^0.15J( _θ )-J( π)>0.15J ( πitalic_θ⋆ ) - J ( over start_ARG π end_ARG ) > 0.15 with a constant probability of at least (e(1+e))−1superscript11(e(1+e))^-1( e ( 1 + e ) )- 1. We defer the proof of the above proposition to Appendix C.1 and offer some intuition here. Figure 1(a) illustrates a failure instance, where the preference data has high coverage over the high-reward choice A but poor coverage on the low-reward choice C. Due to the Bradley-Terry model and the stochastic nature of human preferences, regions with poor coverage are prone to high statistical errors. Consequently, the low-reward choice C might be marked as preferred purely by chance.111For example, if the reward gap between choices C and C is one, the probability of preferring C over A is approximately 27%percent2727\%27 % according to the BT model. Proposition 1 demonstrates that algorithms such as DPO and SimPO overfit the untrustworthy preferences, as their objectives aim at increasing the parameter gap θ(C)−θ(A)θ(C)-θ(A)θ ( C ) - θ ( A ), despite the preference being untrustworthy due to inadequate coverage. This can ultimately lead to a final policy that places significant weight on poor choices. Type I Reward Hacking and the failure result in Proposition 1 are closely connected to the challenge of partial data coverage in offline RL Levine et al. (2020), which can be robustly addressed through the principle of pessimism in the face of uncertainty. Pessimism can be applied in various ways such as reducing the rewards (values) of poorly covered actions Kumar et al. (2020); Cheng et al. (2022) or keeping the learned policy close to data collection policy Nachum et al. (2019). Although divergence-based methods DPO, IPO, and χPO aim at keeping the learned policy close to the initial policy, Proposition 1 shows that maintaining a small divergence from initial model does not induce a sufficient amount of pessimism to prevent Type I Reward Hacking.222Proposition 1 does not contradict guarantees of Huang et al. (2024) as this work assumes that preference data are collected from the initial policy. However, this assumption is restrictive, as it prevents using existing preference datasets collected from other models, which is a common approach in practical pipelines such as Wang et al. (2024e) and Tunstall et al. (2023). Remark 1 (Comparison with previous theoretical results on failure of DPO). Failure result in Proposition 1 is rigorous and constructed under a realistic setting close to practice: the policy class is a softmax with bounded rewards and the KL divergence between initial and best-in-class policy is bounded. This makes Proposition 1 stronger than prior arguments on overoptimization in DPO, which rely on unbounded rewards Azar et al. (2024), updates to model parameters despite receiving no samples (hence, conclusion breaking in gradient-based optimization) Huang et al. (2024), or events with probabilities approaching zero Song et al. (2024). 3.2 Type I Reward Hacking Type I Reward Hacking can occur when poorly covered, good choices in the dataset appear to be less favorable than their true value due to statistical errors, leading to the deterioration of the initial model after preference optimization. In the following proposition, we prove that many preference optimization methods are susceptible to Type I Reward Hacking. Proposition 2 (Type I Reward Hacking in ⋆ ⋆PO). Consider the multi-armed bandits setting with the softmax policy class ΠθsubscriptΠ _θΠitalic_θ, as defined in (4). Let πθ⋆=maxπ∈ΠθJ(π)subscriptsuperscript⋆subscriptsubscriptΠ _θ = _π∈ _θJ(π)πitalic_θ⋆ = maxitalic_π ∈ Π start_POSTSUBSCRIPT θ end_POSTSUBSCRIPT J ( π ) represent the best-in-class policy. There exists a three-armed bandit problem with ΠθsubscriptΠ _θΠitalic_θ parameterization and πθ0=πθ⋆subscriptsubscript0subscriptsuperscript⋆ _ _0= _θ πitalic_θ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT = πitalic_θ⋆, such that for any N≥33N≥ 3N ≥ 3, β>0,η≥0,γformulae-sequence00β>0,η≥ 0,γβ > 0 , η ≥ 0 , γ and policy π^∈π^DPO,π^DPO+SFT,π^SimPO^subscript^DPOsubscript^DPO+SFTsubscript^SimPO π∈\ π_DPO, π_DPO+SFT, π_% SimPO\over start_ARG π end_ARG ∈ over start_ARG π end_ARGDPO , over start_ARG π end_ARGDPO+SFT , over start_ARG π end_ARGSimPO or π^∈π^χPO,π^IPO^subscript^POsubscript^IPO π∈\ π_ , π_IPO\over start_ARG π end_ARG ∈ over start_ARG π end_ARGχ PO , over start_ARG π end_ARGIPO for 0<β,τ≤1formulae-sequence010<β,τ≤ 10 < β , τ ≤ 1, the following holds with a constant probability of at least (e(1+e))−1superscript11(e(1+e))^-1( e ( 1 + e ) )- 1: J(πθ⋆)−J(π^)>0.1.subscriptsuperscript⋆^0.1 J( _θ )-J( π)>0.1.J ( πitalic_θ⋆ ) - J ( over start_ARG π end_ARG ) > 0.1 . Proof of the above proposition can be found in Appendix C.2. An example of this type of reward hacking is illustrated in Figure 1(b), where the initial policy has a high probability on the high-reward choice C. Yet, due to its low coverage, C can appear unfavorable simply by chance. In such a scenario, the preference optimization methods analyzed in Proposition 2 drastically reduce the likelihood of the high-reward choice C from its initial likelihood. Proposition 2 states that even with a strong initial model, a preference dataset that poorly covers high-reward actions can lead to substantial deterioration of the initial model in existing approaches, even in methods such as DPO, IPO, and χPO that incorporate divergence-minimization. We note that the above setting is beyond the guarantees of traditional pessimistic offline RL, as these techniques typically do not consider access to an initial model and guarantee competing with the best covered policy in the data. Despite this, as we see in Section 5, there may be hope to mitigate degradation of the initial model and better control the trade-off between Type I and Type I Reward Hacking. 4 Against Type I Reward Hacking: Weighted Entropy Robust Rewards 4.1 Weighted Entropy Reward Maximization We demonstrated that approaches involving divergence minimization remain vulnerable to reward hacking. Moreover, maintaining a small divergence can inadvertently lead to underoptimizing the preference dataset, as it may risk overlooking policies that, although well-covered, deviate significantly from the initial policy.333Simply reducing β to alleviate underoptimization may not always be viable. For example, reducing β may reduce underoptimization in one state while inadvertently amplify overoptimization in another state. These reasons motivate us to explore an alternative route and consider regularizing the reward maximization objective with the concept of weighted entropy. Definition 1 (Weighted Entropy; Guiaşu (1971)). The weighted entropy of a (discrete) distribution p(⋅)⋅p(·)p ( ⋅ ) with non-negative weights w(y)w(y)w ( y ) is defined as Hw(p)≔−∑yw(y)p(y)logp(y).≔subscriptsubscriptH_w(p) - _yw(y)p(y) p(y).Hitalic_w ( p ) ≔ - ∑y w ( y ) p ( y ) log p ( y ) . Weighted entropy extends Shannon’s entropy by incorporating weights associated with each outcome, reflecting attributes such as favorableness or utility toward a specific goal Guiaşu (1971). Building on this, we consider a weighted-entropy reward (WER) maximization objective: maxπ∈Πx∼ρ,y∼π[r(x,y)]+βHw(π),subscriptΠsubscriptformulae-sequencesimilar-tosimilar-todelimited-[]subscript _π∈ E_x ρ,y π [r(x,y) % ]+β H_w(π),maxitalic_π ∈ Π italic_x ∼ ρ , y ∼ π [ r ( x , y ) ] + β Hitalic_w ( π ) , (5) where Hw(π)≔x∼ρ[Hw(π(⋅|x))]H_w(π) _x ρ[H_w(π(·|x))]Hitalic_w ( π ) ≔ italic_x ∼ ρ [ Hitalic_w ( π ( ⋅ | x ) ) ]. This objective expresses the principle of maximum (weighted) entropy Jaynes (1957); Guiasu and Shenitzer (1985), promoting the selection of policies with maximum entropy—thus favoring the most uniform or unbiased policies—among those compatible with the constraints, such as achieving high rewards. Objective (5) extends the well-established maximum entropy framework in RL, used in various settings such as exploration Haarnoja et al. (2018), inverse RL Ziebart et al. (2008), and robust RL Eysenbach and Levine (2022). 4.2 POWER: Preference Optimization with Weighted Entropy Robust Rewards To mitigate reward hacking, we integrate the WER objective (5) with a robust (adversarial) reward framework, inspired by favorable theoretical guarantees Liu et al. (2024); Cen et al. (2024); Fisch et al. (2024). Specifically, we find a policy that maximizes WER (5) against an adversarial reward, which seeks to minimize WER while fitting the preference dataset: maxπ∈Πminr∈ℛLBT(r)⏟negative log-likelihood+η(x∼ρ,y∼π[r(x,y)]−x∼ρ,y′∼π′[r(x,y′)]+βHw(π))⏟WER minus baseline,subscriptΠsubscriptℛsubscript⏟subscriptBTnegative log-likelihoodsubscript⏟subscriptformulae-sequencesimilar-tosimilar-todelimited-[]subscriptformulae-sequencesimilar-tosimilar-tosuperscript′delimited-[]superscript′subscriptWER minus baseline _π∈ _r L_BT(r% )_negative log-likelihood+η (E_x % ρ,y π [r(x,y) ]-E_x ρ,y π^% [r(x,y ) ]+β H_w(π) )_WER minus% baseline,maxitalic_π ∈ Π minitalic_r ∈ R under⏟ start_ARG LBT ( r ) end_ARGnegative log-likelihood + η under⏟ start_ARG ( italic_x ∼ ρ , y ∼ π [ r ( x , y ) ] - italic_x ∼ ρ , y′ ∼ π′ [ r ( x , y′ ) ] + β Hitalic_w ( π ) ) end_ARGWER minus baseline , (6) where η≥00η≥ 0η ≥ 0, ℛRR is a reward function class, and LBT(r)subscriptBTL_BT(r)LBT ( r ) is the negative log-likelihood of the BT model. We subtracted a baseline x∼ρ,y′∼π′[r(x,y′)]subscriptformulae-sequencesimilar-tosimilar-tosuperscript′delimited-[]superscript′E_x ρ,y π [r(x,y ) ]italic_x ∼ ρ , y′ ∼ π′ [ r ( x , y′ ) ] that computes the average reward with respect to some policy π′π π′, as the preference data under the BT model only reveal information on reward differences Zhan et al. (2023a). This baseline plays a crucial role in simplifying the objective and establishing finite-sample guarantees, and we subsequently discuss reasonable choices for π′π π′. Under some regularity conditions (detailed in Appendix D.2), the objective (6) can be equivalently expressed as a minimax problem, which leads to a single-step preference objective presented in the proposition below; see Appendix D.3 for the derivation and proof. Proposition 3 (POWER Objective). Let w(y)>00w(y)>0w ( y ) > 0 denote the weights in the weighted entropy Hw(π)subscriptH_w(π)Hitalic_w ( π ) and πrsubscript _rπitalic_r denote the policy that maximizes the objective (5). Under certain regularity conditions on ℛRR (Assumption 1) and for any β>00β>0β > 0, solving the maximin objective (6) is equivalent to solving the following optimization problem: minr∈ℛLBT(β(w(y)logπr(y|x)+w(y)))−ηx∼ρ,y′∼π′[βw(y′)logπr(y′|x)].subscriptℛsubscriptBTsubscriptconditionalsubscriptformulae-sequencesimilar-tosimilar-tosuperscript′delimited-[]superscript′subscriptconditionalsuperscript′ split _r L_BT (β % (w(y) _r(y|x)+w(y) ) )- _x ρ,y % π [βw(y ) _r(y |x) ].% splitstart_ROW start_CELL minitalic_r ∈ R LBT ( β ( w ( y ) log πitalic_r ( y | x ) + w ( y ) ) ) - η italic_x ∼ ρ , y′ ∼ π′ [ β w ( y′ ) log πitalic_r ( y′ | x ) ] . end_CELL end_ROW (7) We call the above objective preference optimization via weighted entropy robust rewards, or POWER. The first term in the above objective is the Bradley-Terry loss with rewards set to w(y)logπr(y|x)+w(y)subscriptconditionalw(y) _r(y|x)+w(y)w ( y ) log πitalic_r ( y | x ) + w ( y ), resulting in a reward gap expressed as a weighted difference of log probabilities of chosen and rejected responses. The second expectation is a weighted negative log-likelihood (a.k.a. supervised fine-tuning, or SFT) regularizer over the baseline policy π′π π′. Remark 2. Liu et al. (2024) propose an adversarial objective similar to (6) that uses KL divergence instead of weighted entropy. From a theoretical perspective, our approach with weighted entropy improves over this work, such as through mitigating underoptimization; see Section 4.3 for details. Moreover, our final algorithm presented in Algorithm 1 is considerably different from the DPO+SFT objective Liu et al. (2024); Pal et al. (2024) and significantly outperforms empirically as shown in Section 6. 4.3 Finite-Sample Guarantees and Theoretical Benefits of POWER The following theorem shows that POWER enjoys finite-sample guarantees on performance. Theorem 1 (Finite-Sample Performance Guarantees of POWER). Given a competing policy π∈Ππ∈ π ∈ Π, assume a bounded concentrability coefficient Cμπ(ℛ,π′)<∞subscriptsuperscriptℛsuperscript′C^π_μ(R,π )<∞Citalic_πitalic_μ ( R , π′ ) < ∞ as defined in Definition 2. Furthermore, assume realizability of the true reward function r⋆∈ℛsuperscript⋆ℛr ⋆ ∈ R, boundedness of rewards ∀r∈ℛ:r(x,y)∈[0,R]:for-allℛ0∀ r :r(x,y)∈[0,R]∀ r ∈ R : r ( x , y ) ∈ [ 0 , R ], and that the reward function class has a finite ϵitalic-ϵεϵ-covering number ϵsubscriptitalic-ϵN_εNitalic_ϵ under the infinity norm. Define R~≔1+exp(R),ϵ≍(R~N)−1formulae-sequence≔~1asymptotically-equalsitalic-ϵsuperscript~1 R 1+ (R),ε ( RN)^-1over~ start_ARG R end_ARG ≔ 1 + exp ( R ) , ϵ ≍ ( over~ start_ARG R end_ARG N )- 1, ι=log(ϵ)/δsubscriptitalic-ϵ = (N_ε)/δι = square-root start_ARG log ( Nitalic_ϵ ) / δ end_ARG. Set η≍ι/(R~2N)asymptotically-equalssuperscript~2η /( R^2 N)η ≍ ι / ( over~ start_ARG R end_ARG2 square-root start_ARG N end_ARG ) and β=1/N1β=1/ Nβ = 1 / square-root start_ARG N end_ARG in the objective (6). Then with a probability of at least 1−δ11- 1 - δ, policy π^POWERsubscript^POWER π_POWERover start_ARG π end_ARGPOWER that solves (6) satisfies the following J(π^POWER)≳J(π)−1N(([Cμπ(ℛ,π′)]2+1)R~2ι+Hw(π)).greater-than-or-equivalent-tosubscript^POWER1superscriptdelimited-[]subscriptsuperscriptℛsuperscript′21superscript~2subscript J( π_POWER) J(π)- 1 N% ( ( [C^π_μ(R,π ) ]^2+1 )% R^2 +H_w(π) ).J ( over start_ARG π end_ARGPOWER ) ≳ J ( π ) - divide start_ARG 1 end_ARG start_ARG square-root start_ARG N end_ARG end_ARG ( ( [ Citalic_πitalic_μ ( R , π′ ) ]2 + 1 ) over~ start_ARG R end_ARG2 ι + Hitalic_w ( π ) ) . Furthermore, let L denote the maximum response length. Selecting w(y)=1/|y|1w(y)=1/|y|w ( y ) = 1 / | y | to be the inverse response length, one has J(π^POWER)≳J(π)−1N(([Cμπ(ℛ,π′)]2+1)R~2ι+log(L||)).greater-than-or-equivalent-tosubscript^POWER1superscriptdelimited-[]subscriptsuperscriptℛsuperscript′21superscript~2 J( π_POWER) J(π)- 1 N% ( ( [C^π_μ(R,π ) ]^2+1 )% R^2 + (L|V|) ).J ( over start_ARG π end_ARGPOWER ) ≳ J ( π ) - divide start_ARG 1 end_ARG start_ARG square-root start_ARG N end_ARG end_ARG ( ( [ Citalic_πitalic_μ ( R , π′ ) ]2 + 1 ) over~ start_ARG R end_ARG2 ι + log ( L | V | ) ) . Proof of the above theorem can be found in Appendix E.2. Below, we discuss the implications of the above theorem and the guidelines it offers for practical choices. Guarantees against Type I Reward Hacking. Theorem 1 shows that the policy learned by POWER competes with the best policy covered in the dataset, where the notion of coverage is characterized by single-policy concentrability, considered the gold standard in offline RL Rashidinejad et al. (2021); Xie et al. (2021); Zhan et al. (2023a). This implies that as long as a favorable policy is covered in the preference data, POWER is robust to existence of poorly covered, subpar policies and thus mitigates Type I Reward Hacking. Moreover, as Theorem 1 does not impose a parametric form on the class ℛRR, guarantees hold for general function classes. Benefits of weighted entropy and choice of weights. A non-zero weighted entropy term (β>0)0(β>0)( β > 0 ) is essential in obtaining the one-step optimization problem in (7) and establishing its equivalence to the maximin problem, as this term induces strict concavity in the objective (5). Moreover, using KL-regularization leads to rates that grow with the divergence of competing policy and initial policy Liu et al. (2024), which can be large or even unbounded. However, weighted entropy ensures bounded rates and thus mitigates underoptimization. Theorem 1 suggests that a particularly appealing choice for weights is the inverse response length w(y)=1/|y|1w(y)=1/|y|w ( y ) = 1 / | y |, which intuitively discourages learning policies that generate lengthy responses. Theoretically, while using Shannon entropy (w(y)=11w(y)=1w ( y ) = 1) results in a convergence rate that grows linearly with response length, with weights w(y)=1/|y|1w(y)=1/|y|w ( y ) = 1 / | y | convergence rate only depends on the logarithm of the vocabulary (token) size and logarithm of response length. Other choices for weights include response preference scores and per-sample importance weights. Choice of the baseline policy. The rate in Theorem 1 is influenced by the concentrability coefficient Cμπ(ℛ,π′)subscriptsuperscriptℛsuperscript′C^π_μ(R,π )Citalic_πitalic_μ ( R , π′ ), which is impacted by the baseline policy π′π π′. Inspecting the definition of concentrability coefficient in Definition 2, a reasonable choice to make this coefficient small is selecting the distribution of chosen responses in the dataset Zhan et al. (2023a); Liu et al. (2024). With the above choices applied to the objective (7), a practically appealing version of POWER becomes: maxθ[logσ[β[logπθ(y+|x)|y+|−logπθ(y−|x)|y−|+1|y+|−1|y−|]]]+ηβ[logπθ(y+|x)|y+|]subscriptsubscriptdelimited-[]delimited-[]delimited-[]subscriptconditionalsuperscriptsuperscriptsubscriptconditionalsuperscriptsuperscript1superscript1superscriptsubscriptdelimited-[]subscriptconditionalsuperscriptsuperscript _θE_D [ σ [% β [ _θ(y^+|x)|y^+|- _θ(y^% -|x)|y^-|+ 1|y^+|- 1|y^-| ] ] ]+% η _D [ _θ(y^+|x)|y^+% | ]maxitalic_θ caligraphic_D [ log σ [ β [ divide start_ARG log πitalic_θ ( y+ | x ) end_ARG start_ARG | y+ | end_ARG - divide start_ARG log πitalic_θ ( y- | x ) end_ARG start_ARG | y- | end_ARG + divide start_ARG 1 end_ARG start_ARG | y+ | end_ARG - divide start_ARG 1 end_ARG start_ARG | y- | end_ARG ] ] ] + η β caligraphic_D [ divide start_ARG log πitalic_θ ( y+ | x ) end_ARG start_ARG | y+ | end_ARG ] (8) Remark 3. Objective (8) shares similarities to SimPO Meng et al. (2024) but has important differences. First, objective (8) includes a length-normalized SFT term, which is key in mitigating Type I Reward Hacking (Theorem 1, Proposition 4), from which SimPO suffers (Proposition 1). Second, our approach analytically leads to the margin 1/|y+|−1/|y−|1superscript1superscript1/|y^+|-1/|y^-|1 / | y+ | - 1 / | y- | while SimPO uses a fixed hyperparameter. Lastly, our objective is rooted in theory and enjoys finite-sample guarantees. 4.4 POWER Faced with Hard Instances and the Role of Partition Function In the following proposition, we analyze the POWER objective (8) in the hard reward hacking instances of Proposition 1 and Proposition 2. Proof is presented in Appendix C.3. Proposition 4. (I) Consider the three-armed bandit instance in Proposition 1. Then, for any β>00β>0β > 0 and η>(2+e)N−(2+e)22η> (2+e)N-(2+e)η > divide start_ARG ( 2 + e ) end_ARG start_ARG N - ( 2 + e ) end_ARG, POWER policy π^POWERsubscript^POWER π_POWERover start_ARG π end_ARGPOWER that solves the objective (8) is the best-in-class policy: π^POWER=πθ⋆subscript^POWERsubscriptsuperscript⋆ π_POWER= _θ over start_ARG π end_ARGPOWER = πitalic_θ⋆. (I) Consider the three-armed bandit instance in Proposition 2. Then, for any β>0,η≥0formulae-sequence00β>0,η≥ 0β > 0 , η ≥ 0, policy π^POWERsubscript^POWER π_POWERover start_ARG π end_ARGPOWER that solves the objective (8) suffers from a constant suboptimality J(πθ⋆)−J(π^POWER)>0.2.subscriptsuperscript⋆subscript^POWER0.2J( _θ )-J( π_POWER)>0.2.J ( πitalic_θ⋆ ) - J ( over start_ARG π end_ARGPOWER ) > 0.2 . Proposition 4 confirms that POWER robustly (for any β>00β>0β > 0 and η≳1/Ngreater-than-or-equivalent-to1η 1/Nη ≳ 1 / N) prevents Type I Reward Hacking in the hard instance of Proposition 1, where other preference optimization algorithms DPO, SimPO, IPO, and χPO fail. Yet, the above proposition shows that POWER suffers from Type I Reward Hacking, which the design dynamic labels in the following section. 5 Against Type I Reward Hacking: Dynamic Labels We now turn our focus to mitigating Type I Reward Hacking, based on the following intuition: keeping the model’s internal preferences close to initialization in the low-coverage regions (trust the preference labels less) while learning from the data in the high-coverage regions (trust the preference labels more). For this purpose, we analyze the learning dynamics of preference optimization in the bandits setting with softmax parameterization πθ(y)∝exp(θ(y))proportional-tosubscript _θ(y) (θ(y))πitalic_θ ( y ) ∝ exp ( θ ( y ) ). We denote the dataset by =(y0,y1,l)superscript0superscript1D=\(y^0,y^1,l)\D = ( y0 , y1 , l ) with labels l∼ℙr⋆(⋅|y0,y1)l P_r (·|y^0,y^1)l ∼ ℙitalic_r⋆ ( ⋅ | y0 , y1 ). We use μ^0,1subscript^01 μ_0,1over start_ARG μ end_ARG0 , 1 to indicate the empirical probability of comparing y0superscript0y^0y0 with y1superscript1y^1y1, and μ^1≻0subscript^succeeds10 μ_1 0over start_ARG μ end_ARG1 ≻ 0 the empirical probability of preferring y1superscript1y^1y1 over y0superscript0y^0y0. To simplify presentation, we consider POWER with w(y)=1,η=0,β=1formulae-sequence1formulae-sequence01w(y)=1,η=0,β=1w ( y ) = 1 , η = 0 , β = 1; a similar analysis can be extended to other objectives. Reverse engineering label updates based on learning dynamics. Rather than using static preference labels, we allow the labels ltsubscriptl_tlitalic_t to evolve across gradient updates. Denote the parameter gap corresponding to two actions y0,y1superscript0superscript1y^0,y^1y0 , y1 by dθt(y1,y0)≔θt(y1)−θt(y0)≔subscriptsubscriptsuperscript1superscript0subscriptsuperscript1subscriptsuperscript0d_ _t(y^1,y^0) _t(y^1)- _t(y^0)ditalic_θ start_POSTSUBSCRIPT t end_POSTSUBSCRIPT ( y1 , y0 ) ≔ θitalic_t ( y1 ) - θitalic_t ( y0 ). We show in Appendix F.1 that isolated (batch) gradient updates on y0superscript0y^0y0 and y1superscript1y^1y1 is: dθt+1(y1,y0)=dθt(y1,y0)+αμ^0,1[(μ^1≻0−μ^0≻1)lt−(σ(dθt(y1,y0))−μ^0≻1)]subscriptsubscript1superscript1superscript0subscriptsubscriptsuperscript1superscript0subscript^01delimited-[]subscript^succeeds10subscript^succeeds01subscriptsubscriptsubscriptsuperscript1superscript0subscript^succeeds01 split&d_ _t+1(y^1,y^0)=d_ _t(y^1,% y^0)+α μ_0,1 [( μ_1 0- μ_0 1)% l_t- (σ (d_ _t(y^1,y^0) )- μ_0 1% ) ] splitstart_ROW start_CELL end_CELL start_CELL ditalic_θ start_POSTSUBSCRIPT t + 1 end_POSTSUBSCRIPT ( y1 , y0 ) = ditalic_θ start_POSTSUBSCRIPT t end_POSTSUBSCRIPT ( y1 , y0 ) + α over start_ARG μ end_ARG0 , 1 [ ( over start_ARG μ end_ARG1 ≻ 0 - over start_ARG μ end_ARG0 ≻ 1 ) litalic_t - ( σ ( ditalic_θ start_POSTSUBSCRIPT t end_POSTSUBSCRIPT ( y1 , y0 ) ) - over start_ARG μ end_ARG0 ≻ 1 ) ] end_CELL end_ROW (9) We design labels ltsubscriptl_tlitalic_t so that gradient updates are rapidly diminished for poorly covered preference pairs, ensuring that preferences for such pairs remain close to initialization. To achieve this, we first directly set the gradient in (9) to zero and derive a “stationary” label l¯tsubscript¯ l_tover¯ start_ARG l end_ARGt: (μ^1≻0−μ^0≻1)lt−(σ(dθt(y1,y0))−μ^0≻1)=0⇒l¯t=σ(dθt(y1,y0))−μ^0≻1μ^1≻0−μ^0≻1.formulae-sequencesubscript^succeeds10subscript^succeeds01subscriptsubscriptsubscriptsuperscript1superscript0subscript^succeeds010⇒subscript¯subscriptsubscriptsuperscript1superscript0subscript^succeeds01subscript^succeeds10subscript^succeeds01 ( μ_1 0- μ_0 1)l_t- (σ(d_% _t(y^1,y^0))- μ_0 1 )=0 % l_t= σ (d_ _t(y^1,y^0) )- μ_0% 1 μ_1 0- μ_0 1.( over start_ARG μ end_ARG1 ≻ 0 - over start_ARG μ end_ARG0 ≻ 1 ) litalic_t - ( σ ( ditalic_θ start_POSTSUBSCRIPT t end_POSTSUBSCRIPT ( y1 , y0 ) ) - over start_ARG μ end_ARG0 ≻ 1 ) = 0 ⇒ over¯ start_ARG l end_ARGt = divide start_ARG σ ( ditalic_θ start_POSTSUBSCRIPT t end_POSTSUBSCRIPT ( y1 , y0 ) ) - over start_ARG μ end_ARG0 ≻ 1 end_ARG start_ARG over start_ARG μ end_ARG1 ≻ 0 - over start_ARG μ end_ARG0 ≻ 1 end_ARG . (10) l¯tsubscript¯ l_tover¯ start_ARG l end_ARGt represents the ratio between a learned preference gap and the empirical preference gap, and we have l¯t=1subscript¯1 l_t=1over¯ start_ARG l end_ARGt = 1 when learned and empirical preferences are equal. To implement dynamic preference labels, we employ the following update rule for ltsubscriptl_tlitalic_t, where γ ranges between 0 and 1: lt+1=(1−γ)lt+γl¯t,l0=1.formulae-sequencesubscript11subscriptsubscript¯subscript01 l_t+1=(1-γ)l_t+γ l_t, l_0=1.litalic_t + 1 = ( 1 - γ ) litalic_t + γ over¯ start_ARG l end_ARGt , l0 = 1 . (11) In the following theorem, we analyze the coupled dynamical systems described by equations (11) and (9); see Appendix F.2 for the proof. Theorem 2 (Learning Dynamics with Label Updates). Consider the following set of differential equations with initial values l0=1subscript01l_0=1l0 = 1 and any d0subscript0d_0d0: d˙t=αμ^0,1((μ^1≻0−μ^0≻1)lt−(σ(dt)−μ^0≻1))l˙t=−γμ^1≻0−μ^0≻1((μ^1≻0−μ^0≻1)lt−(σ(dt)−μ^0≻1))subscript˙subscript^01subscript^succeeds10subscript^succeeds01subscriptsubscriptsubscript^succeeds01subscript˙subscript^succeeds10subscript^succeeds01subscript^succeeds10subscript^succeeds01subscriptsubscriptsubscript^succeeds01 split d_t&=α μ_0,1 (( μ_% 1 0- μ_0 1)l_t-(σ(d_t)- μ_0 1) % )\\ l_t&=- γ μ_1 0- μ_0 1 ((% μ_1 0- μ_0 1)l_t-(σ(d_t)- μ_0% 1) ) splitstart_ROW start_CELL over˙ start_ARG d end_ARGt end_CELL start_CELL = α over start_ARG μ end_ARG0 , 1 ( ( over start_ARG μ end_ARG1 ≻ 0 - over start_ARG μ end_ARG0 ≻ 1 ) litalic_t - ( σ ( ditalic_t ) - over start_ARG μ end_ARG0 ≻ 1 ) ) end_CELL end_ROW start_ROW start_CELL over˙ start_ARG l end_ARGt end_CELL start_CELL = - divide start_ARG γ end_ARG start_ARG over start_ARG μ end_ARG1 ≻ 0 - over start_ARG μ end_ARG0 ≻ 1 end_ARG ( ( over start_ARG μ end_ARG1 ≻ 0 - over start_ARG μ end_ARG0 ≻ 1 ) litalic_t - ( σ ( ditalic_t ) - over start_ARG μ end_ARG0 ≻ 1 ) ) end_CELL end_ROW (12) Assume μ^1≻0>1/2subscript^succeeds1012 μ_1 0>1/2over start_ARG μ end_ARG1 ≻ 0 > 1 / 2 and let c=minσ(d0)(1−σ(d0),μ^1≻0μ^0≻1c= \σ(d_0)(1-σ(d_0), μ_1 0 μ_0 1\c = min σ ( d0 ) ( 1 - σ ( d0 ) , over start_ARG μ end_ARG1 ≻ 0 over start_ARG μ end_ARG0 ≻ 1 . For any ϵl≪1much-less-thansubscriptitalic-ϵ1 _l 1ϵitalic_l ≪ 1, fix μl,μh,T,γ,μ^0,1,αsubscriptsubscriptℎsubscript^01 _l, _h,T,γ, μ_0,1,αμitalic_l , μitalic_h , T , γ , over start_ARG μ end_ARG0 , 1 , α such that αμl/ϵl≤γ≤1/2exp(−1/4)αμh≤1subscriptsubscriptitalic-ϵ1214subscriptℎ1α _l/ _l≤γ≤ 1/2 (-1/4)α _h≤ 1α μitalic_l / ϵitalic_l ≤ γ ≤ 1 / 2 exp ( - 1 / 4 ) α μitalic_h ≤ 1 and αμ^1≻0T≥1subscript^succeeds101α μ_1 0T≥ 1α over start_ARG μ end_ARG1 ≻ 0 T ≥ 1. 1. (Low Coverage Case) When μ^0,1≤μlsubscript^01subscript μ_0,1≤ _lover start_ARG μ end_ARG0 , 1 ≤ μitalic_l, we have |σ(dT)−σ(d0)|≤|dT−d0|≤ϵlsubscriptsubscript0subscriptsubscript0subscriptitalic-ϵ|σ(d_T)-σ(d_0)|≤|d_T-d_0|≤ _l| σ ( ditalic_T ) - σ ( d0 ) | ≤ | ditalic_T - d0 | ≤ ϵitalic_l. 2. (High Coverage Case) When μ^0,1≥μhsubscript^01subscriptℎ μ_0,1≥ _hover start_ARG μ end_ARG0 , 1 ≥ μitalic_h, we have (σ(dT)−μ^1≻0)2≤exp(−αcμ^0,1T)superscriptsubscriptsubscript^succeeds102subscript^01(σ(d_T)- μ_1 0)^2≤ (-α c μ_0,1% T )( σ ( ditalic_T ) - over start_ARG μ end_ARG1 ≻ 0 )2 ≤ exp ( - α c over start_ARG μ end_ARG0 , 1 T ). The above theorem shows that for poorly covered pairs (small μ^1,0subscript^10 μ_1,0over start_ARG μ end_ARG1 , 0), learned preferences σ(dT)subscriptσ(d_T)σ ( ditalic_T ) remain close to initialization, while for high coverage pairs (large μ^1,0subscript^10 μ_1,0over start_ARG μ end_ARG1 , 0), learned preferences converge to empirical preferences. In a sense, γ determines the level of conservatism, adjusting the threshold of what considered poor coverage. Moreover, the convergence rate in the high-coverage case is impacted by empirical preferences through c. In the case of nearly equal preferences μ^1≻0≈1/2subscript^succeeds1012 μ_1 0≈ 1/2over start_ARG μ end_ARG1 ≻ 0 ≈ 1 / 2, the convergence rate is faster, whereas in the case of strong preference with μ^1≻0→1→subscript^succeeds101 μ_1 0→ 1over start_ARG μ end_ARG1 ≻ 0 → 1, the convergence rate is slower suggesting that more updates are required to further distinguish the two choices. Remark 4 (Related work on soft labels in RLHF). In preference optimization, Mitchell (2024) considers noisy preference labels and incorporates constant soft labels through linear interpolation. Concurrent work by Furuta et al. (2024) develop a geometric averaging approach, in which samples are weighted according to the preference gap using scores from a reward model. In the context of reward learning, Zhu et al. (2024) propose iterative data smoothing that updates labels toward learned preferences. In contrast to these methods, our approach is rooted in updating labels to shrink gradients of poorly covered pairs via a general recipe whereby dynamic labels are smoothly updated toward labels that set the gradient to zero. This approach goes beyond constant soft labels and does not require scores from an extra reward or preference model. Moreover, label updates in prior works do not guarantee remaining close to a (non-uniform) initial model in the low coverage areas, which aims at mitigating Type I Reward Hacking in the offline alignment setting. POWER with Dynamic Labels. Our final algorithm POWER-DL (Algorithm 1) integrates the POWER objective with dynamic labels against reward hacking. In untrustworthy regions, POWER-DL interpolates between the initial model and robust rewards, allowing to trade off the two types of reward hacking through adjusting conservatism parameters η and γ, reflecting relative quality of the initial model compared to preference data and up to removing conservatism completely by setting η=γ=00η=γ=0η = γ = 0. We highlight the fact that divergence-based methods aim at keeping the learned model close to the initial model wherever the learned model has a decent probability, regardless of data coverage. In contrast, the dynamic label procedure aims at keeping the learned model close to the initialization only in the untrustworthy regions while learning from the data in high coverage region, which can alleviate potential over-pessimism. All these factors can lead to a better performance, as supported by our empirical evaluations in Section 6. See Appendix B.3 for further discussion. 6 Experiments 6.1 Experimental Setup We conduct experiments to assess different preference optimization methods on aligning LLMs across four settings, varying in dataset size and level of distribution shift between the initial model and data. We follow two pipelines: Helpsteer2 Wang et al. (2024e), which employs smaller datasets, and Zephyr Tunstall et al. (2023) with significantly larger datasets. We implement two distinct setups similar to Meng et al. (2024): the base setup that uses an existing preference dataset and the instruct setup that constructs a preference dataset by sampling from the initial model. These two setups allow evaluating across different levels of distribution shift between the initial model and preference data. Helpsteer2 setups. In the base setup, we train Llama-3-8B on the OpenAssistant2 dataset Köpf et al. (2024) to create the initial model. We conduct preference optimization using the Helpsteer2 dataset Wang et al. (2024e), selecting responses based on helpfulness scores and discarding ties, yielding about 7K samples. In the instruct setup, we use Llama-3-8B-Instruct as the initial model and generate a preference dataset from Helpsteer2 prompts. Following Wang et al. (2024e), we generate 10 responses per prompt with temperature 0.7. We then score them with Armo reward model Wang et al. (2024c) and select the highest and lowest score responses as y+superscripty^+y+ and y−superscripty^-y-, respectively. Zephyr setups. In the base setup, we obtain the initial model by training Llama-3-8B base model on the UltraChat-200K dataset Ding et al. (2023). We then perform preference optimization on the UltraFeedback dataset Cui et al. (2024), comprising approximately 61K samples. In the instruct setup and following Meng et al. (2024), we start from Llama-3-8B-Instruct and generate 5 responses with temperature 0.8 per prompt in the UltraFeedback dataset. As before, the highest and lowest score responses are selected as preference response pairs. Evaluation benchmarks. We primarily assess preference methods by evaluating the trained models on standard instruction-following benchmarks: AlpacaEval 2.0 Li et al. (2023a); Dubois et al. (2024) and Arena-Hard Li et al. (2024), which evaluate the quality of the model responses. Following standard guidelines, for Arena-Hard, we report the win rate (WR) of the model’s responses against responses from GPT-4-Turbo. For AlpacaEval 2.0, in addition to the WR against GPT-4-Turbo, we report the length-controlled (LC) win rate, designed to mitigate bias toward verbosity. We further evaluate the performance of models on MT-Bench Zheng et al. (2023) and downstream tasks such as mathematics, reasoning, truthfulness, and instruction-following Beeching et al. (2023). Preference optimization methods. We compare POWER-DL against various baselines; see Appendix H.1 for details. These include divergence-base methods DPO Rafailov et al. (2024b), IPO Azar et al. (2024), offline SPPO Wu et al. (2024b), and χPO Huang et al. (2024), along with robust variants such as conservative DPO (cDPO) Mitchell (2024), robust preference optimization (ROPO) Liang et al. (2024), R-DPO Park et al. (2024), and DPO+SFT Pal et al. (2024); Liu et al. (2024). We also evaluate against reference-free methods CPO Xu et al. (2024a), SLiC-HF Zhao et al. (2023), RRHF Yuan et al. (2024a), ORPO Hong et al. (2024), and SimPO Meng et al. (2024). 6.2 Benchmark Results POWER-DL outperforms SoTA methods on alignment benchmarks. Table 2 presents the results on alignment benchmarks. POWER-DL consistently outperforms other methods in both Helpsteer2 and Zephyr pipelines and across base and instruct settings. These improvements can largely be attributed to the integration of weighted entropy, which effectively counters underoptimization, and mitigation of reward hacking. Notably, POWER-DL surpasses other robust methods such as cDPO and ROPO demonstrating its efficacy in handling poorly covered samples. Additionally, POWER-DL improvements are more pronounced in the base setting, which is more susceptible to reward hacking due to higher levels of distribution shift. Comparing POWER-DL with POWER shows that incorporating dynamic labels further improves performance. In Appendix I and Appendix J, we provide additional experimental results on the MT-Bench, Mistral family, iterative preference optimization, sample responses, and hyperparameter robustness analysis. Table 2: AlpacaEval 2 and Arena-Hard results on Helpsteer2 and Zephyr settings. Method Helpsteer2 Zephyr Llama3-8B-Base Llama3-8B-Instruct Llama3-8B-Base Llama3-8B-Instruct AlpacaEval Arena-Hard AlpacaEval Arena-Hard AlpacaEval Arena-Hard AlpacaEval Arena-Hard LC(%) WR(%) WR(%) LC(%) WR(%) WR(%) LC(%) WR(%) WR(%) LC(%) WR(%) WR(%) Initial Model 8.02 5.42 2.4 33.41 32.40 23.0 4.76 2.83 2.0 33.41 32.40 23.0 DPO 18.52 14.99 10.0 40.87 39.05 29.6 22.53 17.84 13.3 44.20 43.63 38.4 DPO+SFT 18.33 12.93 7.9 39.85 37.51 27.0 19.11 14.69 9.5 45.98 44.07 39.0 cDPO 19.06 14.65 8.5 42.27 40.36 34.4 21.06 16.33 11.4 44.96 44.37 39.5 R-DPO 11.03 15.20 8.3 33.67 33.89 25.7 18.66 17.88 9.5 44.13 44.94 37.5 IPO 20.11 14.60 9.4 42.95 40.76 30.8 10.55 8.04 7.2 36.63 35.30 24.5 χPO 11.06 7.67 5.1 42.10 39.65 35.8 13.16 10.87 8.9 44.25 42.41 34.7 SPPO 26.23 18.12 11.8 42.01 39.46 29.5 16.08 15.52 9.1 42.64 39.68 35.9 CPO 15.07 16.78 8.3 35.90 35.20 26.8 7.01 6.84 3.0 36.39 35.40 22.8 RRHF 8.25 7.15 5.8 35.15 34.07 25.7 6.61 6.39 3.0 35.56 34.56 23.1 SLiC-HF 15.19 18.77 10.1 37.76 39.68 32.2 19.35 21.81 11.2 41.74 45.05 38.2 ORPO 23.99 16.91 11.2 43.01 35.68 27.1 23.20 19.43 14.7 45.51 40.95 33.3 SimPO 25.35 19.30 13.7 43.23 36.89 32.6 24.38 21.21 16.4 43.24 37.34 26.8 ROPO 21.24 17.66 9.5 41.03 36.32 31.5 22.91 19.67 10.9 45.55 45.58 33.7 POWER-DL 31.52 31.44 21.5 47.16 43.08 34.8 27.00 22.57 17.3 48.97 43.75 41.5 POWER 29.57 30.00 19.0 43.52 40.19 31.5 23.72 21.26 16.0 46.93 42.02 38.0 POWER-DL improves or maintains performance on downstream tasks. One of the challenges of the alignment step is possible degradation of performance on downstream tasks, which can be attributed to reward hacking Xu et al. (2024b). We evaluate the trained models on the LLM Leaderboard Beeching et al. (2023), which encompass a variety of tasks, including language understanding and knowledge benchmarks MMLU Hendrycks et al. (2020), MMLU-PRO Wang et al. (2024d), and ARC-Challenge Clark et al. (2018), commonsense reasoning assessments like HellaSwag Zellers et al. (2019) and Winogrande Sakaguchi et al. (2021), factual accuracy evaluations on TruthfulQA Lin et al. (2022), instruction-following capabilities measured on IFEval Zhou et al. (2023), and mathematical reasoning evaluated on the GSM8K dataset Cobbe et al. (2021). The downstream tasks results are presented in Tables 4 and 5 in Appendix I. POWER-DL consistently improves or maintains performance across all tasks and effective mitigates reward hacking. Notably, while preference optimization methods vary in results on the GSM8K benchmark, with some like SimPO significantly degrading the initial model, POWER-DL consistently maintains or enhances performance, achieving up to a 7.0 point gain. Other tasks with notable variation include IFEval and TruthfulQA benchmarks. In TruthfulQA, POWER-DL significantly outperforms DPO, with up to a 12.8 point improvement over the initial model. In the IFEval, methods like DPO and SLiC-HF sometimes degrade performance of the initial model, whereas POWER-DL consistently maintains or improves it by up to 11.7 points. 7 Discussion We studied reward hacking in offline preference optimization. We identified two types of reward hacking stemming from statistical fluctuations in preference data. We demonstrated that many existing methods are vulnerable to both types of reward hacking, despite maintaining a small divergence from the initial model. To mitigate reward hacking, we introduced POWER-DL, a practical algorithm based on a weighted entropy robust reward framework augmented with dynamic preference labels. POWER-DL enjoys theoretical guarantees and achieves strong empirical performance. Future research directions include applications of dynamic labels to out-of-distribution robustness and investigating the interplay between statistical errors and reward misspecification in reward hacking. References Achiam et al. (2023) Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. GPT-4 technical report. arXiv preprint arXiv:2303.08774, 2023. Amodei et al. (2016) Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in AI safety. arXiv preprint arXiv:1606.06565, 2016. Azar et al. (2024) Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm to understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics, pages 4447–4455. PMLR, 2024. Bansal et al. (2024) Hritik Bansal, John Dang, and Aditya Grover. Peering through preferences: Unraveling feedback acquisition for aligning large language models. In The Twelfth International Conference on Learning Representations, 2024. Beeching et al. (2023) Edward Beeching, Clémentine Fourrier, Nathan Habib, Sheon Han, Nathan Lambert, Nazneen Rajani, Omar Sanseviero, Lewis Tunstall, and Thomas Wolf. Open LLM leaderboard. Hugging Face, 2023. Casper et al. (2023) Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, et al. Open problems and fundamental limitations of reinforcement learning from human feedback. arXiv preprint arXiv:2307.15217, 2023. Cen et al. (2024) Shicong Cen, Jincheng Mei, Katayoon Goshvadi, Hanjun Dai, Tong Yang, Sherry Yang, Dale Schuurmans, Yuejie Chi, and Bo Dai. Value-incentivized preference optimization: A unified approach to online and offline RLHF. arXiv preprint arXiv:2405.19320, 2024. Chakraborty et al. (2024) Souradip Chakraborty, Jiahao Qiu, Hui Yuan, Alec Koppel, Furong Huang, Dinesh Manocha, Amrit Singh Bedi, and Mengdi Wang. MaxMin-RLHF: Towards equitable alignment of large language models with diverse human preferences. arXiv preprint arXiv:2402.08925, 2024. Chen et al. (2024a) Lichang Chen, Chen Zhu, Jiuhai Chen, Davit Soselia, Tianyi Zhou, Tom Goldstein, Heng Huang, Mohammad Shoeybi, and Bryan Catanzaro. ODIN: Disentangled reward mitigates hacking in RLHF. In Forty-first International Conference on Machine Learning, 2024a. Chen et al. (2022) Xiaoyu Chen, Han Zhong, Zhuoran Yang, Zhaoran Wang, and Liwei Wang. Human-in-the-loop: Provably efficient preference-based reinforcement learning with general function approximation. In International Conference on Machine Learning, pages 3773–3793. PMLR, 2022. Chen et al. (2024b) Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning converts weak language models to strong language models. arXiv preprint arXiv:2401.01335, 2024b. Cheng et al. (2022) Ching-An Cheng, Tengyang Xie, Nan Jiang, and Alekh Agarwal. Adversarially trained actor critic for offline reinforcement learning. In International Conference on Machine Learning, pages 3852–3878. PMLR, 2022. Choi et al. (2024) Eugene Choi, Arash Ahmadian, Matthieu Geist, Oilvier Pietquin, and Mohammad Gheshlaghi Azar. Self-improving robust preference optimization. arXiv preprint arXiv:2406.01660, 2024. Christiano et al. (2017) Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017. Clark et al. (2018) Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018. Cobbe et al. (2021) Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. Coste et al. (2024) Thomas Coste, Usman Anwar, Robert Kirk, and David Krueger. Reward model ensembles help mitigate overoptimization. In The Twelfth International Conference on Learning Representations, 2024. Cui et al. (2024) Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ultrafeedback: Boosting language models with high-quality feedback. International Conference on Machine Learning, 2024. Ding et al. (2023) Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality instructional conversations. In Conference on Empirical Methods in Natural Language Processing, 2023. Dong et al. (2023) Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, SHUM KaShun, and Tong Zhang. RAFT: Reward ranked finetuning for generative foundation model alignment. Transactions on Machine Learning Research, 2023. Dong et al. (2024) Hanze Dong, Wei Xiong, Bo Pang, Haoxiang Wang, Han Zhao, Yingbo Zhou, Nan Jiang, Doyen Sahoo, Caiming Xiong, and Tong Zhang. RLHF workflow: From reward modeling to online RLHF. arXiv preprint arXiv:2405.07863, 2024. Dubey et al. (2024) Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The Llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. Dubois et al. (2024) Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto. Length-controlled alpacaeval: A simple way to debias automatic evaluators. arXiv preprint arXiv:2404.04475, 2024. Eisenstein et al. (2023) Jacob Eisenstein, Chirag Nagpal, Alekh Agarwal, Ahmad Beirami, Alex D’Amour, DJ Dvijotham, Adam Fisch, Katherine Heller, Stephen Pfohl, Deepak Ramachandran, et al. Helping or herding? Reward model ensembles mitigate but do not eliminate reward hacking. arXiv preprint arXiv:2312.09244, 2023. Eysenbach and Levine (2022) Benjamin Eysenbach and Sergey Levine. Maximum entropy RL (provably) solves some robust RL problems. In International Conference on Learning Representations, 2022. Fan (1953) Ky Fan. Minimax theorems. Proceedings of the National Academy of Sciences, 39(1):42–47, 1953. Fisch et al. (2024) Adam Fisch, Jacob Eisenstein, Vicky Zayats, Alekh Agarwal, Ahmad Beirami, Chirag Nagpal, Pete Shaw, and Jonathan Berant. Robust preference optimization through reward model distillation. arXiv preprint arXiv:2405.19316, 2024. Furuta et al. (2024) Hiroki Furuta, Kuang-Huei Lee, Shixiang Shane Gu, Yutaka Matsuo, Aleksandra Faust, Heiga Zen, and Izzeddin Gur. Geometric-averaged preference optimization for soft preference labels. arXiv preprint arXiv:2409.06691, 2024. Gao et al. (2023) Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, pages 10835–10866. PMLR, 2023. Grinsztajn et al. (2024) Nathan Grinsztajn, Yannis Flet-Berliac, Mohammad Gheshlaghi Azar, Florian Strub, Bill Wu, Eugene Choi, Chris Cremer, Arash Ahmadian, Yash Chandak, Olivier Pietquin, et al. Averaging log-likelihoods in direct alignment. arXiv preprint arXiv:2406.19188, 2024. Guiaşu (1971) Silviu Guiaşu. Weighted entropy. Reports on Mathematical Physics, 2(3):165–179, 1971. Guiasu and Shenitzer (1985) Silviu Guiasu and Abe Shenitzer. The principle of maximum entropy. The mathematical intelligencer, 7:42–48, 1985. Haarnoja et al. (2018) Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pages 1861–1870. PMLR, 2018. Hadfield-Menell et al. (2017) Dylan Hadfield-Menell, Smitha Milli, Pieter Abbeel, Stuart J Russell, and Anca Dragan. Inverse reward design. Advances in neural information processing systems, 30, 2017. Hendrycks et al. (2020) Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2020. Hong et al. (2024) Jiwoo Hong, Noah Lee, and James Thorne. ORPO: Monolithic preference optimization without reference model. arXiv preprint arXiv:2403.07691, 2(4):5, 2024. Hu et al. (2024) Jian Hu, Xibin Wu, Weixun Wang, Xianyu, Dehao Zhang, and Yu Cao. OpenRLHF: An easy-to-use, scalable and high-performance RLHF framework. arXiv preprint arXiv:2405.11143, 2024. Huang et al. (2024) Audrey Huang, Wenhao Zhan, Tengyang Xie, Jason D Lee, Wen Sun, Akshay Krishnamurthy, and Dylan J Foster. Correcting the mythos of KL-regularization: Direct alignment without overparameterization via Chi-squared preference optimization. arXiv preprint arXiv:2407.13399, 2024. Ibarz et al. (2018) Borja Ibarz, Jan Leike, Tobias Pohlen, Geoffrey Irving, Shane Legg, and Dario Amodei. Reward learning from human preferences and demonstrations in atari. Advances in neural information processing systems, 31, 2018. Jaynes (1957) Edwin T Jaynes. Information theory and statistical mechanics. Physical review, 106(4):620, 1957. Jiang et al. (2023) Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7B. arXiv preprint arXiv:2310.06825, 2023. Kadavath et al. (2022) Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, et al. Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221, 2022. Knox et al. (2023) W Bradley Knox, Alessandro Allievi, Holger Banzhaf, Felix Schmitt, and Peter Stone. Reward (mis)design for autonomous driving. Artificial Intelligence, 316:103829, 2023. Köpf et al. (2024) Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi Rui Tam, Keith Stevens, Abdullah Barhoum, Duc Nguyen, Oliver Stanley, Richárd Nagyfi, et al. Openassistant conversations-democratizing large language model alignment. Advances in Neural Information Processing Systems, 36, 2024. Kumar et al. (2020) Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative Q-learning for offline reinforcement learning. arXiv preprint arXiv:2006.04779, 2020. Lambert and Calandra (2023) Nathan Lambert and Roberto Calandra. The alignment ceiling: Objective mismatch in reinforcement learning from human feedback. arXiv preprint arXiv:2311.00168, 2023. Leng et al. (2024) Jixuan Leng, Chengsong Huang, Banghua Zhu, and Jiaxin Huang. Taming overconfidence in LLMs: Reward calibration in RLHF. arXiv preprint arXiv:2410.09724, 2024. Levine et al. (2020) Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643, 2020. Lewis et al. (2017) Mike Lewis, Denis Yarats, Yann Dauphin, Devi Parikh, and Dhruv Batra. Deal or no deal? End-to-end learning of negotiation dialogues. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2443–2453, 2017. Li et al. (2024) Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Banghua Zhu, Joseph E Gonzalez, and Ion Stoica. From live data to high-quality benchmarks: The arena-hard pipeline, 2024. Li et al. (2023a) Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. AlpacaEval: An automatic evaluator of instruction-following models, 2023a. Li et al. (2023b) Zihao Li, Zhuoran Yang, and Mengdi Wang. Reinforcement learning with human feedback: Learning dynamic choices via pessimism. arXiv preprint arXiv:2305.18438, 2023b. Liang et al. (2024) Xize Liang, Chao Chen, Jie Wang, Yue Wu, Zhihang Fu, Zhihao Shi, Feng Wu, and Jieping Ye. Robust preference optimization with provable noise tolerance for LLMs. arXiv preprint arXiv:2404.04102, 2024. Lin et al. (2022) Stephanie Lin, Jacob Hilton, and Owain Evans. TruthfulQA: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3214–3252, 2022. Liu et al. (2024) Zhihan Liu, Miao Lu, Shenao Zhang, Boyi Liu, Hongyi Guo, Yingxiang Yang, Jose Blanchet, and Zhaoran Wang. Provably mitigating overoptimization in RLHF: Your SFT loss is implicitly an adversarial regularizer. arXiv preprint arXiv:2405.16436, 2024. Meng et al. (2024) Yu Meng, Mengzhou Xia, and Danqi Chen. SimPO: Simple preference optimization with a reference-free reward. arXiv preprint arXiv:2405.14734, 2024. Menick et al. (2022) Jacob Menick, Maja Trebacz, Vladimir Mikulik, John Aslanides, Francis Song, Martin Chadwick, Mia Glaese, Susannah Young, Lucy Campbell-Gillingham, Geoffrey Irving, et al. Teaching language models to support answers with verified quotes. arXiv preprint arXiv:2203.11147, 2022. Michaud et al. (2020) Eric J Michaud, Adam Gleave, and Stuart Russell. Understanding learned reward functions. arXiv preprint arXiv:2012.05862, 2020. Mitchell (2024) Eric Mitchell. A note on dpo with noisy preferences and relationship to IPO. 2024. https://ericmitchell.ai/cdpo.pdf. Moskovitz et al. (2024) Ted Moskovitz, Aaditya K Singh, DJ Strouse, Tuomas Sandholm, Ruslan Salakhutdinov, Anca Dragan, and Stephen Marcus McAleer. Confronting reward model overoptimization with constrained RLHF. In The Twelfth International Conference on Learning Representations, 2024. Munos et al. (2023) Remi Munos, Michal Valko, Daniele Calandriello, Mohammad Gheshlaghi Azar, Mark Rowland, Zhaohan Daniel Guo, Yunhao Tang, Matthieu Geist, Thomas Mesnard, Côme Fiegel, et al. Nash learning from human feedback. In Forty-first International Conference on Machine Learning, 2023. Nachum et al. (2019) Ofir Nachum, Yinlam Chow, Bo Dai, and Lihong Li. DualDICE: Behavior-agnostic estimation of discounted stationary distribution corrections. In Advances in Neural Information Processing Systems, pages 2315–2325, 2019. Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744, 2022. Pal et al. (2024) Arka Pal, Deep Karkhanis, Samuel Dooley, Manley Roberts, Siddartha Naidu, and Colin White. Smaug: Fixing failure modes of preference optimisation with DPO-Positive. arXiv preprint arXiv:2402.13228, 2024. Pan et al. (2022) Alexander Pan, Kush Bhatia, and Jacob Steinhardt. The effects of reward misspecification: Mapping and mitigating misaligned models. In International Conference on Learning Representations, 2022. Park et al. (2024) Ryan Park, Rafael Rafailov, Stefano Ermon, and Chelsea Finn. Disentangling length from quality in direct preference optimization. arXiv preprint arXiv:2403.19159, 2024. Paulus et al. (2018) Romain Paulus, Caiming Xiong, and Richard Socher. A deep reinforced model for abstractive summarization. International Conference on Learning Representations, 2018. Peng et al. (2023) Baolin Peng, Linfeng Song, Ye Tian, Lifeng Jin, Haitao Mi, and Dong Yu. Stabilizing RLHF through advantage model and selective rehearsal. arXiv preprint arXiv:2309.10202, 2023. Rafailov et al. (2024a) Rafael Rafailov, Yaswanth Chittepu, Ryan Park, Harshit Sikchi, Joey Hejna, Bradley Knox, Chelsea Finn, and Scott Niekum. Scaling laws for reward model overoptimization in direct alignment algorithms. arXiv preprint arXiv:2406.02900, 2024a. Rafailov et al. (2024b) Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024b. Rame et al. (2024) Alexandre Rame, Nino Vieillard, Leonard Hussenot, Robert Dadashi, Geoffrey Cideron, Olivier Bachem, and Johan Ferret. WARM: On the benefits of weight averaged reward models. In Forty-first International Conference on Machine Learning, 2024. Rashidinejad et al. (2021) Paria Rashidinejad, Banghua Zhu, Cong Ma, Jiantao Jiao, and Stuart Russell. Bridging offline reinforcement learning and imitation learning: A tale of pessimism. Advances in Neural Information Processing Systems, 34:11702–11716, 2021. Rita et al. (2024) Mathieu Rita, Florian Strub, Rahma Chaabouni, Paul Michel, Emmanuel Dupoux, and Olivier Pietquin. Countering reward over-optimization in LLM with demonstration-guided reinforcement learning. arXiv preprint arXiv:2404.19409, 2024. Rosset et al. (2024) Corby Rosset, Ching-An Cheng, Arindam Mitra, Michael Santacroce, Ahmed Awadallah, and Tengyang Xie. Direct Nash optimization: Teaching language models to self-improve with general preferences. arXiv preprint arXiv:2404.03715, 2024. Röttger et al. (2024) Paul Röttger, Hannah Kirk, Bertie Vidgen, Giuseppe Attanasio, Federico Bianchi, and Dirk Hovy. XSTest: A test suite for identifying exaggerated safety behaviours in large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 5377–5400, 2024. Russell (2022) Stuart Russell. Human-compatible artificial intelligence., 2022. Sakaguchi et al. (2021) Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99–106, 2021. Schulman (2023) John Schulman. Proxy objectives in reinforcement learning from human feedback. Invited Talk at the International Conference on MachineLearning (ICML), 2023. https://icml.c/virtual/2023/invited-talk/21549. Schulman et al. (2017) John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. Shen et al. (2024) Lingfeng Shen, Sihao Chen, Linfeng Song, Lifeng Jin, Baolin Peng, Haitao Mi, Daniel Khashabi, and Dong Yu. The trickle-down impact of reward inconsistency on RLHF. In The Twelfth International Conference on Learning Representations, 2024. Shen et al. (2023) Wei Shen, Rui Zheng, Wenyu Zhan, Jun Zhao, Shihan Dou, Tao Gui, Qi Zhang, and Xuanjing Huang. Loose lips sink ships: Mitigating length bias in reinforcement learning from human feedback. In Conference on Empirical Methods in Natural Language Processing, 2023. Singhal et al. (2023) Prasann Singhal, Tanya Goyal, Jiacheng Xu, and Greg Durrett. A long way to go: Investigating length correlations in RLHF. arXiv preprint arXiv:2310.03716, 2023. Skalse et al. (2022) Joar Skalse, Nikolaus Howe, Dmitrii Krasheninnikov, and David Krueger. Defining and characterizing reward gaming. Advances in Neural Information Processing Systems, 35:9460–9471, 2022. Song et al. (2024) Yuda Song, Gokul Swamy, Aarti Singh, Drew Bagnell, and Wen Sun. The importance of online data: Understanding preference fine-tuning via coverage. ICML Workshop: Aligning Reinforcement Learning Experimentalists and Theorists, 2024. Stiennon et al. (2020) Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008–3021, 2020. Swamy et al. (2024) Gokul Swamy, Christoph Dann, Rahul Kidambi, Steven Wu, and Alekh Agarwal. A minimaximalist approach to reinforcement learning from human feedback. In Forty-first International Conference on Machine Learning, 2024. Tang et al. (2024) Yunhao Tang, Zhaohan Daniel Guo, Zeyu Zheng, Daniele Calandriello, Remi Munos, Mark Rowland, Pierre Harvey Richemond, Michal Valko, Bernardo Avila Pires, and Bilal Piot. Generalized preference optimization: A unified approach to offline alignment. In Forty-first International Conference on Machine Learning, 2024. Tien et al. (2022) Jeremy Tien, Jerry Zhi-Yang He, Zackory Erickson, Anca Dragan, and Daniel S Brown. Causal confusion and reward misidentification in preference-based reward learning. In The Eleventh International Conference on Learning Representations, 2022. Tien et al. (2023) Jeremy Tien, Jerry Zhi-Yang He, Zackory Erickson, Anca Dragan, and Daniel S Brown. Causal confusion and reward misidentification in preference-based reward learning. In The Eleventh International Conference on Learning Representations, 2023. Tunstall et al. (2023) Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, et al. Zephyr: Direct distillation of LM alignment. arXiv preprint arXiv:2310.16944, 2023. VanderWeele (2011) Tyler J VanderWeele. Controlled direct and mediated effects: Definition, identification and bounds. Scandinavian Journal of Statistics, 38(3):551–563, 2011. Wang et al. (2024a) Binghai Wang, Rui Zheng, Lu Chen, Yan Liu, Shihan Dou, Caishuang Huang, Wei Shen, Senjie Jin, Enyu Zhou, Chenyu Shi, et al. Secrets of RLHF in large language models part I: Reward modeling. arXiv preprint arXiv:2401.06080, 2024a. Wang et al. (2024b) Chaoqi Wang, Yibo Jiang, Chenghao Yang, Han Liu, and Yuxin Chen. Beyond reverse KL: Generalizing direct preference optimization with diverse divergence constraints. In The Twelfth International Conference on Learning Representations, 2024b. Wang et al. (2024c) Haoxiang Wang, Wei Xiong, Tengyang Xie, Han Zhao, and Tong Zhang. Interpretable preferences via multi-objective reward modeling and mixture-of-experts. arXiv preprint arXiv:2406.12845, 2024c. Wang et al. (2023a) Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Chandu, David Wadden, Kelsey MacMillan, Noah A Smith, Iz Beltagy, et al. How far can camels go? Exploring the state of instruction tuning on open resources. Advances in Neural Information Processing Systems, 36:74764–74786, 2023a. Wang et al. (2023b) Yuanhao Wang, Qinghua Liu, and Chi Jin. Is RLHF more difficult than standard RL? arXiv preprint arXiv:2306.14111, 2023b. Wang et al. (2024d) Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. MMLU-PRO: A more robust and challenging multi-task language understanding benchmark. arXiv preprint arXiv:2406.01574, 2024d. Wang et al. (2024e) Zhilin Wang, Yi Dong, Olivier Delalleau, Jiaqi Zeng, Gerald Shen, Daniel Egert, Jimmy J Zhang, Makesh Narsimhan Sreedhar, and Oleksii Kuchaiev. HelpSteer2: Open-source dataset for training top-performing reward models. arXiv preprint arXiv:2406.08673, 2024e. Wei et al. (2024) Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. Jailbroken: How does LLM safety training fail? Advances in Neural Information Processing Systems, 36, 2024. Wirth et al. (2017) Christian Wirth, Riad Akrour, Gerhard Neumann, and Johannes Fürnkranz. A survey of preference-based reinforcement learning methods. Journal of Machine Learning Research, 18(136):1–46, 2017. Wu et al. (2021) Jeff Wu, Long Ouyang, Daniel M Ziegler, Nisan Stiennon, Ryan Lowe, Jan Leike, and Paul Christiano. Recursively summarizing books with human feedback. arXiv preprint arXiv:2109.10862, 2021. Wu et al. (2024a) Junkang Wu, Yuexiang Xie, Zhengyi Yang, Jiancan Wu, Jinyang Gao, Bolin Ding, Xiang Wang, and Xiangnan He. β-dpo: Direct preference optimization with dynamic β. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024a. Wu et al. (2024b) Yue Wu, Zhiqing Sun, Huizhuo Yuan, Kaixuan Ji, Yiming Yang, and Quanquan Gu. Self-play preference optimization for language model alignment. arXiv preprint arXiv:2405.00675, 2024b. Xie et al. (2021) Tengyang Xie, Ching-An Cheng, Nan Jiang, Paul Mineiro, and Alekh Agarwal. Bellman-consistent pessimism for offline reinforcement learning. Advances in neural information processing systems, 34:6683–6694, 2021. Xie et al. (2024) Tengyang Xie, Dylan J Foster, Akshay Krishnamurthy, Corby Rosset, Ahmed Awadallah, and Alexander Rakhlin. Exploratory preference optimization: Harnessing implicit Q*-approximation for sample-efficient RLHF. arXiv preprint arXiv:2405.21046, 2024. Xiong et al. (2023) Wei Xiong, Hanze Dong, Chenlu Ye, Han Zhong, Nan Jiang, and Tong Zhang. Gibbs sampling from human feedback: A provable KL-constrained framework for RLHF. arXiv preprint arXiv:2312.11456, 2023. Xu et al. (2024a) Haoran Xu, Amr Sharaf, Yunmo Chen, Weiting Tan, Lingfeng Shen, Benjamin Van Durme, Kenton Murray, and Young Jin Kim. Contrastive preference optimization: Pushing the boundaries of LLM performance in machine translation. In Forty-first International Conference on Machine Learning, 2024a. Xu et al. (2023) Jing Xu, Andrew Lee, Sainbayar Sukhbaatar, and Jason Weston. Some things are more cringe than others: Preference optimization with the pairwise cringe loss. arXiv preprint arXiv:2312.16682, 2023. Xu et al. (2024b) Tengyu Xu, Eryk Helenowski, Karthik Abinav Sankararaman, Di Jin, Kaiyan Peng, Eric Han, Shaoliang Nie, Chen Zhu, Hejia Zhang, Wenxuan Zhou, et al. The perfect blend: Redefining RLHF with mixture of judges. arXiv preprint arXiv:2409.20370, 2024b. Yuan et al. (2024a) Hongyi Yuan, Zheng Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang. RRHF: Rank responses to align language models with human feedback. Advances in Neural Information Processing Systems, 36, 2024a. Yuan et al. (2024b) Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, and Jason E Weston. Self-rewarding language models. In Forty-first International Conference on Machine Learning, 2024b. Zellers et al. (2019) Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791–4800, 2019. Zhai et al. (2023) Yuanzhao Zhai, Han Zhang, Yu Lei, Yue Yu, Kele Xu, Dawei Feng, Bo Ding, and Huaimin Wang. Uncertainty-penalized reinforcement learning from human feedback with diverse reward LoRA ensembles. arXiv preprint arXiv:2401.00243, 2023. Zhan et al. (2023a) Wenhao Zhan, Masatoshi Uehara, Nathan Kallus, Jason D Lee, and Wen Sun. Provable offline preference-based reinforcement learning. International Conference on Learning Representations, 2023a. Zhan et al. (2023b) Wenhao Zhan, Masatoshi Uehara, Wen Sun, and Jason D Lee. How to query human feedback efficiently in RL? In ICML 2023 Workshop The Many Facets of Preference-Based Learning, 2023b. Zhang et al. (2024) Xiaoying Zhang, Jean-Francois Ton, Wei Shen, Hongning Wang, and Yang Liu. Overcoming reward overoptimization via adversarial policy optimization with lightweight uncertainty estimation. arXiv preprint arXiv:2403.05171, 2024. Zhao et al. (2023) Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J Liu. SLiC-HF: Sequence likelihood calibration with human feedback. arXiv preprint arXiv:2305.10425, 2023. Zheng et al. (2023) Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. Advances in Neural Information Processing Systems, 36:46595–46623, 2023. Zhou et al. (2023) Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-following evaluation for large language models. arXiv preprint arXiv:2311.07911, 2023. Zhu et al. (2023) Banghua Zhu, Michael Jordan, and Jiantao Jiao. Principled reinforcement learning with human feedback from pairwise or k-wise comparisons. In International Conference on Machine Learning, pages 43037–43067. PMLR, 2023. Zhu et al. (2024) Banghua Zhu, Michael Jordan, and Jiantao Jiao. Iterative data smoothing: Mitigating reward overfitting and overoptimization in RLHF. In Forty-first International Conference on Machine Learning, 2024. Ziebart et al. (2008) Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, Anind K Dey, et al. Maximum entropy inverse reinforcement learning. In AAAI, volume 8, pages 1433–1438. Chicago, IL, USA, 2008. Ziegler et al. (2019) Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593, 2019. Appendix A Additional Notation We use calligraphy letters to denote sets, e.g., ,X,YX , Y. Given a response y, we write |y||y|| y | to denote the length of the response in the number of tokens. We denote by VV the vocabulary set and write |||V|| V | to denote the cardinality of the token space. We write x≲yless-than-or-similar-tox yx ≲ y when there exists a constant c such that x≤cyx≤ cyx ≤ c y and similarly, write x≍yasymptotically-equalsx yx ≍ y when there exists a constant c such that x=cyx=cyx = c y. We write y1≻y0succeedssuperscript1superscript0y^1 y^0y1 ≻ y0 denoting that y1superscript1y^1y1 is preferred over y0superscript0y^0y0 in the dataset. For any two discrete probability distributions π and π′π π′ over YY, we define the KL divergence DKL(π∥π′)≔y∼π[logπ(y)π′(y)]≔subscriptKLconditionalsuperscript′subscriptsimilar-todelimited-[]superscript′D_KL(π\|π ) _y π[ π(y% )π (y)]DKL ( π ∥ π′ ) ≔ italic_y ∼ π [ log divide start_ARG π ( y ) end_ARG start_ARG π′ ( y ) end_ARG ]. The probability simplex over a set XX is denoted by Δ()Δ (X)Δ ( X ). We write x=cdouble-struck-1\x=c\blackboard_ x = c to denote the indicator function, which is equal to 1111 when x=cx=cx = c and zero otherwise. We write subscriptE_Dcaligraphic_D to denote the empirical average over data. Appendix B Related Work B.1 RLHF and Preference Optimization Earlier works on reinforcement learning from human preferences mainly focused on the continuous control domain Wirth et al. (2017) such as Atari games Christiano et al. (2017). Recently, RLHF has been extensively applied in the natural language domain Ziegler et al. (2019) to improve alignment of LLMs with human preferences in various areas such as summarization Stiennon et al. (2020); Wu et al. (2021), information accuracy Menick et al. (2022), and instruction following Ouyang et al. (2022). Classical RLHF pipeline includes two steps of reward learning and policy optimization using RL, commonly using variants of proximal policy optimization (PPO) algorithm Schulman et al. (2017) that involves on-policy sampling. Direct preference optimization Rafailov et al. (2024b) simplifies the two-step process into a single-step offline optimization of the policy, reducing computational burden and training instabilities of PPO. DPO has inspired development of new preference optimization objectives from a practical perspective such as IPO Azar et al. (2024), RRHF Yuan et al. (2024a), SLiC-HF Zhao et al. (2023), CPO Xu et al. (2024a), ORPO Hong et al. (2024), R-DPO Park et al. (2024), SimPO Meng et al. (2024), and general preference optimization Tang et al. (2024) and theoretical perspective such as χPO Huang et al. (2024) and RPO (DPO+SFT) Liu et al. (2024). Our approach also falls under the category of offline preference optimization. We theoretically analyzed several of the mentioned methods and demonstrated theoretical benefits offered by our approach POWER-DL. We also showed that POWER-DL outperforms prior methods empirically across a variety of settings. Going beyond the Bradley-Terry model of human preferences, some works consider general preference models Munos et al. (2023); Swamy et al. (2024); Rosset et al. (2024); Choi et al. (2024); Wu et al. (2024b), and develop algorithms aiming at finding the Nash equilibrium. Recently Huang et al. (2024) showed that this generalization comes at a cost of an information-theoretic limit, where no statistically efficient algorithm exists to solve RLHF with general preferences under single-policy concentrability. Another line of work focuses on iterative, online improvement of language models through self-play Chen et al. (2024b); Wu et al. (2024b); Xu et al. (2023); Yuan et al. (2024b). B.2 Understanding Reward Hacking The phenomenon of reward hacking in training AI models has been observed in a variety of domains, ranging from games Ibarz et al. (2018) to natural language Paulus et al. (2018) to autonomous driving Knox et al. (2023). In the language modeling domain, existing RLHF algorithms are observed to be susceptible to reward hacking Gao et al. (2023); Casper et al. (2023); Amodei et al. (2016); Lambert and Calandra (2023). Reward hacking in LLMs manifests in different ways such as verbosity Shen et al. (2023); Singhal et al. (2023); Wang et al. (2023a), refusing to follow instructions Röttger et al. (2024), lazy generations Lambert and Calandra (2023), emergence of language Lewis et al. (2017), degradation of performance on downstream tasks such as reasoning Xu et al. (2024b), and other problems such as hedging and self-doubt Schulman (2023). Origins of reward hacking. In RL/RLHF, reward hacking can originate from various factors such as reward misspecification Amodei et al. (2016); Hadfield-Menell et al. (2017); Knox et al. (2023), diversity and inconsistencies in human preferences Chakraborty et al. (2024), labeling noise Wang et al. (2024a), human labeler bias Bansal et al. (2024), and statistical errors Liu et al. (2024). Pan et al. (2022) study reward hacking due to human misspecification of the reward model and empirically assess the impact of model size, optimization, and training on reward hacking, given synthetic misspecified reward models. Wei et al. (2024) attribute failure modes of LLM safety training to conflicts between model’s capabilities and safety goals. Bansal et al. (2024) study mismatch arising from annotator bias in different types of human rating data. Peng et al. (2023) explain that variations of reward distribution across different tasks can lead to reward hacking. Rame et al. (2024) attribute reward hacking in classical RLHF to distribution shift and human preference inconsistencies. Tien et al. (2023) conduct an empirical study, revealing that non-causal distractor features, human bias and noise, and partial observability exacerbate reward misidentification. Lambert and Calandra (2023) argue that objective mismatch in RLHF originates from learning reward model, policy training, and evaluation, and links between each pair, and suggest further research is needed to understand objective mismatch in preference optimization due to entanglement of policy and reward. Rafailov et al. (2024a) conduct an empirical study of reward hacking in direct preference optimization methods, showing that in larger KL regimes, preference optimization methods suffer from degradations reminiscent of overoptimization in RLHF. In contrast to the above works, we focus on reward hacking in offline preference optimization that originates from statistical fluctuations due to partial data coverage. Furthermore, the mentioned works conduct empirical studies, whereas here we present statistical learning theory characterizations of reward hacking. B.3 Reward Hacking Types and Comparison with Pessimism in Offline RL We now highlight the differences between the setting considered in this paper and conventional offline RL, explaining usefulness of defining two types of reward hacking. In the practice of RLHF fine-tuning of LLMs, we typically have access an initial model, which already has decent performance on many downstream tasks, and a previously-collected preference data, which may not have been sampled from initial model Xu et al. (2024b); Wang et al. (2024e). In this setting, we face two sources of distribution shift: one between the final model and data distribution, and the other between the initial model and data distribution. This setting is different from conventional offline RL, which considers access to an offline dataset (with possibly known data collection policy) and is only concerned with distribution shift between final model and data distribution Levine et al. (2020). Due to the existence of two sources of distribution shift, we find it useful to define Type I and Type I Reward Hacking. These definitions motivate the design of our algorithm that achieves strong empirical performance. Furthermore, our empirical results removing dynamic labels (POWER vs. POWER-DL) as well as removing the SFT term (Appendix I.4) show that the two components contribute achieving the best empirical performance. Liu et al. (2024) and Huang et al. (2024) also consider reward hacking due to partial data coverage. However, Huang et al. (2024) assume preference data are collected from the initial model, which eliminates the distribution shift between initial model and data distribution. Liu et al. (2024) propose DPO+SFT to handle the distribution shift between the final model and data distribution, yet; reward hacking due to degradation of the initial model is not considered. In this paper, we present separate analysis for POWER (Theorem 1) and dynamic labels (Theorem 2). Combining these two results into a unified analysis of POWER-DL is challenging due to extending the analysis of dynamic labels to general function approximation, which we leave for future work. B.4 Mitigating Reward Hacking Various approaches have been proposed to mitigate reward hacking from applied and theoretical perspectives. Michaud et al. (2020) propose using interpretability techniques for probing whether learned rewards are aligned with human preferences. To mitigate reward hacking, several methods leverage multiple reward models. Moskovitz et al. (2024); Xu et al. (2024b) develop constrained policy optimization frameworks that leverage multiple reward models and assign weights to each of the reward models to mitigate reward hacking. Reward model ensembles Coste et al. (2024); Zhai et al. (2023) aim to characterize uncertainty and can alleviate reward hacking; however, empirical investigations observe that they are not sufficient Eisenstein et al. (2023). Rame et al. (2024) propose averaging the weights of multiple trained reward models instead of ensembles to improve efficiency and performance. In contrast to these methods, our approach does not require access to or training multiple reward models. To reduce the computational costs of ensemble methods, Zhang et al. (2024) construct lightweight uncertainty estimation via linear approximation that yields a pessimistic reward model. However, such uncertainty quantification requires restrictive neural tangent kernel and approximately linear assumptions while the guarantees for our approach hold under general function approximation. Other works use additional data to reduce reward hacking. Rita et al. (2024) leverage additional human demonstrations to calibrate the reward model and Shen et al. (2024) propose methods that leverage data augmentation to improve consistency of the reward model. RaFT Dong et al. (2023) reduces instabilities of RLHF through iterative supervised finetuning that only keeps the highest ranked responses from a reward model. Peng et al. (2023) propose using advantage models instead of values. We propose theoretically-founded methods to mitigate reward hacking, focusing on the offline setting. Our approaches are implemented with simple modifications to objective of DPO and directly leverage previously-collected dataset; without requiring training any additional models, generating or augmenting data, or computationally expensive operation. Among preference optimization methods, the most common approach is divergence regularization that aims at keeping the learned model close to the initial model, through metrics such as KL divergence Rafailov et al. (2024b), f-divergence Wang et al. (2024b), or Chi-squared divergence Huang et al. (2024). βDPO Wu et al. (2024a) calibrates β, which is the strength of divergence minimization, based on the implicit reward gap and dynamically subsamples each batch to increase robustness with respect to outliers. Other methods such as conservative DPO Mitchell (2024) and ROPO Liang et al. (2024) design robust variants of DPO. We proved that many divergence-based methods still suffer from reward hacking and showed that our proposed methods outperform divergence-based and prior robust methods empirically. B.5 Mitigating Specific Manifestations of Reward Hacking Several works focus on mitigating specific artifacts of reward hacking such as verbosity through various designs. Design of length-controlled winrate Alpaca-Eval Dubois et al. (2024) aims at making the evaluation more robust against length exploitation, through estimation of controlled direct effect VanderWeele (2011). ODIN Chen et al. (2024a) enforces disentanglement of preference estimation and response length by using two linear heads. In preference optimization, length-normalization Meng et al. (2024); Grinsztajn et al. (2024); Yuan et al. (2024a) and length regularization Park et al. (2024) are used to mitigate length exploitation. In this paper, we consider reward hacking due to partial data coverage that can manifest in many different ways and not just length. Our weighted entropy approach provides a general framework for handling specific manifestations of reward hacking by selecting weights w(y)w(y)w ( y ) that are smaller for undesirable response properties, such as inverse response length or preference scores. Furthermore, our approach provides a theoretically-sound way of incorporating such weights into preference optimization objective (Proposition 3), that is different from previous methods. For example, compared to SLiC-HF and SimPO, our approach results in a weight gap in the preference optimization objective and a weighted SFT term. In our practical implementation, we used inverse response length as weights, which we show in Theorem 1 prevents sample complexity to depend linearly on the response length. Another potential benefit of our weighted entropy approach is disentangling entropy (controlled through β) and conservatism components (controlled through η and γ). Adjusting the level of stochasticity of final learned model through β may result in alleviating the notorious overconfidence challenge, in which RLHF-finetuned models become overconfident and have sharpened output probability Leng et al. (2024); Kadavath et al. (2022). In contrast, in DPO, β is the coefficient of the KL divergence, which impacts both pessimism and stochasticity of the learned model. B.6 Theory of RLHF and Preference Optimization A series of works study theoretical foundations for RLHF and preference optimization under different settings Zhu et al. (2023); Xiong et al. (2023); Zhu et al. (2024); Liu et al. (2024); Huang et al. (2024); Song et al. (2024); Fisch et al. (2024). Xiong et al. (2023); Zhu et al. (2023) propose provable pessimistic offline RLHF algorithms either through confidence regions or lower confidence bounds, but are restricted to the linear family of models. Zhu et al. (2023) show that maximum entropy IRL is similar to the maximum likelihood under the Plackett-Luce models; however, entropy in maximum entropy IRL is different from our use of weighted entropy in objective (6), which goes beyond optimizing the Bradley-Terry loss. Other works that develop provable algorithms with general function approximation Chen et al. (2022); Zhan et al. (2023a, b); Wang et al. (2023b); Li et al. (2023b) involve intractable computation. Exceptions include χPO Huang et al. (2024) and DPO+SFT Liu et al. (2024); Cen et al. (2024), which we have compared with POWER-DL from theoretical and empirical fronts. Appendix C Proofs for Reward Hacking C.1 Type I Reward Hacking: Proof of Proposition 1 We first construct two multi-armed bandit (MAB) instances and then analyze each algorithm. C.1.1 MAB Instances for Type I Reward Hacking We construct a three-armed bandit problem, with true rewards r⋆(1)=1superscript⋆11r (1)=1r⋆ ( 1 ) = 1, r⋆(3)=0superscript⋆30r (3)=0r⋆ ( 3 ) = 0, and all actions having length one |y|=11|y|=1| y | = 1. We consider a preference data distribution that has high coverage on the high reward arms, where the probability of comparing arms 1 and 2 is μ1,2=1−1/Nsubscript1211 _1,2=1-1/Nμ1 , 2 = 1 - 1 / N and the probability of comparing arms 1 and 3 is μ1,3=1/Nsubscript131 _1,3=1/Nμ1 , 3 = 1 / N. In this scenario, there is a constant probability that arm 3 is compared with arm 1 exactly once. To demonstrate this, let N(i,j)N(i,j)N ( i , j ) denote the number of comparisons between arms i and j. We have ℙ(N(1,3)=1)=N(1−μ1,3)N−1μ1,3=(1−1/N)N−1.ℙ131superscript1subscript131subscript13superscript111 P(N(1,3)=1)=N(1- _1,3)^N-1 _1,3=(1-1/N)^N-1.ℙ ( N ( 1 , 3 ) = 1 ) = N ( 1 - μ1 , 3 )N - 1 μ1 , 3 = ( 1 - 1 / N )N - 1 . For any N≥22N≥ 2N ≥ 2, the above probability is bounded below according to ℙ(N(1,3)=1)=(1−1/N)N−1≥1/e.ℙ131superscript1111 P(N(1,3)=1)=(1-1/N)^N-1≥ 1/e.ℙ ( N ( 1 , 3 ) = 1 ) = ( 1 - 1 / N )N - 1 ≥ 1 / e . (13) Conditioned on the event N(1,3)=1131N(1,3)=1N ( 1 , 3 ) = 1, there is a constant probability that arm 3 is preferred over arm 1 according to the Bradley-Terry model: ℙr⋆(3≻1)=σ(r⋆(3)−r⋆(1))=σ(−1)=1/(1+e).subscriptℙsuperscript⋆succeeds31superscript⋆3superscript⋆1111 P_r (3 1)=σ(r (% 3)-r (1))=σ(-1)=1/(1+e).ℙitalic_r⋆ ( 3 ≻ 1 ) = σ ( r⋆ ( 3 ) - r⋆ ( 1 ) ) = σ ( - 1 ) = 1 / ( 1 + e ) . (14) Throughout the rest of the proof, we condition on the event ℰ=N(1,3)=1 and 3≻1ℰ131 and 3succeeds1E=\N(1,3)=1 and 3 1\E = N ( 1 , 3 ) = 1 and 3 ≻ 1 which occurs with a probability of at least 1/e(1+e)111/e(1+e)1 / e ( 1 + e ). We further consider two special instances of the above MAB problem, with the following specifications for the initial model parameters and the reward of the second arm: • Instance 1: True reward of the second arm is r⋆(2)=0superscript⋆20r (2)=0r⋆ ( 2 ) = 0 and initial parameters are θ0(1)=θ0(2)=θ0(3)=1subscript01subscript02subscript031 _0(1)= _0(2)= _0(3)=1θ0 ( 1 ) = θ0 ( 2 ) = θ0 ( 3 ) = 1. In this case, the best-in-class softmax policy has parameters θ⋆(1)=1,θ⋆(2)=θ⋆(3)=0formulae-sequencesuperscript⋆11superscript⋆2superscript⋆30θ (1)=1,θ (2)=θ (3)=0θ⋆ ( 1 ) = 1 , θ⋆ ( 2 ) = θ⋆ ( 3 ) = 0. • Instance 2: True reward of the second arm is r⋆(2)=1superscript⋆21r (2)=1r⋆ ( 2 ) = 1 and initial parameters are θ0(1)=θ0(2)=1,θ0(3)=0formulae-sequencesubscript01subscript021subscript030 _0(1)= _0(2)=1, _0(3)=0θ0 ( 1 ) = θ0 ( 2 ) = 1 , θ0 ( 3 ) = 0. In this case, the best-in-class softmax policy has parameters θ⋆(1)=θ⋆(2)=1,θ⋆(3)=0formulae-sequencesuperscript⋆1superscript⋆21superscript⋆30θ (1)=θ (2)=1,θ (3)=0θ⋆ ( 1 ) = θ⋆ ( 2 ) = 1 , θ⋆ ( 3 ) = 0. We additionally consider a favorable scenario, in which an oracle reveals the best-in-class values of the first two arms θ⋆(1),θ⋆(2)superscript⋆1superscript⋆2θ (1),θ (2)θ⋆ ( 1 ) , θ⋆ ( 2 ). This simplifies the preference optimization objectives as it remains for the preference optimization algorithm to find θ(3)3θ(3)θ ( 3 ). C.1.2 Analysis of ⋆ ⋆PO Methods in the MAB Instances in Section C.1.1 We first record the following expression for the difference of the log probabilities of any two arms in the softmax policy class: logπθ(y)−logπθ(y′)=logexp(θ(y))−logZθ−logexp(θ(y′))+logZθ=θ(y)−θ(y′).subscriptsubscriptsuperscript′subscriptsuperscript′subscriptsuperscript′ split _θ(y)- _θ(y )&=% (θ(y))- Z_θ- (θ(y ))+ Z_% θ\\ &=θ(y)-θ(y ). splitstart_ROW start_CELL log πitalic_θ ( y ) - log πitalic_θ ( y′ ) end_CELL start_CELL = log exp ( θ ( y ) ) - log Zitalic_θ - log exp ( θ ( y′ ) ) + log Zitalic_θ end_CELL end_ROW start_ROW start_CELL end_CELL start_CELL = θ ( y ) - θ ( y′ ) . end_CELL end_ROW (15) Suboptimality of DPO. We show that DPO fails in both instances constructed in Section C.1.1. Since parameters θ(1)1θ(1)θ ( 1 ) and θ(2)2θ(2)θ ( 2 ) are revealed by the oracle, the optimization problem solved by DPO simplifies to maxθ(3)∈[0,1]1N[logσ(β(logπθ(3)πθ0(3)−logπθ(1)πθ0(1)))]subscript3011delimited-[]subscript3subscriptsubscript03subscript1subscriptsubscript01 _θ(3)∈[0,1] 1N [ σ (β% ( _θ(3) _ _0(3)- _θ(1% ) _ _0(1) ) ) ]maxitalic_θ ( 3 ) ∈ [ 0 , 1 ] divide start_ARG 1 end_ARG start_ARG N end_ARG [ log σ ( β ( log divide start_ARG πitalic_θ ( 3 ) end_ARG start_ARG πitalic_θ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ( 3 ) end_ARG - log divide start_ARG πitalic_θ ( 1 ) end_ARG start_ARG πitalic_θ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ( 1 ) end_ARG ) ) ] = == maxθ(3)∈[0,1]logσ(β(θ(3)−θ0(3)−θ(1)−θ0(1)))subscript3013subscript031subscript01 _θ(3)∈[0,1] σ (β (θ(3)-% _0(3)-θ(1)- _0(1) ) )maxitalic_θ ( 3 ) ∈ [ 0 , 1 ] log σ ( β ( θ ( 3 ) - θ0 ( 3 ) - θ ( 1 ) - θ0 ( 1 ) ) ) = == maxθ(3)∈[0,1]logσ(β(θ(3)−θ0(3)))subscript3013subscript03 _θ(3)∈[0,1] σ (β (θ(3)-% _0(3) ) )maxitalic_θ ( 3 ) ∈ [ 0 , 1 ] log σ ( β ( θ ( 3 ) - θ0 ( 3 ) ) ) The first equality is due to (15) and the second equality is because θ0(1)=θ(1)=1subscript0111 _0(1)=θ(1)=1θ0 ( 1 ) = θ ( 1 ) = 1. For β>00β>0β > 0 and regardless of θ0(3)subscript03 _0(3)θ0 ( 3 ), the above function is increasing in θ(3)3θ(3)θ ( 3 ), and thus the maximum occurs at θ(3)=131θ(3)=1θ ( 3 ) = 1. As a result, DPO fails in both instances constructed in Section C.1.1, learning a policy with constant suboptimality: Instance 1: J(π⋆)−J(π^DPO)=y∼πθ⋆[r⋆(y)]−y∼π^DPO[r⋆(y)]=e2+e−e1+2e>0.15superscript⋆subscript^DPOsubscriptsimilar-tosubscriptsuperscript⋆delimited-[]superscript⋆subscriptsimilar-tosubscript^DPOdelimited-[]superscript⋆2120.15 J(π )-J( π_DPO)=E_y _% θ [r (y)]-E_y π_DPO[r^% (y)]= e2+e- e1+2e>0.15J ( π⋆ ) - J ( over start_ARG π end_ARGDPO ) = italic_y ∼ π start_POSTSUBSCRIPT θ⋆ end_POSTSUBSCRIPT [ r⋆ ( y ) ] - italic_y ∼ over start_ARG π end_ARG start_POSTSUBSCRIPT DPO end_POSTSUBSCRIPT [ r⋆ ( y ) ] = divide start_ARG e end_ARG start_ARG 2 + e end_ARG - divide start_ARG e end_ARG start_ARG 1 + 2 e end_ARG > 0.15 Instance 2: J(π⋆)−J(π^DPO)=y∼πθ⋆[r⋆(y)]−y∼π^DPO[r⋆(y)]=2e1+2e−23>0.15superscript⋆subscript^DPOsubscriptsimilar-tosubscriptsuperscript⋆delimited-[]superscript⋆subscriptsimilar-tosubscript^DPOdelimited-[]superscript⋆212230.15 J(π )-J( π_DPO)=E_y _% θ [r (y)]-E_y π_DPO[r^% (y)]= 2e1+2e- 23>0.15J ( π⋆ ) - J ( over start_ARG π end_ARGDPO ) = italic_y ∼ π start_POSTSUBSCRIPT θ⋆ end_POSTSUBSCRIPT [ r⋆ ( y ) ] - italic_y ∼ over start_ARG π end_ARG start_POSTSUBSCRIPT DPO end_POSTSUBSCRIPT [ r⋆ ( y ) ] = divide start_ARG 2 e end_ARG start_ARG 1 + 2 e end_ARG - divide start_ARG 2 end_ARG start_ARG 3 end_ARG > 0.15 Suboptimality of IPO. We show that IPO fails in Instance 1 constructed in Section C.1.1. Leveraging uniform initialization and the logit gap expression (15), the IPO objective can be simplified as follows: minθ[(logπθ(y+)πθ0(y+)−logπθ(y−)πθ0(y−)−12τ)2]=minθ[(θ(y+)−θ(y−)−12τ)2]subscriptsubscriptdelimited-[]superscriptsubscriptsuperscriptsubscriptsubscript0superscriptsubscriptsuperscriptsubscriptsubscript0superscript122subscriptsubscriptdelimited-[]superscriptsuperscriptsuperscript122 _θE_D [ ( _% θ(y^+) _ _0(y^+)- _θ(y^-) _% _0(y^-)- 12τ )^2 ]= _θE_% D [ (θ(y^+)-θ(y^-)- 12τ )^2% ]minitalic_θ caligraphic_D [ ( log divide start_ARG πitalic_θ ( y+ ) end_ARG start_ARG πitalic_θ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ( y+ ) end_ARG - log divide start_ARG πitalic_θ ( y- ) end_ARG start_ARG πitalic_θ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ( y- ) end_ARG - divide start_ARG 1 end_ARG start_ARG 2 τ end_ARG )2 ] = minitalic_θ caligraphic_D [ ( θ ( y+ ) - θ ( y- ) - divide start_ARG 1 end_ARG start_ARG 2 τ end_ARG )2 ] Since θ(1)=θ(2)12θ(1)=θ(2)θ ( 1 ) = θ ( 2 ) are revealed by an oracle, the IPO objective can be further simplified to minθ(3)∈[0,1]1N(θ(3)−θ(1)−12τ)2=minθ(3)∈[0,1](θ(3)−1−12τ)2 _θ(3)∈[0,1] 1N (θ(3)-θ(1)-% 12τ )^2= _θ(3)∈[0,1] (θ(3)-1- 1% 2τ )^2minitalic_θ ( 3 ) ∈ [ 0 , 1 ] divide start_ARG 1 end_ARG start_ARG N end_ARG ( θ ( 3 ) - θ ( 1 ) - divide start_ARG 1 end_ARG start_ARG 2 τ end_ARG )2 = minitalic_θ ( 3 ) ∈ [ 0 , 1 ] ( θ ( 3 ) - 1 - divide start_ARG 1 end_ARG start_ARG 2 τ end_ARG )2 Over the interval of θ(3)∈[0,1]301θ(3)∈[0,1]θ ( 3 ) ∈ [ 0 , 1 ] and for any τ>00τ>0τ > 0, this objective is decreasing in θ(3)3θ(3)θ ( 3 ) and therefore the optimum is found at θ(3)=131θ(3)=1θ ( 3 ) = 1. Thus, the policy found by IPO suffers from the following subpoptimality: J(πθ⋆)−J(π^IPO)subscriptsuperscript⋆subscript^IPO J( _θ )-J( π_IPO)J ( πitalic_θ⋆ ) - J ( over start_ARG π end_ARGIPO ) =y∼πθ⋆[r⋆(y)]−y∼π^IPO[r⋆(y)]=e2+e−e1+2e>0.15absentsubscriptsimilar-tosubscriptsuperscript⋆delimited-[]superscript⋆subscriptsimilar-tosubscript^IPOdelimited-[]superscript⋆2120.15 =E_y _θ [r (y)]-E_% y π_IPO[r (y)]= e2+e- e1+2e>0.15= italic_y ∼ π start_POSTSUBSCRIPT θ⋆ end_POSTSUBSCRIPT [ r⋆ ( y ) ] - italic_y ∼ over start_ARG π end_ARG start_POSTSUBSCRIPT IPO end_POSTSUBSCRIPT [ r⋆ ( y ) ] = divide start_ARG e end_ARG start_ARG 2 + e end_ARG - divide start_ARG e end_ARG start_ARG 1 + 2 e end_ARG > 0.15 Suboptimality of SimPO. We show that SimPO suffers from Type I Reward Hacking in both instances detailed in Section C.1.1. Following the same steps as in our analysis of DPO and since θ(1)=θ(2)=1121θ(1)=θ(2)=1θ ( 1 ) = θ ( 2 ) = 1 are assumed to be revealed by an oracle, SimPO objective simplifies to maxθ(3)∈[0,1]1Nlogσ(β(θ(3)−θ(1)−γ)).subscript301131 _θ(3)∈[0,1] 1N σ (β (% θ(3)-θ(1)-γ ) ).maxitalic_θ ( 3 ) ∈ [ 0 , 1 ] divide start_ARG 1 end_ARG start_ARG N end_ARG log σ ( β ( θ ( 3 ) - θ ( 1 ) - γ ) ) . Regardless of the value of γ and for any β>00β>0β > 0, the function logσ(β(θ(3)−θ(1)−γ))31 σ (β (θ(3)-θ(1)-γ ) )log σ ( β ( θ ( 3 ) - θ ( 1 ) - γ ) ) is increasing in θ(3)3θ(3)θ ( 3 ) and thus optimizing this objective over θ(3)∈[0,1]301θ(3)∈[0,1]θ ( 3 ) ∈ [ 0 , 1 ] finds θ(3)=131θ(3)=1θ ( 3 ) = 1. Therefore, policies found by SimPO in both instances in Section C.1.1 suffer from constant suboptimality: Instance 1: J(π⋆)−J(π^SimPO)=y∼πθ⋆[r⋆(y)]−y∼π^SimPO[r⋆(y)]=e2+e−e1+2e>0.15superscript⋆subscript^SimPOsubscriptsimilar-tosubscriptsuperscript⋆delimited-[]superscript⋆subscriptsimilar-tosubscript^SimPOdelimited-[]superscript⋆2120.15 J(π )-J( π_SimPO)=E_y _% θ [r (y)]-E_y π_SimPO[r^% (y)]= e2+e- e1+2e>0.15J ( π⋆ ) - J ( over start_ARG π end_ARGSimPO ) = italic_y ∼ π start_POSTSUBSCRIPT θ⋆ end_POSTSUBSCRIPT [ r⋆ ( y ) ] - italic_y ∼ over start_ARG π end_ARG start_POSTSUBSCRIPT SimPO end_POSTSUBSCRIPT [ r⋆ ( y ) ] = divide start_ARG e end_ARG start_ARG 2 + e end_ARG - divide start_ARG e end_ARG start_ARG 1 + 2 e end_ARG > 0.15 Instance 2: J(π⋆)−J(π^SimPO)=y∼πθ⋆[r⋆(y)]−y∼π^SimPO[r⋆(y)]=2e1+2e−23>0.15superscript⋆subscript^SimPOsubscriptsimilar-tosubscriptsuperscript⋆delimited-[]superscript⋆subscriptsimilar-tosubscript^SimPOdelimited-[]superscript⋆212230.15 J(π )-J( π_SimPO)=E_y _% θ [r (y)]-E_y π_SimPO[r^% (y)]= 2e1+2e- 23>0.15J ( π⋆ ) - J ( over start_ARG π end_ARGSimPO ) = italic_y ∼ π start_POSTSUBSCRIPT θ⋆ end_POSTSUBSCRIPT [ r⋆ ( y ) ] - italic_y ∼ over start_ARG π end_ARG start_POSTSUBSCRIPT SimPO end_POSTSUBSCRIPT [ r⋆ ( y ) ] = divide start_ARG 2 e end_ARG start_ARG 1 + 2 e end_ARG - divide start_ARG 2 end_ARG start_ARG 3 end_ARG > 0.15 Suboptimality of χPO. We analyze χPO for Instance 2 constructed in Section C.1.1. The χPO objective is given by maxθ(3)∈[0,1](1−1N)[μ^1≻2log(σ(2[β(logπθ(1)πθ0(1)−logπθ(2)πθ0(2)+πθ(1)πθ0(1)−πθ(2)πθ0(2))]))+μ^2≻1log(σ(2[β(logπθ(2)πθ0(2)−logπθ(1)πθ0(1)+πθ(2)πθ0(2)−πθ(1)πθ0(1))]))]+1N[log(σ(2[β(logπθ(3)πθ0(3)−logπθ(1)πθ0(1)+πθ(3)πθ0(3)−πθ(1)πθ0(1))]))]subscript30111delimited-[]subscript^succeeds12subscript2delimited-[]subscript1subscriptsubscript01subscript2subscriptsubscript02subscript1subscriptsubscript01subscript2subscriptsubscript02subscript^succeeds21subscript2delimited-[]subscript2subscriptsubscript02subscript1subscriptsubscript01subscript2subscriptsubscript02subscript1subscriptsubscript011delimited-[]subscript2delimited-[]subscript3subscriptsubscript03subscript1subscriptsubscript01subscript3subscriptsubscript03subscript1subscriptsubscript01 split _θ(3)∈[0,1] (1- 1N )% [& μ_1 2 (σ ( clip_2 [% β ( _θ(1) _ _0(1)- _% θ(2) _ _0(2)+ _θ(1) _ _0(1)-% _θ(2) _ _0(2) ) ] ) )\\ &+ μ_2 1 (σ ( clip_2 [β (% _θ(2) _ _0(2)- _θ(1)π% _ _0(1)+ _θ(2) _ _0(2)- _% θ(1) _ _0(1) ) ] ) ) ]\\ + 1N& [ (σ ( clip_2 [β (% _θ(3) _ _0(3)- _θ(1)π% _ _0(1)+ _θ(3) _ _0(3)- _% θ(1) _ _0(1) ) ] ) ) ] splitstart_ROW start_CELL maxitalic_θ ( 3 ) ∈ [ 0 , 1 ] ( 1 - divide start_ARG 1 end_ARG start_ARG N end_ARG ) [ end_CELL start_CELL over start_ARG μ end_ARG1 ≻ 2 log ( σ ( sansserif_clip2 [ β ( log divide start_ARG πitalic_θ ( 1 ) end_ARG start_ARG πitalic_θ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ( 1 ) end_ARG - log divide start_ARG πitalic_θ ( 2 ) end_ARG start_ARG πitalic_θ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ( 2 ) end_ARG + divide start_ARG πitalic_θ ( 1 ) end_ARG start_ARG πitalic_θ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ( 1 ) end_ARG - divide start_ARG πitalic_θ ( 2 ) end_ARG start_ARG πitalic_θ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ( 2 ) end_ARG ) ] ) ) end_CELL end_ROW start_ROW start_CELL end_CELL start_CELL + over start_ARG μ end_ARG2 ≻ 1 log ( σ ( sansserif_clip2 [ β ( log divide start_ARG πitalic_θ ( 2 ) end_ARG start_ARG πitalic_θ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ( 2 ) end_ARG - log divide start_ARG πitalic_θ ( 1 ) end_ARG start_ARG πitalic_θ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ( 1 ) end_ARG + divide start_ARG πitalic_θ ( 2 ) end_ARG start_ARG πitalic_θ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ( 2 ) end_ARG - divide start_ARG πitalic_θ ( 1 ) end_ARG start_ARG πitalic_θ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ( 1 ) end_ARG ) ] ) ) ] end_CELL end_ROW start_ROW start_CELL + divide start_ARG 1 end_ARG start_ARG N end_ARG end_CELL start_CELL [ log ( σ ( sansserif_clip2 [ β ( log divide start_ARG πitalic_θ ( 3 ) end_ARG start_ARG πitalic_θ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ( 3 ) end_ARG - log divide start_ARG πitalic_θ ( 1 ) end_ARG start_ARG πitalic_θ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ( 1 ) end_ARG + divide start_ARG πitalic_θ ( 3 ) end_ARG start_ARG πitalic_θ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ( 3 ) end_ARG - divide start_ARG πitalic_θ ( 1 ) end_ARG start_ARG πitalic_θ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ( 1 ) end_ARG ) ] ) ) ] end_CELL end_ROW (16) By construction, θ0(1)=θ0(2),πθ0(1)=πθ0(2)formulae-sequencesubscript01subscript02subscriptsubscript01subscriptsubscript02 _0(1)= _0(2), _ _0(1)= _ _0(2)θ0 ( 1 ) = θ0 ( 2 ) , πitalic_θ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ( 1 ) = πitalic_θ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ( 2 ), and θ(1)=θ(2)=1121θ(1)=θ(2)=1θ ( 1 ) = θ ( 2 ) = 1 are known, and therefore the first two terms in (16) are equal to zero: logπθ(1)πθ0(1)−logπθ(2)πθ0(2)+πθ(1)πθ0(1)−πθ(2)πθ0(2)subscript1subscriptsubscript01subscript2subscriptsubscript02subscript1subscriptsubscript01subscript2subscriptsubscript02 _θ(1) _ _0(1)- _% θ(2) _ _0(2)+ _θ(1) _ _0(1)-% _θ(2) _ _0(2)log divide start_ARG πitalic_θ ( 1 ) end_ARG start_ARG πitalic_θ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ( 1 ) end_ARG - log divide start_ARG πitalic_θ ( 2 ) end_ARG start_ARG πitalic_θ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ( 2 ) end_ARG + divide start_ARG πitalic_θ ( 1 ) end_ARG start_ARG πitalic_θ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ( 1 ) end_ARG - divide start_ARG πitalic_θ ( 2 ) end_ARG start_ARG πitalic_θ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ( 2 ) end_ARG =θ(1)−θ0(1)−θ(2)+θ0(2)+1πθ0(1)(exp(θ(1))Zθ−exp(θ(2))Zθ)=0.absent1subscript012subscript021subscriptsubscript011subscript2subscript0 =θ(1)- _0(1)-θ(2)+ _0(2)+ 1% _ _0(1) ( (θ(1))Z_θ- (θ% (2))Z_θ )=0.= θ ( 1 ) - θ0 ( 1 ) - θ ( 2 ) + θ0 ( 2 ) + divide start_ARG 1 end_ARG start_ARG πitalic_θ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ( 1 ) end_ARG ( divide start_ARG exp ( θ ( 1 ) ) end_ARG start_ARG Zitalic_θ end_ARG - divide start_ARG exp ( θ ( 2 ) ) end_ARG start_ARG Zitalic_θ end_ARG ) = 0 . Thus the maximization problem in (16) is simplified to maxθ(3)∈[0,1]logσ2β(logπθ(3)πθ0(3)−logπθ(1)πθ0(1)+πθ(3)πθ0(3)−πθ(1)πθ0(1))subscript301subscript2subscript3subscriptsubscript03subscript1subscriptsubscript01subscript3subscriptsubscript03subscript1subscriptsubscript01 _θ(3)∈[0,1] σ clip_2β (% _θ(3) _ _0(3)- _θ(1)π% _ _0(1)+ _θ(3) _ _0(3)- _% θ(1) _ _0(1) )maxitalic_θ ( 3 ) ∈ [ 0 , 1 ] log σ sansserif_clip2 β ( log divide start_ARG πitalic_θ ( 3 ) end_ARG start_ARG πitalic_θ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ( 3 ) end_ARG - log divide start_ARG πitalic_θ ( 1 ) end_ARG start_ARG πitalic_θ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ( 1 ) end_ARG + divide start_ARG πitalic_θ ( 3 ) end_ARG start_ARG πitalic_θ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ( 3 ) end_ARG - divide start_ARG πitalic_θ ( 1 ) end_ARG start_ARG πitalic_θ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ( 1 ) end_ARG ) =maxθ(3)∈[0,1]logσ2β(θ(3)−θ0(3)−θ(1)+θ0(1)+(1+2e)(exp(θ(3))Zθ−exp(θ(1))eZθ))absentsubscript301subscript23subscript031subscript01123subscript1subscript = _θ(3)∈[0,1] σ clip_2β% (θ(3)- _0(3)-θ(1)+ _0(1)+(1+2e) ( (% θ(3))Z_θ- (θ(1))eZ_θ ) )= maxitalic_θ ( 3 ) ∈ [ 0 , 1 ] log σ sansserif_clip2 β ( θ ( 3 ) - θ0 ( 3 ) - θ ( 1 ) + θ0 ( 1 ) + ( 1 + 2 e ) ( divide start_ARG exp ( θ ( 3 ) ) end_ARG start_ARG Zitalic_θ end_ARG - divide start_ARG exp ( θ ( 1 ) ) end_ARG start_ARG e Zitalic_θ end_ARG ) ) =maxθ(3)∈[0,1]logσ2β(θ(3)+(1+2e)(exp(θ(3))−1/eexp(θ(3))+2e))absentsubscript301subscript23123132 = _θ(3)∈[0,1] σ clip_2β% (θ(3)+(1+2e) ( (θ(3))-1/e (θ(3))+2e% ) )= maxitalic_θ ( 3 ) ∈ [ 0 , 1 ] log σ sansserif_clip2 β ( θ ( 3 ) + ( 1 + 2 e ) ( divide start_ARG exp ( θ ( 3 ) ) - 1 / e end_ARG start_ARG exp ( θ ( 3 ) ) + 2 e end_ARG ) ) It is straightforward to check that the above function is strictly increasing over θ(3)∈[0,1]301θ(3)∈[0,1]θ ( 3 ) ∈ [ 0 , 1 ] and the maximization step finds θ(3)=131θ(3)=1θ ( 3 ) = 1 when 0<β≤1/30130<β≤ 1/30 < β ≤ 1 / 3. This results in χPO finding the uniform policy θ(1)=θ(2)=θ(3)=11231θ(1)=θ(2)=θ(3)=1θ ( 1 ) = θ ( 2 ) = θ ( 3 ) = 1 and thus suffering from the following suboptimality: J(πθ⋆)−J(π^χPO)subscriptsuperscript⋆subscript^PO J( _θ )-J( π_ )J ( πitalic_θ⋆ ) - J ( over start_ARG π end_ARGχ PO ) =y∼πθ⋆[r⋆(y)]−y∼π^χPO[r⋆(y)]=2e2e+1−23>0.15.absentsubscriptsimilar-tosubscriptsuperscript⋆delimited-[]superscript⋆subscriptsimilar-tosubscript^POdelimited-[]superscript⋆221230.15 =E_y _θ [r (y)]-E_% y π_ [r (y)]= 2e2e+1- 23>0.15.= italic_y ∼ π start_POSTSUBSCRIPT θ⋆ end_POSTSUBSCRIPT [ r⋆ ( y ) ] - italic_y ∼ over start_ARG π end_ARG start_POSTSUBSCRIPT χ PO end_POSTSUBSCRIPT [ r⋆ ( y ) ] = divide start_ARG 2 e end_ARG start_ARG 2 e + 1 end_ARG - divide start_ARG 2 end_ARG start_ARG 3 end_ARG > 0.15 . C.2 Type I Reward Hacking: Proof of Proposition 2 C.2.1 MAB Instance for Type I Reward Hacking We construct a three-armed bandit problem with the following true reward structure: r⋆(1)=r⋆(2)=0,r⋆(3)=1formulae-sequencesuperscript⋆1superscript⋆20superscript⋆31r (1)=r (2)=0,r (3)=1r⋆ ( 1 ) = r⋆ ( 2 ) = 0 , r⋆ ( 3 ) = 1. This reward function implies the following parameters for the best-in-class policy: θ⋆(1)=θ⋆(2)=0,θ⋆(3)=1formulae-sequencesuperscript⋆1superscript⋆20superscript⋆31θ (1)=θ (2)=0,θ (3)=1θ⋆ ( 1 ) = θ⋆ ( 2 ) = 0 , θ⋆ ( 3 ) = 1, leading to the following policy: πθ⋆(1)=πθ⋆(2)=12+e,πθ⋆(3)=e2+e.formulae-sequencesubscriptsuperscript⋆1subscriptsuperscript⋆212subscriptsuperscript⋆32 _θ (1)= _θ (2)= 12+e,π% _θ (3)= e2+e.πitalic_θ⋆ ( 1 ) = πitalic_θ⋆ ( 2 ) = divide start_ARG 1 end_ARG start_ARG 2 + e end_ARG , πitalic_θ⋆ ( 3 ) = divide start_ARG e end_ARG start_ARG 2 + e end_ARG . The performance of the above policy is J(πθ⋆)=r⋆(3)πθ⋆(3)=e/(2+e)subscriptsuperscript⋆superscript⋆3subscriptsuperscript⋆32J( _θ )=r (3) _θ (3)=e/(2+e)J ( πitalic_θ⋆ ) = r⋆ ( 3 ) πitalic_θ⋆ ( 3 ) = e / ( 2 + e ). We further consider the following initialization: θ0(1)=θ0(2)=0,θ0(3)=1formulae-sequencesubscript01subscript020subscript031 _0(1)= _0(2)=0, _0(3)=1θ0 ( 1 ) = θ0 ( 2 ) = 0 , θ0 ( 3 ) = 1. Suppose that the comparison probabilities between the arms are μ1,2=1−1/Nsubscript1211 _1,2=1-1/Nμ1 , 2 = 1 - 1 / N and μ1,3=1/Nsubscript131 _1,3=1/Nμ1 , 3 = 1 / N. Following the same argument as in Section C.1.1, there is a constant probability that arm 3 is compared with arm 1 exactly once and that arm 1 is preferred to arm 3. Throughout the rest of the proof, we condition on the event N(1,3)=1,1≻3formulae-sequence131succeeds13\N(1,3)=1,1 3\ N ( 1 , 3 ) = 1 , 1 ≻ 3 . We further consider a favorable case where the optimal parameters corresponding to arms 1 and 2 are revealed by an oracle θ(1)=θ(2)=0120θ(1)=θ(2)=0θ ( 1 ) = θ ( 2 ) = 0, leaving only θ(3)3θ(3)θ ( 3 ) to be estimated. C.2.2 Analysis of ⋆ ⋆PO Methods in the MAB Instance C.2.1 Suboptimality of the DPO+SFT policy. We show that DPO+SFT suffers from reward hacking for any η≥00η≥ 0η ≥ 0, and hence the argument also shows reward hacking in DPO as a special case. The objective of DPO+SFT is given by maxθ[logσ(β(logπθ(y+)πθ0(y+)−logπθ(y−)πθ0(y−)))]+ηβ[logπθ(y+)]subscriptdelimited-[]subscriptsuperscriptsubscriptsubscript0superscriptsubscriptsuperscriptsubscriptsubscript0superscriptdelimited-[]subscriptsuperscript _θE [ σ (β ( % _θ(y^+) _ _0(y^+)- _θ(y^-)% _ _0(y^-) ) ) ]+η [ π% _θ(y^+) ]maxitalic_θ roman_ [ log σ ( β ( log divide start_ARG πitalic_θ ( y+ ) end_ARG start_ARG πitalic_θ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ( y+ ) end_ARG - log divide start_ARG πitalic_θ ( y- ) end_ARG start_ARG πitalic_θ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ( y- ) end_ARG ) ) ] + η β roman_ [ log πitalic_θ ( y+ ) ] Since θ(1)=θ(2)=0120θ(1)=θ(2)=0θ ( 1 ) = θ ( 2 ) = 0 are revealed by an oracle, we focus on the terms that involve θ(3)3θ(3)θ ( 3 ) in the objective: maxθ(3)∈[0,1]1N[logσ(β(logπθ(1)πθ0(1)−logπθ(3)πθ0(3)))]≔T1≔subscript3011delimited-[]subscript1subscriptsubscript01subscript3subscriptsubscript03subscript1 _θ(3)∈[0,1] 1N [ σ (β% ( _θ(1) _ _0(1)- _θ(3% ) _ _0(3) ) ) ] T_1maxitalic_θ ( 3 ) ∈ [ 0 , 1 ] divide start_ARG 1 end_ARG start_ARG N end_ARG [ log σ ( β ( log divide start_ARG πitalic_θ ( 1 ) end_ARG start_ARG πitalic_θ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ( 1 ) end_ARG - log divide start_ARG πitalic_θ ( 3 ) end_ARG start_ARG πitalic_θ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ( 3 ) end_ARG ) ) ] ≔ T1 +ηβ(1−1N)[μ^1≻2logπθ(1)+μ^2≻1logπθ(2)]+η1Nlogπθ(3)≔T2≔11delimited-[]subscript^succeeds12subscript1subscript^succeeds21subscript21subscript3subscript2 +ηβ (1- 1N ) [ μ_1 2% _θ(1)+ μ_2 1 _θ(2) ]+η % 1N _θ(3) T_2+ η β ( 1 - divide start_ARG 1 end_ARG start_ARG N end_ARG ) [ over start_ARG μ end_ARG1 ≻ 2 log πitalic_θ ( 1 ) + over start_ARG μ end_ARG2 ≻ 1 log πitalic_θ ( 2 ) ] + η divide start_ARG 1 end_ARG start_ARG N end_ARG log πitalic_θ ( 3 ) ≔ T2 Applying the softmax policy parameterization and using the fact that θ(1)=θ0(1)=01subscript010θ(1)= _0(1)=0θ ( 1 ) = θ0 ( 1 ) = 0 and θ0(3)=1subscript031 _0(3)=1θ0 ( 3 ) = 1, term T1subscript1T_1T1 simplifies to T1subscript1 T_1T1 =1Nlog(σ(β(θ(1)−θ0(1)−θ(3)+θ0(3))))=1Nlog(σ(β(1−θ(3))))absent11subscript013subscript03113 = 1N (σ (β (θ(1)- _0% (1)-θ(3)+ _0(3) ) ) )= 1N (σ% (β (1-θ(3) ) ) )= divide start_ARG 1 end_ARG start_ARG N end_ARG log ( σ ( β ( θ ( 1 ) - θ0 ( 1 ) - θ ( 3 ) + θ0 ( 3 ) ) ) ) = divide start_ARG 1 end_ARG start_ARG N end_ARG log ( σ ( β ( 1 - θ ( 3 ) ) ) ) The term T2subscript2T_2T2 can also be simplified by substituting values θ(1)=θ(2)=0120θ(1)=θ(2)=0θ ( 1 ) = θ ( 2 ) = 0: T2subscript2 T_2T2 =ηβ(1−1N)[μ^1≻2(θ(1)−logZθ)+μ^2≻1(θ(2)−logZθ)]+ηβ1N(θ(3)−logZθ)absent11delimited-[]subscript^succeeds121subscriptsubscript^succeeds212subscript13subscript =ηβ (1- 1N ) [ μ_1 2(% θ(1)- Z_θ)+ μ_2 1(θ(2)- Z_θ)% ]+ηβ 1N(θ(3)- Z_θ)= η β ( 1 - divide start_ARG 1 end_ARG start_ARG N end_ARG ) [ over start_ARG μ end_ARG1 ≻ 2 ( θ ( 1 ) - log Zitalic_θ ) + over start_ARG μ end_ARG2 ≻ 1 ( θ ( 2 ) - log Zitalic_θ ) ] + η β divide start_ARG 1 end_ARG start_ARG N end_ARG ( θ ( 3 ) - log Zitalic_θ ) =ηβ(1Nθ(3)−logZθ)absent13subscript =ηβ ( 1Nθ(3)- Z_θ )= η β ( divide start_ARG 1 end_ARG start_ARG N end_ARG θ ( 3 ) - log Zitalic_θ ) =ηβ(1Nθ(3)−log(exp(θ(1))+exp(θ(2))+exp(θ(3))))absent13123 =ηβ ( 1Nθ(3)- ( (θ(1))+ (% θ(2))+ (θ(3))) )= η β ( divide start_ARG 1 end_ARG start_ARG N end_ARG θ ( 3 ) - log ( exp ( θ ( 1 ) ) + exp ( θ ( 2 ) ) + exp ( θ ( 3 ) ) ) ) =ηβ(1Nθ(3)−log(exp(θ(3))+2))absent1332 =ηβ ( 1Nθ(3)- ( (θ(3))+2) )= η β ( divide start_ARG 1 end_ARG start_ARG N end_ARG θ ( 3 ) - log ( exp ( θ ( 3 ) ) + 2 ) ) Combining terms T1subscript1T_1T1 and T2subscript2T_2T2, the objective becomes maxθ(3)∈[0,1]1Nlog(σ(β(1−θ(3))))+ηβ(1Nθ(3)−log(exp(θ(3))+2))subscript3011131332 _θ(3)∈[0,1] 1N (σ (β% (1-θ(3) ) ) )+ηβ ( 1Nθ(3)-% ( (θ(3))+2) )maxitalic_θ ( 3 ) ∈ [ 0 , 1 ] divide start_ARG 1 end_ARG start_ARG N end_ARG log ( σ ( β ( 1 - θ ( 3 ) ) ) ) + η β ( divide start_ARG 1 end_ARG start_ARG N end_ARG θ ( 3 ) - log ( exp ( θ ( 3 ) ) + 2 ) ) We show that for any N>33N>3N > 3 the above function is decreasing in θ(3)3θ(3)θ ( 3 ) for any β,ηβ,ηβ , η and θ(3)∈[0,1]301θ(3)∈[0,1]θ ( 3 ) ∈ [ 0 , 1 ]. Since the first function 1Nlog(σ(β(1−θ(3))))113 1N (σ (β (1-θ(3) ) ) )divide start_ARG 1 end_ARG start_ARG N end_ARG log ( σ ( β ( 1 - θ ( 3 ) ) ) ) is decreasing in θ(3)3θ(3)θ ( 3 ), it is sufficient to show 1Nθ(3)−log(exp(θ(3))+2)1332 1Nθ(3)- ( (θ(3))+2)divide start_ARG 1 end_ARG start_ARG N end_ARG θ ( 3 ) - log ( exp ( θ ( 3 ) ) + 2 ) is decreasing in θ(3)3θ(3)θ ( 3 ). Derivative of this function is over θ(3)∈[0,1]301θ(3)∈[0,1]θ ( 3 ) ∈ [ 0 , 1 ] and N≥33N≥ 3N ≥ 3 is bounded by 1N−exp(θ(3))exp(θ(3))+2≤1N−13<0.13321130 1N- (θ(3)) (θ(3))+2≤ 1% N- 13<0.divide start_ARG 1 end_ARG start_ARG N end_ARG - divide start_ARG exp ( θ ( 3 ) ) end_ARG start_ARG exp ( θ ( 3 ) ) + 2 end_ARG ≤ divide start_ARG 1 end_ARG start_ARG N end_ARG - divide start_ARG 1 end_ARG start_ARG 3 end_ARG < 0 . Therefore, optimizing over θ(3)∈[0,1]301θ(3)∈[0,1]θ ( 3 ) ∈ [ 0 , 1 ] finds θ(3)=030θ(3)=0θ ( 3 ) = 0. This leads DPO+SFT to find a uniform policy, which suffers from a constant suboptimality: J(πθ⋆)−J(π^DPO+SFT)=e2+e−13>0.2.subscriptsuperscript⋆subscript^DPO+SFT2130.2 J( _θ )-J( π_DPO+SFT)= e2% +e- 13>0.2.J ( πitalic_θ⋆ ) - J ( over start_ARG π end_ARGDPO+SFT ) = divide start_ARG e end_ARG start_ARG 2 + e end_ARG - divide start_ARG 1 end_ARG start_ARG 3 end_ARG > 0.2 . Suboptimality of the IPO policy. Similar to the analysis of DPO+SFT, for the IPO objective, we only focus on the terms that include θ(3)3θ(3)θ ( 3 ): minθ(3)∈[0,1]1N(logπθ(1)πθ0(1)−logπθ(3)πθ0(3)−12τ)2=minθ(3)∈[0,1](1−12τ−θ(3))2 _θ(3)∈[0,1] 1N ( _θ(1% ) _ _0(1)- _θ(3) _ _0(3)- % 12τ )^2= _θ(3)∈[0,1] (1- 12τ-θ(3)% )^2minitalic_θ ( 3 ) ∈ [ 0 , 1 ] divide start_ARG 1 end_ARG start_ARG N end_ARG ( log divide start_ARG πitalic_θ ( 1 ) end_ARG start_ARG πitalic_θ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ( 1 ) end_ARG - log divide start_ARG πitalic_θ ( 3 ) end_ARG start_ARG πitalic_θ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ( 3 ) end_ARG - divide start_ARG 1 end_ARG start_ARG 2 τ end_ARG )2 = minitalic_θ ( 3 ) ∈ [ 0 , 1 ] ( 1 - divide start_ARG 1 end_ARG start_ARG 2 τ end_ARG - θ ( 3 ) )2 Since τ>00τ>0τ > 0, solution to the above minimization is θ(3)=max0,1−12τ30112 θ(3)= \0,1- 12τ \θ ( 3 ) = max 0 , 1 - divide start_ARG 1 end_ARG start_ARG 2 τ end_ARG Therefore, the suboptimality of the IPO policy is given by J(πθ⋆)−J(π^IPO)=e2+e−exp(max0,1−1/(2τ))2+exp(max0,1−1/(2τ))subscriptsuperscript⋆subscript^IPO2011220112 J( _θ )-J( π_IPO)= e2+e-% ( \0,1-1/(2τ)\)2+ ( \0,1-1/(2τ)\)J ( πitalic_θ⋆ ) - J ( over start_ARG π end_ARGIPO ) = divide start_ARG e end_ARG start_ARG 2 + e end_ARG - divide start_ARG exp ( max 0 , 1 - 1 / ( 2 τ ) ) end_ARG start_ARG 2 + exp ( max 0 , 1 - 1 / ( 2 τ ) ) end_ARG And for the regime with τ<11τ<1τ < 1, we have J(πθ⋆)−J(π^IPO)>0.1subscriptsuperscript⋆subscript^IPO0.1J( _θ )-J( π_IPO)>0.1J ( πitalic_θ⋆ ) - J ( over start_ARG π end_ARGIPO ) > 0.1. Suboptimality of the SimPO policy. The objective optimized by SimPO over the term that involve θ(3)3θ(3)θ ( 3 ) is given by maxθ(3)∈[0,1]1Nlogσ(β(logπθ(1)−logπθ(3)−γ))subscript3011subscript1subscript3 _θ(3)∈[0,1] 1N σ (β (% _θ(1)- _θ(3)-γ ) )maxitalic_θ ( 3 ) ∈ [ 0 , 1 ] divide start_ARG 1 end_ARG start_ARG N end_ARG log σ ( β ( log πitalic_θ ( 1 ) - log πitalic_θ ( 3 ) - γ ) ) =maxθ(3)∈[0,1]logσ(β(−θ(3)−γ))absentsubscript3013 = _θ(3)∈[0,1] σ (β (-θ(3)-% γ ) )= maxitalic_θ ( 3 ) ∈ [ 0 , 1 ] log σ ( β ( - θ ( 3 ) - γ ) ) The above function is decreasing in θ(3)3θ(3)θ ( 3 ) therefore SimPO finds θ(3)=030θ(3)=0θ ( 3 ) = 0 and suffers from the followingg suboptimality J(πθ⋆)−J(π^SimPO)=e2+e−13>0.2.subscriptsuperscript⋆subscript^SimPO2130.2 J( _θ )-J( π_SimPO)= e2+e% - 13>0.2.J ( πitalic_θ⋆ ) - J ( over start_ARG π end_ARGSimPO ) = divide start_ARG e end_ARG start_ARG 2 + e end_ARG - divide start_ARG 1 end_ARG start_ARG 3 end_ARG > 0.2 . Suboptimality of the χPO policy. The objective optimized by χPO is given by maxθ(3)∈[0,1](1−1N)[μ^1≻2log(σ(2[β(logπθ(1)πθ0(1)−logπθ(2)πθ0(2)+πθ(1)πθ0(1)−πθ(2)πθ0(2))]))+μ^2≻1log(σ(2[β(logπθ(2)πθ0(2)−logπθ(1)πθ0(1)+πθ(2)πθ0(2)−πθ(1)πθ0(1))]))]+1N[log(σ(2[β(logπθ(1)πθ0(1)−logπθ(3)πθ0(3)+πθ(1)πθ0(1)−πθ(3)πθ0(3))]))]subscript30111delimited-[]subscript^succeeds12subscript2delimited-[]subscript1subscriptsubscript01subscript2subscriptsubscript02subscript1subscriptsubscript01subscript2subscriptsubscript02subscript^succeeds21subscript2delimited-[]subscript2subscriptsubscript02subscript1subscriptsubscript01subscript2subscriptsubscript02subscript1subscriptsubscript011delimited-[]subscript2delimited-[]subscript1subscriptsubscript01subscript3subscriptsubscript03subscript1subscriptsubscript01subscript3subscriptsubscript03 split _θ(3)∈[0,1] (1- 1N )% [& μ_1 2 (σ ( clip_2 [% β ( _θ(1) _ _0(1)- _% θ(2) _ _0(2)+ _θ(1) _ _0(1)-% _θ(2) _ _0(2) ) ] ) )\\ &+ μ_2 1 (σ ( clip_2 [β (% _θ(2) _ _0(2)- _θ(1)π% _ _0(1)+ _θ(2) _ _0(2)- _% θ(1) _ _0(1) ) ] ) ) ]\\ + 1N& [ (σ ( clip_2 [β (% _θ(1) _ _0(1)- _θ(3)π% _ _0(3)+ _θ(1) _ _0(1)- _% θ(3) _ _0(3) ) ] ) ) ] splitstart_ROW start_CELL maxitalic_θ ( 3 ) ∈ [ 0 , 1 ] ( 1 - divide start_ARG 1 end_ARG start_ARG N end_ARG ) [ end_CELL start_CELL over start_ARG μ end_ARG1 ≻ 2 log ( σ ( sansserif_clip2 [ β ( log divide start_ARG πitalic_θ ( 1 ) end_ARG start_ARG πitalic_θ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ( 1 ) end_ARG - log divide start_ARG πitalic_θ ( 2 ) end_ARG start_ARG πitalic_θ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ( 2 ) end_ARG + divide start_ARG πitalic_θ ( 1 ) end_ARG start_ARG πitalic_θ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ( 1 ) end_ARG - divide start_ARG πitalic_θ ( 2 ) end_ARG start_ARG πitalic_θ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ( 2 ) end_ARG ) ] ) ) end_CELL end_ROW start_ROW start_CELL end_CELL start_CELL + over start_ARG μ end_ARG2 ≻ 1 log ( σ ( sansserif_clip2 [ β ( log divide start_ARG πitalic_θ ( 2 ) end_ARG start_ARG πitalic_θ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ( 2 ) end_ARG - log divide start_ARG πitalic_θ ( 1 ) end_ARG start_ARG πitalic_θ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ( 1 ) end_ARG + divide start_ARG πitalic_θ ( 2 ) end_ARG start_ARG πitalic_θ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ( 2 ) end_ARG - divide start_ARG πitalic_θ ( 1 ) end_ARG start_ARG πitalic_θ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ( 1 ) end_ARG ) ] ) ) ] end_CELL end_ROW start_ROW start_CELL + divide start_ARG 1 end_ARG start_ARG N end_ARG end_CELL start_CELL [ log ( σ ( sansserif_clip2 [ β ( log divide start_ARG πitalic_θ ( 1 ) end_ARG start_ARG πitalic_θ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ( 1 ) end_ARG - log divide start_ARG πitalic_θ ( 3 ) end_ARG start_ARG πitalic_θ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ( 3 ) end_ARG + divide start_ARG πitalic_θ ( 1 ) end_ARG start_ARG πitalic_θ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ( 1 ) end_ARG - divide start_ARG πitalic_θ ( 3 ) end_ARG start_ARG πitalic_θ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ( 3 ) end_ARG ) ] ) ) ] end_CELL end_ROW (17) By construction, πθ0(1)=πθ0(2)=1/(2+e)subscriptsubscript01subscriptsubscript0212 _ _0(1)= _ _0(2)=1/(2+e)πitalic_θ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ( 1 ) = πitalic_θ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ( 2 ) = 1 / ( 2 + e ), and θ(1)=θ(2)=0120θ(1)=θ(2)=0θ ( 1 ) = θ ( 2 ) = 0 revealed by an oracle, and therefore the first two lines in (16) are equal to zero: logπθ(1)πθ0(1)−logπθ(2)πθ0(2)+πθ(1)πθ0(1)−πθ(2)πθ0(2)subscript1subscriptsubscript01subscript2subscriptsubscript02subscript1subscriptsubscript01subscript2subscriptsubscript02 _θ(1) _ _0(1)- _% θ(2) _ _0(2)+ _θ(1) _ _0(1)-% _θ(2) _ _0(2)log divide start_ARG πitalic_θ ( 1 ) end_ARG start_ARG πitalic_θ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ( 1 ) end_ARG - log divide start_ARG πitalic_θ ( 2 ) end_ARG start_ARG πitalic_θ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ( 2 ) end_ARG + divide start_ARG πitalic_θ ( 1 ) end_ARG start_ARG πitalic_θ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ( 1 ) end_ARG - divide start_ARG πitalic_θ ( 2 ) end_ARG start_ARG πitalic_θ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ( 2 ) end_ARG =θ(1)−θ0(1)−θ(2)+θ0(2)+(2+e)(exp(θ(1))Zθ−exp(θ(2))Zθ)=0.absent1subscript012subscript0221subscript2subscript0 =θ(1)- _0(1)-θ(2)+ _0(2)+(2+e) (% (θ(1))Z_θ- (θ(2))Z_θ )=0.= θ ( 1 ) - θ0 ( 1 ) - θ ( 2 ) + θ0 ( 2 ) + ( 2 + e ) ( divide start_ARG exp ( θ ( 1 ) ) end_ARG start_ARG Zitalic_θ end_ARG - divide start_ARG exp ( θ ( 2 ) ) end_ARG start_ARG Zitalic_θ end_ARG ) = 0 . The objective (17) can therefore be simplified to maxθ(3)∈[0,1]1N[log(σ(2[β(logπθ(1)πθ0(1)−logπθ(3)πθ0(3)+πθ(1)πθ0(1)−πθ(3)πθ0(3))]))]subscript3011delimited-[]subscript2delimited-[]subscript1subscriptsubscript01subscript3subscriptsubscript03subscript1subscriptsubscript01subscript3subscriptsubscript03 _θ(3)∈[0,1] 1N [ (σ (% clip_2 [β ( _θ(1) _ _0(% 1)- _θ(3) _ _0(3)+ _θ(1)π% _ _0(1)- _θ(3) _ _0(3) ) ]% ) ) ]maxitalic_θ ( 3 ) ∈ [ 0 , 1 ] divide start_ARG 1 end_ARG start_ARG N end_ARG [ log ( σ ( sansserif_clip2 [ β ( log divide start_ARG πitalic_θ ( 1 ) end_ARG start_ARG πitalic_θ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ( 1 ) end_ARG - log divide start_ARG πitalic_θ ( 3 ) end_ARG start_ARG πitalic_θ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ( 3 ) end_ARG + divide start_ARG πitalic_θ ( 1 ) end_ARG start_ARG πitalic_θ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ( 1 ) end_ARG - divide start_ARG πitalic_θ ( 3 ) end_ARG start_ARG πitalic_θ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ( 3 ) end_ARG ) ] ) ) ] =maxθ(3)∈[0,1]log(σ(2[β(θ(1)−θ0(1)−θ(3)+θ0(3)+πθ(1)πθ0(1)−πθ(3)πθ0(3))]))absentsubscript301subscript2delimited-[]1subscript013subscript03subscript1subscriptsubscript01subscript3subscriptsubscript03 = _θ(3)∈[0,1] (σ ( clip_2% [β (θ(1)- _0(1)-θ(3)+ _0(3)+ _% θ(1) _ _0(1)- _θ(3) _ _0(3)% ) ] ) )= maxitalic_θ ( 3 ) ∈ [ 0 , 1 ] log ( σ ( sansserif_clip2 [ β ( θ ( 1 ) - θ0 ( 1 ) - θ ( 3 ) + θ0 ( 3 ) + divide start_ARG πitalic_θ ( 1 ) end_ARG start_ARG πitalic_θ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ( 1 ) end_ARG - divide start_ARG πitalic_θ ( 3 ) end_ARG start_ARG πitalic_θ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ( 3 ) end_ARG ) ] ) ) =maxθ(3)∈[0,1]log(σ(2[β(1−θ(3)+πθ(1)πθ0(1)−πθ(3)πθ0(3))]))absentsubscript301subscript2delimited-[]13subscript1subscriptsubscript01subscript3subscriptsubscript03 = _θ(3)∈[0,1] (σ ( clip_2% [β (1-θ(3)+ _θ(1) _ _0(1)- % _θ(3) _ _0(3) ) ] ) )= maxitalic_θ ( 3 ) ∈ [ 0 , 1 ] log ( σ ( sansserif_clip2 [ β ( 1 - θ ( 3 ) + divide start_ARG πitalic_θ ( 1 ) end_ARG start_ARG πitalic_θ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ( 1 ) end_ARG - divide start_ARG πitalic_θ ( 3 ) end_ARG start_ARG πitalic_θ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ( 3 ) end_ARG ) ] ) ) =maxθ(3)∈[0,1]logσ(2[β(1−θ(3)+2+e2+exp(θ(3))(1−exp(θ(3)−1)))])absentsubscript301subscript2delimited-[]13223131 = _θ(3)∈[0,1] σ ( clip_2 [% β (1-θ(3)+ 2+e2+ (θ(3)) (1- (θ(3)-1)% ) ) ] )= maxitalic_θ ( 3 ) ∈ [ 0 , 1 ] log σ ( sansserif_clip2 [ β ( 1 - θ ( 3 ) + divide start_ARG 2 + e end_ARG start_ARG 2 + exp ( θ ( 3 ) ) end_ARG ( 1 - exp ( θ ( 3 ) - 1 ) ) ) ] ) The last equation applies the definition of πθsubscript _θπitalic_θ and substitutes values for θ0(1),θ0(3),θ(1)subscript01subscript031 _0(1), _0(3),θ(1)θ0 ( 1 ) , θ0 ( 3 ) , θ ( 1 ). The function β(1−θ(3)+2+e2+exp(θ(3))(1−exp(θ(3)−1)))13223131 β (1-θ(3)+ 2+e2+ (θ(3)) (1- (% θ(3)-1) ) )β ( 1 - θ ( 3 ) + divide start_ARG 2 + e end_ARG start_ARG 2 + exp ( θ ( 3 ) ) end_ARG ( 1 - exp ( θ ( 3 ) - 1 ) ) ) is decreasing in θ(3)3θ(3)θ ( 3 ) for 0<β≤1010<β≤ 10 < β ≤ 1 and that for θ(3)∈[0,1]301θ(3)∈[0,1]θ ( 3 ) ∈ [ 0 , 1 ], the above function remains between 0 and 2. As a result, the maximization problem leads to θ(3)=030θ(3)=0θ ( 3 ) = 0. Thus, χPO finds the uniform policy π^χPO=(1,2,3)subscript^PO123 π_ = Unif(\1,2,3\)over start_ARG π end_ARGχ PO = sansserif_Unif ( 1 , 2 , 3 ), which suffers from a constant suboptimality: J(πθ⋆)−J(π^χPO)=e2+e−13>0.2.subscriptsuperscript⋆subscript^PO2130.2 J( _θ )-J( π_ )= e2+% e- 13>0.2.J ( πitalic_θ⋆ ) - J ( over start_ARG π end_ARGχ PO ) = divide start_ARG e end_ARG start_ARG 2 + e end_ARG - divide start_ARG 1 end_ARG start_ARG 3 end_ARG > 0.2 . C.3 Proof of Proposition 4 Policy learned by POWER in the MAB instances in C.1.1. We show that when faced with the Type I Reward Hacking MAB instances of C.1.1, a more general variant of POWER mitigates reward hacking. Let g(x)g(x)g ( x ) be an increasing function with bounded derivative: g′(x−1)≤Bgsuperscript′1subscriptg (x-1)≤ B_gg′ ( x - 1 ) ≤ Bitalic_g for any x∈[0,1]01x∈[0,1]x ∈ [ 0 , 1 ]. We consider the following objective: maxθ[g(w(y+)logπθ(y+|x)−w(y−)logπθ(y−|x)+w(y+)−w(y−))]+ηβ[w(y)logπθ(y)]subscriptsubscriptdelimited-[]superscriptsubscriptconditionalsuperscriptsuperscriptsubscriptconditionalsuperscriptsuperscriptsuperscriptsubscriptdelimited-[]subscript _θE_D [g (w(y^+) π% _θ(y^+|x)-w(y^-) _θ(y^-|x)+w(y^+)-w(y^-) )% ]+η _D [w(y) _θ(y) ]maxitalic_θ caligraphic_D [ g ( w ( y+ ) log πitalic_θ ( y+ | x ) - w ( y- ) log πitalic_θ ( y- | x ) + w ( y+ ) - w ( y- ) ) ] + η β caligraphic_D [ w ( y ) log πitalic_θ ( y ) ] The POWER objective is a special case of the above objective with g(⋅)=logσ(⋅)⋅g(·)= σ(·)g ( ⋅ ) = log σ ( ⋅ ). Note that we have g′(x−1)=σ(−(x−1))≤1superscript′111g (x-1)=σ(-(x-1))≤ 1g′ ( x - 1 ) = σ ( - ( x - 1 ) ) ≤ 1, and therefore the bounded derivative assumption is satisfied. For the MAB instances in C.1.1, the optimization problem simplifies to maxθ(3)∈[0,1]1N(g(θ(3)−1)+ηθ(3))−ηlog(exp(θ(3))+e+1).subscript301131331 _θ(3)∈[0,1] 1N (g(θ(3)-1)+η% θ(3) )-η ( (θ(3))+e+1 ).maxitalic_θ ( 3 ) ∈ [ 0 , 1 ] divide start_ARG 1 end_ARG start_ARG N end_ARG ( g ( θ ( 3 ) - 1 ) + η θ ( 3 ) ) - η log ( exp ( θ ( 3 ) ) + e + 1 ) . We show that for any η>Bg(2+e)N−(2+e)subscript22η> B_g(2+e)N-(2+e)η > divide start_ARG Bitalic_g ( 2 + e ) end_ARG start_ARG N - ( 2 + e ) end_ARG the derivative of above function is negative. This is because for any θ(3)∈[0,1]301θ(3)∈[0,1]θ ( 3 ) ∈ [ 0 , 1 ], we have 1N(g′(θ(3)−1)+η)−ηexp(θ(3))exp(θ(3))+e+11superscript′31331 1N(g (θ(3)-1)+η)-η (θ(3% )) (θ(3))+e+1divide start_ARG 1 end_ARG start_ARG N end_ARG ( g′ ( θ ( 3 ) - 1 ) + η ) - η divide start_ARG exp ( θ ( 3 ) ) end_ARG start_ARG exp ( θ ( 3 ) ) + e + 1 end_ARG ≤1N(Bg+η)−η2+eabsent1subscript2 ≤ 1N(B_g+η)- η2+e≤ divide start_ARG 1 end_ARG start_ARG N end_ARG ( Bitalic_g + η ) - divide start_ARG η end_ARG start_ARG 2 + e end_ARG =BgN−η(12+e−N)absentsubscript12 = B_gN-η ( 12+e-N )= divide start_ARG Bitalic_g end_ARG start_ARG N end_ARG - η ( divide start_ARG 1 end_ARG start_ARG 2 + e end_ARG - N ) <BgN−Bg(2+e)N−(2+e)(12+e−N)absentsubscriptsubscript2212 < B_gN- B_g(2+e)N-(2+e) ( 12+e-N )< divide start_ARG Bitalic_g end_ARG start_ARG N end_ARG - divide start_ARG Bitalic_g ( 2 + e ) end_ARG start_ARG N - ( 2 + e ) end_ARG ( divide start_ARG 1 end_ARG start_ARG 2 + e end_ARG - N ) ≤0.absent0 ≤ 0.≤ 0 . Because the function is decreasing in θ(3)3θ(3)θ ( 3 ) the optimum is at θ(3)=030θ(3)=0θ ( 3 ) = 0 and thus the algorithm finds π^=πθ⋆^subscriptsuperscript⋆ π= _θ over start_ARG π end_ARG = πitalic_θ⋆. Finally, the conclusion also holds for POWER as a special case and thus π^POWER=πθ⋆subscript^POWERsubscriptsuperscript⋆ π_POWER= _θ over start_ARG π end_ARGPOWER = πitalic_θ⋆. Policy learned by POWER in the MAB instance in C.2.1. The POWER objective with response lengths equal to one is given by maxθ[logσ(β(logπθ(y+)−logπθ(y−)))]+ηβ[logπθ(y+)]subscriptdelimited-[]subscriptsuperscriptsubscriptsuperscriptdelimited-[]subscriptsuperscript _θE [ σ (β ( _% θ(y^+)- _θ(y^-) ) ) ]+ηβ% E [ _θ(y^+) ]maxitalic_θ roman_ [ log σ ( β ( log πitalic_θ ( y+ ) - log πitalic_θ ( y- ) ) ) ] + η β roman_ [ log πitalic_θ ( y+ ) ] With a similar argument as in our analysis of DPO+SFT, we obtain the following objective: maxθ(3)∈[0,1]1Nlog(σ(β(−θ(3))))+ηβ(1Nθ(3)−log(exp(θ(3))+2))subscript301131332 _θ(3)∈[0,1] 1N (σ (β% (-θ(3) ) ) )+ηβ ( 1Nθ(3)- % ( (θ(3))+2) )maxitalic_θ ( 3 ) ∈ [ 0 , 1 ] divide start_ARG 1 end_ARG start_ARG N end_ARG log ( σ ( β ( - θ ( 3 ) ) ) ) + η β ( divide start_ARG 1 end_ARG start_ARG N end_ARG θ ( 3 ) - log ( exp ( θ ( 3 ) ) + 2 ) ) It is easy to check that the above function is decreasing in θ(3)3θ(3)θ ( 3 ). Therefore, optimization leads to θ(3)=030θ(3)=0θ ( 3 ) = 0 and the policy learned by POWER suffers from a constant suboptimality J(πθ⋆)−J(π^POWER)=e2+e−13>0.2.subscriptsuperscript⋆subscript^POWER2130.2 J( _θ )-J( π_POWER)= e2+e% - 13>0.2.J ( πitalic_θ⋆ ) - J ( over start_ARG π end_ARGPOWER ) = divide start_ARG e end_ARG start_ARG 2 + e end_ARG - divide start_ARG 1 end_ARG start_ARG 3 end_ARG > 0.2 . Appendix D POWER Objective Derivation This section is organized as follows. In Section D.1 we record a useful proposition that captures properties of the optimal policy to the WER objective. This result comes in handy for deriving our preference optimization objective—which relies on the equivalence between minimax and maximin objectives as well as finding a closed-form solution for the inner maximization problem. In Section D.2, we prove that under certain regularity conditions on reward class ℛRR, the maximization and minimization steps in objective (6) can be interchanged. With these two results at hand, we prove Proposition 5 in Section D.3, which gives the POWER objective. D.1 Optimal Policy for Weighted Entropy Reward Maximization For the WER objective, we have the following proposition which shows the uniqueness of the optimal WER policy on the support of prompt distribution, and connects this policy to the reward gap. Proposition 5 (WER Policy). For any β>00β>0β > 0, reward function r, any x∈x ∈ X with ρ(x)>00ρ(x)>0ρ ( x ) > 0, and any action pairs y,y′∈superscript′y,y , y′ ∈ Y, the policy πrsubscript _rπitalic_r that maximizes the WER objective (5) satisfies the following statements: 1. For any β>00β>0β > 0, policy πrsubscript _rπitalic_r is unique on the support of ρ. 2. Policy πrsubscript _rπitalic_r satisfies the following equation: r(x,y)−r(x,y′)=β(w(y)logπr(y|x)−w(y′)logπr(y′|x)+(w(y)−w(y′)))superscript′subscriptconditionalsuperscript′subscriptconditionalsuperscript′ r(x,y)-r(x,y )=β (w(y) _r(y|x)-w(y^% ) _r(y |x)+ (w(y)-w(y ) ) )r ( x , y ) - r ( x , y′ ) = β ( w ( y ) log πitalic_r ( y | x ) - w ( y′ ) log πitalic_r ( y′ | x ) + ( w ( y ) - w ( y′ ) ) ) (18) Proof of Proposition 5. The WER objective solves the following optimization problem: maxπx∼ρ[∑yπ(y|x)[r(x,y)−βw(y)logπ(y|x)]]∑yπ(y|x)=1∀x∈formulae-sequencesubscriptsubscriptsimilar-todelimited-[]subscriptconditionaldelimited-[]conditionalsubscriptconditional1for-all split& _πE_x ρ [ _yπ(y% |x) [r(x,y)-β w(y) π(y|x) ] ]\\ & _yπ(y|x)=1 ∀ x splitstart_ROW start_CELL end_CELL start_CELL maxitalic_π italic_x ∼ ρ [ ∑y π ( y | x ) [ r ( x , y ) - β w ( y ) log π ( y | x ) ] ] end_CELL end_ROW start_ROW start_CELL end_CELL start_CELL ∑y π ( y | x ) = 1 ∀ x ∈ X end_CELL end_ROW (19) We find the optimal policy for each context in the support of ρ independently. For any such x, we rewrite the constrained optimization problem using Lagrange multipliers: ∑yπ(y|x)[r(x,y)−βw(y)logπ(y|x)]−λx(∑yπ(y|x)−1)subscriptconditionaldelimited-[]conditionalsubscriptsubscriptconditional1 _yπ(y|x) [r(x,y)-β w(y) π(y|x) ]-% _x ( _yπ(y|x)-1 )∑y π ( y | x ) [ r ( x , y ) - β w ( y ) log π ( y | x ) ] - λitalic_x ( ∑y π ( y | x ) - 1 ) Notice that the above function is concave in π(y|x)conditionalπ(y|x)π ( y | x ) due to w(y),β>00w(y),β>0w ( y ) , β > 0 and thus the solution is unique on the support of ρ and the stationary point is the maximizer. Taking the derivative with respect to π(y|x)conditionalπ(y|x)π ( y | x ) and setting it to zero finds an equation governing the optimal policy πrsubscript _rπitalic_r r(x,y)=βw(y)logπr(y|x)+βw(y)+λxsubscriptconditionalsubscript r(x,y)=βw(y) _r(y|x)+βw(y)+ _xr ( x , y ) = β w ( y ) log πitalic_r ( y | x ) + β w ( y ) + λitalic_x Thus for any y,y′,y y , y′, one has r(x,y)−r(x,y′)=β(w(y)logπr(y|x)+w(y)−w(y′)logπr(y′|x)−w(y′)),superscript′subscriptconditionalsuperscript′subscriptconditionalsuperscript′ r(x,y)-r(x,y )=β (w(y) _r(y|x)+w(y)-w(y% ) _r(y |x)-w(y ) ),r ( x , y ) - r ( x , y′ ) = β ( w ( y ) log πitalic_r ( y | x ) + w ( y ) - w ( y′ ) log πitalic_r ( y′ | x ) - w ( y′ ) ) , which concludes the proof. ∎ D.2 Minimax Objective Equivalence to Maximin Objective In this section, we show that that the maximin objective (6) can be written as a minimax objective under certain regularity conditions. Define the following notation to denote the weighted-entropy robust reward objective for any π∈Ππ∈ π ∈ Π and r∈ℛr ∈ R: ϕ(π,r)≔LBT(r)+η(x∼ρ,y∼π[r(x,y)]−x∼ρ,y′∼π′[r(x,y′)]+βHw(π))≔italic-ϕsubscriptBTsubscriptformulae-sequencesimilar-tosimilar-todelimited-[]subscriptformulae-sequencesimilar-tosimilar-tosuperscript′delimited-[]superscript′subscript φ(π,r) L_BT(r)+η (E_x% ρ,y π [r(x,y) ]-E_x ρ,y π^% [r(x,y ) ]+β H_w(π) )ϕ ( π , r ) ≔ LBT ( r ) + η ( italic_x ∼ ρ , y ∼ π [ r ( x , y ) ] - italic_x ∼ ρ , y′ ∼ π′ [ r ( x , y′ ) ] + β Hitalic_w ( π ) ) (20) We follow the approach of Liu et al. (2024) and impose regularity conditions on the class ℛRR, which is contingent upon our definition of ϕitalic-ϕφϕ, to show the maximin and minimax equivalence. Formally, we make the following assumption. Assumption 1 (Regularity of the Reward Class). We assume that class ℛRR satisfies the following: 1. The space ℛRR is a non-empty compact topological space; 2. The function ϕitalic-ϕφϕ defined in (20) is convex-like in ℛRR; that is, for any r1,r2∈ℛ,π∈Πformulae-sequencesubscript1subscript2ℛΠr_1,r_2 ,π∈ 1 , r2 ∈ R , π ∈ Π, and α∈[0,1]01α∈[0,1]α ∈ [ 0 , 1 ], there exists r3∈ℛsubscript3ℛr_3 3 ∈ R such that ϕ(π,r3)≤αϕ(π,r1)+(1−α)ϕ(π,r2).italic-ϕsubscript3italic-ϕsubscript11italic-ϕsubscript2 φ(π,r_3)≤αφ(π,r_1)+(1-α)φ(π,r_2).ϕ ( π , r3 ) ≤ α ϕ ( π , r1 ) + ( 1 - α ) ϕ ( π , r2 ) . The above condition is satisfied in several special cases. For example, it is satisfied when ℛRR is convex such as a linear class Xiong et al. (2023); Fisch et al. (2024). As a more general case, if ℛRR is a Lipschitz continuous class, we can conclude function ϕ(π,⋅)italic-ϕ⋅φ(π,·)ϕ ( π , ⋅ ) to be convex over ℛRR as ϕ(π,⋅)italic-ϕ⋅φ(π,·)ϕ ( π , ⋅ ) is a sum of a linear term in r and a convex term LBT(r)subscriptBTL_BT(r)LBT ( r ). Under this assumption, we have the following proposition showing the equivalence between maximin and minimax objectives. Proposition 6 (Equivalence of Maximin and Minimax Algorithms). For the policy class Π=π:→Δ()Πconditional-set→Δ =\π:X→ (Y)\Π = π : X → Δ ( Y ) and reward class ℛRR satisfying Assumption 1, define policy πr^subscript _ rπover start_ARG r end_ARG to be the optimal policy corresponding to the minimax reward function, i.e., πr^∈argmaxπ∈Πϕ(π,r^)wherer^∈argminr∈ℛmaxπ∈Πϕ(π,r)formulae-sequencesubscript^subscriptargmaxΠitalic-ϕ^where^subscriptargminℛsubscriptΠitalic-ϕ split _ r&∈ *argmax_π∈ % φ(π, r) r∈ *argmin_r∈% R _π∈ φ(π,r) splitstart_ROW start_CELL πover start_ARG r end_ARG end_CELL start_CELL ∈ argmaxitalic_π ∈ Π ϕ ( π , over start_ARG r end_ARG ) where over start_ARG r end_ARG ∈ argminitalic_r ∈ R maxitalic_π ∈ Π ϕ ( π , r ) end_CELL end_ROW Then, policy πr^subscript _ rπover start_ARG r end_ARG is also the optimal solution to the maximin objective, i.e. πr^∈argmaxπ∈Πminr∈ℛϕ(π,r).subscript^subscriptargmaxΠsubscriptℛitalic-ϕ _ r∈ *argmax_π∈ _r∈% Rφ(π,r).πover start_ARG r end_ARG ∈ argmaxitalic_π ∈ Π minitalic_r ∈ R ϕ ( π , r ) . Proof. We begin by recording the following lemma that shows the equivalence of the maximin and minimax objectives under the assumptions on ℛRR. Proof of this lemma is deferred to the end of this section. Lemma 1 (Equivalence of Maximin and Minimax Objectives). Given the policy class Π:π:→Δ():Πconditional-set→Δ :\π:X→ (Y)\Π : π : X → Δ ( Y ) and reward class ℛRR satisfying Assumption 1, the following statement holds for ϕitalic-ϕφϕ defined in (20): maxπ∈Πminr∈ℛϕ(π,r)=minr∈ℛmaxπ∈Πϕ(π,r).subscriptΠsubscriptℛitalic-ϕsubscriptℛsubscriptΠitalic-ϕ _π∈ _r φ(π,r)= _r∈% R _π∈ φ(π,r).maxitalic_π ∈ Π minitalic_r ∈ R ϕ ( π , r ) = minitalic_r ∈ R maxitalic_π ∈ Π ϕ ( π , r ) . Denote the policy solving the maximin problem by π^∈argmaxπ∈Πminr∈ℛϕ(π,r)^subscriptargmaxΠsubscriptℛitalic-ϕ π∈ *argmax_π∈ _r φ(π,r)over start_ARG π end_ARG ∈ argmaxitalic_π ∈ Π minitalic_r ∈ R ϕ ( π , r ). The duality gap of r^,π^^ r, πover start_ARG r end_ARG , over start_ARG π end_ARG is given by dual(r^,π^)≔maxπ∈Πϕ(π,r^)−minr∈ℛϕ(π^,r)=maxπ∈Πϕ(π,r^)−minr∈ℛmaxπ∈Πϕ(π,r)+minr∈ℛmaxπ∈Πϕ(π,r)−minr∈ℛϕ(π^,r)=maxπ∈Πϕ(π,r^)−minr∈ℛmaxπ∈Πϕ(π,r)+maxπ∈Πminr∈ℛϕ(π,r)−minr∈ℛϕ(π^,r)=0≔dual^^subscriptΠitalic-ϕ^subscriptℛitalic-ϕ^subscriptΠitalic-ϕ^subscriptℛsubscriptΠitalic-ϕsubscriptℛsubscriptΠitalic-ϕsubscriptℛitalic-ϕ^subscriptΠitalic-ϕ^subscriptℛsubscriptΠitalic-ϕsubscriptΠsubscriptℛitalic-ϕsubscriptℛitalic-ϕ^0 splitdual( r, π)& _π∈% φ(π, r)- _r φ( π,r)\\ &= _π∈ φ(π, r)- _r _π∈ φ% (π,r)+ _r _π∈ φ(π,r)- _r % φ( π,r)\\ &= _π∈ φ(π, r)- _r _π∈ φ% (π,r)+ _π∈ _r φ(π,r)- _r % φ( π,r)\\ &=0 splitstart_ROW start_CELL dual ( over start_ARG r end_ARG , over start_ARG π end_ARG ) end_CELL start_CELL ≔ maxitalic_π ∈ Π ϕ ( π , over start_ARG r end_ARG ) - minitalic_r ∈ R ϕ ( over start_ARG π end_ARG , r ) end_CELL end_ROW start_ROW start_CELL end_CELL start_CELL = maxitalic_π ∈ Π ϕ ( π , over start_ARG r end_ARG ) - minitalic_r ∈ R maxitalic_π ∈ Π ϕ ( π , r ) + minitalic_r ∈ R maxitalic_π ∈ Π ϕ ( π , r ) - minitalic_r ∈ R ϕ ( over start_ARG π end_ARG , r ) end_CELL end_ROW start_ROW start_CELL end_CELL start_CELL = maxitalic_π ∈ Π ϕ ( π , over start_ARG r end_ARG ) - minitalic_r ∈ R maxitalic_π ∈ Π ϕ ( π , r ) + maxitalic_π ∈ Π minitalic_r ∈ R ϕ ( π , r ) - minitalic_r ∈ R ϕ ( over start_ARG π end_ARG , r ) end_CELL end_ROW start_ROW start_CELL end_CELL start_CELL = 0 end_CELL end_ROW (21) In the penultimate equation, we applied Lemma 1 and the last equation uses the definition of r^ rover start_ARG r end_ARG and π^ πover start_ARG π end_ARG. The duality gap is also equal to dual(r^,π^)=maxπ∈Πϕ(π,r^)−ϕ(π^,r^)+ϕ(π^,r^)−minr∈ℛϕ(π^,r)dual^^subscriptΠitalic-ϕ^italic-ϕ^^italic-ϕ^^subscriptℛitalic-ϕ ( r, π)= _π∈ φ(π, r)-% φ( π, r)+φ( π, r)- _r φ(% π,r)dual ( over start_ARG r end_ARG , over start_ARG π end_ARG ) = maxitalic_π ∈ Π ϕ ( π , over start_ARG r end_ARG ) - ϕ ( over start_ARG π end_ARG , over start_ARG r end_ARG ) + ϕ ( over start_ARG π end_ARG , over start_ARG r end_ARG ) - minitalic_r ∈ R ϕ ( over start_ARG π end_ARG , r ) (22) Comparing (21) and (22), we conclude that maxπϕ(π,r^)=ϕ(π^,r^)subscriptitalic-ϕ^italic-ϕ^ _πφ(π, r)=φ( π, r)maxitalic_π ϕ ( π , over start_ARG r end_ARG ) = ϕ ( over start_ARG π end_ARG , over start_ARG r end_ARG ) which means π^∈argmaxπ∈Πϕ(r^,π)^subscriptargmaxΠitalic-ϕ π∈ *argmax_π∈ φ( r,π)over start_ARG π end_ARG ∈ argmaxitalic_π ∈ Π ϕ ( over start_ARG r end_ARG , π ). Recall that by definition, we also have πr^∈argmaxπ∈Πϕ(r^,π)subscript^subscriptargmaxΠitalic-ϕ _ r∈ *argmax_π∈ φ( r,π)πover start_ARG r end_ARG ∈ argmaxitalic_π ∈ Π ϕ ( over start_ARG r end_ARG , π ). By the uniqueness of the WER optimal policy over ρ as established in Proposition 5, we conclude that πr^(⋅|x)=π^(⋅|x) _ r(·|x)= π(·|x)πover start_ARG r end_ARG ( ⋅ | x ) = over start_ARG π end_ARG ( ⋅ | x ) for any x with ρ(x)>00ρ(x)>0ρ ( x ) > 0. Since ϕ(π,r)italic-ϕφ(π,r)ϕ ( π , r ) depends on π only through its value on the support of ρ, we conclude that πr^∈argmaxπ∈Πminr∈ℛϕ(π,r)subscript^subscriptargmaxΠsubscriptℛitalic-ϕ _ r∈ *argmax_π∈ _r φ(% π,r)πover start_ARG r end_ARG ∈ argmaxitalic_π ∈ Π minitalic_r ∈ R ϕ ( π , r ), which completes the proof. ∎ Proof of Lemma 1. This result relies on a minimax theorem by Fan (1953) presented in Lemma 2. We prove that all the requirements of this theorem are satisfied. First, by definition, policy class Π Π is a non-empty convex set and by Assumption 1, the reward class ℛRR is a non-empty compact topological space. Second, function ϕ(π,r)italic-ϕφ(π,r)ϕ ( π , r ) is concave on Π Π because it is a sum of a linear function in π and (weighted) entropy of π. Lastly, by Assumption 1, function ϕ(π,r)italic-ϕφ(π,r)ϕ ( π , r ) is continuous and convex-like on ℛRR. Therefore, we apply Lemma 2 to conclude the equivalence of maximin and minimax problems on ϕitalic-ϕφϕ. ∎ D.3 Proof of Proposition 3 We start by deriving the objective (8) by changing the order of maximization and minimization in the maximin objective (6), which is valid on the account of Proposition 6. Writing the minimax objective and rearranging some terms yields minrLBT(r)+ηmaxπ(x∼ρ,y∼π,y′∼π′[r(x,y)−r(x,y′)]+βHw(π))subscriptsubscriptBTsubscriptsubscriptformulae-sequencesimilar-toformulae-sequencesimilar-tosimilar-tosuperscript′delimited-[]superscript′subscript _rL_BT(r)+η _π (E_x % ρ,y π,y π [r(x,y)-r(x,y ) ]+% β H_w(π) )minitalic_r LBT ( r ) + η maxitalic_π ( italic_x ∼ ρ , y ∼ π , y′ ∼ π′ [ r ( x , y ) - r ( x , y′ ) ] + β Hitalic_w ( π ) ) (23) The inner maximization problem over π is the same as the weighted entropy reward maximization objective (5) minus a baseline term, which is independent of π. We apply the reward gap expression provided by Proposition 5 that governs the maximizer policy πrsubscript _rπitalic_r as well as the definition of weighted entropy in Definition 1 to find the maximum value of the inner optimization problem: maxπx∼ρ,y∼π,y′∼π′[r(x,y)−r(x,y′)]+βHw(π)=βx∼ρ,y∼π,y′∼π′[w(y)logπr(y|x)−w(y′)logπr(y′|x)+(w(y)−w(y′))−w(y)logπr(y|x)]=−βx∼ρ,y∼π,y′∼π′[w(y′)logπr(y′|x)−(w(y)−w(y′))]subscriptsubscriptformulae-sequencesimilar-toformulae-sequencesimilar-tosimilar-tosuperscript′delimited-[]superscript′subscriptsubscriptformulae-sequencesimilar-toformulae-sequencesimilar-tosimilar-tosuperscript′delimited-[]subscriptconditionalsuperscript′subscriptconditionalsuperscript′subscriptconditionalsubscriptformulae-sequencesimilar-toformulae-sequencesimilar-tosimilar-tosuperscript′delimited-[]superscript′subscriptconditionalsuperscript′ split& _πE_x ρ,y π,y % π [r(x,y)-r(x,y ) ]+β H_w(π)\\ & = _x ρ,y π,y π [w% (y) _r(y|x)-w(y ) _r(y |x)+ (w(y)-w(y^% ) )-w(y) _r(y|x) ]\\ & =- _x ρ,y π,y π [% w(y ) _r(y |x)- (w(y)-w(y ) ) ]% splitstart_ROW start_CELL end_CELL start_CELL maxitalic_π italic_x ∼ ρ , y ∼ π , y′ ∼ π′ [ r ( x , y ) - r ( x , y′ ) ] + β Hitalic_w ( π ) end_CELL end_ROW start_ROW start_CELL end_CELL start_CELL = β italic_x ∼ ρ , y ∼ π , y′ ∼ π′ [ w ( y ) log πitalic_r ( y | x ) - w ( y′ ) log πitalic_r ( y′ | x ) + ( w ( y ) - w ( y′ ) ) - w ( y ) log πitalic_r ( y | x ) ] end_CELL end_ROW start_ROW start_CELL end_CELL start_CELL = - β italic_x ∼ ρ , y ∼ π , y′ ∼ π′ [ w ( y′ ) log πitalic_r ( y′ | x ) - ( w ( y ) - w ( y′ ) ) ] end_CELL end_ROW We substitute the above expression back in the minimax objective (23): minrLBT(r)−ηβx∼ρ,y∼π,y′∼π′[w(y′)logπr(y′|x)−(w(y)−w(y′))]subscriptsubscriptBTsubscriptformulae-sequencesimilar-toformulae-sequencesimilar-tosimilar-tosuperscript′delimited-[]superscript′subscriptconditionalsuperscript′ _rL_BT(r)-η _x ρ,y π,% y π [w(y ) _r(y |x)-% (w(y)-w(y ) ) ]minitalic_r LBT ( r ) - η β italic_x ∼ ρ , y ∼ π , y′ ∼ π′ [ w ( y′ ) log πitalic_r ( y′ | x ) - ( w ( y ) - w ( y′ ) ) ] (24) =minrLBT(r)−ηβx∼ρ,y′∼π′[w(y′)logπr(y′|x)]absentsubscriptsubscriptBTsubscriptformulae-sequencesimilar-tosimilar-tosuperscript′delimited-[]superscript′subscriptconditionalsuperscript′ = _rL_BT(r)-η _x ρ,y^% π [w(y ) _r(y |x) ]= minitalic_r LBT ( r ) - η β italic_x ∼ ρ , y′ ∼ π′ [ w ( y′ ) log πitalic_r ( y′ | x ) ] (25) The above equation uses the fact that (w(y)−w(y′))superscript′ (w(y)-w(y ) )( w ( y ) - w ( y′ ) ) is independent of r. To obtain the final objective, we replace the reward gap expression from Proposition 5 in LBT(r)subscriptBTL_BT(r)LBT ( r ), which cocludes the proof. D.4 Auxiliary Lemmas Lemma 2 (Minimax Theorem; Fan (1953)). Let XX be a nonempty (not necessarily topologized) set and YY be a nonempty compact topological space. Let f:×→ℝ:→ℝf:X×Y : X × Y → ℝ be lower semicontinuous on Y. Suppose that f is concave-like on XX and convex-like on YY, i.e., for any x1,x2∈,y∈formulae-sequencesubscript1subscript2x_1,x_2 ,y 1 , x2 ∈ X , y ∈ Y, α∈[0,1]01α∈[0,1]α ∈ [ 0 , 1 ], there exists x3∈subscript3x_3 3 ∈ X such that f(x3,y)≥α⋅f(x1,y)+(1−α)⋅f(x2,y),subscript3⋅subscript1⋅1subscript2 f(x_3,y)≥α· f(x_1,y)+(1-α)· f(x_2,y),f ( x3 , y ) ≥ α ⋅ f ( x1 , y ) + ( 1 - α ) ⋅ f ( x2 , y ) , and for any y1,y2∈,x∈formulae-sequencesubscript1subscript2y_1,y_2 ,x 1 , y2 ∈ Y , x ∈ X, β∈[0,1]01β∈[0,1]β ∈ [ 0 , 1 ], there exists y3∈Ysubscript3y_3∈ Yy3 ∈ Y such that f(x,y3)≤β⋅f(x,y1)+(1−β)⋅f(x,y2).subscript3⋅subscript1⋅1subscript2 f(x,y_3)≤β· f(x,y_1)+(1-β)· f(x,y_2).f ( x , y3 ) ≤ β ⋅ f ( x , y1 ) + ( 1 - β ) ⋅ f ( x , y2 ) . Then the following holds: maxx∈miny∈f(x,y)=miny∈maxx∈f(x,y).subscriptsubscriptsubscriptsubscript _x _y f(x,y)= _y∈% Y _x f(x,y).maxitalic_x ∈ X minitalic_y ∈ Y f ( x , y ) = minitalic_y ∈ Y maxitalic_x ∈ X f ( x , y ) . Appendix E Finite-Sample Analysis of POWER This section is organized as follows. We begin by presenting the definition of single-policy concentrability for offline preference optimization, which characterizes the coverage of the competing policy in the dataset, in Section E.1. In Section E.2, we prove the finite-sample guarantees for POWER. E.1 Single-Policy Concentrability in Preference Optimization Definition 2 (Single-Policy Concentrability; Zhan et al. (2023a)). Given a policy π and ground truth reward r⋆superscript⋆r r⋆, the concentrability coefficient of offline data distribution μ with respect to the reward model class ℛRR and the baseline policy π′π π′ is defined as Cμπ(ℛ,π′)≔max0,supr∈ℛx∼ρ,y∼π,y′∼π′[r⋆(x,y)−r⋆(x,y′)−(r(x,y)−r(x,y′))]x,y,y′∼μ[(r⋆(x,y)−r⋆(x,y′)−(r(x,y)−r(x,y′)))2].≔subscriptsuperscriptℛsuperscript′0subscriptsupremumℛsubscriptformulae-sequencesimilar-toformulae-sequencesimilar-tosimilar-tosuperscript′delimited-[]superscript⋆superscript⋆superscript′subscriptsimilar-tosuperscript′delimited-[]superscriptsuperscript⋆superscript⋆superscript′2 C^π_μ(R,π ) \0,% _r E_x ρ,y π,y π^% [r (x,y)-r (x,y )-(r(x,y)-r(x,y ))% ] E_x,y,y μ [ (r (x,y)-r^% (x,y )-(r(x,y)-r(x,y )) )^2 ] \.Citalic_πitalic_μ ( R , π′ ) ≔ max 0 , supitalic_r ∈ R divide start_ARG italic_x ∼ ρ , y ∼ π , y′ ∼ π′ [ r⋆ ( x , y ) - r⋆ ( x , y′ ) - ( r ( x , y ) - r ( x , y′ ) ) ] end_ARG start_ARG square-root start_ARG italic_x , y , y′ ∼ μ [ ( r⋆ ( x , y ) - r⋆ ( x , y′ ) - ( r ( x , y ) - r ( x , y′ ) ) )2 ] end_ARG end_ARG . (26) Single-policy concentrability coefficient in offline RL quantifies the extent to which a target competing policy π is covered by an offline data collection distribution μ. In the offline RLHF setting, single-policy concentrability as defined in the work Zhan et al. (2023a) also depends on a baseline policy π′π π′. E.2 Proof of Theorem 1 To prove finite-sample guarantees, we use a similar argument to Liu et al. (2024), adapted to the weighted-entropy objective and combined with the bounds on weighted entropy and the special weights as inverse response lengths. Suboptimality of the learned policy π^≔π^POWER≔^subscript^POWER π π_POWERover start_ARG π end_ARG ≔ over start_ARG π end_ARGPOWER with respect to a competing policy π can be decomposed into three terms: J(π)−J(π^)=x∼ρ,y∼π[r⋆(x,y)]−x∼ρ,y∼π^[r⋆(x,y)]=T1+T2+T3.^subscriptformulae-sequencesimilar-tosimilar-todelimited-[]superscript⋆subscriptformulae-sequencesimilar-tosimilar-to^delimited-[]superscript⋆subscript1subscript2subscript3 J(π)-J( π)=E_x ρ,y π [r^% (x,y) ]-E_x ρ,y π [r (x,y)% ]=T_1+T_2+T_3.J ( π ) - J ( over start_ARG π end_ARG ) = italic_x ∼ ρ , y ∼ π [ r⋆ ( x , y ) ] - italic_x ∼ ρ , y ∼ over start_ARG π end_ARG [ r⋆ ( x , y ) ] = T1 + T2 + T3 . where T1subscript1T_1T1 is defined as T1≔x∼ρ,y∼π,y′∼π′[r⋆(x,y)−r⋆(x,y′)−βHw(π)]−η−1minr∈ℛηx∼ρ,y∼π^,y′∼π′[r(x,y)−r(x,y′)−βHw(π)]+LBT(r),≔subscript1subscriptformulae-sequencesimilar-toformulae-sequencesimilar-tosimilar-tosuperscript′delimited-[]superscript⋆superscript⋆superscript′subscriptsuperscript1subscriptℛsubscriptformulae-sequencesimilar-toformulae-sequencesimilar-to^similar-tosuperscript′delimited-[]superscript′subscriptsubscriptBT splitT_1& _x ρ,y π,y^% π [r (x,y)-r (x,y )-β H_% w(π) ]\\ & -η^-1 _r \ _x ρ,y% π,y π [r(x,y)-r(x,y )-β H_% w(π) ]+L_BT(r) \, splitstart_ROW start_CELL T1 end_CELL start_CELL ≔ italic_x ∼ ρ , y ∼ π , y′ ∼ π′ [ r⋆ ( x , y ) - r⋆ ( x , y′ ) - β Hitalic_w ( π ) ] end_CELL end_ROW start_ROW start_CELL end_CELL start_CELL - η- 1 minitalic_r ∈ R η italic_x ∼ ρ , y ∼ over start_ARG π end_ARG , y′ ∼ π′ [ r ( x , y ) - r ( x , y′ ) - β Hitalic_w ( π ) ] + LBT ( r ) , end_CELL end_ROW (27) T2subscript2T_2T2 is defined as T2≔η−1minr∈ℛηx∼ρ,y∼π^,y′∼π′[r(x,y)−r(x,y′)−βHw(π)]+LBT(r)−x∼ρ,y∼π^,y′∼π′[r⋆(x,y)−r⋆(x,y′)−βHw(π)],≔subscript2superscript1subscriptℛsubscriptformulae-sequencesimilar-toformulae-sequencesimilar-to^similar-tosuperscript′delimited-[]superscript′subscriptsubscriptBTsubscriptformulae-sequencesimilar-toformulae-sequencesimilar-to^similar-tosuperscript′delimited-[]superscript⋆superscript⋆superscript′subscript splitT_2& η^-1 _r % \ _x ρ,y π,y π [% r(x,y)-r(x,y )-β H_w(π) ]+L_BT(r) \\\ & -E_x ρ,y π,y π [r% (x,y)-r (x,y )-β H_w(π) ], splitstart_ROW start_CELL T2 end_CELL start_CELL ≔ η- 1 minitalic_r ∈ R η italic_x ∼ ρ , y ∼ over start_ARG π end_ARG , y′ ∼ π′ [ r ( x , y ) - r ( x , y′ ) - β Hitalic_w ( π ) ] + LBT ( r ) end_CELL end_ROW start_ROW start_CELL end_CELL start_CELL - italic_x ∼ ρ , y ∼ over start_ARG π end_ARG , y′ ∼ π′ [ r⋆ ( x , y ) - r⋆ ( x , y′ ) - β Hitalic_w ( π ) ] , end_CELL end_ROW (28) and T3subscript3T_3T3 is defined as T3≔β[Hw(π)−Hw(π^)].≔subscript3delimited-[]subscriptsubscript splitT_3& β [H_w(π)-H_w( π% ) ]. splitstart_ROW start_CELL T3 end_CELL start_CELL ≔ β [ Hitalic_w ( π ) - Hitalic_w ( over start_ARG π end_ARG ) ] . end_CELL end_ROW (29) We will prove in the subsequent section that for weighted entropy Hw(π)subscriptH_w(π)Hitalic_w ( π ) with general weights, terms T1+T2subscript1subscript2T_1+T_2T1 + T2 and T3subscript3T_3T3 are bounded according to: T1+T2subscript1subscript2 T_1+T_2T1 + T2 ≲(Cμπ(ℛ,π′)+1)R~2ιN,less-than-or-similar-toabsentsubscriptsuperscriptℛsuperscript′1superscript~2 (C^π_μ(R,π )+1) R% ^2 N,≲ divide start_ARG ( Citalic_πitalic_μ ( R , π′ ) + 1 ) over~ start_ARG R end_ARG2 ι end_ARG start_ARG square-root start_ARG N end_ARG end_ARG , (30) T3subscript3 T_3T3 ≲Hw(π)N.less-than-or-similar-toabsentsubscript H_w(π) N.≲ divide start_ARG Hitalic_w ( π ) end_ARG start_ARG square-root start_ARG N end_ARG end_ARG . (31) Summing the above bounds, we conclude the first claim: J(π)−J(π^)≲1N(([Cμπ(ℛ,π′)]2+1)R~2ι+Hw(π)).less-than-or-similar-to^1superscriptdelimited-[]subscriptsuperscriptℛsuperscript′21superscript~2subscript J(π)-J( π) 1 N ( ( [% C^π_μ(R,π ) ]^2+1 ) R^2 +% H_w(π) ).J ( π ) - J ( over start_ARG π end_ARG ) ≲ divide start_ARG 1 end_ARG start_ARG square-root start_ARG N end_ARG end_ARG ( ( [ Citalic_πitalic_μ ( R , π′ ) ]2 + 1 ) over~ start_ARG R end_ARG2 ι + Hitalic_w ( π ) ) . Furthermore, in the special case of w(y)=1/|y|1w(y)=1/|y|w ( y ) = 1 / | y |, we have the following bound on T3subscript3T_3T3: T3≲log||N,less-than-or-similar-tosubscript3 T_3 |V| N,T3 ≲ divide start_ARG log | V | end_ARG start_ARG square-root start_ARG N end_ARG end_ARG , (32) The above bound combined with (30) leads to the following rate J(π)−J(π^)≲1N(([Cμπ(ℛ,π′)]2+1)R~2ι+log||).less-than-or-similar-to^1superscriptdelimited-[]subscriptsuperscriptℛsuperscript′21superscript~2 J(π)-J( π) 1 N ( ( [% C^π_μ(R,π ) ]^2+1 ) R^2 +% |V| ).J ( π ) - J ( over start_ARG π end_ARG ) ≲ divide start_ARG 1 end_ARG start_ARG square-root start_ARG N end_ARG end_ARG ( ( [ Citalic_πitalic_μ ( R , π′ ) ]2 + 1 ) over~ start_ARG R end_ARG2 ι + log | V | ) . E.2.1 Proof of the Bound (30) on T1+T2subscript1subscript2T_1+T_2T1 + T2 Bounding T1subscript1T_1T1. π^ πover start_ARG π end_ARG is the maximizer to the following objective π^∈argmaxπ∈Πminr∈ℛηx∼ρ,y∼π^,y′∼π′[r(x,y)−r(x,y′)−βHw(π)]+LBT(r)^subscriptargmaxΠsubscriptℛsubscriptformulae-sequencesimilar-toformulae-sequencesimilar-to^similar-tosuperscript′delimited-[]superscript′subscriptsubscriptBT π∈ *argmax_π∈ _r∈% R _x ρ,y π,y π [r% (x,y)-r(x,y )-β H_w(π) ]+L_BT(r)over start_ARG π end_ARG ∈ argmaxitalic_π ∈ Π minitalic_r ∈ R η italic_x ∼ ρ , y ∼ over start_ARG π end_ARG , y′ ∼ π′ [ r ( x , y ) - r ( x , y′ ) - β Hitalic_w ( π ) ] + LBT ( r ) We use this fact to bound the term T1subscript1T_1T1 according to T1subscript1 T_1T1 ≤x∼ρ,y∼π,y′∼π′[r⋆(x,y)−r⋆(x,y′)−βHw(π)]absentsubscriptformulae-sequencesimilar-toformulae-sequencesimilar-tosimilar-tosuperscript′delimited-[]superscript⋆superscript⋆superscript′subscript _x ρ,y π,y π % [r (x,y)-r (x,y )-β H_w(π) ]≤ italic_x ∼ ρ , y ∼ π , y′ ∼ π′ [ r⋆ ( x , y ) - r⋆ ( x , y′ ) - β Hitalic_w ( π ) ] −η−1minr∈ℛηx∼ρ,y∼π,y′∼π′[r(x,y)−r(x,y′)−βHw(π)]+LBT(r),superscript1subscriptℛsubscriptformulae-sequencesimilar-toformulae-sequencesimilar-tosimilar-tosuperscript′delimited-[]superscript′subscriptsubscriptBT -η^-1 _r \ _x% ρ,y π,y π [r(x,y)-r(x,y )-% β H_w(π) ]+L_BT(r) \,- η- 1 minitalic_r ∈ R η italic_x ∼ ρ , y ∼ π , y′ ∼ π′ [ r ( x , y ) - r ( x , y′ ) - β Hitalic_w ( π ) ] + LBT ( r ) , =maxr∈ℛx∼ρ,y∼π,y′∼π′[r⋆(x,y)−r⋆(x,y′)−(r(x,y)−r(x,y′))]−η−1LBT(r)absentsubscriptℛsubscriptformulae-sequencesimilar-toformulae-sequencesimilar-tosimilar-tosuperscript′delimited-[]superscript⋆superscript⋆superscript′superscript1subscriptBT = _r \E_x ρ,y π,y^% π [r (x,y)-r (x,y )-(r(x,y)-r% (x,y )) ]-η^-1L_BT(r) \= maxitalic_r ∈ R italic_x ∼ ρ , y ∼ π , y′ ∼ π′ [ r⋆ ( x , y ) - r⋆ ( x , y′ ) - ( r ( x , y ) - r ( x , y′ ) ) ] - η- 1 LBT ( r ) (33) Bounding T2subscript2T_2T2. By realizability of the true reward function r⋆∈ℛsuperscript⋆ℛr ⋆ ∈ R we bound the term T2subscript2T_2T2: T2subscript2 T_2T2 ≤x∼ρ,y∼π,y′∼π′[r⋆(x,y)−r⋆(x,y′)−βHw(π)]+η−1LBT(r⋆)absentsubscriptformulae-sequencesimilar-toformulae-sequencesimilar-tosimilar-tosuperscript′delimited-[]superscript⋆superscript⋆superscript′subscriptsuperscript1subscriptBTsuperscript⋆ _x ρ,y π,y π % [r (x,y)-r (x,y )-β H_w(π) ]+η^-1% L_BT(r )≤ italic_x ∼ ρ , y ∼ π , y′ ∼ π′ [ r⋆ ( x , y ) - r⋆ ( x , y′ ) - β Hitalic_w ( π ) ] + η- 1 LBT ( r⋆ ) −x∼ρ,y∼π^,y′∼π′[r⋆(x,y)−r⋆(x,y′)−βHw(π)]subscriptformulae-sequencesimilar-toformulae-sequencesimilar-to^similar-tosuperscript′delimited-[]superscript⋆superscript⋆superscript′subscript -E_x ρ,y π,y π^% [r (x,y)-r (x,y )-β H_w(π) ]- italic_x ∼ ρ , y ∼ over start_ARG π end_ARG , y′ ∼ π′ [ r⋆ ( x , y ) - r⋆ ( x , y′ ) - β Hitalic_w ( π ) ] =η−1LBT(r⋆)absentsuperscript1subscriptBTsuperscript⋆ =η^-1L_BT(r )= η- 1 LBT ( r⋆ ) (34) Bounding T1+T2subscript1subscript2T_1+T_2T1 + T2. Combing the bound (33) on T1subscript1T_1T1 and to bound (34) on T2subscript2T_2T2, it remains the bound the following: T1+T2≤maxr∈ℛx∼ρ,y∼π,y′∼π′[r⋆(x,y)−r⋆(x,y′)−(r(x,y)−r(x,y′))]≔T1,1+η−1(LBT(r⋆)−LBT(r))≔T1,2subscript1subscript2subscriptℛ≔subscriptformulae-sequencesimilar-toformulae-sequencesimilar-tosimilar-tosuperscript′delimited-[]superscript⋆superscript⋆superscript′subscript11superscript1subscriptBTsuperscript⋆subscriptBT≔subscript12 splitT_1+T_2≤ _r \&% E_x ρ,y π,y π [r (x,y)% -r (x,y )-(r(x,y)-r(x,y )) ] T_1,1\\ & +η^-1 (L_BT(r )-L_BT(r) )% T_1,2 \ splitstart_ROW start_CELL T1 + T2 ≤ maxitalic_r ∈ R end_CELL start_CELL italic_x ∼ ρ , y ∼ π , y′ ∼ π′ [ r⋆ ( x , y ) - r⋆ ( x , y′ ) - ( r ( x , y ) - r ( x , y′ ) ) ] ≔ T1 , 1 end_CELL end_ROW start_ROW start_CELL end_CELL start_CELL + η- 1 ( LBT ( r⋆ ) - LBT ( r ) ) ≔ T1 , 2 end_CELL end_ROW (35) Define the following notation: Δr≔x,y,y′∼μ[(r⋆(x,y)−r⋆(x,y′))−(r(x,y)−r(x,y′))2]≔subscriptΔsubscriptsimilar-tosuperscript′delimited-[]superscript⋆superscript⋆superscript′superscript′2 _r E_x,y,y μ [% (r (x,y)-r (x,y ) )- (r(x,y)-r(x,y^% ) )^2 ]Δitalic_r ≔ square-root start_ARG italic_x , y , y′ ∼ μ [ ( r⋆ ( x , y ) - r⋆ ( x , y′ ) ) - ( r ( x , y ) - r ( x , y′ ) )2 ] end_ARG (36) Term T1,1subscript11T_1,1T1 , 1 is directly bounded by Cμπ(ℛ,π′)Δrsubscriptsuperscriptℛsuperscript′subscriptΔC^π_μ(R,π ) _rCitalic_πitalic_μ ( R , π′ ) Δitalic_r based on the Definition 2 of single-policy concentrability, and we subsequently prove a bound on T1,2subscript12T_1,2T1 , 2 according to: T1,1subscript11 T_1,1T1 , 1 ≤Cμπ(ℛ,π′)Δrabsentsubscriptsuperscriptℛsuperscript′subscriptΔ ≤ C^π_μ(R,π ) _r≤ Citalic_πitalic_μ ( R , π′ ) Δitalic_r (37) T1,2subscript12 T_1,2T1 , 2 ≤−2Δr2ηR~2+3ιηN,absent2subscriptsuperscriptΔ2superscript~23 ≤-2 ^2_rη R^2+ 3 η N,≤ - 2 divide start_ARG Δ2italic_r end_ARG start_ARG η over~ start_ARG R end_ARG2 end_ARG + divide start_ARG 3 ι end_ARG start_ARG η N end_ARG , (38) where R~=1+exp(R)~1 R=1+ (R)over~ start_ARG R end_ARG = 1 + exp ( R ) and ι=log(ϵ)/δsubscriptitalic-ϵ = (N_ε)/δι = square-root start_ARG log ( Nitalic_ϵ ) / δ end_ARG. Adding the bounds on T1,1subscript11T_1,1T1 , 1 and T1,2subscript12T_1,2T1 , 2 and taking the maximum over r, the bound on T1+T2subscript1subscript2T_1+T_2T1 + T2: T1+T2subscript1subscript2 T_1+T_2T1 + T2 ≤maxrCμπ(ℛ,π′)Δr−2Δr2ηR~2+3ιηNabsentsubscriptsubscriptsuperscriptℛsuperscript′subscriptΔ2superscriptsubscriptΔ2superscript~23 ≤ _r \C^π_μ(R,π ) _% r-2 _r^2η R^2 \+ 3 η N≤ maxitalic_r Citalic_πitalic_μ ( R , π′ ) Δitalic_r - 2 divide start_ARG Δitalic_r2 end_ARG start_ARG η over~ start_ARG R end_ARG2 end_ARG + divide start_ARG 3 ι end_ARG start_ARG η N end_ARG (39) ≤[Cμπ(ℛ,π′)]2ηR~28+3ιηNabsentsuperscriptdelimited-[]subscriptsuperscriptℛsuperscript′2superscript~283 ≤ [C^π_μ(R,π ) ]^2% η R^28+ 3 η N≤ divide start_ARG [ Citalic_πitalic_μ ( R , π′ ) ]2 η over~ start_ARG R end_ARG2 end_ARG start_ARG 8 end_ARG + divide start_ARG 3 ι end_ARG start_ARG η N end_ARG (40) The last inequality uses the fact that az−bz2≤a2/4bsuperscript2superscript24az-bz^2≤ a^2/4ba z - b z2 ≤ a2 / 4 b for any z∈ℝz ∈ ℝ. By the choice of η=6ι/(R~2N)6superscript~2η= 6 /( R^2 N)η = square-root start_ARG 6 end_ARG ι / ( over~ start_ARG R end_ARG2 square-root start_ARG N end_ARG ), the above bound becomes: T1+T2≲([Cμπ(ℛ,π′)]2+1)R~2ιN.less-than-or-similar-tosubscript1subscript2superscriptdelimited-[]subscriptsuperscriptℛsuperscript′21superscript~2 T_1+T_2 ( [C^π_μ(R,π^% ) ]^2+1) R^2 N.T1 + T2 ≲ divide start_ARG ( [ Citalic_πitalic_μ ( R , π′ ) ]2 + 1 ) over~ start_ARG R end_ARG2 ι end_ARG start_ARG square-root start_ARG N end_ARG end_ARG . Proof of the bound (38) on T1,2subscript12T_1,2T1 , 2.. In the view of the uniform concentration result in Liu et al. (2024, Lemma A.1), with probability at least 1−δ11- 1 - δ setting ϵ=(6R~N)−1italic-ϵsuperscript6~1ε=(6 RN)^-1ϵ = ( 6 over~ start_ARG R end_ARG N )- 1, the following bound holds for any r∈ℛr ∈ R LBT(r⋆)−LBT(r)subscriptBTsuperscript⋆subscriptBT L_BT(r )-L_BT(r)LBT ( r⋆ ) - LBT ( r ) ≤−2x,y,y′∼μ[DHellinger2(ℙr⋆(⋅|x,y,y′)∥ℙr(⋅|x,y,y′))]+3ιN, ≤-2E_x,y,y μ [D^2_% Hellinger ( P_r (·|x,y,y )\|% P_r(·|x,y,y ) ) ]+ 3 % N,≤ - 2 italic_x , y , y′ ∼ μ [ D2Hellinger ( ℙitalic_r⋆ ( ⋅ | x , y , y′ ) ∥ ℙitalic_r ( ⋅ | x , y , y′ ) ) ] + divide start_ARG 3 ι end_ARG start_ARG N end_ARG , (41) where ℙr(⋅|x,y,y′) P_r(·|x,y,y )ℙitalic_r ( ⋅ | x , y , y′ ) is the Bradley-Terry preference probability given a reward model r as defined in (2). The Hellinger distance can be bounded by total variation (TV) distance according to DHellinger2(ℙr⋆(⋅|x,y,y′)∥ℙr(⋅|x,y,y′)) D^2_Hellinger ( P_r^% (·|x,y,y )\| P_r(·|x,y,y^% ) )D2Hellinger ( ℙitalic_r⋆ ( ⋅ | x , y , y′ ) ∥ ℙitalic_r ( ⋅ | x , y , y′ ) ) ≥DTV2(ℙr⋆(⋅|x,y,y′)∥ℙr(⋅|x,y,y′)) ≥ D^2_TV ( P_r % (·|x,y,y )\| P_r(·|x,y,y ) )≥ D2TV ( ℙitalic_r⋆ ( ⋅ | x , y , y′ ) ∥ ℙitalic_r ( ⋅ | x , y , y′ ) ) =12|σ(r⋆(x,y)−r⋆(x,y′))−σ(r(x,y)−r(x,y′))|absent12superscript⋆superscript⋆superscript′ = 12 |σ (r (x,y)-r (x,y^% ) )-σ (r(x,y)-r(x,y ) ) |= divide start_ARG 1 end_ARG start_ARG 2 end_ARG | σ ( r⋆ ( x , y ) - r⋆ ( x , y′ ) ) - σ ( r ( x , y ) - r ( x , y′ ) ) | +12|σ(r⋆(x,y′)−r⋆(x,y))−σ(r(x,y′)−r(x,y))|12superscript⋆superscript′⋆superscript′ + 12 |σ (r (x,y )-r^% (x,y) )-σ (r(x,y )-r(x,y) ) |+ divide start_ARG 1 end_ARG start_ARG 2 end_ARG | σ ( r⋆ ( x , y′ ) - r⋆ ( x , y ) ) - σ ( r ( x , y′ ) - r ( x , y ) ) | =|σ(r⋆(x,y)−r⋆(x,y′))−σ(r(x,y)−r(x,y′))|absentsuperscript⋆superscript⋆superscript′ = |σ (r (x,y)-r (x,y ) )% -σ (r(x,y)-r(x,y ) ) |= | σ ( r⋆ ( x , y ) - r⋆ ( x , y′ ) ) - σ ( r ( x , y ) - r ( x , y′ ) ) | ≥1R~2|(r⋆(x,y)−r⋆(x,y′))−(r(x,y)−r(x,y′))|absent1superscript~2superscript⋆superscript⋆superscript′ ≥ 1 R^2 | (r (x,y)-r % (x,y ) )- (r(x,y)-r(x,y ) ) |≥ divide start_ARG 1 end_ARG start_ARG over~ start_ARG R end_ARG2 end_ARG | ( r⋆ ( x , y ) - r⋆ ( x , y′ ) ) - ( r ( x , y ) - r ( x , y′ ) ) | (42) The penultimate equation uses the fact that σ(−x)=1−σ(x)1σ(-x)=1-σ(x)σ ( - x ) = 1 - σ ( x ) and the last inequality is due to bi-Lipschitz continuity of the sigmoid function over [−R,R][-R,R][ - R , R ]; see e.g., Liu et al. (2024, Lemma A.2). Applying the bound in (42) to (41), we have LBT(r⋆)−LBT(r)subscriptBTsuperscript⋆subscriptBT L_BT(r )-L_BT(r)LBT ( r⋆ ) - LBT ( r ) ≤−2x,y,y′∼μ[|(r⋆(x,y)−r⋆(x,y′))−(r(x,y)−r(x,y′))|2]+3ιNabsent2subscriptsimilar-tosuperscript′delimited-[]superscriptsuperscript⋆superscript⋆superscript′23 ≤-2E_x,y,y μ [ | (r^% (x,y)-r (x,y ) )- (r(x,y)-r(x,y ) )% |^2 ]+ 3 N≤ - 2 italic_x , y , y′ ∼ μ [ | ( r⋆ ( x , y ) - r⋆ ( x , y′ ) ) - ( r ( x , y ) - r ( x , y′ ) ) |2 ] + divide start_ARG 3 ι end_ARG start_ARG N end_ARG =−2ΔrR~2+3ιN.absent2subscriptΔsuperscript~23 =- 2 _r R^2+ 3 N.= - divide start_ARG 2 Δitalic_r end_ARG start_ARG over~ start_ARG R end_ARG2 end_ARG + divide start_ARG 3 ι end_ARG start_ARG N end_ARG . where the last equation uses the definition of ΔrsubscriptΔ _rΔitalic_r provided in (36), completing the proof. ∎ E.2.2 Proof of the Bounds (31) and (32) on T3subscript3T_3T3 In this section, we prove the bounds on T3subscript3T_3T3 as delineated in inequalities (31) and (32) through bounding weighted entropy. The key bounds are encapsulated in the following lemma which asserts that weighted entropy is non-negative and for special case of weights w(y)=1/|y|1w(y)=1/|y|w ( y ) = 1 / | y | it can be bounded from above. The proof of this lemma is presented at the end of this section. Lemma 3 (Bounds on Weighted Entropy). For any weight function w(y)≥00w(y)≥ 0w ( y ) ≥ 0 and any probability distribution p(y)p(y)p ( y ), the weighted entropy satisfies Hw(p)≥0subscript0H_w(p)≥ 0Hitalic_w ( p ) ≥ 0. Furthermore, when weights are assigned according to w(y)=1/|y|1w(y)=1/|y|w ( y ) = 1 / | y |, with |y||y|| y | denoting the response length, the weighted entropy is bounded by Hw(p)≤log||subscriptH_w(p)≤ |V|Hitalic_w ( p ) ≤ log | V |, where |||V|| V | is the size of the vocabulary. Based on Lemma 3, weighted entropy is nonnegative. Setting β≍1/Nasymptotically-equals1β 1/ Nβ ≍ 1 / square-root start_ARG N end_ARG immediately gives the bound (31) on T3subscript3T_3T3: T3=β[Hw(π)−Hw(π^)]≤Hw(π)Nsubscript3delimited-[]subscriptsubscript^subscript splitT_3&=β [H_w(π)-H_w( π) ]% ≤ H_w(π) N splitstart_ROW start_CELL T3 end_CELL start_CELL = β [ Hitalic_w ( π ) - Hitalic_w ( over start_ARG π end_ARG ) ] ≤ divide start_ARG Hitalic_w ( π ) end_ARG start_ARG square-root start_ARG N end_ARG end_ARG end_CELL end_ROW (43) Moreover, when w(y)=1/|y|1w(y)=1/|y|w ( y ) = 1 / | y | by Lemma 3, we have T3≤Hw(π)N≤log||N.subscript3subscript T_3≤ H_w(π) N≤ |V|% N.T3 ≤ divide start_ARG Hitalic_w ( π ) end_ARG start_ARG square-root start_ARG N end_ARG end_ARG ≤ divide start_ARG log | V | end_ARG start_ARG square-root start_ARG N end_ARG end_ARG . Proof of Lemma 3. First consider the case for any general non-negative weight function w(y)>00w(y)>0w ( y ) > 0. This ensures that the weighted entropy is non-negative because: Hw(p)=−∑yw(y)p(y)logp(y)=∑yw(y)p(y)log1p(y)≥0.subscriptsubscriptsubscript10 H_w(p)=- _yw(y)p(y) p(y)= _yw(y)p(y) 1% p(y)≥ 0.Hitalic_w ( p ) = - ∑y w ( y ) p ( y ) log p ( y ) = ∑y w ( y ) p ( y ) log divide start_ARG 1 end_ARG start_ARG p ( y ) end_ARG ≥ 0 . Next, we provide an upper bound on the weighted entropy when w(y)=1/|y|1w(y)=1/|y|w ( y ) = 1 / | y |. Define z to be a random variable denoting the length of a response. The weighted entropy can be decomposed as follows −∑y1|y|p(y)logp(y)subscript1 - _y 1|y|p(y) p(y)- ∑y divide start_ARG 1 end_ARG start_ARG | y | end_ARG p ( y ) log p ( y ) =−∑y1|y|pz(z=|y|)p(y∣|y|=z)logpz(z=|y|)p(y∣|y|=z) =- _y 1|y|p_z(z=|y|)p(y |y|=z) p_z(z=|y|)% p(y |y|=z)= - ∑y divide start_ARG 1 end_ARG start_ARG | y | end_ARG pitalic_z ( z = | y | ) p ( y ∣ | y | = z ) log pitalic_z ( z = | y | ) p ( y ∣ | y | = z ) =−∑y1|y|pz(z=|y|)p(y∣|y|=z)[logpz(z=|y|)+logp(y∣|y|=z)] =- _y 1|y|p_z(z=|y|)p(y |y|=z) [ p_z% (z=|y|)+ p(y |y|=z) ]= - ∑y divide start_ARG 1 end_ARG start_ARG | y | end_ARG pitalic_z ( z = | y | ) p ( y ∣ | y | = z ) [ log pitalic_z ( z = | y | ) + log p ( y ∣ | y | = z ) ] =−∑z1|y|pz(z=|y|)∑y s.t. |y|=zp(y∣|y|=z)logp(y∣|y|=z)≔T3,1 =- _z 1|y|p_z(z=|y|) _y s.t. |y|=zp(y% |y|=z) p(y |y|=z) T_3,1= - ∑z divide start_ARG 1 end_ARG start_ARG | y | end_ARG pitalic_z ( z = | y | ) ∑y s.t. | y | = z p ( y ∣ | y | = z ) log p ( y ∣ | y | = z ) ≔ T3 , 1 −∑z1|y|pz(z=|y|)logpz(z=|y|)∑y s.t. |y|=zp(y∣|y|=z)≔T3,2 - _z 1|y|p_z(z=|y|) p_z(z=|y|) _y% s.t. |y|=zp(y |y|=z) T_3,2- ∑z divide start_ARG 1 end_ARG start_ARG | y | end_ARG pitalic_z ( z = | y | ) log pitalic_z ( z = | y | ) ∑y s.t. | y | = z p ( y ∣ | y | = z ) ≔ T3 , 2 For the term T3,1subscript31T_3,1T3 , 1, we have T3,1=∑zpz(z=|y|)1|y|⋅−∑y s.t. |y|=zpy|z(y∣z=|y|)logpy|z(y∣z=|y|) T_3,1= _zp_z(z=|y|) 1|y|·- _y s.t% . |y|=zp_y|z(y z=|y|) p_y|z(y z=|y|)T3 , 1 = ∑z pitalic_z ( z = | y | ) divide start_ARG 1 end_ARG start_ARG | y | end_ARG ⋅ - ∑y s.t. | y | = z pitalic_y | z ( y ∣ z = | y | ) log pitalic_y | z ( y ∣ z = | y | ) The sum −∑ypy|z(y∣z=|y|)logpy|z(y∣z=|y|)subscriptsubscriptconditionalconditionalsubscriptconditionalconditional- _yp_y|z(y z=|y|) p_y|z(y z=|y|)- ∑y pitalic_y | z ( y ∣ z = | y | ) log pitalic_y | z ( y ∣ z = | y | ) is the Shannon entropy of a conditional distribution, which reaches its maximum when the distribution is uniform. Consequently, the maximum of this conditional entropy for a fixed length |y||y|| y | is given by log|||y|=|y|log||superscript |V|^|y|=|y| |V|log | V || y | = | y | log | V |. Substituting this bound back to T1subscript1T_1T1 gives: T3,1≤∑zpz(z=|y|)1|y||y|log||=log||subscript31subscriptsubscript1 T_3,1≤ _zp_z(z=|y|) 1|y||y| |V|% = |V|T3 , 1 ≤ ∑z pitalic_z ( z = | y | ) divide start_ARG 1 end_ARG start_ARG | y | end_ARG | y | log | V | = log | V | For the term T3,2subscript32T_3,2T3 , 2, first note that ∑y∣|y|=zp(y∣|y|=z)=1 _y |y|=zp(y |y|=z)=1∑y ∣ | y | = z p ( y ∣ | y | = z ) = 1. Therefore, with the maximum response length denoted by L and since |y|≥11|y|≥ 1| y | ≥ 1, we have T3,2=∑z1|y|pz(z=|y|)logpz(z=|y|)≤∑zpz(z=|y|)logpz(z=|y|)≤logL.subscript32subscript1subscriptsubscriptsubscriptsubscriptsubscript T_3,2= _z 1|y|p_z(z=|y|) p_z(z=|y|)≤% _zp_z(z=|y|) p_z(z=|y|)≤ L.T3 , 2 = ∑z divide start_ARG 1 end_ARG start_ARG | y | end_ARG pitalic_z ( z = | y | ) log pitalic_z ( z = | y | ) ≤ ∑z pitalic_z ( z = | y | ) log pitalic_z ( z = | y | ) ≤ log L . Combining the bounds on T3,1subscript31T_3,1T3 , 1 and T3,2subscript32T_3,2T3 , 2, the upper bound on weighted entropy when using inverse response length as weights is log||+logL |V|+ Llog | V | + log L, which concludes the proof. ∎ Appendix F Derivations and Proofs for Dynamic Labels F.1 Derivation of the Learning Dynamics In this section we compute the learning dynamics of POWER over preference dataset =(x,y0,y1,l)superscript0superscript1D=\(x,y^0,y^1,l)\D = ( x , y0 , y1 , l ) with the label notation defined in 2.1. Here, l=00l=0l = 0 indicates that y0superscript0y^0y0 was preferred and l=11l=1l = 1 indicates that l=11l=1l = 1 was preferred. With this notation, the POWER objective from Proposition (3) is given by maxπ[llogσ(β[w(y1)logπθ(y1|x)−w(y0)logπθ(y0|x)+(w(y1)−w(y0))])(1−l)logσ(β[w(y0)logπθ(y0|x)−w(y1)logπθ(y1|x)+(w(y0)−w(y1))])]+ηβ[lw(y1)logπθ(y1|x)+(1−l)w(y0)logπθ(y0|x)]subscriptsubscriptdelimited-[]delimited-[]superscript1subscriptconditionalsuperscript1superscript0subscriptconditionalsuperscript0superscript1superscript01delimited-[]superscript0subscript|superscript0superscript1subscript|superscript1superscript0superscript1subscriptdelimited-[]superscript1subscriptconditionalsuperscript11superscript0subscriptconditionalsuperscript0 split _πE_D [&l % σ (β [w(y^1) _θ(y^1|x)-w(y^0) _% θ(y^0|x)+ (w(y^1)-w(y^0) ) ] )\\ &(1-l) σ (β [w(y^0) _θ(y^0|x)-w(y^1)% _θ(y^1|x)+ (w(y^0)-w(y^1) ) ] ) % ]\\ +&η _D [lw(y^1) _θ(y^1|x)+% (1-l)w(y^0) _θ(y^0|x) ] splitstart_ROW start_CELL maxitalic_π caligraphic_D [ end_CELL start_CELL l log σ ( β [ w ( y1 ) log πitalic_θ ( y1 | x ) - w ( y0 ) log πitalic_θ ( y0 | x ) + ( w ( y1 ) - w ( y0 ) ) ] ) end_CELL end_ROW start_ROW start_CELL end_CELL start_CELL ( 1 - l ) log σ ( β [ w ( y0 ) log πitalic_θ ( y0 | x ) - w ( y1 ) log πitalic_θ ( y1 | x ) + ( w ( y0 ) - w ( y1 ) ) ] ) ] end_CELL end_ROW start_ROW start_CELL + end_CELL start_CELL η β caligraphic_D [ l w ( y1 ) log πitalic_θ ( y1 | x ) + ( 1 - l ) w ( y0 ) log πitalic_θ ( y0 | x ) ] end_CELL end_ROW To simplify presentation, we consider the softmax MAB setting with β=1,w(y)=1,η=0formulae-sequence1formulae-sequence10β=1,w(y)=1,η=0β = 1 , w ( y ) = 1 , η = 0. In this case, the objective simplifies to minθ−[llogσ(logπθ(y1)−logπθ(y0))+(1−l)logσ(logπθ(y0)−logπθ(y1))]subscriptsubscriptdelimited-[]subscriptsuperscript1subscriptsuperscript01subscriptsuperscript0subscriptsuperscript1 _θ-E_D [l σ( _% θ(y^1)- _θ(y^0))+(1-l) σ( _θ(y^0% )- _θ(y^1)) ]minitalic_θ - caligraphic_D [ l log σ ( log πitalic_θ ( y1 ) - log πitalic_θ ( y0 ) ) + ( 1 - l ) log σ ( log πitalic_θ ( y0 ) - log πitalic_θ ( y1 ) ) ] =minθ−[llogσ(logexp(θ(y1))−logZθ−logexp(θ(y0))+logZθ) = _θ-E_D [l σ(% (θ(y^1))- Z_θ- (θ(y^0))+ Z_θ)= minitalic_θ - caligraphic_D [ l log σ ( log exp ( θ ( y1 ) ) - log Zitalic_θ - log exp ( θ ( y0 ) ) + log Zitalic_θ ) +(1−l)logσ(logexp(θ(y0))−logZθ−logexp(θ(y1))+logZθ] +(1-l) σ( (θ(y^0))- Z_% θ- (θ(y^1))+ Z_θ ]+ ( 1 - l ) log σ ( log exp ( θ ( y0 ) ) - log Zitalic_θ - log exp ( θ ( y1 ) ) + log Zitalic_θ ] =minθ−[llogσ(θ(y1)−θ(y0))+(1−l)logσ(θ(y0)−θ(y1))].absentsubscriptsubscriptdelimited-[]superscript1superscript01superscript0superscript1 = _θ-E_D [l σ(% θ(y^1)-θ(y^0))+(1-l) σ(θ(y^0)-θ(y^1))% ].= minitalic_θ - caligraphic_D [ l log σ ( θ ( y1 ) - θ ( y0 ) ) + ( 1 - l ) log σ ( θ ( y0 ) - θ ( y1 ) ) ] . To understand the updates to the model parameters, consider the empirical probabilities derived from dataset comparisons: μ^0,1subscript^01 μ_0,1over start_ARG μ end_ARG0 , 1 is the empirical probability of comparing y0superscript0y^0y0 and y1superscript1y^1y1, and μ^1≻0subscript^succeeds10 μ_1 0over start_ARG μ end_ARG1 ≻ 0 is the empirical probability of preferring y1superscript1y^1y1 over y0superscript0y^0y0, conditioned on their comparison. Isolate the updates to the parameters θ(y1)superscript1θ(y^1)θ ( y1 ) and θ(y0)superscript0θ(y^0)θ ( y0 ) based on comparisons between y0superscript0y^0y0 and y1superscript1y^1y1 and through batch gradient descent. We allow the preference labels ltsubscriptl_tlitalic_t to change across gradient steps and thus updates to the parameter gap at step t and with a learning rate α is given by θt+1(y1)−θt+1(y0)subscript1superscript1subscript1superscript0 _t+1(y^1)- _t+1(y^0)θitalic_t + 1 ( y1 ) - θitalic_t + 1 ( y0 ) =θt(y1)−θt(y0)+αμ^0,1[ = _t(y^1)- _t(y^0)+α μ_0,1% [= θitalic_t ( y1 ) - θitalic_t ( y0 ) + α over start_ARG μ end_ARG0 , 1 [ (μ^1≻0lt+μ^0≻1(1−lt))σ(θt(y0)−θt(y1))subscript^succeeds10subscriptsubscript^succeeds011subscriptsubscriptsuperscript0subscriptsuperscript1 ( μ_1 0l_t+ μ_0 1(1-l_t)% )σ( _t(y^0)- _t(y^1))( over start_ARG μ end_ARG1 ≻ 0 litalic_t + over start_ARG μ end_ARG0 ≻ 1 ( 1 - litalic_t ) ) σ ( θitalic_t ( y0 ) - θitalic_t ( y1 ) ) −(μ^0≻1lt+μ^1≻0(1−lt))σ(θt(y1)−θt(y0))]. - ( μ_0 1l_t+ μ_1 0(1-l_% t) )σ( _t(y^1)- _t(y^0)) ].- ( over start_ARG μ end_ARG0 ≻ 1 litalic_t + over start_ARG μ end_ARG1 ≻ 0 ( 1 - litalic_t ) ) σ ( θitalic_t ( y1 ) - θitalic_t ( y0 ) ) ] . We used the fact that ∂(logσ(x))/∂x=σ(−x)∂( σ(x))/∂ x=σ(-x)∂ ( log σ ( x ) ) / ∂ x = σ ( - x ). Since σ(x)=1−σ(−x)1σ(x)=1-σ(-x)σ ( x ) = 1 - σ ( - x ) and μ^1≻0+μ^0≻1=1subscript^succeeds10subscript^succeeds011 μ_1 0+ μ_0 1=1over start_ARG μ end_ARG1 ≻ 0 + over start_ARG μ end_ARG0 ≻ 1 = 1, the learning dynamics simplify to: θt+1(y1)−θt+1(y0)subscript1superscript1subscript1superscript0 _t+1(y^1)- _t+1(y^0)θitalic_t + 1 ( y1 ) - θitalic_t + 1 ( y0 ) =θt(y1)−θt(y0)+αμ^0,1[(μ^1≻0−μ^0≻1)l−(σ(θt(y1)−θt(y0))−μ^0≻1)].absentsubscriptsuperscript1subscriptsuperscript0subscript^01delimited-[]subscript^succeeds10subscript^succeeds01subscriptsuperscript1subscriptsuperscript0subscript^succeeds01 = _t(y^1)- _t(y^0)+α μ_0,1% [( μ_1 0- μ_0 1)l- (σ ( _% t(y^1)- _t(y^0) )- μ_0 1 ) ].= θitalic_t ( y1 ) - θitalic_t ( y0 ) + α over start_ARG μ end_ARG0 , 1 [ ( over start_ARG μ end_ARG1 ≻ 0 - over start_ARG μ end_ARG0 ≻ 1 ) l - ( σ ( θitalic_t ( y1 ) - θitalic_t ( y0 ) ) - over start_ARG μ end_ARG0 ≻ 1 ) ] . F.2 Proof of Theorem 2 The proof is organized as follows. We start by establishing a lower bound on the dynamic labels. We subsequently use this lower bound to prove bounds on the parameter gap in low-coverage and high-coverage cases separately. Lower bound on dynamic labels. The dynamics of labels are described by the following equation: l˙tsubscript˙ l_tover˙ start_ARG l end_ARGt =γ(σ(dt)−(1−μ^1≻0)μ^1≻0−(1−μ^1≻0)−lt)absentsubscript1subscript^succeeds10subscript^succeeds101subscript^succeeds10subscript =γ ( σ(d_t)-(1- μ_1 0) % μ_1 0-(1- μ_1 0)-l_t )= γ ( divide start_ARG σ ( ditalic_t ) - ( 1 - over start_ARG μ end_ARG1 ≻ 0 ) end_ARG start_ARG over start_ARG μ end_ARG1 ≻ 0 - ( 1 - over start_ARG μ end_ARG1 ≻ 0 ) end_ARG - litalic_t ) (44) Without loss of generality, we assumed that μ^1≻0>1/2subscript^succeeds1012 μ_1 0>1/2over start_ARG μ end_ARG1 ≻ 0 > 1 / 2. This condition is easily met by appropriately ordering the responses. Define: κ≔1−μ^1≻0μ^1≻0−(1−μ^1≻0)≔1subscript^succeeds10subscript^succeeds101subscript^succeeds10 κ 1- μ_1 0 μ_1 0% -(1- μ_1 0)κ ≔ divide start_ARG 1 - over start_ARG μ end_ARG1 ≻ 0 end_ARG start_ARG over start_ARG μ end_ARG1 ≻ 0 - ( 1 - over start_ARG μ end_ARG1 ≻ 0 ) end_ARG (45) Given that 0≤σ(dt)≤10subscript10≤σ(d_t)≤ 10 ≤ σ ( ditalic_t ) ≤ 1 and the assumption μ^1≻0−(1−μ^1≻0)=2μ^1≻0−1>0subscript^succeeds101subscript^succeeds102subscript^succeeds1010 μ_1 0-(1- μ_1 0)=2 μ_1 0-1>0over start_ARG μ end_ARG1 ≻ 0 - ( 1 - over start_ARG μ end_ARG1 ≻ 0 ) = 2 over start_ARG μ end_ARG1 ≻ 0 - 1 > 0, we find the following lower bound on l˙tsubscript˙ l_tover˙ start_ARG l end_ARGt: l˙tsubscript˙ l_tover˙ start_ARG l end_ARGt =γ(σ(dt)−(1−μ^1≻0)μ^1≻0−(1−μ^1≻0)−lt)≥γ(−(1−μ^1≻0)μ^1≻0−(1−μ^1≻0)−lt)=γ(−κ−lt)absentsubscript1subscript^succeeds10subscript^succeeds101subscript^succeeds10subscript1subscript^succeeds10subscript^succeeds101subscript^succeeds10subscriptsubscript =γ ( σ(d_t)-(1- μ_1 0) % μ_1 0-(1- μ_1 0)-l_t )≥γ ( -% (1- μ_1 0) μ_1 0-(1- μ_1 0)-l_t% )=γ (-κ-l_t )= γ ( divide start_ARG σ ( ditalic_t ) - ( 1 - over start_ARG μ end_ARG1 ≻ 0 ) end_ARG start_ARG over start_ARG μ end_ARG1 ≻ 0 - ( 1 - over start_ARG μ end_ARG1 ≻ 0 ) end_ARG - litalic_t ) ≥ γ ( divide start_ARG - ( 1 - over start_ARG μ end_ARG1 ≻ 0 ) end_ARG start_ARG over start_ARG μ end_ARG1 ≻ 0 - ( 1 - over start_ARG μ end_ARG1 ≻ 0 ) end_ARG - litalic_t ) = γ ( - κ - litalic_t ) The subsequent lemma establishes a lower bound on ltsubscriptl_tlitalic_t using Grönwall’s inequality, with its proof provided at the end of this section. Lemma 4. Suppose that ltsubscriptl_tlitalic_t satisfies the following inequality l˙t≥γ(−κ−lt)subscript˙subscript l_t≥γ(-κ-l_t)over˙ start_ARG l end_ARGt ≥ γ ( - κ - litalic_t ) with initial value l0=1subscript01l_0=1l0 = 1. Then, we have the following lower bound lt≥−κ+(κ+1)exp(−γt).subscript1l_t≥-κ+(κ+1) (-γ t).litalic_t ≥ - κ + ( κ + 1 ) exp ( - γ t ) . We proceed by separately analyzing the scenarios of low coverage and high coverage. Low coverage case. The coupled dynamical system in (12) satisfies the following equation: γ2μ^1≻0−1d˙t+αμ^0,1l˙t=02subscript^succeeds101subscript˙subscript^01subscript˙0 γ2 μ_1 0-1 d_t+α μ% _0,1 l_t=0divide start_ARG γ end_ARG start_ARG 2 over start_ARG μ end_ARG1 ≻ 0 - 1 end_ARG over˙ start_ARG d end_ARGt + α over start_ARG μ end_ARG0 , 1 over˙ start_ARG l end_ARGt = 0 Upon integrating the equation above and considering the initial condition l0=1subscript01l_0=1l0 = 1, it follows that γ2μ^1≻0−1(dt−d0)=αμ^0,1(1−lt)2subscript^succeeds101subscriptsubscript0subscript^011subscript γ2 μ_1 0-1 (d_t-d_0 )=% α μ_0,1 (1-l_t )divide start_ARG γ end_ARG start_ARG 2 over start_ARG μ end_ARG1 ≻ 0 - 1 end_ARG ( ditalic_t - d0 ) = α over start_ARG μ end_ARG0 , 1 ( 1 - litalic_t ) Applying the lower bound from Lemma 4 yields: γ2μ^1≻0−1(dt−d0)2subscript^succeeds101subscriptsubscript0 γ2 μ_1 0-1 (d_t-d_0 )divide start_ARG γ end_ARG start_ARG 2 over start_ARG μ end_ARG1 ≻ 0 - 1 end_ARG ( ditalic_t - d0 ) =αμ^0,1(1−lt)absentsubscript^011subscript =α μ_0,1 (1-l_t )= α over start_ARG μ end_ARG0 , 1 ( 1 - litalic_t ) ≤αμ^0,1(1+κ−(1+κ)exp(−γt))absentsubscript^0111 ≤α μ_0,1(1+κ-(1+κ) (-γ t))≤ α over start_ARG μ end_ARG0 , 1 ( 1 + κ - ( 1 + κ ) exp ( - γ t ) ) Consequently, we conclude that |dt−d0|subscriptsubscript0 |d_t-d_0|| ditalic_t - d0 | ≤αμ^0,1(2μ^1≻0−1)γ(1+κ−(1+κ)exp(−γt))≤αμ^0,1μ^1≻0γ≤αμlγ≤ϵl,absentsubscript^012subscript^succeeds10111subscript^01subscript^succeeds10subscriptsubscriptitalic-ϵ ≤ α μ_0,1(2 μ_1 0-1)γ% (1+κ-(1+κ) (-γ t) )≤ α μ_0,1% μ_1 0γ≤ α _lγ≤ _l,≤ divide start_ARG α over start_ARG μ end_ARG0 , 1 ( 2 over start_ARG μ end_ARG1 ≻ 0 - 1 ) end_ARG start_ARG γ end_ARG ( 1 + κ - ( 1 + κ ) exp ( - γ t ) ) ≤ divide start_ARG α over start_ARG μ end_ARG0 , 1 over start_ARG μ end_ARG1 ≻ 0 end_ARG start_ARG γ end_ARG ≤ divide start_ARG α μitalic_l end_ARG start_ARG γ end_ARG ≤ ϵitalic_l , where we used the definition of κ and the fact that by assumption αμl/ϵl≤γsubscriptsubscriptitalic-ϵα _l/ _l≤γα μitalic_l / ϵitalic_l ≤ γ. High coverage case. We extend the argument by Zhu et al. (2024) for a general initialization d0subscript0d_0d0 and establish the final convergence rate for proper choices of hyperparameters. Consider a Lyapunov function Vt=(σ(dt)−μ^1≻0)2subscriptsuperscriptsubscriptsubscript^succeeds102V_t= (σ(d_t)- μ_1 0 )^2Vitalic_t = ( σ ( ditalic_t ) - over start_ARG μ end_ARG1 ≻ 0 )2. Derivative of VtsubscriptV_tVitalic_t is given by V˙tsubscript˙ V_tover˙ start_ARG V end_ARGt =2(σ(dt)−μ^1≻0)σ(dt)σ(−dt)d˙tabsent2subscriptsubscript^succeeds10subscriptsubscriptsubscript˙ =2 (σ(d_t)- μ_1 0 )σ(d_t)% σ(-d_t) d_t= 2 ( σ ( ditalic_t ) - over start_ARG μ end_ARG1 ≻ 0 ) σ ( ditalic_t ) σ ( - ditalic_t ) over˙ start_ARG d end_ARGt =2αμ^0,1(σ(dt)−μ^1≻0)σ(dt)σ(−dt)((2μ^1≻0−1)lt+1−μ^1≻0−σ(dt))absent2subscript^01subscriptsubscript^succeeds10subscriptsubscript2subscript^succeeds101subscript1subscript^succeeds10subscript =2α μ_0,1 (σ(d_t)- μ_1 0% )σ(d_t)σ(-d_t) ((2 μ_1 0-1)l_t+1- % μ_1 0-σ(d_t) )= 2 α over start_ARG μ end_ARG0 , 1 ( σ ( ditalic_t ) - over start_ARG μ end_ARG1 ≻ 0 ) σ ( ditalic_t ) σ ( - ditalic_t ) ( ( 2 over start_ARG μ end_ARG1 ≻ 0 - 1 ) litalic_t + 1 - over start_ARG μ end_ARG1 ≻ 0 - σ ( ditalic_t ) ) =−2αμ^0,1σ(dt)σ(−dt)(σ(dt)−μ^1≻0)2+2αnσ(dt)σ(−dt)(σ(dt)−μ^1≻0)(2μ^1≻0−1)(lt−1)absent2subscript^01subscriptsubscriptsuperscriptsubscriptsubscript^succeeds1022subscriptsubscriptsubscriptsubscript^succeeds102subscript^succeeds101subscript1 =-2α μ_0,1σ(d_t)σ(-d_t) (σ(% d_t)- μ_1 0 )^2+2α nσ(d_t)σ(-d_t)% (σ(d_t)- μ_1 0 )(2 μ_1 0-1)(l_t% -1)= - 2 α over start_ARG μ end_ARG0 , 1 σ ( ditalic_t ) σ ( - ditalic_t ) ( σ ( ditalic_t ) - over start_ARG μ end_ARG1 ≻ 0 )2 + 2 α n σ ( ditalic_t ) σ ( - ditalic_t ) ( σ ( ditalic_t ) - over start_ARG μ end_ARG1 ≻ 0 ) ( 2 over start_ARG μ end_ARG1 ≻ 0 - 1 ) ( litalic_t - 1 ) =2αμ^0,1σ(dt)σ(−dt)(Vt−(σ(dt)−μ^1≻0)(2μ^1≻0−1)(lt−1))absent2subscript^01subscriptsubscriptsubscriptsubscriptsubscript^succeeds102subscript^succeeds101subscript1 =2α μ_0,1σ(d_t)σ(-d_t) (V_t-(% σ(d_t)- μ_1 0)(2 μ_1 0-1)(l_t-1) )= 2 α over start_ARG μ end_ARG0 , 1 σ ( ditalic_t ) σ ( - ditalic_t ) ( Vitalic_t - ( σ ( ditalic_t ) - over start_ARG μ end_ARG1 ≻ 0 ) ( 2 over start_ARG μ end_ARG1 ≻ 0 - 1 ) ( litalic_t - 1 ) ) Let ϵ=ϵ0αμ^0,1T≥γTitalic-ϵsubscriptitalic-ϵ0subscript^01ε= _0α μ_0,1T≥γ Tϵ = ϵ0 α over start_ARG μ end_ARG0 , 1 T ≥ γ T. We find an upper bound on V˙tsubscript˙ V_tover˙ start_ARG V end_ARGt by applying the bound on ltsubscriptl_tlitalic_t given in Lemma 4 and using the fact that μ^1≻0,σ∈[0,1]subscript^succeeds1001 μ_1 0,σ∈[0,1]over start_ARG μ end_ARG1 ≻ 0 , σ ∈ [ 0 , 1 ]: V˙tsubscript˙ V_tover˙ start_ARG V end_ARGt ≤αμ^0,1σ(dt)σ(−dt)(−Vt+(κ+1)(1−exp(−γt)))absentsubscript^01subscriptsubscriptsubscript11 ≤α μ_0,1σ(d_t)σ(-d_t) (-V_t% +(κ+1)(1- (-γ t)) )≤ α over start_ARG μ end_ARG0 , 1 σ ( ditalic_t ) σ ( - ditalic_t ) ( - Vitalic_t + ( κ + 1 ) ( 1 - exp ( - γ t ) ) ) (46) ≤αμ^0,1σ(dt)σ(−dt)(−Vt+(κ+1)(1−exp(−ϵ)))absentsubscript^01subscriptsubscriptsubscript11italic-ϵ ≤α μ_0,1σ(d_t)σ(-d_t) (-V_t% +(κ+1)(1- (-ε)) )≤ α over start_ARG μ end_ARG0 , 1 σ ( ditalic_t ) σ ( - ditalic_t ) ( - Vitalic_t + ( κ + 1 ) ( 1 - exp ( - ϵ ) ) ) (47) Now consider two cases. The first case is that for any 0≤t≤T00≤ t≤ T0 ≤ t ≤ T, we have Vt≥2(κ+1)(1−exp(−ϵ))subscript211italic-ϵV_t≥ 2(κ+1)(1- (-ε))Vitalic_t ≥ 2 ( κ + 1 ) ( 1 - exp ( - ϵ ) ). In such a scenario, VtsubscriptV_tVitalic_t is a non-increasing function because V˙tsubscript˙ V_tover˙ start_ARG V end_ARGt ≤2αμ^0,1σ(dt)σ(−dt)(−Vt+(κ+1)(1−exp(−ϵ)))≤−αμ^0,1σ(dt)σ(−dt)Vt≤0.absent2subscript^01subscriptsubscriptsubscript11italic-ϵsubscript^01subscriptsubscriptsubscript0 ≤ 2α μ_0,1σ(d_t)σ(-d_t) (-V_% t+(κ+1)(1- (-ε)) )≤-α μ_0,1σ(d_t)% σ(-d_t)V_t≤ 0.≤ 2 α over start_ARG μ end_ARG0 , 1 σ ( ditalic_t ) σ ( - ditalic_t ) ( - Vitalic_t + ( κ + 1 ) ( 1 - exp ( - ϵ ) ) ) ≤ - α over start_ARG μ end_ARG0 , 1 σ ( ditalic_t ) σ ( - ditalic_t ) Vitalic_t ≤ 0 . (48) Next, we analyze the term σ(dt)σ(−dt)subscriptsubscriptσ(d_t)σ(-d_t)σ ( ditalic_t ) σ ( - ditalic_t ). We establish a bound on σ(dt)σ(−dt)subscriptsubscriptσ(d_t)σ(-d_t)σ ( ditalic_t ) σ ( - ditalic_t ) for the case of σ(d0)≤μ^1≻0subscript0subscript^succeeds10σ(d_0)≤ μ_1 0σ ( d0 ) ≤ over start_ARG μ end_ARG1 ≻ 0; the case of σ(d0)≥μ^1≻0subscript0subscript^succeeds10σ(d_0)≥ μ_1 0σ ( d0 ) ≥ over start_ARG μ end_ARG1 ≻ 0 can be proved with a similar argument. We prove that when σ(d0)≤μ^1≻0subscript0subscript^succeeds10σ(d_0)≤ μ_1 0σ ( d0 ) ≤ over start_ARG μ end_ARG1 ≻ 0, then we must have σ(dt)≤μ^1≻0subscriptsubscript^succeeds10σ(d_t)≤ μ_1 0σ ( ditalic_t ) ≤ over start_ARG μ end_ARG1 ≻ 0 for any t. Assume otherwise that there exists some t0subscript0t_0t0 where σ(dt0)>μ^1≻0subscriptsubscript0subscript^succeeds10σ(d_t_0)> μ_1 0σ ( ditalic_t start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ) > over start_ARG μ end_ARG1 ≻ 0. By continuity of σ(dt)subscriptσ(d_t)σ ( ditalic_t ) and since σ(d0)≤μ^1≻0subscript0subscript^succeeds10σ(d_0)≤ μ_1 0σ ( d0 ) ≤ over start_ARG μ end_ARG1 ≻ 0, there exists t1≤t0subscript1subscript0t_1≤ t_0t1 ≤ t0 such that σ(dt1)=μ^1≻0subscriptsubscript1subscript^succeeds10σ(d_t_1)= μ_1 0σ ( ditalic_t start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT ) = over start_ARG μ end_ARG1 ≻ 0. However, this results in Vt1=0<Vt0subscriptsubscript10subscriptsubscript0V_t_1=0<V_t_0Vitalic_t start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT = 0 < Vitalic_t start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT, which contradicts the fact that VtsubscriptV_tVitalic_t is non-increasing. Therefore, we have σ(dt)≤μ^1≻0subscriptsubscript^succeeds10σ(d_t)≤ μ_1 0σ ( ditalic_t ) ≤ over start_ARG μ end_ARG1 ≻ 0. Moreover, since VtsubscriptV_tVitalic_t is non-increasing, we also have σ(d0)≤σ(dt)subscript0subscriptσ(d_0)≤σ(d_t)σ ( d0 ) ≤ σ ( ditalic_t ). Thus, we have σ(d0)≤σ(dt)≤μ^1≻0subscript0subscriptsubscript^succeeds10σ(d_0)≤σ(d_t)≤ μ_1 0σ ( d0 ) ≤ σ ( ditalic_t ) ≤ over start_ARG μ end_ARG1 ≻ 0 and we use this fact to find the following bound: σ(dt)σ(−dt)=σ(dt)(1−σ(dt))≥minσ(d0)(1−σ(d0)),μ^1≻0(1−μ^1≻0)=c.subscriptsubscriptsubscript1subscriptsubscript01subscript0subscript^succeeds101subscript^succeeds10 σ(d_t)σ(-d_t)=σ(d_t)(1-σ(d_t))≥% \σ(d_0)(1-σ(d_0)), μ_1 0(1- μ_1% 0) \=c.σ ( ditalic_t ) σ ( - ditalic_t ) = σ ( ditalic_t ) ( 1 - σ ( ditalic_t ) ) ≥ min σ ( d0 ) ( 1 - σ ( d0 ) ) , over start_ARG μ end_ARG1 ≻ 0 ( 1 - over start_ARG μ end_ARG1 ≻ 0 ) = c . (49) Applying bound (49) to (48) and integrating over t on both sides, we have Vtsubscript V_tVitalic_t ≤exp(−cαμ^0,1t)V0≤exp(−cαμ^0,1t).absentsubscript^01subscript0subscript^01 ≤ (-cα μ_0,1t)V_0≤ (-cα μ% _0,1t).≤ exp ( - c α over start_ARG μ end_ARG0 , 1 t ) V0 ≤ exp ( - c α over start_ARG μ end_ARG0 , 1 t ) . (50) We now consider the case where for some t0subscript0t_0t0, we have Vt0<2(κ+1)(1−exp(−ϵ))subscriptsubscript0211italic-ϵV_t_0<2(κ+1)(1- (-ε))Vitalic_t start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT < 2 ( κ + 1 ) ( 1 - exp ( - ϵ ) ). We show that in this case we must have VT≤2(κ+1)(1−exp(−ϵ))subscript211italic-ϵV_T≤ 2(κ+1)(1- (-ε))Vitalic_T ≤ 2 ( κ + 1 ) ( 1 - exp ( - ϵ ) ). Assume otherwise that VT>2(κ+1)(1−exp(−ϵ)V_T>2(κ+1)(1- (-ε)Vitalic_T > 2 ( κ + 1 ) ( 1 - exp ( - ϵ ), then by continuity there must be t1subscript1t_1t1 such that Vt1=2(κ+1)(1−exp(−ϵ))subscriptsubscript1211italic-ϵV_t_1=2(κ+1)(1- (-ε))Vitalic_t start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT = 2 ( κ + 1 ) ( 1 - exp ( - ϵ ) ). However, inequality (48) implies that for any t∈[t1,T]subscript1t∈[t_1,T]t ∈ [ t1 , T ], VtsubscriptV_tVitalic_t is non-increasing, leading to VT≤2(κ+1)(1−exp(−ϵ))subscript211italic-ϵV_T≤ 2(κ+1)(1- (-ε))Vitalic_T ≤ 2 ( κ + 1 ) ( 1 - exp ( - ϵ ) ), which contradicts our assumption. Therefore, we know that VT≤2(κ+1)(1−exp(−ϵ))subscript211italic-ϵ V_T≤ 2(κ+1)(1- (-ε))Vitalic_T ≤ 2 ( κ + 1 ) ( 1 - exp ( - ϵ ) ) (51) Combining the two bounds (50) and (51) on VTsubscriptV_TVitalic_T, we have: VT≤max2(κ+1)(1−exp(−ϵ)),exp(−cαμ^0,1T).subscript211italic-ϵsubscript^01 V_T≤ \2(κ+1)(1- (-ε)), (-cα% μ_0,1T) \.Vitalic_T ≤ max 2 ( κ + 1 ) ( 1 - exp ( - ϵ ) ) , exp ( - c α over start_ARG μ end_ARG0 , 1 T ) . (52) Subsequently, we show that 2(κ+1)(1−exp(−ϵ))≤exp(−cαμ^0,1T)211italic-ϵsubscript^012(κ+1)(1- (-ε))≤ (-cα μ_0,1T)2 ( κ + 1 ) ( 1 - exp ( - ϵ ) ) ≤ exp ( - c α over start_ARG μ end_ARG0 , 1 T ) provided that ϵ0≤exp(−1/4)2(κ+1)subscriptitalic-ϵ01421 _0≤ (-1/4)2(κ+1)ϵ0 ≤ divide start_ARG exp ( - 1 / 4 ) end_ARG start_ARG 2 ( κ + 1 ) end_ARG. Utilizing the inequality exp(−x)≥1−x1 (-x)≥ 1-xexp ( - x ) ≥ 1 - x, we have exp(−ϵ0)≥1−ϵ0≥1−exp(−1/4)2(κ+1)exp(−1/4)subscriptitalic-ϵ01subscriptitalic-ϵ01142114 (- _0)≥ 1- _0≥ 1- (-1/4)2(% κ+1) (-1/4)exp ( - ϵ0 ) ≥ 1 - ϵ0 ≥ 1 - divide start_ARG exp ( - 1 / 4 ) end_ARG start_ARG 2 ( κ + 1 ) end_ARG exp ( - 1 / 4 ) Using the fact that c≤1/4,κ≥0formulae-sequence140c≤ 1/4,κ≥ 0c ≤ 1 / 4 , κ ≥ 0 and that μ^0,1αT≥1subscript^011 μ_0,1α T≥ 1over start_ARG μ end_ARG0 , 1 α T ≥ 1, we conclude 2(κ+1)21 2(κ+1)2 ( κ + 1 ) ≤2(κ+1)exp(−ϵ0)+exp(−1/4)absent21subscriptitalic-ϵ014 ≤ 2(κ+1) (- _0)+ (-1/4)≤ 2 ( κ + 1 ) exp ( - ϵ0 ) + exp ( - 1 / 4 ) ≤2(κ+1)exp(−ϵ0)+exp(−c)absent21subscriptitalic-ϵ0 ≤ 2(κ+1) (- _0)+ (-c)≤ 2 ( κ + 1 ) exp ( - ϵ0 ) + exp ( - c ) ≤2(κ+1)exp(−ϵ0αμ^0,1T)+exp(−cαμ^0,1T)absent21subscriptitalic-ϵ0subscript^01subscript^01 ≤ 2(κ+1) (- _0α μ_0,1T)+ (-c% α μ_0,1T)≤ 2 ( κ + 1 ) exp ( - ϵ0 α over start_ARG μ end_ARG0 , 1 T ) + exp ( - c α over start_ARG μ end_ARG0 , 1 T ) ≤2(κ+1)exp(−ϵ0αμ^0,1T)+exp(−cαμ^0,1T).absent21subscriptitalic-ϵ0subscript^01subscript^01 ≤ 2(κ+1) (- _0α μ_0,1T)+ (-c% α μ_0,1T).≤ 2 ( κ + 1 ) exp ( - ϵ0 α over start_ARG μ end_ARG0 , 1 T ) + exp ( - c α over start_ARG μ end_ARG0 , 1 T ) . This leads to the bound VT≤exp(−cαμ^0,1T)subscriptsubscript^01V_T≤ (-cα μ_0,1T)Vitalic_T ≤ exp ( - c α over start_ARG μ end_ARG0 , 1 T ). Proof of Lemma 4. The result can be shown using Grönwall’s inequality. Rewriting the inequality as l˙t+γlt≥−γκsubscript˙subscript l_t+γ l_t≥-γ ˙ start_ARG l end_ARGt + γ litalic_t ≥ - γ κ and multiplying both sides by exp(γt) (γ t)exp ( γ t ) gives: exp(γt)l˙t+γexp(γt)lt≥−γκexp(γt).subscript˙subscript (γ t) l_t+γ (γ t)l_t≥-γ% κ (γ t).exp ( γ t ) over˙ start_ARG l end_ARGt + γ exp ( γ t ) litalic_t ≥ - γ κ exp ( γ t ) . Notice that the left-hand side is the derivative of ltexp(γt)subscriptl_t (γ t)litalic_t exp ( γ t ). This yields ∂t(ltexp(γt))≥−γκexp(γt)⇒∫0t∂t(ltexp(γt))≥∫0t−γκexp(γt).subscript⇒superscriptsubscript0subscriptsuperscriptsubscript0 ∂ t(l_t (γ t))≥-γκ% (γ t) _0^t ∂ t(l_t (% γ t))≥ _0^t-γκ (γ t).divide start_ARG ∂ end_ARG start_ARG ∂ t end_ARG ( litalic_t exp ( γ t ) ) ≥ - γ κ exp ( γ t ) ⇒ ∫0t divide start_ARG ∂ end_ARG start_ARG ∂ t end_ARG ( litalic_t exp ( γ t ) ) ≥ ∫0t - γ κ exp ( γ t ) . (53) The left-hand side and using the fact that l0=1subscript01l_0=1l0 = 1 is given by ∫0t∂t(ltexp(γt))=ltexp(γt)−l0exp(γ⋅0)=ltexp(γt)−1.superscriptsubscript0subscriptsubscriptsubscript0⋅0subscript1 _0^t ∂ t(l_t (γ t))=l_t% (γ t)-l_0 (γ· 0)=l_t (γ t)-1.∫0t divide start_ARG ∂ end_ARG start_ARG ∂ t end_ARG ( litalic_t exp ( γ t ) ) = litalic_t exp ( γ t ) - l0 exp ( γ ⋅ 0 ) = litalic_t exp ( γ t ) - 1 . The right-hand side is ∫0t−γκexp(γt)=−κ(exp(γt)−1).superscriptsubscript01 _0^t-γκ (γ t)=-κ( (γ t)-1).∫0t - γ κ exp ( γ t ) = - κ ( exp ( γ t ) - 1 ) . (54) Substituting equations (54) and (54) in (53), we have ltexp(γt)−1≥−κ(exp(γt)−1)⇒ltexp(γt)≥−κexp(γt)+κ+1.subscript11⇒subscript1 l_t (γ t)-1≥-κ( (γ t)-1) l_% t (γ t)≥-κ (γ t)+κ+1.litalic_t exp ( γ t ) - 1 ≥ - κ ( exp ( γ t ) - 1 ) ⇒ litalic_t exp ( γ t ) ≥ - κ exp ( γ t ) + κ + 1 . (55) Multiplying both sides with exp(−γt) (-γ t)exp ( - γ t ) gives the final lower bound on ltsubscriptl_tlitalic_t. ∎ Appendix G POWER-DL Pseudocode Algorithm 1 POWER with Dynamic Labels Inputs: Dataset =(xi,yi+,yi−)i=1nsuperscriptsubscriptsubscriptsubscriptsuperscriptsubscriptsuperscript1D=\(x_i,y^+_i,y^-_i)\_i=1^nD = ( xitalic_i , y+i , y-i ) i = 1n, hyperparameters η,β,γη,β,γη , β , γ, learning rate α, weight function w(y)w(y)w ( y ) (e.g., w(y)=1/|y|1w(y)=1/|y|w ( y ) = 1 / | y |), initial model πθ0subscriptsubscript0 _ _0πitalic_θ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT Initialize li0=0subscriptsuperscript00l^0_i=0l0italic_i = 0 for all (xi,yi+,yi−)subscriptsubscriptsuperscriptsubscriptsuperscript(x_i,y^+_i,y^-_i)( xitalic_i , y+i , y-i ). for t∈1,…,T1…t∈\1,…,T\t ∈ 1 , … , T do Objective function: L−(lt;θ)subscriptsuperscript L_ POWER-DL(l^t;θ)Lsansserif_POWER - sansserif_DL ( litalic_t ; θ ) =[litL(x,yi+,yi−)+(1−lit)L(x,yi−,yi+)]absentsubscriptdelimited-[]subscriptsuperscriptsubscriptsubscriptsuperscriptsubscriptsuperscript1subscriptsuperscriptsubscriptsubscriptsuperscriptsubscriptsuperscript =E_D [l^t_iL_ POWER(x,y^+% _i,y^-_i)+(1-l^t_i)L_ POWER(x,y^-_i,y^+_i) ]= caligraphic_D [ litalic_titalic_i Lsansserif_POWER ( x , y+i , y-i ) + ( 1 - litalic_titalic_i ) Lsansserif_POWER ( x , y-i , y+i ) ] L(x,y+,y−)subscriptsuperscriptsuperscript L_ POWER(x,y^+,y^-)Lsansserif_POWER ( x , y+ , y- ) ≔log(σ(β[w(y+)logπθ(y+|x)−w(y−)logπθ(y−|x)+w(y+)−w(y−)]) (σ (β [w(y^+) _% θ(y^+|x)-w(y^-) _θ(y^-|x)+w(y^+)-w(y^-) ]% )≔ log ( σ ( β [ w ( y+ ) log πitalic_θ ( y+ | x ) - w ( y- ) log πitalic_θ ( y- | x ) + w ( y+ ) - w ( y- ) ] ) +ηβw(y+)logπ(y+|x)superscriptconditionalsuperscript +ηβ w(y^+) π(y^+|x)+ η β w ( y+ ) log π ( y+ | x ) Update policy with stop gradient on labels: θt←θt−1+α∇(lt)L−(lt;θ)←subscriptsubscript1subscript∇superscriptsubscriptsuperscript _t← _t-1+α _ sg(l^t)L_ % POWER-DL(l^t;θ)θitalic_t ← θitalic_t - 1 + α ∇sansserif_sg ( litalic_t ) Lsansserif_POWER - sansserif_DL ( litalic_t ; θ ) Update dynamic labels: lit=(1−γ)lit−1+γσ(w(yi+)logπθ(y+|x)−w(yi−)logπθ(y−|x)+w(yi+)−w(yi−))−μ^yi−≻yi+μ^yi+≻yi−μ^yi−≻yi+subscriptsuperscript1subscriptsuperscript1subscriptsuperscriptsubscriptconditionalsuperscriptsubscriptsuperscriptsubscriptconditionalsuperscriptsubscriptsuperscriptsubscriptsuperscriptsubscript^succeedssubscriptsuperscriptsubscriptsuperscriptsubscript^succeedssubscriptsuperscriptsubscriptsuperscriptsubscript^succeedssubscriptsuperscriptsubscriptsuperscript l^t_i=(1-γ)l^t-1_i+γ σ (w(y^+% _i) _θ(y^+|x)-w(y^-_i) _θ(y^-|x)+w(y^+_% i)-w(y^-_i) )- μ_y^-_i y^+_i μ_y^+% _i y^-_i- μ_y^-_i y^+_ilitalic_titalic_i = ( 1 - γ ) litalic_t - 1i + γ divide start_ARG σ ( w ( y+i ) log πitalic_θ ( y+ | x ) - w ( y-i ) log πitalic_θ ( y- | x ) + w ( y+i ) - w ( y-i ) ) - over start_ARG μ end_ARGy- start_POSTSUBSCRIPT i ≻ y+i end_POSTSUBSCRIPT end_ARG start_ARG over start_ARG μ end_ARGy+ start_POSTSUBSCRIPT i ≻ y-i end_POSTSUBSCRIPT - over start_ARG μ end_ARGy- start_POSTSUBSCRIPT i ≻ y+i end_POSTSUBSCRIPT end_ARG Return: πθTsubscriptsubscript _ _Tπitalic_θ start_POSTSUBSCRIPT T end_POSTSUBSCRIPT Appendix H Experimental Details H.1 Preference Optimization Objectives We compare POWER with a variety of preference optimization methods as baselines, summarized in Table 3. Several of these methods include divergence minimization (typically KL divergence) against the reference models, such as DPO Rafailov et al. (2024b), IPO Azar et al. (2024), which considers a different loss function on preferences, and χPO Huang et al. (2024), which combines Chi-Squared with KL divergence for stronger pessimism. We also implement an offline variant of SPPO Wu et al. (2024b), derived based on a self-play mechanism to improve upon the initial model. We additionally consider several objectives that do not include the reference model. These include CPO Xu et al. (2024a), which considers DPO with a uniform initial model; SLiC-HF Zhao et al. (2023), which uses a hinge loss; RRHF Yuan et al. (2024a), which applies a hinge loss with length-normalization on the contrastive term; and ORPO Hong et al. (2024), which proposes odd ratio terms without initial models to contrast the chosen and rejected responses. Finally, SimPO Meng et al. (2024) that removes the reference model from DPO and adds length-normalization and a margin. We implement POWER-DL by combining the objective in (8) with dynamic labels (11), where we estimate the empirical preferences with μ^1≻0=lsubscript^succeeds10 μ_1 0=lover start_ARG μ end_ARG1 ≻ 0 = l. We also compare against POWER, which corresponds to γ=00γ=0γ = 0, removing the dynamic labels. Table 3: Different preference optimization objectives and hyperparameter search range. Method Objective (Min) Hyperparameter Range DPO −logσ(βlogπθ(y+|x)πθ0(y+|x)−βlogπθ(y−|x)πθ0(y−|x))subscriptconditionalsuperscriptsubscriptsubscript0conditionalsuperscriptsubscriptconditionalsuperscriptsubscriptsubscript0conditionalsuperscript- σ (β _θ(y^+|x) _ _0(y^+|% x)-β _θ(y^-|x) _ _0(y^-|x) )- log σ ( β log divide start_ARG πitalic_θ ( y+ | x ) end_ARG start_ARG πitalic_θ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ( y+ | x ) end_ARG - β log divide start_ARG πitalic_θ ( y- | x ) end_ARG start_ARG πitalic_θ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ( y- | x ) end_ARG ) β∈0.001,0.005,0.01,0.05,0.10.0010.0050.010.050.1β∈\0.001,0.005,0.01,0.05,0.1\β ∈ 0.001 , 0.005 , 0.01 , 0.05 , 0.1 DPO+SFT −logσ(βlogπθ(y+|x)πθ0(y+|x)−βlogπθ(y−|x)πθ0(y−|x))−λlogπθ(y+|x)subscriptconditionalsuperscriptsubscriptsubscript0conditionalsuperscriptsubscriptconditionalsuperscriptsubscriptsubscript0conditionalsuperscriptsubscriptconditionalsuperscript- σ (β _θ(y^+|x) _ _0(y^+|% x)-β _θ(y^-|x) _ _0(y^-|x) )-% λ _θ(y^+|x)- log σ ( β log divide start_ARG πitalic_θ ( y+ | x ) end_ARG start_ARG πitalic_θ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ( y+ | x ) end_ARG - β log divide start_ARG πitalic_θ ( y- | x ) end_ARG start_ARG πitalic_θ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ( y- | x ) end_ARG ) - λ log πitalic_θ ( y+ | x ) β∈0.001,0.005,0.01,0.05,0.10.0010.0050.010.050.1β∈\0.001,0.005,0.01,0.05,0.1\β ∈ 0.001 , 0.005 , 0.01 , 0.05 , 0.1 λ∈0.0005,0.001,0.01,0.1,10.00050.0010.010.11λ∈\0.0005,0.001,0.01,0.1,1\λ ∈ 0.0005 , 0.001 , 0.01 , 0.1 , 1 cDPO −(1−c)logσ(βlogπθ(y+|x)πθ0(y+|x)−βlogπθ(y−|x)πθ0(y−|x))1subscriptconditionalsuperscriptsubscriptsubscript0conditionalsuperscriptsubscriptconditionalsuperscriptsubscriptsubscript0conditionalsuperscript-(1-c) σ (β _θ(y^+|x) _ _0(y% ^+|x)-β _θ(y^-|x) _ _0(y^-|x) )- ( 1 - c ) log σ ( β log divide start_ARG πitalic_θ ( y+ | x ) end_ARG start_ARG πitalic_θ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ( y+ | x ) end_ARG - β log divide start_ARG πitalic_θ ( y- | x ) end_ARG start_ARG πitalic_θ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ( y- | x ) end_ARG ) β∈0.001,0.005,0.01,0.05,0.10.0010.0050.010.050.1β∈\0.001,0.005,0.01,0.05,0.1\β ∈ 0.001 , 0.005 , 0.01 , 0.05 , 0.1 −clogσ(βlogπθ(y−|x)πθ0(y−|x)−βlogπθ(y+|x)πθ0(y+|x))subscriptconditionalsuperscriptsubscriptsubscript0conditionalsuperscriptsubscriptconditionalsuperscriptsubscriptsubscript0conditionalsuperscript -c σ (β _θ(y^-|x) _ _0(% y^-|x)-β _θ(y^+|x) _ _0(y^+|x) )- c log σ ( β log divide start_ARG πitalic_θ ( y- | x ) end_ARG start_ARG πitalic_θ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ( y- | x ) end_ARG - β log divide start_ARG πitalic_θ ( y+ | x ) end_ARG start_ARG πitalic_θ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ( y+ | x ) end_ARG ) c∈0.05,0.1,0.15,0.2,0.30.050.10.150.20.3c∈\0.05,0.1,0.15,0.2,0.3\c ∈ 0.05 , 0.1 , 0.15 , 0.2 , 0.3 R-DPO −logσ(βlogπθ(y+|x)πθ0(y+|x)−βlogπθ(y−|x)πθ0(y−|x)−(α|y+|−α|y−|))subscriptconditionalsuperscriptsubscriptsubscript0conditionalsuperscriptsubscriptconditionalsuperscriptsubscriptsubscript0conditionalsuperscriptsuperscriptsuperscript- σ (β _θ(y^+|x) _ _0(y^+|% x)-β _θ(y^-|x) _ _0(y^-|x)-(α|y% ^+|-α|y^-|) )- log σ ( β log divide start_ARG πitalic_θ ( y+ | x ) end_ARG start_ARG πitalic_θ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ( y+ | x ) end_ARG - β log divide start_ARG πitalic_θ ( y- | x ) end_ARG start_ARG πitalic_θ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ( y- | x ) end_ARG - ( α | y+ | - α | y- | ) ) α∈0.005,0.01,0.05,0.1,0.50.0050.010.050.10.5α∈\0.005,0.01,0.05,0.1,0.5\α ∈ 0.005 , 0.01 , 0.05 , 0.1 , 0.5 β∈0.001,0.005,0.01,0.05,0.10.0010.0050.010.050.1β∈\0.001,0.005,0.01,0.05,0.1\β ∈ 0.001 , 0.005 , 0.01 , 0.05 , 0.1 IPO (logπθ(y+|x)πθ0(y+|x)−logπθ(y−|x)πθ0(y−|x)−12τ)2superscriptsubscriptconditionalsuperscriptsubscriptsubscript0conditionalsuperscriptsubscriptconditionalsuperscriptsubscriptsubscript0conditionalsuperscript122 ( _θ(y^+|x) _ _0(y^+|x)- % _θ(y^-|x) _ _0(y^-|x)- 12τ )^2( log divide start_ARG πitalic_θ ( y+ | x ) end_ARG start_ARG πitalic_θ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ( y+ | x ) end_ARG - log divide start_ARG πitalic_θ ( y- | x ) end_ARG start_ARG πitalic_θ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ( y- | x ) end_ARG - divide start_ARG 1 end_ARG start_ARG 2 τ end_ARG )2 τ∈0.001,0.005,0.01,0.1,1.00.0010.0050.010.11.0τ∈\0.001,0.005,0.01,0.1,1.0\τ ∈ 0.001 , 0.005 , 0.01 , 0.1 , 1.0 χPO −log(σ(2R[βϕ(πθ(y+|x)πθ0(y+|x))−βϕ(πθ(y−|x)πθ0(y−|x))]))subscript2delimited-[]italic-ϕsubscriptconditionalsuperscriptsubscriptsubscript0conditionalsuperscriptitalic-ϕsubscriptconditionalsuperscriptsubscriptsubscript0conditionalsuperscript- (σ ( clip_2R [βφ ( _% θ(y^+|x) _ _0(y^+|x) )-βφ ( _% θ(y^-|x) _ _0(y^-|x) ) ] ) )- log ( σ ( sansserif_clip2 R [ β ϕ ( divide start_ARG πitalic_θ ( y+ | x ) end_ARG start_ARG πitalic_θ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ( y+ | x ) end_ARG ) - β ϕ ( divide start_ARG πitalic_θ ( y- | x ) end_ARG start_ARG πitalic_θ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ( y- | x ) end_ARG ) ] ) ) β∈0.001,0.01,0.10.0010.010.1β∈\0.001,0.01,0.1\β ∈ 0.001 , 0.01 , 0.1 ϕ(z)≔z+log(z)≔italic-ϕφ(z) z+ (z)ϕ ( z ) ≔ z + log ( z ) R∈0.1,0.5,1,5,100.10.51510R∈\0.1,0.5,1,5,10\R ∈ 0.1 , 0.5 , 1 , 5 , 10 SPPO (offline) (βlogπθ(y+|x)πθ0(y+|x)−12)2+(βlogπθ(y−|x)πθ0(y−|x)+12)2superscriptsubscriptconditionalsuperscriptsubscriptsubscript0conditionalsuperscript122superscriptsubscriptconditionalsuperscriptsubscriptsubscript0conditionalsuperscript122\; (β _θ(y^+|x) _ _0(y^+|x)- % 12 )^2+ (β _θ(y^-|x) _ _0% (y^-|x)+ 12 )^2( β log divide start_ARG πitalic_θ ( y+ | x ) end_ARG start_ARG πitalic_θ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ( y+ | x ) end_ARG - divide start_ARG 1 end_ARG start_ARG 2 end_ARG )2 + ( β log divide start_ARG πitalic_θ ( y- | x ) end_ARG start_ARG πitalic_θ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ( y- | x ) end_ARG + divide start_ARG 1 end_ARG start_ARG 2 end_ARG )2 β∈0.1,1,10,100,1000,100000.1110100100010000β∈\0.1,1,10,100,1000,10000\β ∈ 0.1 , 1 , 10 , 100 , 1000 , 10000 CPO −logσ(βlogπθ(y+|x)−βlogπθ(y−|x))−λlogπθ(y+|x)subscriptconditionalsuperscriptsubscriptconditionalsuperscriptsubscriptconditionalsuperscript- σ (β _θ(y^+|x)-β _θ(y^-|x)% )-λ _θ(y^+|x)- log σ ( β log πitalic_θ ( y+ | x ) - β log πitalic_θ ( y- | x ) ) - λ log πitalic_θ ( y+ | x ) λ=1.01.0λ=1.0λ = 1.0 β∈0.001,0.01,0.1,1,100.0010.010.1110β∈\0.001,0.01,0.1,1,10\β ∈ 0.001 , 0.01 , 0.1 , 1 , 10 RRHF max0,−1|y+|logπθ(y+|x)+1|y−|logπθ(y−|x)01superscriptsubscriptconditionalsuperscript1superscriptsubscriptconditionalsuperscript \0,- 1|y^+| _θ(y^+|x)+ 1|y^-| % _θ(y^-|x) \max 0 , - divide start_ARG 1 end_ARG start_ARG | y+ | end_ARG log πitalic_θ ( y+ | x ) + divide start_ARG 1 end_ARG start_ARG | y- | end_ARG log πitalic_θ ( y- | x ) −λlogπθ(y+|x)subscriptconditionalsuperscript-λ _θ(y^+|x)- λ log πitalic_θ ( y+ | x ) λ∈0.01,0.05,0.1,0.5,1,100.010.050.10.5110λ∈\0.01,0.05,0.1,0.5,1,10\λ ∈ 0.01 , 0.05 , 0.1 , 0.5 , 1 , 10 SLiC-HF max(0,β−logπθ(y+|x)+logπθ(y−|x))−λlogπθ(y+|x)0subscriptconditionalsuperscriptsubscriptconditionalsuperscriptsubscriptconditionalsuperscript (0,β- _θ(y^+|x)+ _θ(y^-|x) )-% λ _θ(y^+|x)max ( 0 , β - log πitalic_θ ( y+ | x ) + log πitalic_θ ( y- | x ) ) - λ log πitalic_θ ( y+ | x ) λ∈0.01,0.05,0.1,0.5,1,100.010.050.10.5110λ∈\0.01,0.05,0.1,0.5,1,10\λ ∈ 0.01 , 0.05 , 0.1 , 0.5 , 1 , 10 β∈0.1,0.5,1.0,2.00.10.51.02.0β∈\0.1,0.5,1.0,2.0\β ∈ 0.1 , 0.5 , 1.0 , 2.0 ORPO −logpθ(y+|x)−λlogσ(logpθ(y−|x)1−pθ(y−|x)−logpθ(y+|x)1−pθ(y+|x))subscriptconditionalsuperscriptsubscriptconditionalsuperscript1subscriptconditionalsuperscriptsubscriptconditionalsuperscript1subscriptconditionalsuperscript- p_θ(y^+|x)-λ σ ( p_θ(y^-|x)% 1-p_θ(y^-|x)- p_θ(y^+|x)1-p_θ(y^+|x) )- log pitalic_θ ( y+ | x ) - λ log σ ( log divide start_ARG pitalic_θ ( y- | x ) end_ARG start_ARG 1 - pitalic_θ ( y- | x ) end_ARG - log divide start_ARG pitalic_θ ( y+ | x ) end_ARG start_ARG 1 - pitalic_θ ( y+ | x ) end_ARG ) λ∈0.1,0.5,1.0,2.00.10.51.02.0λ∈\0.1,0.5,1.0,2.0\λ ∈ 0.1 , 0.5 , 1.0 , 2.0 pθ(y|x)≔exp(1|y−|logπθ(y−|x))≔subscriptconditional1superscriptsubscriptconditionalsuperscript p_θ(y|x) ( 1|y^-| _θ(y^-% |x) )pitalic_θ ( y | x ) ≔ exp ( divide start_ARG 1 end_ARG start_ARG | y- | end_ARG log πitalic_θ ( y- | x ) ) SimPO −log(β|y−|logπθ(y+|x)−β|y−|logπθ(y−|x)−γ)superscriptsubscriptconditionalsuperscriptsuperscriptsubscriptconditionalsuperscript- ( β|y^-| _θ(y^+|x)- β|y^-|% _θ(y^-|x)-γ )- log ( divide start_ARG β end_ARG start_ARG | y- | end_ARG log πitalic_θ ( y+ | x ) - divide start_ARG β end_ARG start_ARG | y- | end_ARG log πitalic_θ ( y- | x ) - γ ) β∈1,2,10,20121020β∈\1,2,10,20\β ∈ 1 , 2 , 10 , 20 γ∈0.3,0.5,0.8,1.00.30.50.81.0γ∈\0.3,0.5,0.8,1.0\γ ∈ 0.3 , 0.5 , 0.8 , 1.0 ROPO −αlogσ(βlogπθ(y+|x)πθ0(y+|x)−βlogπθ(y−|x)πθ0(y−|x))subscriptconditionalsuperscriptsubscriptsubscript0conditionalsuperscriptsubscriptconditionalsuperscriptsubscriptsubscript0conditionalsuperscript-α σ (β _θ(y^+|x) _ _0(% y^+|x)-β _θ(y^-|x) _ _0(y^-|x) )- α log σ ( β log divide start_ARG πitalic_θ ( y+ | x ) end_ARG start_ARG πitalic_θ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ( y+ | x ) end_ARG - β log divide start_ARG πitalic_θ ( y- | x ) end_ARG start_ARG πitalic_θ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ( y- | x ) end_ARG ) β∈0.001,0.005,0.01,0.05,0.10.0010.0050.010.050.1β∈\0.001,0.005,0.01,0.05,0.1\β ∈ 0.001 , 0.005 , 0.01 , 0.05 , 0.1 +γσ(βlogπθ(y−|x)πθ0(y−|x)−βlogπθ(y+|x)πθ0(y+|x))subscriptconditionalsuperscriptsubscriptsubscript0conditionalsuperscriptsubscriptconditionalsuperscriptsubscriptsubscript0conditionalsuperscript +γσ (β _θ(y^-|x) _ _0% (y^-|x)-β _θ(y^+|x) _ _0(y^+|x) )+ γ σ ( β log divide start_ARG πitalic_θ ( y- | x ) end_ARG start_ARG πitalic_θ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ( y- | x ) end_ARG - β log divide start_ARG πitalic_θ ( y+ | x ) end_ARG start_ARG πitalic_θ start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ( y+ | x ) end_ARG ) γ=0.1,α∈0.2,2,20,200,2000formulae-sequence0.10.22202002000γ=0.1,α∈\0.2,2,20,200,2000\γ = 0.1 , α ∈ 0.2 , 2 , 20 , 200 , 2000 H.2 Instruction-Following Benchmarks Benchmark details. We select the default choices in benchmark as baselines. In particular, for AlpacaEval 2.0 benchmark, we use GPT-4-Preview-1106 as comparison baseline model, and GPT-4 as the judge model. AlpacaEval 2.0 then compares responses generated by our PO trained models with responses from the baseline model, and length-controlled (LC) and raw winrate (WR) are computed as metrics. For Arena Hard benchmark, we use the GPT-4-0314 as the baseline model, and GPT-4 as the judge model, and the reported metric is winrate againts the baseline model. Decoding hyperparameters. We follow Meng et al. (2024) and use a sampling decoding strategy with a temperature 0.9 on AlpacaEval 2.0 for all methods. For Arena Hard, we use the default approach of greedy generation. H.3 Training and Hyperparameter Details Hyperparameters for training reference models in the base setting. We train initial reference models in the base setups. In the Helpsteer2 setting Wang et al. (2024e), we train a model through supervised instruction finetuning on the (English only) OpenAssistant2 dataset Köpf et al. (2024). In the Zephyr setting Tunstall et al. (2023), we conduct supervised finetuning on the UltraChat-200K dataset Ding et al. (2023). We use the following hyperparameters for both cases: a train batch size of 256, learning rate of 2e-5 with a cosine learning rate schedule with 10% warmup, right padding, and a max sequence length of 2048. We train the models with Adam optimizer for 1 epoch. General hyperparameters for preference optimization. We use a fixed batch size of 128 and a maximum sequence length of 2048 for all methods. For learning rate, we search over 3e-7,5e-7 separately for each method and use a cosine learning rate schedule with a 10% warmup. We use Adam optimizer for all the approaches. In the Helpsteer2 setting and following Wang et al. (2024e), we train the models for up to 7 epochs, and in the Zephyr setting, we train the models for up to 3 epochs, and select the best number of epochs for each method according to validation. We use right padding for preference optimization following the recommendation of Hu et al. (2024). Specific hyperparameters for preference optimization. For the hyperparameters specific to each preference optimization objective, we conduct hyperparameter search according to the values in Table 3. In each setting, we select the best model according to the ranking performance on the validation set. For POWER-DL, we conduct hyperparameter search over β∈1,2,10,20121020β∈\1,2,10,20\β ∈ 1 , 2 , 10 , 20 , η∈0.0005,0.0010.00050.001η∈\0.0005,0.001\η ∈ 0.0005 , 0.001 , and γ∈0.1,0.30.10.3γ∈\0.1,0.3\γ ∈ 0.1 , 0.3 . For Helpsteer2, we select β=10,η=0.001,γ=0.1formulae-sequence10formulae-sequence0.0010.1β=10,η=0.001,γ=0.1β = 10 , η = 0.001 , γ = 0.1, 5 epochs, and learning rate of 5e-7 in the base setting and β=20,η=0.0005,γ=0.1formulae-sequence20formulae-sequence0.00050.1β=20,η=0.0005,γ=0.1β = 20 , η = 0.0005 , γ = 0.1, 4 epochs and learning rate of 3e-7 in the instruct setting. For Zephyr, we select β=1,η=0.001,γ=0.1formulae-sequence1formulae-sequence0.0010.1β=1,η=0.001,γ=0.1β = 1 , η = 0.001 , γ = 0.1, 2 epochs, and learning rate of 3e-7 in the base setting, and β=2,η=0.0005,γ=0.3formulae-sequence2formulae-sequence0.00050.3β=2,η=0.0005,γ=0.3β = 2 , η = 0.0005 , γ = 0.3, 2 epochs, and learning rate of 3e-7 in the instruct setting. Computation environment. All experiments are conducted on 8×A100 GPUs based on the OpenRLHF repository Hu et al. (2024). Appendix I Additional Experimental Results on the Llama Family I.1 Performance on Academic Benchmarks Tables 4 and 5 present the benchmark scores for the Helpsteer2 and Zephyr pipelines across a variety of downstream tasks. Considering the average benchmark score, POWER-DL consistently improves over the initial model and ranks within the top two or three methods across all four settings, despite significantly outperforming other methods in alignment benchmarks AlpacaEval 2.0 and Arena-Hard as detailed in Table 2. Achieving a high score on instruction-following benchmarks AlpacaEval 2.0 and Arena-Hard while maintaining a good performance on downstream tasks is considered key empirical evidence for mitigating reward hacking in practice Xu et al. (2024b). Table 4: Downstream task evaluation results of the models trained on the Helpsteer2 pipeline. The arrows show improvement or degradation of performance with respect to the initial model. The top three average scores are shown in bold. Task MMLU ARC HellaSwag TruthfulQA Winogrande GSM8K IFEval MMLU-PRO Average Helpsteer2 Llama3-8B-Base Initial Model 62.6 ↓ ↓0.0 58.4 ↓ ↓0.0 80.0 ↓ ↓0.0 49.5 ↓ ↓0.0 77.4 ↓ ↓0.0 37.1 ↓ ↓0.0 33.6 ↓ ↓0.0 29.8 ↓ ↓0.0 53.5 ↓ ↓0.0 DPO 61.2 ↓ ↓1.4 57.5 ↓ ↓0.9 81.1 ↑ ↑1.1 51.0 ↑ ↑1.5 75.6 ↓ ↓1.8 32.5 ↓ ↓4.6 41.1 ↑ ↑7.5 29.8 ↓ ↓0.0 53.7 ↑ ↑0.2 DPO+SFT 62.1 ↓ ↓0.5 60.5 ↑ ↑2.1 81.9 ↑ ↑1.9 52.9 ↑ ↑3.4 77.0 ↓ ↓0.4 42.1 ↑ ↑5.0 42.5 ↑ ↑8.9 30.5 ↑ ↑0.7 56.2 ↑ ↑2.7 cDPO 62.0 ↓ ↓0.6 60.9 ↑ ↑2.5 82.9 ↑ ↑2.9 53.9 ↑ ↑4.4 76.6 ↓ ↓0.8 37.9 ↑ ↑0.8 42.6 ↑ ↑9.0 30.5 ↑ ↑0.7 55.9 ↑ ↑2.4 R-DPO 62.8 ↑ ↑0.2 58.2 ↓ ↓0.2 80.3 ↑ ↑0.3 52.9 ↑ ↑3.4 77.4 ↓ ↓0.0 43.2 ↑ ↑6.1 38.7 ↑ ↑5.1 30.1 ↑ ↑0.3 55.5 ↑ ↑2.0 IPO 62.7 ↑ ↑0.1 60.6 ↑ ↑2.2 81.7 ↑ ↑1.7 52.5 ↑ ↑3.0 78.1 ↑ ↑0.7 43.8 ↑ ↑6.7 42.3 ↑ ↑8.7 30.3 ↑ ↑0.5 56.5 ↑ ↑3.0 χPO 62.9 ↑ ↑0.3 59.0 ↑ ↑0.6 81.0 ↑ ↑1.0 51.7 ↑ ↑2.2 78.1 ↑ ↑0.7 41.7 ↑ ↑4.6 38.0 ↑ ↑4.4 30.1 ↑ ↑0.3 55.3 ↑ ↑1.8 SPPO 62.8 ↑ ↑0.2 60.9 ↑ ↑2.5 83.0 ↑ ↑3.0 54.2 ↑ ↑4.7 77.1 ↓ ↓0.3 38.8 ↑ ↑1.7 42.7 ↑ ↑9.1 30.6 ↑ ↑0.8 56.3 ↑ ↑2.8 CPO 62.6 ↓ ↓0.0 59.0 ↑ ↑0.6 80.2 ↑ ↑0.2 54.2 ↑ ↑4.7 77.0 ↓ ↓0.4 44.2 ↑ ↑7.1 41.4 ↑ ↑7.8 30.4 ↑ ↑0.6 56.1 ↑ ↑2.6 RRHF 62.6 ↓ ↓0.0 57.7 ↓ ↓0.7 79.0 ↓ ↓1.0 51.1 ↑ ↑1.6 77.2 ↓ ↓0.2 37.2 ↑ ↑0.1 34.2 ↑ ↑0.6 29.8 ↓ ↓0.0 53.6 ↓ ↓0.1 SLiCHF 62.6 ↓ ↓0.0 59.0 ↑ ↑0.6 79.9 ↓ ↓0.1 55.0 ↑ ↑5.5 76.8 ↓ ↓0.6 43.7 ↑ ↑6.6 40.1 ↑ ↑6.5 30.2 ↑ ↑0.4 55.9 ↑ ↑2.4 ORPO 61.5 ↓ ↓1.1 57.6 ↓ ↓0.8 79.0 ↓ ↓1.0 61.4 ↑ ↑11.9 77.7 ↑ ↑0.3 15.6 ↓ ↓21.5 40.4 ↑ ↑6.8 29.6 ↓ ↓0.2 52.9 ↓ ↓0.7 SimPO 61.3 ↓ ↓1.3 59.0 ↑ ↑0.6 80.6 ↑ ↑0.6 59.6 ↑ ↑10.1 77.7 ↑ ↑0.3 23.4 ↓ ↓13.7 40.5 ↑ ↑6.9 30.2 ↑ ↑0.4 54.0 ↑ ↑0.5 ROPO 61.5 ↓ ↓1.1 60.9 ↑ ↑2.5 82.1 ↑ ↑2.1 52.5 ↑ ↑3.0 76.6 ↓ ↓0.8 37.5 ↑ ↑0.4 41.4 ↑ ↑7.8 30.1 ↑ ↑0.3 55.3 ↑ ↑1.8 POWER-DL 62.0 ↓ ↓0.6 59.6 ↑ ↑1.2 82.0 ↑ ↑2.0 61.0 ↑ ↑11.5 78.1 ↑ ↑0.7 36.5 ↓ ↓0.6 40.3 ↑ ↑6.7 30.5 ↑ ↑0.7 56.3 ↑ ↑2.8 POWER 61.9 ↓ ↓0.7 60 ↑ ↑1.6 82.0 ↑ ↑2.0 61.0 ↑ ↑11.5 77.9 ↑ ↑0.5 35.3 ↓ ↓1.8 40.1 ↑ ↑6.5 30.4 ↑ ↑0.6 56.1 ↑ ↑2.6 Helpsteer2 Llama3-8B-Instruct Initial Model 65.7 ↓ ↓0.0 62.0 ↓ ↓0.0 78.8 ↓ ↓0.0 51.7 ↓ ↓0.0 76.0 ↓ ↓0.0 75.3 ↓ ↓0.0 54.4 ↓ ↓0.0 36.0 ↓ ↓0.0 62.5 ↓ ↓0.0 DPO 65.9 ↑ ↑0.2 63.4 ↑ ↑1.4 79.5 ↑ ↑0.7 52.7 ↑ ↑1.0 75.9 ↓ ↓0.1 76.4 ↑ ↑1.1 54.1 ↓ ↓0.3 36.3 ↑ ↑0.3 63.0 ↑ ↑0.5 DPO+SFT 65.9 ↑ ↑0.2 61.0 ↓ ↓1.0 73.6 ↓ ↓5.2 54.7 ↑ ↑3.0 71.2 ↓ ↓4.8 74.7 ↓ ↓0.6 48.2 ↓ ↓6.2 37.0 ↑ ↑1.0 60.8 ↓ ↓1.7 cDPO 66.0 ↑ ↑0.3 65.4 ↑ ↑3.4 79.5 ↑ ↑0.7 57.0 ↑ ↑5.3 74.6 ↓ ↓1.4 76.9 ↑ ↑1.6 49.9 ↓ ↓4.5 36.9 ↑ ↑0.9 63.3 ↑ ↑0.8 R-DPO 65.8 ↑ ↑0.1 62.8 ↑ ↑0.8 74.8 ↓ ↓4.0 54.6 ↑ ↑2.9 72.6 ↓ ↓3.4 77.7 ↑ ↑2.4 51.0 ↓ ↓3.4 36.6 ↑ ↑0.6 62.0 ↓ ↓0.5 IPO 66.0 ↑ ↑0.3 64.9 ↑ ↑2.9 79.2 ↑ ↑0.4 58.4 ↑ ↑6.7 73.8 ↓ ↓2.2 75.8 ↑ ↑0.5 49.8 ↓ ↓4.6 37.1 ↑ ↑1.1 63.1 ↑ ↑0.6 χPO 65.8 ↑ ↑0.1 63.7 ↑ ↑1.7 75.6 ↓ ↓3.2 59.1 ↑ ↑7.4 72.4 ↓ ↓3.6 75.2 ↓ ↓0.1 52.4 ↓ ↓2.0 37.2 ↑ ↑1.2 62.7 ↑ ↑0.2 SPPO 66.0 ↑ ↑0.3 63.0 ↑ ↑1.0 76.7 ↓ ↓2.1 55.7 ↑ ↑4.0 72.9 ↓ ↓3.1 75.7 ↑ ↑0.4 49.6 ↓ ↓4.8 37.1 ↑ ↑1.1 62.1 ↓ ↓0.4 CPO 65.7 ↓ ↓0.0 62.2 ↑ ↑0.2 78.0 ↓ ↓0.8 52.2 ↑ ↑0.5 74.1 ↓ ↓1.9 75.7 ↑ ↑0.4 49.9 ↓ ↓4.5 36.0 ↓ ↓0.0 61.9 ↓ ↓0.6 RRHF 65.9 ↑ ↑0.2 62.0 ↓ ↓0.0 77.5 ↓ ↓1.3 51.5 ↓ ↓0.2 73.8 ↓ ↓2.2 76.7 ↑ ↑1.4 51.2 ↓ ↓3.2 36.4 ↑ ↑0.4 61.9 ↓ ↓0.6 SLiCHF 65.7 ↓ ↓0.0 62.9 ↑ ↑0.9 78.0 ↓ ↓0.8 53.9 ↑ ↑2.2 74.2 ↓ ↓1.8 76.7 ↑ ↑1.4 47.6 ↓ ↓6.8 36.1 ↑ ↑0.1 62.3 ↓ ↓0.2 ORPO 65.7 ↓ ↓0.0 62.1 ↑ ↑0.1 74.0 ↓ ↓4.8 56.7 ↑ ↑5.0 71.5 ↓ ↓4.5 75.9 ↑ ↑0.6 51.3 ↓ ↓3.1 37.2 ↑ ↑1.2 61.9 ↓ ↓0.6 SimPO 65.8 ↑ ↑0.1 61.9 ↓ ↓0.1 75.0 ↓ ↓3.8 58.3 ↑ ↑6.6 72.3 ↓ ↓3.7 74.3 ↓ ↓1.0 54.1 ↓ ↓0.3 37.2 ↑ ↑1.2 62.3 ↓ ↓0.2 ROPO 65.4 ↓ ↓0.3 62.0 ↓ ↓0.0 76.9 ↓ ↓1.9 54.1 ↑ ↑2.4 73.7 ↓ ↓2.3 73.9 ↓ ↓1.4 50.2 ↓ ↓4.2 36.2 ↑ ↑0.2 61.8 ↓ ↓0.7 POWER-DL 66.0 ↑ ↑0.3 64.3 ↑ ↑2.3 79.5 ↑ ↑0.7 53.1 ↑ ↑1.4 76.0 ↓ ↓0.0 76.3 ↑ ↑1.0 53.5 ↓ ↓0.9 36.6 ↑ ↑0.6 63.2 ↑ ↑0.7 POWER 65.8 ↑ ↑0.1 63.9 ↑ ↑1.9 79.6 ↑ ↑0.8 53.1 ↑ ↑1.4 76.1 ↑ ↑0.1 76.6 ↑ ↑1.3 52.5 ↓ ↓1.9 36.4 ↑ ↑0.4 63.0 ↑ ↑0.5 Table 5: Downstream task evaluation results of the model trained on the Zephyr pipeline. The arrows show improvement or degradation of performance with respect to the initial model. The top three average scores are shown in bold. Task MMLU ARC HellaSwag TruthfulQA Winogrande GSM8K IFEval MMLU-PRO Average Zephyr Llama3-8B-Base Initial Model 61.6 ↓ ↓0.0 58.2 ↓ ↓0.0 78.6 ↓ ↓0.0 52.1 ↓ ↓0.0 75.9 ↓ ↓0.0 47.3 ↓ ↓0.0 38.1 ↓ ↓0.0 29.4 ↓ ↓0.0 55.2 ↓ ↓0.0 DPO 61.9 ↑ ↑0.3 62.4 ↑ ↑4.2 81.6 ↑ ↑3.0 63.0 ↑ ↑11.0 74.4 ↓ ↓1.4 52.8 ↑ ↑5.5 50.6 ↑ ↑12.5 31.2 ↑ ↑1.7 59.7 ↑ ↑4.6 DPO+SFT 62.0 ↑ ↑0.3 62.0 ↑ ↑3.8 80.9 ↑ ↑2.3 61.6 ↑ ↑9.5 75.1 ↓ ↓0.7 55.0 ↑ ↑7.7 48.7 ↑ ↑10.6 31.0 ↑ ↑1.6 59.5 ↑ ↑4.4 cDPO 61.9 ↑ ↑0.3 61.9 ↑ ↑3.7 81.4 ↑ ↑2.7 62.3 ↑ ↑10.2 75.3 ↓ ↓0.5 54.7 ↑ ↑7.4 49.3 ↑ ↑11.2 31.0 ↑ ↑1.6 59.7 ↑ ↑4.6 R-DPO 61.9 ↑ ↑0.2 59.5 ↑ ↑1.3 80.1 ↑ ↑1.5 62.0 ↑ ↑9.9 75.0 ↓ ↓0.9 52.8 ↑ ↑5.5 45.3 ↑ ↑7.2 30.5 ↑ ↑1.1 58.4 ↑ ↑3.2 IPO 61.8 ↑ ↑0.1 58.3 ↑ ↑0.1 78.7 ↑ ↑0.1 53.5 ↑ ↑1.4 76.7 ↑ ↑0.9 46.8 ↓ ↓0.5 39.5 ↑ ↑1.3 29.7 ↑ ↑0.3 55.6 ↓ ↓0.0 χPO 62.1 ↑ ↑0.4 60.8 ↑ ↑2.6 80.2 ↑ ↑1.6 57.9 ↑ ↑5.8 75.3 ↓ ↓0.5 54.8 ↑ ↑7.5 48.8 ↑ ↑10.7 30.2 ↑ ↑0.7 58.7 ↑ ↑3.6 SPPO 61.5 ↓ ↓0.1 61.9 ↑ ↑3.7 81.2 ↑ ↑2.6 62.0 ↑ ↑10.0 73.4 ↓ ↓2.4 51.9 ↑ ↑4.6 50.7 ↑ ↑12.6 30.7 ↑ ↑1.3 59.2 ↑ ↑4.0 CPO 61.6 ↓ ↓0.0 56.1 ↓ ↓2.1 77.8 ↓ ↓0.8 51.5 ↓ ↓0.6 75.9 ↓ ↓0.0 40.2 ↓ ↓7.1 37.3 ↓ ↓0.8 29.3 ↓ ↓0.2 53.4 ↓ ↓1.8 RRHF 61.7 ↑ ↑0.1 55.8 ↓ ↓2.4 77.7 ↓ ↓0.9 51.3 ↓ ↓0.8 75.6 ↓ ↓0.2 39.1 ↓ ↓8.2 36.6 ↓ ↓1.6 29.3 ↓ ↓0.2 53.4 ↓ ↓1.8 SLiCHF 61.8 ↑ ↑0.2 57.9 ↓ ↓0.3 79.4 ↑ ↑0.8 60.2 ↑ ↑8.2 75.6 ↓ ↓0.2 49.4 ↑ ↑2.1 42.3 ↑ ↑4.2 30.3 ↑ ↑0.9 57.1 ↑ ↑2.0 ORPO 61.8 ↑ ↑0.1 62.5 ↑ ↑4.3 81.2 ↑ ↑2.6 63.9 ↑ ↑11.9 76.7 ↑ ↑0.9 48.9 ↑ ↑1.6 56.1 ↑ ↑18.0 30.8 ↑ ↑1.4 60.2 ↑ ↑5.1 SimPO 61.9 ↑ ↑0.3 62.4 ↑ ↑4.2 81.6 ↑ ↑3.0 63.0 ↑ ↑11.0 74.4 ↓ ↓1.4 42.8 ↓ ↓4.5 50.6 ↑ ↑12.5 31.2 ↑ ↑1.7 58.5 ↑ ↑3.3 ROPO 61.5 ↓ ↓0.2 61.7 ↑ ↑3.5 81.4 ↑ ↑2.8 64.8 ↑ ↑12.8 73.8 ↓ ↓2.1 54.4 ↑ ↑7.1 49.9 ↑ ↑11.8 30.9 ↑ ↑1.4 59.8 ↑ ↑4.6 POWER-DL 61.5 ↓ ↓0.2 61.7 ↑ ↑3.5 81.4 ↑ ↑2.8 64.8 ↑ ↑12.8 73.8 ↓ ↓2.1 54.4 ↑ ↑7.1 49.9 ↑ ↑11.8 30.9 ↑ ↑1.4 59.8 ↑ ↑4.6 POWER 61.8 ↑ ↑0.2 61.8 ↑ ↑3.6 80.6 ↑ ↑2.0 59.8 ↑ ↑7.7 76.6 ↑ ↑0.8 46.3 ↓ ↓1.1 53.7 ↑ ↑15.6 30.4 ↑ ↑1.0 58.9 ↑ ↑3.7 Zephyr Llama3-8B-Instruct Initial Model 65.7 ↓ ↓0.0 62.0 ↓ ↓0.0 78.8 ↓ ↓0.0 51.7 ↓ ↓0.0 76.0 ↓ ↓0.0 75.3 ↓ ↓0.0 54.4 ↓ ↓0.0 36.0 ↓ ↓0.0 62.5 ↓ ↓0.0 DPO 66.0 ↑ ↑0.3 63.0 ↑ ↑0.9 76.7 ↓ ↓2.1 55.7 ↑ ↑4.0 72.9 ↓ ↓3.1 75.7 ↑ ↑0.4 49.6 ↓ ↓4.8 37.1 ↑ ↑1.1 62.1 ↑ ↑0.5 DPO+SFT 66.0 ↑ ↑0.3 66.6 ↑ ↑4.5 79.2 ↑ ↑0.5 59.9 ↑ ↑8.2 75.0 ↓ ↓1.0 75.7 ↑ ↑0.4 51.8 ↓ ↓2.6 37.3 ↑ ↑1.3 64.2 ↑ ↑1.7 cDPO 66.0 ↑ ↑0.3 67.8 ↑ ↑5.8 80.5 ↑ ↑1.8 59.0 ↑ ↑7.3 75.1 ↓ ↓1.0 76.9 ↑ ↑1.6 51.2 ↓ ↓3.2 37.3 ↑ ↑1.4 64.2 ↑ ↑1.7 R-DPO 65.7 ↑ ↑0.0 66.0 ↑ ↑4.0 78.2 ↓ ↓0.6 58.9 ↑ ↑7.2 74.5 ↓ ↓1.5 75.8 ↑ ↑0.5 50.7 ↓ ↓3.7 36.9 ↑ ↑0.9 63.3 ↑ ↑0.8 IPO 65.8 ↑ ↑0.1 62.1 ↑ ↑0.1 78.7 ↓ ↓0.0 51.8 ↑ ↑0.1 75.9 ↓ ↓0.2 75.8 ↑ ↑0.5 54.0 ↓ ↓0.5 35.8 ↓ ↓0.1 62.2 ↑ ↑0.3 χPO 66.2 ↑ ↑0.5 65.4 ↑ ↑3.3 80.5 ↑ ↑1.7 54.2 ↑ ↑2.6 76.2 ↑ ↑0.1 76.8 ↑ ↑1.5 54.6 ↑ ↑0.1 37.1 ↑ ↑1.1 64.0 ↑ ↑1.2 SPPO 66.1 ↑ ↑0.4 65.8 ↑ ↑3.8 78.7 ↓ ↓0.1 58.6 ↑ ↑6.9 74.1 ↓ ↓1.9 74.0 ↓ ↓1.3 55.2 ↑ ↑0.7 37.4 ↑ ↑1.4 63.7 ↑ ↑1.1 CPO 65.4 ↓ ↓0.3 61.9 ↓ ↓0.2 77.8 ↓ ↓0.9 52.3 ↑ ↑0.6 75.5 ↓ ↓0.5 75.4 ↑ ↑0.1 53.8 ↓ ↓0.6 35.9 ↓ ↓0.1 62.2 ↓ ↓0.2 RRHF 65.4 ↓ ↓0.3 61.9 ↓ ↓0.2 77.7 ↓ ↓1.1 52.3 ↑ ↑0.6 75.3 ↓ ↓0.7 75.3 ↓ ↓0.0 54.2 ↓ ↓0.2 35.8 ↓ ↓0.2 62.2 ↓ ↓0.2 SLiCHF 65.6 ↓ ↓0.0 63.1 ↑ ↑1.0 79.1 ↑ ↑0.3 56.0 ↑ ↑4.3 75.4 ↓ ↓0.6 76.8 ↑ ↑1.5 48.8 ↓ ↓5.6 36.4 ↑ ↑0.5 62.9 ↑ ↑0.4 ORPO 65.9 ↑ ↑0.2 64.5 ↑ ↑2.5 78.5 ↓ ↓0.3 57.6 ↑ ↑5.9 75.3 ↓ ↓0.7 77.2 ↑ ↑1.9 52.3 ↓ ↓2.2 36.7 ↑ ↑0.8 63.7 ↑ ↑1.1 SimPO 65.8 ↑ ↑0.1 62.1 ↑ ↑0.1 74.2 ↓ ↓4.6 57.4 ↑ ↑5.7 71.1 ↓ ↓4.9 72.4 ↓ ↓2.9 54.1 ↓ ↓0.4 37.0 ↑ ↑1.0 61.8 ↓ ↓0.7 ROPO 66.2 ↑ ↑0.5 63.6 ↑ ↑1.5 76.4 ↓ ↓2.4 58.1 ↑ ↑6.4 72.9 ↓ ↓3.1 73.0 ↓ ↓2.3 55.6 ↑ ↑1.2 37.5 ↑ ↑1.5 62.9 ↑ ↑0.4 POWER-DL 65.8 ↑ ↑0.1 64.7 ↑ ↑2.7 76.9 ↓ ↓1.8 59.5 ↑ ↑7.9 73.6 ↓ ↓2.4 76.2 ↑ ↑0.9 55.6 ↑ ↑1.2 37.2 ↑ ↑1.3 63.7 ↑ ↑1.1 POWER 65.7 ↓ ↓0.0 63.1 ↑ ↑1.1 75.0 ↓ ↓3.7 59.1 ↑ ↑7.4 71.7 ↓ ↓4.3 75.7 ↑ ↑0.5 54.6 ↑ ↑0.1 37.1 ↑ ↑1.1 62.9 ↑ ↑0.4 I.2 Performance on MT-Bench We evaluate the trained models on MT-Bench, which includes 80 questions across 8 categories. We report the average score in Table 6 as evaluated by GPT-4 as the judge model. The highest score in trained models is shown in bold and the second highest score is underlined. POWER-DL consistently outperforms other preference optimization methods. Similar to Meng et al. (2024), we observe that variations across different methods in MT-Bench scores are small compared to AlpacaEval 2.0 and Arena-Hard. Table 6: Llama3 MT-Bench results on Helpsteer2 and Zephyr settings. Helpsteer2 Zephyr Llama3-8B-Base Llama3-8B-Instruct Llama3-8B-Base Llama3-8B-Instruct Method GPT-4 Score GPT-4 Score GPT-4 Score GPT-4 Score Initial Model 4.9 8.3 6.1 8.3 DPO 5.6 8.2 7.0 8.3 DPO+SFT 5.5 8.1 7.0 8.0 cDPO 5.8 8.1 6.9 8.2 R-DPO 5.4 8.2 6.9 8.1 IPO 4.9 8.0 6.3 8.0 χPO 5.5 8.0 6.9 8.1 SPPO 5.8 8.2 7.1 8.2 CPO 5.9 7.2 6.3 8.0 RRHF 5.7 7.9 6.2 8.1 SLiC-HF 5.9 7.8 6.8 7.9 ORPO 5.7 8.1 6.6 8.3 SimPO 6.0 8.0 6.8 8.2 ROPO 5.8 8.1 7.1 8.2 POWER-DL 6.0 8.2 7.1 8.2 POWER 5.9 8.2 7.0 8.2 I.3 Experiments on Iterative Preference Optimization The multi-iteration experiments are conducted on the Zephyr pipeline that extends the single-iteration instruct setting into three iteration by splitting the dataset of prompts. We compare our approach with iterative DPO Dong et al. (2024), XPO Xie et al. (2024), which adds negative SFT to DPO, and an iterative variant of SimPO. Table 7 presents the results, demonstrating that the benefits of our approach that extend to the multi-iteration setting. Table 7: Instruction-following benchmark results in the multi-iteration setting. Method AlpacaEval 2.0 (LC%) AlpacaEval 2.0 (WR%) Arena-Hard (WR%) Initial Model 33.41 32.40 23.0 DPO iter=1 40.02 38.58 31.6 DPO iter=2 42.15 40.71 37.80 DPO iter=3 42.38 40.8 38.8 XPO iter=1 40.96 39.50 32.6 XPO iter=2 41.33 39.66 36.3 XPO iter=3 42.42 39.98 35.7 SimPO iter=1 38.91 36.44 28.7 SimPO iter=2 41.15 37.77 28.9 SimPO iter=3 41.97 38.32 32.6 POWER-DL iter=1 41.42 37.64 34.4 POWER-DL iter=2 43.48 40.17 39.8 POWER-DL iter=3 46.55 43.21 42.1 I.4 Hyperparameter Robustness Results We assess the hyperparameter robustness of POWER-DL by examining its performance across a range of values for γ, η, and β in the Helpsteer2 base setting. Figure 3 shows AlpacaEval winrate and length-controlled winrate for various hyperparameters. POWER-DL exhibits a robust behavior with respect to all three hyperparameters, particularly considering the length-controlled winrate. Figure 3: POWER-DL hyperparameter robustness results in the Helpsteer2 base setting. I.5 Examples Responses The following is an example question along with responses from various models from the Helpsteer2 base pipeline. POWER-DL response shows analytical thinking, a deeper reasoning process, and consideration of implicit factors, characteristics reminiscent of System 2 thinking. Question One One was a race horse Two Two was one too. One One won one race. Two Two won one too. How many horses are in this story? Responses from Different Models Initial Model: There are two horses in this story, One One and Two Two. GPT-4: There are two horses in this story. Their names are “One One” and “Two Two.” DPO: Based on the information given in the riddle, there are two horses involved in the story. “The One One” and “The Two Two” are both mentioned as race horses, and each of them has won one race respectively. So, there is one “One One” horse and one “Two Two” horse in this story. Together, they make up the total two horses. cDPO: Based on the information given in the riddle, there are two horses involved in the story. “The One One” and “The Two Two” are both designated as race horses, indicating that there are two horses with those specific monikers. “One One won one race” refers to the horse with the name “One One” winning one race. “And Two Two won one too” refers to the horse with the name “Two Two” winning one race as well, implying that both horses won one race each, not ruling out the possibility of both being race horses. So, there is “One One” horse and “Two Two” horse in this story. χPO: There are two horses in this story. SPPO: Based on the information provided in the riddle, there are two horses involved in the story. “One One” and “Two Two” are both race horses, and the sentence “One One won one race. Two Two won one too.” indicates that both horses won one race each. So, there are two distinct horses that won a race each, namely One One and Two Two. Therefore, the number of horses in this story is 2. SimPO: Based solely on the information provided in the riddle, there could be as many as three horses and as few as one. The first two lines introduce two horses, “One-One” and “Two-Two.” The next two lines indicate that each can win a race, but the information about the races themselves and the horses that competed in them is missing. If “One-One” and “Two-Two” raced each other, then just those two horses would exist in this story. However, if either or both raced a third horse, the number of horses in the story could be three. Therefore, the actual number of horses in this riddle’s story is unknown. POWER-DL (ours): The story mentions two horses, One One and Two Two. However, it’s possible that there could be more horses involved. The statement “One One won one race” does not specify whether the track was crowded with competitors, and it’s possible that the horse had to race against multiple opponents. Therefore, the number of horses in the story could be anywhere from two to many. The statement only specifies the race results of the two named horses. Appendix J Experimental Results on Mistral Family In this section, we provide experimental results on conducting preference optimization on Mistral-7B Jiang et al. (2023). We follow the same experimental details described earlier and finetune mistralai/Mistral-7B-v0.1 in the base setting and mistralai/Mistral-7B-Instruct-v0.2 in the instruct setting. Table 8: AlpacaEval 2 and Arena-Hard results on Helpsteer2 and Zephyr settings for Mistral family. Helpsteer2 Zephyr Mistral-7B-Base Mistral-7B-Instruct Mistral-7B-Base Mistral-7B-Instruct AlpacaEval Arena-Hard AlpacaEval Arena-Hard AlpacaEval Arena-Hard AlpacaEval Arena-Hard Method LC(%) WR(%) WR(%) LC(%) WR(%) WR(%) LC(%) WR(%) WR(%) LC(%) WR(%) WR(%) Initial Model 5.47 4.16 1.5 27.70 22.26 14.8 2.85 2.11 0.6 27.70 22.26 14.8 DPO 11.89 10.39 4.6 36.47 29.41 17.1 15.67 13.26 5.9 36.55 36.28 24.3 DPO+SFT 10.57 8.21 3.4 35.28 27.88 16.9 12.30 10.68 5.4 35.31 35.70 25.1 cDPO 12.12 10.52 3.5 32.07 29.22 16.6 13.57 12.24 5.4 31.42 30.43 20.9 χPO 10.88 8.64 4.1 38.90 37.02 22.5 9.80 8.35 3.4 34.79 35.53 17.0 SimPO 14.56 13.97 7.9 38.28 28.89 14.1 16.08 16.50 7.3 36.23 29.11 22.5 POWER-DL 19.83 15.40 8.2 42.26 34.72 23.7 20.34 19.50 12.1 42.57 42.53 28.0 POWER 19.72 16.04 6.5 39.23 33.06 20.1 17.09 15.25 10.2 38.13 36.85 26.2 Table 9: Mistral MT-Bench results on Helpsteer2 and Zephyr settings. Helpsteer2 Zephyr Mistral-7B-Base Mistral-7B-Instruct Mistral-7B-Base Mistral-7B-Instruct Method GPT-4 Score GPT-4 Score GPT-4 Score GPT-4 Score Initial Model 3.1 6.6 3.6 6.6 DPO 3.4 6.3 5.0 6.6 DPO+SFT 3.4 6.3 5.2 6.3 cDPO 3.5 6.1 5.2 6.5 χPO 3.2 6.6 5.0 6.2 SimPO 2.9 6.3 4.5 6.0 POWER-DL 3.4 6.6 5.2 6.6 POWER 3.2 6.6 5.2 6.5