Paper deep dive

Towards Efficient and Exact Optimization of Language Model Alignment

Haozhe Ji, Cheng Lu, Yilin Niu, Pei Ke, Hongning Wang, Jun Zhu, Jie Tang, Minlie Huang

Year: 2024Venue: arXiv preprintArea: Alignment TrainingType: EmpiricalEmbeddings: 286

Models: ChatGLM2-6B, GPT-2 Large, Pythia-2.8B

Intelligence

Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 95%

Last extracted: 3/12/2026, 7:22:56 PM

Summary

The paper introduces Efficient Exact Optimization (EXO), a method for aligning language models with human preferences. It identifies that Direct Preference Optimization (DPO) effectively minimizes forward KL divergence, leading to mean-seeking behavior, whereas EXO minimizes reverse KL divergence, achieving mode-seeking behavior that aligns better with the optimal policy while avoiding the high variance and complexity of traditional Reinforcement Learning (RL) methods.

Entities (5)

DPO · algorithm · 100%EXO · algorithm · 100%RL · methodology · 100%Forward KL Divergence · mathematical-objective · 95%Reverse KL-divergence · mathematical-objective · 95%

Relation Signals (3)

EXO → minimizes → Reverse KL-divergence

confidence 95% · EXO is equivalent to probability matching... measured by the reverse KL divergence

DPO → minimizes → Forward KL Divergence

confidence 95% · DPO actually corresponds to minimizing the forward KL divegence

EXO → improvesupon → DPO

confidence 90% · demonstrate the advantages of our method over existing approaches

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:The alignment of language models with human preferences is vital for their application in real-world tasks. The problem is formulated as optimizing the model's policy to maximize the expected reward that reflects human preferences with minimal deviation from the initial policy. While considered as a straightforward solution, reinforcement learning (RL) suffers from high variance in policy updates, which impedes efficient policy improvement. Recently, direct preference optimization (DPO) was proposed to directly optimize the policy from preference data. However, we show that DPO derived based on the optimal solution of the problem leads to a compromised mean-seeking approximation of the optimal solution in practice. In this paper, we propose efficient exact optimization (EXO) of the alignment objective. EXO is guaranteed to optimize in the same direction as RL algorithms asymptotically for arbitrary policy parametrization. This leads to the same mode-seeking solution, while enables efficient optimization by circumventing the complexities of RL. We also compare our method to DPO with both theoretical and empirical analyses, and further demonstrate the advantages of our method over existing approaches on realistic human preference data. Code is available at this https URL.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

285,467 characters extracted from source content.

Expand or collapse full text

Towards Efficient Exact Optimization of Language Model Alignment Haozhe Ji Cheng Lu Yilin Niu Pei Ke Hongning Wang Jun Zhu Jie Tang Minlie Huang Abstract The alignment of language models with human preferences is vital for their application in real-world tasks. The problem is formulated as optimizing the model’s policy to maximize the expected reward that reflects human preferences with minimal deviation from the initial policy. While considered as a straightforward solution, reinforcement learning (RL) suffers from high variance in policy updates, which impedes efficient policy improvement. Recently, direct preference optimization (DPO) was proposed to directly optimize the policy from preference data. However, we show that DPO derived based on the optimal solution of the problem leads to a compromised mean-seeking approximation of the optimal solution in practice. In this paper, we propose efficient exact optimization (EXO) of the alignment objective. EXO is guaranteed to optimize in the same direction as RL algorithms asymptotically for arbitrary policy parametrization. This leads to the same mode-seeking solution, while enables efficient optimization by circumventing the complexities of RL. We also compare our method to DPO with both theoretical and empirical analyses, and further demonstrate the advantages of our method over existing approaches on realistic human preference data. Code is available at https://github.com/haozheji/exact-optimization. Machine Learning, ICML 1 Introduction Despite the proficiency of large language models, e.g., GPT-3 (Brown et al., 2020) in complex tasks under minimal supervision, they are still prone to produce harmful (Bai et al., 2022), biased (Bender et al., 2021), and unfaithful (Ji et al., 2023c) responses due to the heterogeneous sources of their pre-training corpora. Ensuring the large language models to generate desired responses that are in line with humans’ ethical standards and quality preferences is crucial for the development of reliable AI systems. The problem, well known as language model (LM) alignment with human preferences (Ouyang et al., 2022), is generally formulated as optimizing the LM policy πθsubscript _θπitalic_θ to maximize the expected reward, which reflects human preferences regarding the completion yitalic_y for a given prompt xitalic_x. The practical recipe is to train a reward model rϕsubscriptitalic-ϕr_φritalic_ϕ to predict the human-chosen response from a set of responses generated by an initial LM policy πinitsubscriptinit _ initπinit. Yet, the challenge of acquiring substantial high-quality preference data often impedes accurate estimation of the ideal reward model. Consequently, this empirically learned reward model may lead to misspecified behaviors, particularly under the distributional shift between its training data and the data generated by πθsubscript _θπitalic_θ (Gao et al., 2023). Therefore, the final objective of alignment additionally involves minimizing the reverse Kullback-Leibler (KL) divergence of πθsubscript _θπitalic_θ from its initial distribution πinitsubscriptinit _ initπinit with an intensity β, besides maximizing the expected reward: maxπθ⁡∼,∼πθ⁢(|)⁢[rϕ⁢(,)]−β⁢KL⁢(πθ∥πinit).subscriptsubscriptsubscriptformulae-sequencesimilar-tosimilar-tosubscriptconditionaldelimited-[]subscriptitalic-ϕsubscriptKLconditionalsubscriptsubscriptinit _ _θE_ x , y% _θ( y| x)[r_φ( x,% y)]- _ KL( _θ\| _ init% ).maxitalic_π start_POSTSUBSCRIPT θ end_POSTSUBSCRIPT blackboard_Eitalic_x ∼ D , italic_y ∼ π start_POSTSUBSCRIPT θ ( italic_y | italic_x ) end_POSTSUBSCRIPT [ ritalic_ϕ ( italic_x , italic_y ) ] - β blackboard_DKL ( πitalic_θ ∥ πinit ) . (1) Due to the discrete nature of content generation from an LM (i.e., sampling yitalic_y from πθ(⋅|) _θ(·| x)πitalic_θ ( ⋅ | italic_x )), the most widely adopted approach to optimize Eq. (1) is reinforcement learning (RL) (Ziegler et al., 2019; Stiennon et al., 2020; Ouyang et al., 2022). Albeit well studied even before the era of large language models, RL solutions are notorious for their poor stability due to the high variance in estimating the policy gradients or value functions, which potentially worsens sample complexity and thus compromises efficient convergence (Papini et al., 2018; Anschel et al., 2017). As a remedy, direct preference optimization (DPO) was recently proposed to replace the RL solutions (Rafailov et al., 2023). Specifically, DPO defines a pair-wise preference loss on the estimated policy πθsubscript _θπitalic_θ by leveraging the following policy-reward mapping in the optimal solution to Eq. (1): πβ∗⁢(|)∝πinit⁢(|)⁢e1β⁢rϕ⁢(,)proportional-tosubscriptsuperscriptconditionalsubscriptinitconditionalsuperscript1subscriptitalic-ϕπ^*_β( y| x) _ init(% y| x)e 1βr_φ( x,% y)π∗italic_β ( italic_y | italic_x ) ∝ πinit ( italic_y | italic_x ) edivide start_ARG 1 end_ARG start_ARG β end_ARG ritalic_ϕ ( italic_x , italic_y ) (2) However, this optimal policy is obtained analytically while not considering the practical parametrization of πθsubscript _θπitalic_θ. When πθsubscript _θπitalic_θ is mis-specified and cannot perfectly capture the target πβ∗superscriptsubscript _β^*πitalic_β∗, our analysis demonstrates that DPO leads to a compromised approximation of πβ∗superscriptsubscript _β^*πitalic_β∗, which only covers the support of πβ∗superscriptsubscript _β^*πitalic_β∗ while failing to capture its modes. In this paper, we present an underexplored perspective of the alignment objective in Eq. (1): We prove that Eq. (1) is equivalent to probability matching between the parametrized policy πθsubscript _θπitalic_θ and the optimal policy πβ∗superscriptsubscript _β^*πitalic_β∗ defined in Eq. (2) measured by the reverse KL divergence KL⁢(πθ∥πβ∗)subscriptKLconditionalsubscriptsuperscriptsubscriptD_ KL( _θ\| _β^*)blackboard_DKL ( πitalic_θ ∥ πitalic_β∗ ). Based on the established equivalence, we propose efficient exact optimization (EXO) of the KL-regularized reward maximization objective. Specifically, we rigorously prove that irrespective of the policy’s parametrization, EXO is guaranteed to improve πθsubscript _θπitalic_θ in the same direction as the RL solutions asymptotically. In practice, we demonstrate that EXO facilitates efficient optimization towards this alignment objective with finite samples while bypassing the complexities of RL. Under this probability matching perspective, we demonstrate that DPO actually corresponds to minimizing the forward KL divegence KL⁢(πβ∗∥πθ)subscriptKLconditionalsuperscriptsubscriptsubscriptD_ KL( _β^*\| _θ)blackboard_DKL ( πitalic_β∗ ∥ πitalic_θ ). Though minimizing both the forward and reverse KL divergences lead to the same analytic solution, it is not necessarily achievable when taking into account the expressivity gap between the model families of πβ∗superscriptsubscript _β^*πitalic_β∗ and πθsubscript _θπitalic_θ (Lin et al., 2021). Under this realistic constraint, minimizing these two divergences converge to parametrized policies with different behaviors (Bishop & Nasrabadi, 2006). As illustrated in Figure 1, minimizing the reverse KL fosters a mode-seeking policy πθRKLsubscriptsubscriptRKL _ _ RKLπitalic_θ start_POSTSUBSCRIPT RKL end_POSTSUBSCRIPT that concentrates to the principal modes of πβ∗superscriptsubscript _β^*πitalic_β∗ (Chan et al., 2022), while minimizing the forward KL results in a mean-seeking policy πθFKLsubscriptsubscriptFKL _ _ FKLπitalic_θ start_POSTSUBSCRIPT FKL end_POSTSUBSCRIPT that places large mass to the mean of different modes in πβ∗superscriptsubscript _β^*πitalic_β∗, which does not necessitate high probabilities under πβ∗superscriptsubscript _β^*πitalic_β∗. In the inference stage, πθRKLsubscriptsubscriptRKL _ _ RKLπitalic_θ start_POSTSUBSCRIPT RKL end_POSTSUBSCRIPT is preferrably better than πθFKLsubscriptsubscriptFKL _ _ FKLπitalic_θ start_POSTSUBSCRIPT FKL end_POSTSUBSCRIPT by capturing the main characteristics of πβ∗superscriptsubscript _β^*πitalic_β∗ (Ji et al., 2023b). Figure 1: Illustration of different characteristics of (a) πθRKLsubscriptsubscriptRKL _ _ RKLπitalic_θ start_POSTSUBSCRIPT RKL end_POSTSUBSCRIPT by minimizing the reverse KL (by EXO) and (b) πθFKLsubscriptsubscriptFKL _ _ FKLπitalic_θ start_POSTSUBSCRIPT FKL end_POSTSUBSCRIPT by minimizing the forward KL (by DPO). We conduct a series of experiments to verify the effectiveness and scalability of EXO. We first systematically evaluate the efficiency of different approaches in trading off maximizing the oracle reward and minimizing the KL divergence during optimization of the alignment objective. Then, we conduct evaluations on the effectiveness of learning from real human preferences in various alignment benchmarks, involving summarization, dialogue generation, and instruction following tasks. Comprehensive empirical analyses substantiate our theoretical findings and demonstrate the advantageous performance of EXO over DPO and PPO. Finally, we summarize our contributions in this paper: • We reveal the underexplored equivalence between KL-regularizd reward maximization and minimizing the reverse KL divergence against the optimal policy for the language model alignment problem. • We propose EXO, an algorithm towards efficient exact optimization of the KL-regularized reward maximization objective for alignment. Both theoretical and empricial results confirm its effectiveness. • We show that DPO corresponds to minimizing the forward KL divergence, which is less effective in capturing the essential characteristics of the optimal policy. 2 Preliminaries We first formally review the formulation and objective of the alignment problem. Then we review existing approaches that solve this problem via reinforcement learning and direct preference optimization, respectively. 2.1 Aligning Language Models with Human Preferences Given a vocabulary VV, a language model defines a probability distribution π⁢()=∏t=1nπ⁢(xt|x1,⋯,xt−1)superscriptsubscriptproduct1conditionalsubscriptsubscript1⋯subscript1π( x)= _t=1^nπ(x_t|x_1,·s,x_t-1)π ( italic_x ) = ∏t = 1n π ( xitalic_t | x1 , ⋯ , xitalic_t - 1 ) over a sequence of tokens =(x1,⋯,xn)subscript1⋯subscript x=(x_1,·s,x_n)italic_x = ( x1 , ⋯ , xitalic_n ). We apply π to a conditional generation task of interest with input space =msuperscriptX=V^mX = Vitalic_m and output space =nsuperscriptY=V^nY = Vitalic_n modeled by π⁢(|)=π⁢(,)/π⁢()conditionalπ( y| x)=π( x, y)/π(% x)π ( italic_y | italic_x ) = π ( italic_x , italic_y ) / π ( italic_x ). The alignment procedure typically starts from supervised fine-tuning (SFT) the language model on a high-quality dataset sftsuperscriptsftD sftDsft via maximum likelihood estimation, which obtains the SFT policy πsftsubscriptsft _ sftπsft. Then a preference dataset prefsuperscriptprefD prefDpref is collected by asking humans to select a better response from (1,2)∼πsft⁢(|)similar-tosubscript1subscript2subscriptsftconditional( y_1, y_2) _ sft( y% | x)( italic_y1 , italic_y2 ) ∼ πsft ( italic_y | italic_x ) given a prompt xitalic_x from the same domain of sftsuperscriptsftD sftDsft. Let wsubscript y_witalic_yitalic_w and lsubscript y_litalic_yitalic_l be the chosen and rejected responses among (1,2)subscript1subscript2( y_1, y_2)( italic_y1 , italic_y2 ) respectively according to human preferences. A reward model rϕ:×→ℝ:subscriptitalic-ϕ→ℝr_φ:X×Y _ϕ : X × Y → blackboard_R is usually learned on prefsuperscriptprefD prefDpref to act as a surrogate to expensive human labeling. The reward model is trained to prioritize wsubscript y_witalic_yitalic_w over lsubscript y_litalic_yitalic_l by minimizing the following pair-wise preference loss: ℒr⁢(rϕ)=(,w,l)∼pref⁢[−log⁡erϕ⁢(,w)erϕ⁢(,w)+erϕ⁢(,l)].subscriptℒsubscriptitalic-ϕsubscriptsimilar-tosubscriptsubscriptsuperscriptprefdelimited-[]superscriptsubscriptitalic-ϕsubscriptsuperscriptsubscriptitalic-ϕsubscriptsuperscriptsubscriptitalic-ϕsubscriptL_r(r_φ)=E_( x, y_w,% y_l) pref [- e^r_% φ( x, y_w)e^r_φ( x,% y_w)+e^r_φ( x, y_l) ].Litalic_r ( ritalic_ϕ ) = blackboard_E( italic_x , italic_y start_POSTSUBSCRIPT w , italic_yitalic_l ) ∼ Dpref end_POSTSUBSCRIPT [ - log divide start_ARG eitalic_ritalic_ϕ ( italic_x , italic_yitalic_w ) end_ARG start_ARG eitalic_ritalic_ϕ ( italic_x , italic_yitalic_w ) + eitalic_ritalic_ϕ ( italic_x , italic_yitalic_l ) end_ARG ] . Finally, a policy πθsubscript _θπitalic_θ is learned to maximize the following alignment objective (Ziegler et al., 2019): lhfβ⁢(πθ)superscriptsubscriptlhfsubscript _ lhf^β( _θ)Jlhfitalic_β ( πitalic_θ ) =∼pref(πθ⁢(|)[rϕ(,)] =E_ x pref (% E_ _θ( y| x)[r_φ( % x, y)]= blackboard_Eitalic_x ∼ Dpref ( blackboard_Eπ start_POSTSUBSCRIPT θ ( italic_y | italic_x ) end_POSTSUBSCRIPT [ ritalic_ϕ ( italic_x , italic_y ) ] −βKL[πθ(|)∥πsft(|)]). - _ KL[ _θ( % y| x)\| _ sft( y| x)] % ).- β blackboard_DKL [ πitalic_θ ( italic_y | italic_x ) ∥ πsft ( italic_y | italic_x ) ] ) . (3) Given a prompt xitalic_x sampled from the dataset prefsuperscriptprefD prefDpref, the objective seeks to find the πθsubscript _θπitalic_θ that maximizes the expected reward while minimizes its reverse KL divergence against the SFT policy πsftsubscriptsft _ sftπsft governed by the coefficient β>00β>0β > 0. The KL penalty keeps πθsubscript _θπitalic_θ from moving too far from πsftsubscriptsft _ sftπsft to avoid over optimization of the reward model. The analytic solution that maximizes lhfβ⁢(πθ)superscriptsubscriptlhfsubscriptJ_ lhf^β( _θ)Jlhfitalic_β ( πitalic_θ ) takes the form of an energy-based model (EBM): πβ∗⁢(|)=πsft⁢(|)⁢e1β⁢rϕ⁢(,)Zβ⁢(),superscriptsubscriptconditionalsubscriptsftconditionalsuperscript1subscriptitalic-ϕsubscript _β^*( y| x)= _ sft(% y| x) e 1βr_φ( x,% y)Z_β( x),πitalic_β∗ ( italic_y | italic_x ) = πsft ( italic_y | italic_x ) divide start_ARG edivide start_ARG 1 end_ARG start_ARG β end_ARG ritalic_ϕ ( italic_x , italic_y ) end_ARG start_ARG Zitalic_β ( italic_x ) end_ARG , (4) where Zβ⁢()=∑′∈πsft⁢(′|)⁢e1β⁢rϕ⁢(,′)subscriptsubscriptsuperscript′subscriptsftconditionalsuperscript′1subscriptitalic-ϕsuperscript′Z_β( x)= _ y _% sft( y | x)e 1βr_% φ( x, y )Zitalic_β ( italic_x ) = ∑italic_y′ ∈ Y πsft ( italic_y′ | italic_x ) edivide start_ARG 1 end_ARG start_ARG β end_ARG ritalic_ϕ ( italic_x , italic_y start_POSTSUPERSCRIPT ′ ) end_POSTSUPERSCRIPT is the partition function. In Eq (4), the coefficient β can be considered as the temperature for controlling the strength of the reward model signal when sampling from πβ∗⁢(|)superscriptsubscriptconditional _β^*( y| x)πitalic_β∗ ( italic_y | italic_x ). 2.2 RL Fine-Tuning Due to the discrete nature of language generation, the objective in Eq. (3) is not differentiable with respect to πθsubscript _θπitalic_θ, which prohibits supervised training. One standard approach is to use RL algorithms to optimize this objective. Ziegler et al. (2019) proposed to search for πθsubscript _θπitalic_θ that maximizes a KL-regularized reward rϕ⁢(,)−β⁢log⁡πθ⁢(|)πsft⁢(|)subscriptitalic-ϕsubscriptconditionalsubscriptsftconditionalr_φ( x, y)-β _θ(% y| x) _ sft( y|% x)ritalic_ϕ ( italic_x , italic_y ) - β log divide start_ARG πitalic_θ ( italic_y | italic_x ) end_ARG start_ARG πsft ( italic_y | italic_x ) end_ARG. This can be achieved by policy gradient methods, such as Proximal Policy Optimization (PPO) (Schulman et al., 2017). 2.3 Direct Preference Optimization To optimize πθsubscript _θπitalic_θ directly using the preference data, Rafailov et al. (2023) rearranged Eq. (4) to express the reward function by the optimal policy πβ∗superscriptsubscript _β^*πitalic_β∗, rϕ⁢(,)=β⁢log⁡πβ∗⁢(|)πsft⁢(|)+β⁢log⁡Zβ⁢().subscriptitalic-ϕsuperscriptsubscriptconditionalsubscriptsftconditionalsubscriptr_φ( x, y)=β _β^*(% y| x) _ sft( y|% x)+β Z_β( x).ritalic_ϕ ( italic_x , italic_y ) = β log divide start_ARG πitalic_β∗ ( italic_y | italic_x ) end_ARG start_ARG πsft ( italic_y | italic_x ) end_ARG + β log Zitalic_β ( italic_x ) . (5) Then they proposed to directly optimize the policy πθsubscript _θπitalic_θ by replacing πβ∗superscriptsubscript _β^*πitalic_β∗ with πθsubscript _θπitalic_θ and substituting the corresponding reward function into a pair-wise preference loss: ℒdpo(πθ)=(,w,l)∼pref[ _ dpo( _θ)=E_( % x, y_w, y_l) pref% [Ldpo ( πitalic_θ ) = blackboard_E( italic_x , italic_y start_POSTSUBSCRIPT w , italic_yitalic_l ) ∼ Dpref end_POSTSUBSCRIPT [ −logσ(βlogπθ⁢(w|)πsft⁢(w|)−βlogπθ⁢(l|)πsft⁢(l|))]. - σ (β _θ( y% _w| x) _ sft( y_w| x)% -β _θ( y_l| x) _ % sft( y_l| x) ) ].- log σ ( β log divide start_ARG πitalic_θ ( italic_yitalic_w | italic_x ) end_ARG start_ARG πsft ( italic_yitalic_w | italic_x ) end_ARG - β log divide start_ARG πitalic_θ ( italic_yitalic_l | italic_x ) end_ARG start_ARG πsft ( italic_yitalic_l | italic_x ) end_ARG ) ] . (6) 3 Methodology In this section, we start with a generalized alignment objective and present its equivalent form under the perspective of probability matching. Then we formally derive efficient exact optimization (EXO) of the generalized alignment objective while bypassing the necessity of employing any RL algorithms. Furthermore, we compare against DPO and demonstrate the advantage of our approach in terms of the distributional characteristics of the learned policy. All the proofs are provided in Appendix A. 3.1 From the Generalized Alignment Objective to Probability Matching We first introduce a generalized alignment objective that distributes the intensity of the KL regularization regarding the SFT policy to both the parametrized policy πθsubscript _θπitalic_θ and the reward model rϕsubscriptitalic-ϕr_φritalic_ϕ, which intuitively connects the regularization setting of DPO (Eq. (2.3)) that only regularizes πθsubscript _θπitalic_θ and PPO (Eq. (3)) that only regularizes rϕsubscriptitalic-ϕr_φritalic_ϕ. In the following theorem, we present the formal definition and the property of the generalized alignment objective. Theorem 3.1. Let βπ>0,βr>0formulae-sequencesubscript0subscript0 _π>0, _r>0βitalic_π > 0 , βitalic_r > 0 and βπ⁢βr=βsubscriptsubscript _π _r=βitalic_π βitalic_r = β. The generalized alignment objective is defined as lhfβr⁢(πθβπ)superscriptsubscriptlhfsubscriptsuperscriptsubscriptsubscript _ lhf _r( _θ _π)Jlhfitalic_βitalic_r ( πitalic_θitalic_βitalic_π ) =∼pref(πθβπ⁢(|)[rϕ(,)] =E_ x pref (% E_ _θ _π( y| x)[r_% φ( x, y)]= blackboard_Eitalic_x ∼ Dpref ( blackboard_Eπ start_POSTSUBSCRIPT θitalic_βitalic_π ( italic_y | italic_x ) end_POSTSUBSCRIPT [ ritalic_ϕ ( italic_x , italic_y ) ] −βrKL[πθβπ(|)∥πsft(|)]), - _rD_ KL[ _θ^% _π( y| x)\| _ sft( % y| x)] ),- βitalic_r blackboard_DKL [ πitalic_θitalic_βitalic_π ( italic_y | italic_x ) ∥ πsft ( italic_y | italic_x ) ] ) , (7) where πθβπ⁢(|)superscriptsubscriptsubscriptconditional _θ _π( y| x)πitalic_θitalic_βitalic_π ( italic_y | italic_x ) satisfies πθβπ⁢(|)∝πθ⁢(|)βπ⁢πsft⁢(|)1−βπ.proportional-tosuperscriptsubscriptsubscriptconditionalsubscriptsuperscriptconditionalsubscriptsubscriptsftsuperscriptconditional1subscript _θ _π( y| x) _θ(% y| x) _π _ sft( % y| x)^1- _π.πitalic_θitalic_βitalic_π ( italic_y | italic_x ) ∝ πitalic_θ ( italic_y | italic_x )βitalic_π πsft ( italic_y | italic_x )1 - βitalic_π . (8) Given unlimited model capacity, the optimal πθ∗subscriptsuperscript _θ^*πitalic_θ∗ that maximizes lhfβr⁢(πθβπ)superscriptsubscriptlhfsubscriptsuperscriptsubscriptsubscriptJ_ lhf _r( _θ _π)Jlhfitalic_βitalic_r ( πitalic_θitalic_βitalic_π ) satisfies πθ∗=πβ∗subscriptsuperscriptsuperscriptsubscript _θ^*= _β^*πitalic_θ∗ = πitalic_β∗. Intuitively, lhfβr⁢(πθβπ)superscriptsubscriptlhfsubscriptsuperscriptsubscriptsubscriptJ_ lhf _r( _θ _π)Jlhfitalic_βitalic_r ( πitalic_θitalic_βitalic_π ) calculates the expectation of the reward regularized with the intensity βrsubscript _rβitalic_r with respect to the policy regularized with the intensity βπsubscript _πβitalic_π. As the total regularization intensity β=βr⁢βπsubscriptsubscriptβ= _r _πβ = βitalic_r βitalic_π is fixed, lhfβr⁢(πθβπ)superscriptsubscriptlhfsubscriptsuperscriptsubscriptsubscriptJ_ lhf _r( _θ _π)Jlhfitalic_βitalic_r ( πitalic_θitalic_βitalic_π ) maintains the same analytic solution πβ∗superscriptsubscript _β^*πitalic_β∗ as lhfβ⁢(πθ)superscriptsubscriptlhfsubscriptJ_ lhf^β( _θ)Jlhfitalic_β ( πitalic_θ ). Furthermore, it interpolates the policy regularization setting (βr=1,βπ=βformulae-sequencesubscript1subscript _r=1, _π=βitalic_r = 1 , βitalic_π = β) in the DPO objective111β⁢log⁡πθ⁢(|)πsft⁢(|)=log⁡πθβ⁢(|)πsft⁢(|)+C⁢()subscriptconditionalsubscriptsftconditionalsuperscriptsubscriptconditionalsubscriptsftconditionalβ _θ( y| x) _ sft% ( y| x)= _θ^β( % y| x) _ sft( y| x)+C(% x)β log divide start_ARG πitalic_θ ( italic_y | italic_x ) end_ARG start_ARG πsft ( italic_y | italic_x ) end_ARG = log divide start_ARG πitalic_θitalic_β ( italic_y | italic_x ) end_ARG start_ARG πsft ( italic_y | italic_x ) end_ARG + C ( italic_x ) where C⁢()C( x)C ( italic_x ) depends only on xitalic_x, which does not effect the DPO objective in Eq. (2.3). and the reward regularization setting (βr=β,βπ=1formulae-sequencesubscriptsubscript1 _r=β, _π=1βitalic_r = β , βitalic_π = 1) in the PPO objective when continuously tunning βrsubscript _rβitalic_r and βπsubscript _πβitalic_π while keeping their product fixed. We also empirically show the effect of βrsubscript _rβitalic_r and βπsubscript _πβitalic_π beyond the impact on their product β=βr⁢βπsubscriptsubscriptβ= _r _πβ = βitalic_r βitalic_π in Appendix C.2. Next, we derive an equivalent form of the generalized alignment objective by rearranging the elements in Eq. (7), which offers a new insight of the alignment problem from the probability matching perspective. The detailed derivation can be found in Appendix A.2. lhfβr⁢(πθβπ)=−βrsuperscriptsubscriptlhfsubscriptsuperscriptsubscriptsubscriptsubscript _ lhf _r( _θ _π% )=- _rJlhfitalic_βitalic_r ( πitalic_θitalic_βitalic_π ) = - βitalic_r ∼pref[KL(πθβπ(|)∥πβr∗(|))] _ x pref [% D_ KL( _θ _π( y| % x)\|π^*_ _r( y| x)) ]blackboard_Eitalic_x ∼ Dpref [ blackboard_DKL ( πitalic_θitalic_βitalic_π ( italic_y | italic_x ) ∥ π∗italic_β start_POSTSUBSCRIPT r end_POSTSUBSCRIPT ( italic_y | italic_x ) ) ] +βr⁢∼pref⁢[log⁡Zβr⁢()],subscriptsubscriptsimilar-tosuperscriptprefdelimited-[]subscriptsubscript + _rE_ x % pref [ Z_ _r( x) ],+ βitalic_r blackboard_Eitalic_x ∼ Dpref [ log Zitalic_β start_POSTSUBSCRIPT r end_POSTSUBSCRIPT ( italic_x ) ] , (9) As the second term is a constant with respect to πθsubscript _θπitalic_θ, Eq. (3.1) reveals that maximizing the generalized alignment objective lhfβr⁢(πθβπ)superscriptsubscriptlhfsubscriptsuperscriptsubscriptsubscriptJ_ lhf _r( _θ _π)Jlhfitalic_βitalic_r ( πitalic_θitalic_βitalic_π ) with respect to πθsubscript _θπitalic_θ is equivalent to minimizing the expected reverse KL divergence KL⁢(πθβπ∥πβr∗)subscriptKLconditionalsuperscriptsubscriptsubscriptsubscriptsuperscriptsubscriptD_ KL( _θ _π\|π^*_ _r)blackboard_DKL ( πitalic_θitalic_βitalic_π ∥ π∗italic_β start_POSTSUBSCRIPT r end_POSTSUBSCRIPT ), where πθβπsuperscriptsubscriptsubscript _θ _πitalic_θitalic_βitalic_π is a function of πθsubscript _θπitalic_θ as defined in Eq. (8). This equivalence implies the possibility of converting the reward maximization problem into a supervised divergence minimization problem, which is able to circumvent the poor stability and low sample efficiency issue caused by high variance in RL solutions (Papini et al., 2018; Anschel et al., 2017). In the following, we introduce our approach towards exact optimization of this generalized alignment objective by practically realizing the probability matching objective. Without loss of generality, our results remain valid for the original alignment objective in Eq. (3). 3.2 Efficient Exact Optimization of the Generalized Alignment Objective We now formally derive EXO which optimizes the generalized alignment objective lhfβr⁢(πθβπ)superscriptsubscriptlhfsubscriptsuperscriptsubscriptsubscriptJ_ lhf _r( _θ _π)Jlhfitalic_βitalic_r ( πitalic_θitalic_βitalic_π ) by realizing the reverse KL divergence KL⁢(πθβπ∥πβr∗)subscriptKLconditionalsuperscriptsubscriptsubscriptsubscriptsuperscriptsubscriptD_ KL( _θ _π\|π^*_ _r)blackboard_DKL ( πitalic_θitalic_βitalic_π ∥ π∗italic_β start_POSTSUBSCRIPT r end_POSTSUBSCRIPT ). We start with the general setting of language model alignment which preassumes the existence of a reward model rϕsubscriptitalic-ϕr_φritalic_ϕ; then we consider the case of learning directly from the preference data. To facilitate policy optimization with straightforward gradient back propagation, we rewrite KL⁢(πθβπ∥πβr∗)subscriptKLconditionalsuperscriptsubscriptsubscriptsuperscriptsubscriptsubscriptD_ KL( _θ _π\| _ _r^*)blackboard_DKL ( πitalic_θitalic_βitalic_π ∥ πitalic_β start_POSTSUBSCRIPT r end_POSTSUBSCRIPT∗ ) under the expectation of the proposal policy πsftsubscriptsft _ sftπsft: KL⁢(πθβπ∥πβr∗)=πsft⁢(|)⁢[πθβπ⁢(|)πsft⁢(|)⁢log⁡πθβπ⁢(|)πβr∗⁢(|)].subscriptKLconditionalsuperscriptsubscriptsubscriptsuperscriptsubscriptsubscriptsubscriptsubscriptsftconditionaldelimited-[]superscriptsubscriptsubscriptconditionalsubscriptsftconditionalsuperscriptsubscriptsubscriptconditionalsuperscriptsubscriptsubscriptconditionalD_ KL( _θ _π\| _ _r^*)=% E_ _ sft( y| x) [ % _θ _π( y| x) _ sft% ( y| x) _θ _π(% y| x) _ _r^*( y|% x) ].blackboard_DKL ( πitalic_θitalic_βitalic_π ∥ πitalic_β start_POSTSUBSCRIPT r end_POSTSUBSCRIPT∗ ) = blackboard_Eπ start_POSTSUBSCRIPT sft ( italic_y | italic_x ) end_POSTSUBSCRIPT [ divide start_ARG πitalic_θitalic_βitalic_π ( italic_y | italic_x ) end_ARG start_ARG πsft ( italic_y | italic_x ) end_ARG log divide start_ARG πitalic_θitalic_βitalic_π ( italic_y | italic_x ) end_ARG start_ARG πitalic_β start_POSTSUBSCRIPT r end_POSTSUBSCRIPT∗ ( italic_y | italic_x ) end_ARG ] . The above expression can be further simplified by defining the log ratio as fθ⁢(,)=log⁡πθβπ⁢(|)−log⁡πsft⁢(|)subscriptsuperscriptsubscriptsubscriptconditionalsubscriptsftconditionalf_θ( x, y)= _θ _π(% y| x)- _ sft( y|% x)fitalic_θ ( italic_x , italic_y ) = log πitalic_θitalic_βitalic_π ( italic_y | italic_x ) - log πsft ( italic_y | italic_x ): KL⁢(πθβπ∥πβr∗)=πsft⁢(|)⁢[efθ⁢(,)⁢log⁡efθ⁢(,)1Zβr⁢()⁢erϕ⁢(,)βr].subscriptKLconditionalsuperscriptsubscriptsubscriptsuperscriptsubscriptsubscriptsubscriptsubscriptsftconditionaldelimited-[]superscriptsubscriptsuperscriptsubscript1subscriptsubscriptsuperscriptsubscriptitalic-ϕsubscriptD_ KL( _θ _π\| _ _r^*)=% E_ _ sft( y| x) [e^f_% θ( x, y) e^f_θ( x,% y) 1Z_ _r( x)e r_φ(% x, y) _r ].blackboard_DKL ( πitalic_θitalic_βitalic_π ∥ πitalic_β start_POSTSUBSCRIPT r end_POSTSUBSCRIPT∗ ) = blackboard_Eπ start_POSTSUBSCRIPT sft ( italic_y | italic_x ) end_POSTSUBSCRIPT [ eitalic_fitalic_θ ( italic_x , italic_y ) log divide start_ARG eitalic_fitalic_θ ( italic_x , italic_y ) end_ARG start_ARG divide start_ARG 1 end_ARG start_ARG Zitalic_β start_POSTSUBSCRIPT r end_POSTSUBSCRIPT ( italic_x ) end_ARG edivide start_ARG ritalic_ϕ ( italic_x , italic_y ) end_ARG start_ARG βitalic_r end_ARG end_ARG ] . As the intractable nature of Zβr⁢()subscriptsubscriptZ_ _r( x)Zitalic_β start_POSTSUBSCRIPT r end_POSTSUBSCRIPT ( italic_x ) hinders direct estimation via Monte Carlo simulation, we propose a practical way to estimate this term by first drawing multiple samples from πsftsubscriptsft _ sftπsft, and then calculating the reverse KL between the probability distributions defined by fθ⁢(,)subscriptf_θ( x, y)fitalic_θ ( italic_x , italic_y ) and 1βr⁢rϕ⁢(,)1subscriptsubscriptitalic-ϕ 1 _rr_φ( x, y)divide start_ARG 1 end_ARG start_ARG βitalic_r end_ARG ritalic_ϕ ( italic_x , italic_y ) over these samples via self-normalization respectively. ∙ ∙ Learning from a reward model. Formally, given K>11K>1K > 1 i.i.d. completions 1:K=1,⋯,Ksubscript:1subscript1⋯subscript y_1:K=\ y_1,·s, y_K\italic_y1 : K = italic_y1 , ⋯ , italic_yitalic_K drawn from πsft⁢(|)subscriptsftconditional _ sft( y| x)πsft ( italic_y | italic_x ), we define an empirical distribution pfθsubscriptsubscriptp_f_θpitalic_f start_POSTSUBSCRIPT θ end_POSTSUBSCRIPT by normalizing the exponential reward efθ⁢(,)superscriptsubscripte^f_θ( x, y)eitalic_fitalic_θ ( italic_x , italic_y ) over the K samples: pfθ⁢(i|1:K,)=efθ⁢(,i)∑j=1Kefθ⁢(,j).subscriptsubscriptconditionalsubscript:1superscriptsubscriptsubscriptsuperscriptsubscript1superscriptsubscriptsubscriptp_f_θ(i| y_1:K, x)= e^f_θ(% x, y_i) _j=1^Ke^f_θ( x% , y_j).pitalic_f start_POSTSUBSCRIPT θ end_POSTSUBSCRIPT ( i | italic_y1 : K , italic_x ) = divide start_ARG eitalic_fitalic_θ ( italic_x , italic_yitalic_i ) end_ARG start_ARG ∑j = 1K eitalic_fitalic_θ ( italic_x , italic_yitalic_j ) end_ARG . (10) Recall that fθ⁢(|)=log⁡πθβπ⁢(|)−log⁡πsft⁢(|)subscriptconditionalsuperscriptsubscriptsubscriptconditionalsubscriptsftconditionalf_θ( y| x)= _θ _π(% y| x)- _ sft( y|% x)fitalic_θ ( italic_y | italic_x ) = log πitalic_θitalic_βitalic_π ( italic_y | italic_x ) - log πsft ( italic_y | italic_x ) and πθβπ⁢(|)∝πθ⁢(|)βπ⁢πsft⁢(|)1−βπproportional-tosuperscriptsubscriptsubscriptconditionalsubscriptsuperscriptconditionalsubscriptsubscriptsftsuperscriptconditional1subscript _θ _π( y| x) _θ(% y| x) _π _ sft( % y| x)^1- _πitalic_θitalic_βitalic_π ( italic_y | italic_x ) ∝ πitalic_θ ( italic_y | italic_x )βitalic_π πsft ( italic_y | italic_x )1 - βitalic_π, Eq. (10) can be rewritten into a form that explicitly depends on πθsubscript _θπitalic_θ: pfθ⁢(i|1:K,)=eβπ⁢log⁡ ⁢πθ⁢(i|)⁢πsft⁢(i|)∑j=1Keβπ⁢log⁡ ⁢πθ⁢(j|)⁢πsft⁢(j|),subscriptsubscriptconditionalsubscript:1superscriptsubscript 1superscriptsubscript _f_θ(i| y_1:K, x)= e _π% _θ( y_i| x) _ sft% ( y_i| x) _j=1^Ke _π % _θ( y_j| x) _ sft(% y_j| x),pitalic_f start_POSTSUBSCRIPT θ end_POSTSUBSCRIPT ( i | italic_y1 : K , italic_x ) = divide start_ARG eitalic_βitalic_π log πitalic_θ ( italic_yitalic_i | italic_x ) πsft ( italic_yitalic_i | italic_x ) end_ARG start_ARG ∑j = 1K eitalic_βitalic_π log πitalic_θ ( italic_yitalic_j | italic_x ) πsft ( italic_yitalic_j | italic_x ) end_ARG , (11) where βπsubscript _πβitalic_π can be regarded as the inverse temperature that modulates the empirical distribution defined by the log ratio between πθsubscript _θπitalic_θ and πsftsubscriptsft _ sftπsft. Similarly, we define a distribution prsubscriptp_rpitalic_r over the K samples modeled by the reward model rϕsubscriptitalic-ϕr_φritalic_ϕ: prϕ⁢(i|1:K,)=e1βr⁢rϕ⁢(,i)∑j=1Ke1βr⁢rϕ⁢(,j).subscriptsubscriptitalic-ϕconditionalsubscript:1superscript1subscriptsubscriptitalic-ϕsubscriptsuperscriptsubscript1superscript1subscriptsubscriptitalic-ϕsubscriptp_r_φ(i| y_1:K, x)= e 1β% _rr_φ( x, y_i) _j=1^Ke 1% _rr_φ( x, y_j).pitalic_r start_POSTSUBSCRIPT ϕ end_POSTSUBSCRIPT ( i | italic_y1 : K , italic_x ) = divide start_ARG edivide start_ARG 1 end_ARG start_ARG βitalic_r end_ARG ritalic_ϕ ( italic_x , italic_yitalic_i ) end_ARG start_ARG ∑j = 1K edivide start_ARG 1 end_ARG start_ARG βitalic_r end_ARG ritalic_ϕ ( italic_x , italic_yitalic_j ) end_ARG . (12) Finally, we translate the original objective of reward maximization lhfβr⁢(πθβπ)superscriptsubscriptlhfsubscriptsuperscriptsubscriptsubscriptJ_ lhf _r( _θ _π)Jlhfitalic_βitalic_r ( πitalic_θitalic_βitalic_π ) into the reverse KL between pfθsubscriptsubscriptp_f_θpitalic_f start_POSTSUBSCRIPT θ end_POSTSUBSCRIPT and prϕsubscriptsubscriptitalic-ϕp_r_φpitalic_r start_POSTSUBSCRIPT ϕ end_POSTSUBSCRIPT over 1:Ksubscript:1 y_1:Kitalic_y1 : K sampled from πsftsubscriptsft _ sftπsft: ℒexo(πθ)=∼prefπsft⁢(1:K|)[ _ exo( _θ)=E_ % x prefE_ _ sft( % y_1:K| x) [Lexo ( πitalic_θ ) = blackboard_Eitalic_x ∼ Dpref blackboard_Eπ start_POSTSUBSCRIPT sft ( italic_y1 : K | italic_x ) end_POSTSUBSCRIPT [ KL(pfθ(⋅|1:K,)∥prϕ(⋅|1:K,))]. _ KL (p_f_θ(·|% y_1:K, x)\|p_r_φ(·| y_1:K,% x) ) ].blackboard_DKL ( pitalic_f start_POSTSUBSCRIPT θ end_POSTSUBSCRIPT ( ⋅ | italic_y1 : K , italic_x ) ∥ pitalic_r start_POSTSUBSCRIPT ϕ end_POSTSUBSCRIPT ( ⋅ | italic_y1 : K , italic_x ) ) ] . (13) The complete form of ℒexosubscriptℒexoL_ exoLexo is presented in Eq. (23) in Appendix A.3. Besides its practical simplicity for implementation, we also elucidate its theoretical attributes by characterizing its connection with the generalized alignment objective in Theorem 3.2. Theorem 3.2. Following πβr∗superscriptsubscriptsubscript _ _r^*πitalic_β start_POSTSUBSCRIPT r end_POSTSUBSCRIPT∗, πθβπsuperscriptsubscriptsubscript _θ _πitalic_θitalic_βitalic_π and lhfβr⁢(πθβπ)superscriptsubscriptlhfsubscriptsuperscriptsubscriptsubscriptJ_ lhf _r( _θ _π)Jlhfitalic_βitalic_r ( πitalic_θitalic_βitalic_π ) defined in Eq. (4), (8), and (7), for K→∞→K→∞K → ∞ and arbitary θ, the gradient of ℒexo⁢(πθ)subscriptℒexosubscriptL_ exo( _θ)Lexo ( πitalic_θ ) satisfies ∇θℒexo⁢(πθ)subscript∇subscriptℒexosubscript _θL_ exo( _θ)∇θ Lexo ( πitalic_θ ) =∇θ∼pref[KL(πθβπ(|)∥πβr∗(|))] \!\!=\!\! _θE_ x % pref\! [D_ KL\!( _θ _π% ( y| x)\|π^*_ _r( y|% x)\!)\! ]\!= ∇θ blackboard_Eitalic_x ∼ Dpref [ blackboard_DKL ( πitalic_θitalic_βitalic_π ( italic_y | italic_x ) ∥ π∗italic_β start_POSTSUBSCRIPT r end_POSTSUBSCRIPT ( italic_y | italic_x ) ) ] (14) =−1βr⁢∇θlhfβr⁢(πθβπ).absent1subscriptsubscript∇superscriptsubscriptlhfsubscriptsuperscriptsubscriptsubscript \!\!=\!- 1 _r _θJ_ % lhf _r( _θ _π).= - divide start_ARG 1 end_ARG start_ARG βitalic_r end_ARG ∇θ Jlhfitalic_βitalic_r ( πitalic_θitalic_βitalic_π ) . (15) Theorem 3.2 suggests the optimization direction for πθsubscript _θπitalic_θ during the intermediate optimization steps for minimizing ℒexo⁢(πθ)subscriptℒexosubscriptL_ exo( _θ)Lexo ( πitalic_θ ) aligns with the direction required to maximize the generalized alignment objective lhfβr⁢(πθβπ)superscriptsubscriptlhfsubscriptsuperscriptsubscriptsubscriptJ_ lhf _r( _θ _π)Jlhfitalic_βitalic_r ( πitalic_θitalic_βitalic_π ) asymptotically, when sufficient sample population is provided. In §4.1, we show that EXO achieves encouraging convergence in practical scenarios with only a finite K. Again, as a special case, ∇θℒexo⁢(πθ)subscript∇subscriptℒexosubscript _θL_ exo( _θ)∇θ Lexo ( πitalic_θ ) aligns with ∇θlhfβ⁢(πθ)subscript∇superscriptsubscriptlhfsubscript _θJ_ lhf^β( _θ)∇θ Jlhfitalic_β ( πitalic_θ ) when βπ=1,βr=βformulae-sequencesubscript1subscript _π=1, _r=βitalic_π = 1 , βitalic_r = β, which effectively connects with Eq. (3). The result in Theorem 3.2 is crucial, as it sheds light on exact optimization of the generalized alignment objective via a simple density matching approach with strong theoretical guarantees. In Appendix A.5, we provide a mechanistic understanding of the gradient ∇θℒexo⁢(πθ)subscript∇subscriptℒexosubscript _θL_ exo( _θ)∇θ Lexo ( πitalic_θ ) which is a weighted sum of the gradients ∇θlog⁡πθ⁢(k|)subscript∇subscriptconditionalsubscript _θ _θ( y_k| x)∇θ log πitalic_θ ( italic_yitalic_k | italic_x ). The weight is proportional to KL⁢(pfθ∥prϕ)subscriptKLconditionalsubscriptsubscriptsubscriptsubscriptitalic-ϕD_ KL(p_f_θ\|p_r_φ)blackboard_DKL ( pitalic_f start_POSTSUBSCRIPT θ end_POSTSUBSCRIPT ∥ pitalic_r start_POSTSUBSCRIPT ϕ end_POSTSUBSCRIPT ) which characterizes the distributional gap biased by a point-wise correction on the sample ksubscript y_kitalic_yitalic_k. ∙ ∙ Learning from human preference data. In situations where only preference data is accessible, we devise an empirical formulation of ℒexosubscriptℒexoL_ exoLexo. Given a preference dataset prefsuperscriptprefD prefDpref where each prompt xitalic_x is paired with wsubscript y_witalic_yitalic_w and lsubscript y_litalic_yitalic_l denoting the chosen and rejected completions. This binary supervision can be effectively transformed into an empirical distribution of prhsubscriptsubscriptℎp_r_hpitalic_r start_POSTSUBSCRIPT h end_POSTSUBSCRIPT defined by the underlying reward rhsubscriptℎr_hritalic_h of human preference. To avoid infinity when calculating KL divergence, we smooth the one-hot distribution into a soft distribution, i.e., prh⁢(w|w,l,)=1−εsubscriptsubscriptℎconditionalsubscriptsubscript1p_r_h(w| y_w, y_l, x)=1- _r start_POSTSUBSCRIPT h end_POSTSUBSCRIPT ( w | italic_yitalic_w , italic_yitalic_l , italic_x ) = 1 - ε and prh⁢(l|w,l,)=εsubscriptsubscriptℎconditionalsubscriptsubscriptp_r_h(l| y_w, y_l, x)= _r start_POSTSUBSCRIPT h end_POSTSUBSCRIPT ( l | italic_yitalic_w , italic_yitalic_l , italic_x ) = ε, where ε>00 >0ε > 0 is a hyperparameter. pfθsubscriptsubscriptp_f_θpitalic_f start_POSTSUBSCRIPT θ end_POSTSUBSCRIPT can still be computed according to Eq. (11) over wsubscript y_witalic_yitalic_w and lsubscript y_litalic_yitalic_l. As a result, we present the EXO objective on the prefence data by setting K=22K=2K = 2 and substituting rϕsubscriptitalic-ϕr_φritalic_ϕ with rhsubscriptℎr_hritalic_h in Eq. (3.2): ℒexo-pref(πθ)=(,w,l)∼pref[ _ exo-pref( _θ)=E_(% x, y_w, y_l) % pref [Lexo-pref ( πitalic_θ ) = blackboard_E( italic_x , italic_y start_POSTSUBSCRIPT w , italic_yitalic_l ) ∼ Dpref end_POSTSUBSCRIPT [ KL(pfθ(⋅|w,l,)∥prh(⋅|w,l,))]. _ KL (p_f_θ(·|% y_w, y_l, x)\|p_r_h(·|% y_w, y_l, x) ) ].blackboard_DKL ( pitalic_f start_POSTSUBSCRIPT θ end_POSTSUBSCRIPT ( ⋅ | italic_yitalic_w , italic_yitalic_l , italic_x ) ∥ pitalic_r start_POSTSUBSCRIPT h end_POSTSUBSCRIPT ( ⋅ | italic_yitalic_w , italic_yitalic_l , italic_x ) ) ] . (16) In practice, πsftsubscriptsft _ sftπsft is fine-tuned on either sftsuperscriptsftD sftDsft which is collected from the same domain as prefsuperscriptprefD prefDpref, or the chosen completions in prefsuperscriptprefD prefDpref when sftsuperscriptsftD sftDsft is not available. This closes the distributional gap between πsftsubscriptsft _ sftπsft and the unknown distribution that generates the preference data prefsuperscriptprefD prefDpref. 3.3 Comparing with DPO under the Perspective of Probability Matching Under the perspective of probability matching, we formally demonstrate that the DPO objective corresponds to the forward KL which is essentially different from the reverse KL required by the alignment objective lhfβ⁢(πθ)superscriptsubscriptlhfsubscriptJ_ lhf^β( _θ)Jlhfitalic_β ( πitalic_θ ) in Eq. (3). We then analyze their differences under realistic constraints of model capacities. We first consider the general form of the DPO objective. Given K completions 1:K=1,⋯,Ksubscript:1subscript1⋯subscript y_1:K=\ y_1,·s, y_K\italic_y1 : K = italic_y1 , ⋯ , italic_yitalic_K drawn from πsft⁢(|)subscriptsftconditional _ sft( y| x)πsft ( italic_y | italic_x ) and a reward model rϕsubscriptitalic-ϕr_φritalic_ϕ that captures human prefence, we generalize ℒdposubscriptℒdpoL_ dpoLdpo by substituting the sigmoid function with softmax over K responses and replacing the one-hot label with a soft distribution defined by rϕsubscriptitalic-ϕr_φritalic_ϕ: ℒdpo-rw(πθ)=∼prefπsft⁢(1:K|)[−∑i=1K _ dpo-rw( _θ)=E_% x prefE_ _ sft% ( y_1:K| x) [- _i=1^K -rw ( πitalic_θ ) = blackboard_Eitalic_x ∼ Dpref blackboard_Eπ start_POSTSUBSCRIPT sft ( italic_y1 : K | italic_x ) end_POSTSUBSCRIPT [ - ∑i = 1K e1βr⁢rϕ⁢(,i)∑j=1Ke1βr⁢rϕ⁢(,j)logeβπ⁢log⁡πθ⁢(i|)πsft⁢(i|)∑j=1Keβπ⁢log⁡πθ⁢(j|)πsft⁢(j|)]. e 1 _rr_φ( x, % y_i) _j=1^Ke 1 _rr_φ( x,% y_j) e _π _θ(% y_i| x) _ sft( y_i|% x) _j=1^Ke _π _θ(% y_j| x) _ sft( y_j|% x) ].divide start_ARG edivide start_ARG 1 end_ARG start_ARG βitalic_r end_ARG ritalic_ϕ ( italic_x , italic_yitalic_i ) end_ARG start_ARG ∑j = 1K edivide start_ARG 1 end_ARG start_ARG βitalic_r end_ARG ritalic_ϕ ( italic_x , italic_yitalic_j ) end_ARG log divide start_ARG eitalic_βitalic_π log divide start_ARG πitalic_θ ( italic_yitalic_i | italic_x ) end_ARG start_ARG πsft ( italic_yitalic_i | italic_x ) end_ARG end_ARG start_ARG ∑j = 1K eitalic_βitalic_π log divide start_ARG πitalic_θ ( italic_yitalic_j | italic_x ) end_ARG start_ARG πsft ( italic_yitalic_j | italic_x ) end_ARG end_ARG ] . (17) Upon substituting rϕsubscriptitalic-ϕr_φritalic_ϕ with rhsubscriptℎr_hritalic_h and setting K=22K=2K = 2, ℒdpo-rwsubscriptℒdpo-rwL_ dpo-rwLdpo-rw simplifies to ℒdposubscriptℒdpoL_ dpoLdpo. In the following, we build connection of ℒdpo-rw⁢(πθ)subscriptℒdpo-rwsubscriptL_ dpo-rw( _θ)Ldpo-rw ( πitalic_θ ) to the forward KL divegence KL⁢(πβr∗∥πθβπ)subscriptKLconditionalsuperscriptsubscriptsubscriptsuperscriptsubscriptsubscriptD_ KL( _ _r^*\| _θ _π)blackboard_DKL ( πitalic_β start_POSTSUBSCRIPT r end_POSTSUBSCRIPT∗ ∥ πitalic_θitalic_βitalic_π ). Theorem 3.3. With πβr∗superscriptsubscriptsubscript _ _r^*πitalic_β start_POSTSUBSCRIPT r end_POSTSUBSCRIPT∗, πθβπsuperscriptsubscriptsubscript _θ _πitalic_θitalic_βitalic_π and prϕsubscriptsubscriptitalic-ϕp_r_φpitalic_r start_POSTSUBSCRIPT ϕ end_POSTSUBSCRIPT defined in Eq. (4), (8) and (12) respectively, for K→∞→K→∞K → ∞ and arbitary θ, the gradient of ℒdpo-rw⁢(πθ)subscriptℒdpo-rwsubscriptL_ dpo-rw( _θ)Ldpo-rw ( πitalic_θ ) satisfies ∇θℒdpo-rw⁢(πθ)subscript∇subscriptℒdpo-rwsubscript _θL_ dpo-rw( _θ)∇θ Ldpo-rw ( πitalic_θ ) =∇θ∼pref[ = _θE_ x ^% pref [= ∇θ blackboard_Eitalic_x ∼ Dpref [ KL(πβr∗(|)∥πθβπ(|))]. _ KL(π^*_ _r( y|% x)\| _θ _π( y| x))% ].blackboard_DKL ( π∗italic_β start_POSTSUBSCRIPT r end_POSTSUBSCRIPT ( italic_y | italic_x ) ∥ πitalic_θitalic_βitalic_π ( italic_y | italic_x ) ) ] . (18) In the following, we consider βπ=1,βr=βformulae-sequencesubscript1subscript _π=1, _r=βitalic_π = 1 , βitalic_r = β to simplify the analysis, while the results still hold for general settings. Putting the results of Theorem 3.2 and Theorem 3.3 together, we readily connect ℒexosubscriptℒexoL_ exoLexo and ℒdpo-rwsubscriptℒdpo-rwL_ dpo-rwLdpo-rw with two divergences, i.e., the reverse KL, KL⁢(πθ∥πβ∗)subscriptKLconditionalsubscriptsubscriptsuperscriptD_ KL( _θ\|π^*_β)blackboard_DKL ( πitalic_θ ∥ π∗italic_β ), and the forward KL, KL⁢(πβ∗∥πθ)subscriptKLconditionalsubscriptsuperscriptsubscriptD_ KL(π^*_β\| _θ)blackboard_DKL ( π∗italic_β ∥ πitalic_θ ), respectively. To provide a clear image, we illustrate the interrelationship among the objectives of EXO, DPO, and the objective of alignment in Figure 5, as presented in Appendix A.6. Although minimizing these two divergences leads to the same analytic solution πβ∗superscriptsubscript _β^*πitalic_β∗, they converge to two distinct solutions when considering the expressivity bottleneck of the practical model parametrization. Specifically, the LM policy πθsubscript _θπitalic_θ is commonly parametrized as an auto-regressive (AR) model, which enables efficient sampling due to the employment of local normalization. However, the optimal policy πβ∗⁢(|)∝πsft⁢(|)⁢exp⁡(1β⁢rϕ⁢(,))proportional-tosuperscriptsubscriptconditionalsubscriptsftconditional1subscriptitalic-ϕ _β^*( y| x) _ sft(% y| x) ( 1βr_φ( x,% y))πitalic_β∗ ( italic_y | italic_x ) ∝ πsft ( italic_y | italic_x ) exp ( divide start_ARG 1 end_ARG start_ARG β end_ARG ritalic_ϕ ( italic_x , italic_y ) ) defined as an EBM trade-offs sampling efficiency with modeling capacity. Notably, Lin et al. (2021) rigorously proved that AR models cannot perfectly capture all possible distributions defined by EBMs in terms of supports or rankings based on the computational complexity theories. From an empirical view, this result is also intuitive because the reward model as a discriminator is more flexible in distribution modeling than the auto-regressive generator. Under the practical constraint that πθsubscript _θπitalic_θ cannot perfectly represent πβ∗superscriptsubscript _β^*πitalic_β∗, minimizing the forward and reverse KL results in two policies πθFKLsubscriptsubscriptFKL _ _ FKLπitalic_θ start_POSTSUBSCRIPT FKL end_POSTSUBSCRIPT and πθRKLsubscriptsubscriptRKL _ _ RKLπitalic_θ start_POSTSUBSCRIPT RKL end_POSTSUBSCRIPT with different properties. One well-known fact is that πθFKLsubscriptsubscriptFKL _ _ FKLπitalic_θ start_POSTSUBSCRIPT FKL end_POSTSUBSCRIPT is mean-seeking while πθRKLsubscriptsubscriptRKL _ _ RKLπitalic_θ start_POSTSUBSCRIPT RKL end_POSTSUBSCRIPT is mode-seeking (Bishop & Nasrabadi, 2006). In Figure 1, we consider an illustrative picture of fitting a unimodal πθsubscript _θπitalic_θ to a multi-modal target πβ∗superscriptsubscript _β^*πitalic_β∗. To maintain a minimal forward KL divergence, πθFKLsubscriptsubscriptFKL _ _ FKLπitalic_θ start_POSTSUBSCRIPT FKL end_POSTSUBSCRIPT must encompass all the modes of πβ∗superscriptsubscript _β^*πitalic_β∗ (regions where πβ∗superscriptsubscript _β^*πitalic_β∗ exhibits significant values). However due to the representational constraints, πθFKLsubscriptsubscriptFKL _ _ FKLπitalic_θ start_POSTSUBSCRIPT FKL end_POSTSUBSCRIPT tends to overpresents the mean of different modes of πβ∗superscriptsubscript _β^*πitalic_β∗, potentially extending into the long tail region of πβ∗superscriptsubscript _β^*πitalic_β∗ (Chan et al., 2022; Ji et al., 2023a). On the other hand, πθRKLsubscriptsubscriptRKL _ _ RKLπitalic_θ start_POSTSUBSCRIPT RKL end_POSTSUBSCRIPT can select one mode of πβ∗superscriptsubscript _β^*πitalic_β∗ without causing the reverse KL to explode, meaning that πθRKLsubscriptsubscriptRKL _ _ RKLπitalic_θ start_POSTSUBSCRIPT RKL end_POSTSUBSCRIPT will effectively capture the major mode of πβ∗superscriptsubscript _β^*πitalic_β∗ under realistic model capacity. In §4.1, we empirically demonstrate the results of optimizing these two divergences in practice. Within the context of language model alignment, reverse KL is preferred for generating samples according to the evaluation of the optimal policy. Conversely, forward KL tradeoffs preference evaluation with sample diversity, which is rational only if the samples are valid under the evaluation. To some extent, the reverse KL can also effectively capture this rational diversity, as it maximizes the policy’s entropy to prevent distributional collapse to a single point. 4 Experiments We verify the effectiveness of EXO via extensive experimentations. In §4.1, we systematically study the frontier of reward maximization and KL minimization achieved by different alignment methods in a controlled text generation task following previous works (Ziegler et al., 2019; Rafailov et al., 2023). We investigate two different settings, including learning directly from preference data governed by a predefined oracle reward model, and 2) learning from a reward model estimated from the preference data. In §4.2, we compare EXO against different approaches on realistic alignment problems including generating human-preferred summaries, helpful dialogue responses, and answers that follow human instructions. Under both settings of learning directly from the preference data and from a reward model, EXO outperforms existing alignment approaches, e.g., DPO and PPO. Next, we briefly describe the experiment settings and leave additional details in Appendix B. Experiment Setting. Our experiments consider two practical settings of LM alignment: (i) Directly train on a preference dataset pref=(j),w(j),l(j)j=1Nsuperscriptprefsuperscriptsubscriptsuperscriptsuperscriptsubscriptsuperscriptsubscript1D pref=\ x^(j), y_w^(j),% y_l^(j)\_j=1^NDpref = italic_x( j ) , italic_yitalic_w( j ) , italic_yitalic_l( j ) j = 1N where wsubscript y_witalic_yitalic_w and lsubscript y_litalic_yitalic_l are the chosen and rejected responses judged by an oracle reward model or human labelers. (i) Train on a reward dataset rw=(j),(1(j),r1(j)),⋯,(K(j),rK(j))j=1Nsuperscriptrwsuperscriptsubscriptsuperscriptsuperscriptsubscript1superscriptsubscript1⋯superscriptsubscriptsuperscriptsubscript1D rw=\ x^(j),( y_1^(j),r_% 1^(j)),·s,( y_K^(j),r_K^(j))\_j=1^NDrw = italic_x( j ) , ( italic_y1( j ) , r1( j ) ) , ⋯ , ( italic_yitalic_K( j ) , ritalic_K( j ) ) j = 1N where ksubscript y_kitalic_yitalic_k is generated by the SFT policy and rksubscriptr_kritalic_k is a scalar reward provided by a reward model estimated on the given preference dataset. In the controlled text generation task, the policy is optimized to generate a completion yitalic_y with positive sentiment given a prefix xitalic_x of a movie review from the IMDB dataset222https://huggingface.co/datasets/imdb. (Maas et al., 2011). To systematically evaluate the alignment performance, we train a binary sentiment classifier on the IMDB dataset and define the oracle reward as its log odds following Ziegler et al. (2019). Both the policy and the reward models are initialized from the GPT-2 large model (Radford et al., 2019). In the summarization task, the policy is required to generate a summary yitalic_y of the post xitalic_x from the Reddit forum that is preferred by human annotators. Following Stiennon et al. (2020), we use the same filtered version333https://huggingface.co/datasets/UCL-DARK/openai-tldr-filtered. of the Reddit TL;DR summarization dataset (Völske et al., 2017) to train the SFT policy and use their preference dataset444https://huggingface.co/datasets/openai/summarize_from_feedback. for the alignment problem. In the dialogue generation task, the policy is learned to generate a helpful response yitalic_y given multi-turn dialogue history between the user and the assistant denoted as xitalic_x. We use the helpfulness subset of the Anthropic Helpful and Harmless dialogue dataset555https://huggingface.co/datasets/Anthropic/h-rlhf. (Bai et al., 2022) as the preference dataset and train the SFT policy using the chosen responses. For summarization and dialogue generation tasks, we initialize both the policy and the reward model from the Pythia-2.8B (Biderman et al., 2023) following Rafailov et al. (2023). To ensure sample quality, we use a temperature of τ=0.80.8τ=0.8τ = 0.8 to divide the logits of the language model in all experiments. Lastly, for the instruction following task, we create a dataset based on instructions with high demand and representativeness from the real-world application scenarios, featuring 83K pairs of preferences annotated by human labelers and 49K prompts for policy training. The average lengths of the instructions and the answers are 47 and 230 respectively. We curate a diverse set of high-quality test instructions to assess a range of capabilities of the learned LM policy, encompassing multilingual ability, creative writing, open-ended question answering, and role playing. Each category takes the same proportion in the test set. Both the policy and the reward models are initialized from ChatGLM2-6B (Du et al., 2022). Evaluation. In the controlled text generation task, we evaluate the frontier of the oracle reward and the KL divergence achieved by different approaches. This enables us to systematically compare the effectiveness of different methods in maxmizing the oracle reward under the same distributional shift constrained by the reverse KL. For experiments on the public preference datasets of summarization and dialogue generation, we use the reward model trained on the preference dataset as an in-domain proxy of the unknown ground-truth reward and also query GPT-4 for zero-shot pair-wise evaluation, which is shown to be consistent with human judgments (Rafailov et al., 2023). The prompts for GPT-4 evaluation are slightly modified based on those used in Rafailov et al. (2023), as detailed in Appendix B. We compare the generated outputs against those generated by the SFT policy and the preferred choice in the preference dataset. For the instruction-following task, we report the win rate of directly comparing our method against various baselines as judged by GPT-4. Additionally, we employ human assessment to evaluate criteria including adherence to instruction, correctness, fluency, safety and helpfulness. Methods for Comparison. We consider the following methods for aligning language models with human preferences under various settings. Under the setting of learning directly from preferences, we consider the special case of EXO for preference learning ℒexo-prefsubscriptℒexo-prefL_ exo-prefLexo-pref (Eq. (3.2)) denoted as EXOprefsubscriptEXOpref EXO_ prefEXOpref, and the standard DPO (Rafailov et al., 2023) that minimizes the pair-wise loss ℒdposubscriptℒdpoL_ dpoLdpo (Eq. (2.3)) on the preference data, which we denoted as DPOprefsubscriptDPOpref DPO_ prefDPOpref. Then we consider the setup of alignment with a reward model estimated from the preference dataset, which includes the RL algorithm PPO (Ziegler et al., 2019) that optimizes the expected reward with a KL penalty (Eq. (3)), the general EXO objective ℒexosubscriptℒexoL_ exoLexo (Eq. (3.2)) that performs probability matching by minimizing reverse KL, which is denoted as EXOrwsubscriptEXOrw EXO_ rwEXOrw, the general DPO objective ℒdpo-rwsubscriptℒdpo-rwL_ dpo-rwLdpo-rw (Eq. (3.3)) that minimizes the forward KL, which is denoted as DPOrwsubscriptDPOrw DPO_ rwDPOrw, and the Best-of-N method which first samples N=128128N=128N = 128 outputs from the SFT policy and then returns the response with the highest score according to the reward model. Note that the Best-of-N baseline is practically inefficient and can be regarded as an upperbound of exploiting the SFT policy according to the reward model in Eq. (4) by trading off the computation. 4.1 Alignment with the Oracle Reward Figure 2: The frontier of oracle reward vs reverse KL to the SFT policy of different methods in the controlled experiment. To avoid undesirable reward overoptimizaion due to distributional shift, a preferred alignment solution should return a policy that obtains high oracle reward while incuring minimum deviation from πsftsubscriptsft _ sftπsft. Thereby, we plot the frontier of the oracle reward against KL divergence in Figure 2. We additionally present the accuracy-KL frontier in Figure 7 in Appendix C.3. Each point represents a checkpoint of the learned policy which is evaluated on 512 prefixes from the test set to complete the response with maximumly 512 tokens. We sample M=44M=4M = 4 completions ii=1Msuperscriptsubscriptsubscript1\ y_i\_i=1^M italic_yitalic_i i = 1M for each given prompt xitalic_x to calculate the average oracle reward as well as to reduce the variance of approximating the sequence-level KL divergence KL⁢(πθ∥πsft)≈1M⁢∑i=1Mlog⁡πθ⁢(i|)−log⁡πsft⁢(i|)subscriptKLconditionalsubscriptsubscriptsft1superscriptsubscript1subscriptconditionalsubscriptsubscriptsftconditionalsubscriptD_ KL( _θ\| _ sft)≈ 1M% _i=1^M _θ( y_i| x)- _% sft( y_i| x)blackboard_DKL ( πitalic_θ ∥ πsft ) ≈ divide start_ARG 1 end_ARG start_ARG M end_ARG ∑i = 1M log πitalic_θ ( italic_yitalic_i | italic_x ) - log πsft ( italic_yitalic_i | italic_x ). Despite aiming to optimize the same alignment objective, the EXO approaches (EXOprefsubscriptEXOpref EXO_ prefEXOpref and EXOrwsubscriptEXOrw EXO_ rwEXOrw) yield the most efficient frontiers in their respective settings , evidenced by consistently achieving higher oracle rewards than baselines under the same KL divergence. Specifically, in the setting of directly learning from preference pairs, EXOprefsubscriptEXOpref EXO_ prefEXOpref outperforms DPOprefsubscriptDPOpref DPO_ prefDPOpref by a large margin, which underscores EXO’s better sample efficiency when learning from a constrained source of preference indicators. As K increases, the frontier of EXOrwsubscriptEXOrw EXO_ rwEXOrw begins to exhibit convergence, indicating an effective progression towards the intended solution within a finite K. Although DPOrwsubscriptDPOrw DPO_ rwDPOrw also improves over DPOprefsubscriptDPOpref DPO_ prefDPOpref when K=44K=4K = 4, the frontier becomes worse when K is further increased to 8. This result substantiates our analysis about the mean-seeking behavior of forward KL, which leads to inaccuracy in capturing the modes of the complex target distribution. Finally, we illustrate the strong optimization efficiency of EXO in Figure 8, evidenced by consistently achieving high and stable oracle rewards within fewer number of training steps compared with PPO and DPO in Appendix C.4. Figure 3: Visualization of the estimated density ratio between the optimal and learned policy by EXO and DPO and the SFT policy on samples from the SFT policy sorted by their log probabilities. Next, we compare DPO and EXO from the probability matching perspective by visualizing the probability density of the policies obtained by these two approaches666We consider the setting of learning directly from preferences.. In Figure 3, we plot the estimated density ratio of the optimal and learned policies by EXO and DPO against πsftsubscriptsft _ sftπsft given a randomly chosen test prompt “This Fox spectacle was a big hit when released in”. Since the probability density of an LM policy is defined over a high dimensional space of =nsuperscriptY=V^nY = Vitalic_n, it is intractable to evaluate every point in this space exhaustively. Thus, we consider the representative data points that are sampled from πsftsubscriptsft _ sftπsft, and sort them in the ascending order of their log probabilities. Then we compute the empirical distribution under the learned policies over these samples. Formally, given M=256256M=256M = 256 samples ii=1Msuperscriptsubscriptsubscript1\ y_i\_i=1^M italic_yitalic_i i = 1M drawn from πsftsubscriptsft _ sftπsft conditioned on the prompt xitalic_x, the empirical distribution π^θsubscript π_θover start_ARG π end_ARGθ is calculated via self-normalized importance sampling over the learned policy πθsubscript _θπitalic_θ: π^θ⁢(i|)=M⁢πθ⁢(i|)∑j=1Mπθ⁢(j|)/πsft⁢(j|).subscript^conditionalsubscriptsubscriptconditionalsubscriptsuperscriptsubscript1subscriptconditionalsubscriptsubscriptsftconditionalsubscript π_θ( y_i| x)= M _θ(% y_i| x) _j=1^M _θ( y_% j| x)/ _ sft( y_j| x).over start_ARG π end_ARGθ ( italic_yitalic_i | italic_x ) = divide start_ARG M πitalic_θ ( italic_yitalic_i | italic_x ) end_ARG start_ARG ∑j = 1M πitalic_θ ( italic_yitalic_j | italic_x ) / πsft ( italic_yitalic_j | italic_x ) end_ARG . For the optimal policy, the empirical distribution reduces to: π^β∗⁢(i|)=M⁢πsft⁢(i|)⁢exp⁡(r⁢(,i)/β)∑j=1Mexp⁡(r⁢(,j)/β).subscriptsuperscript^conditionalsubscriptsubscriptsftconditionalsubscriptsubscriptsuperscriptsubscript1subscript π^*_β( y_i| x)= M _ % sft( y_i| x) (r( x, y% _i)/β) _j=1^M (r( x, y_j)/β% ).over start_ARG π end_ARG∗β ( italic_yitalic_i | italic_x ) = divide start_ARG M πsft ( italic_yitalic_i | italic_x ) exp ( r ( italic_x , italic_yitalic_i ) / β ) end_ARG start_ARG ∑j = 1M exp ( r ( italic_x , italic_yitalic_j ) / β ) end_ARG . Finally, we use kernel density estimation to estimate the probability density π^⁢(|)^conditional π( y| x)over start_ARG π end_ARG ( italic_y | italic_x ) of the empirical distribution and plot the density ratio ρπ^⁢(|)=π^⁢(|)πsft⁢(|)subscript^conditional^conditionalsubscriptsftconditional _ π( y| x)= π( y% | x) _ sft( y| x)ρover start_ARG π end_ARG ( italic_y | italic_x ) = divide start_ARG over start_ARG π end_ARG ( italic_y | italic_x ) end_ARG start_ARG πsft ( italic_y | italic_x ) end_ARG against the log probability of the data points under πsftsubscriptsft _ sftπsft. From the result, the density ratio of the EXO policy closely matches the optimal policy at the high probability region against πsftsubscriptsft _ sftπsft, which reflects its ability in capturing the major modes of the optimal policy. However, the DPO policy overestimates the long tail in πsftsubscriptsft _ sftπsft due to the mean-seeking tendency of forward KL, resulting in a reduced accuracy in fitting the modes of the optimal policy. We present more visualization results in Appendix C.5. 4.2 Alignment with Human Preferences Method Reward Model (%) GPT-4 (%) vs SFT vs Chosen vs SFT vs Chosen w/ Preferences DPOprefsubscriptDPOpref DPO_ prefDPOpref 68.3 23.7 57.0 30.5 EXOprefsubscriptEXOpref EXO_ prefEXOpref 92.5 60.1 83.0 55.0 w/ Reward Model Best-of-N 99.3 75.8 83.5 60.0 PPO 93.2 58.3 77.0 52.0 DPOrwsubscriptDPOrw DPO_ rwDPOrw 82.7 39.8 70.0 41.0 EXOrwsubscriptEXOrw EXO_ rwEXOrw 97.3 76.4 88.5 64.0 Table 1: Win rates against the SFT generated texts and the chosen texts on the TL;DR summarization dataset. Best results from the computationally efficient methods are highlighted in boldface. Next, we undertake a series of experiments focused on learning from real human preferences. In the tasks of summarization and dialogue generation, we consider the two settings of learning directly from preference data and from a reward model respectively. We set the same hyperparameters (e.g., βπsubscript _πβitalic_π, βrsubscript _rβitalic_r) for EXO and DPO across different settings and datasets, and provide the results of tuning these hyperparameters in Appendix C.1 and C.2. Evaluation results on text summaization and dialogue generation are shown in Table 1 and Table 2 respectively. Upon comparison with both the SFT and chosen responses, EXO demonstrates notable improvement over DPO and PPO. This advantage is evident in evaluations using both the in-domain reward model and zero-shot assessment by GPT-4, across both settings of learning from preferences and from a reward model. Notably, EXO is the only practically efficient method to attain a GPT-4 win rate exceeding 60% when compared to the chosen responses that may have been produced by a more advanced language model. Although the Best-of-N baseline achieves comparable or higher reward model win rate by maximizing out the reward model, it suffers from the most significant decline of win rate when assessed by GPT-4. This drop of performance is attributed to its excessive exploitation of the imperfect reward model while neglecting the deviation from the initial SFT policy. We provide examples generated by DPO and EXO on both tasks in Appendix C.6. For the instruction-following task, we report the win rates of EXO compared to various baselines in Figure 4 under the setting of learning from the reward model given its advantageous performance observed so far. From the result, we observe that EXO outperforms all baselines by clear margins, thereby underscoring its scalability in practical applications. Notably, EXO achieves 10% and 5% improvement over its closest competitors as judged by GPT-4 and human annotators respectively. Method Reward Model (%) GPT-4 (%) vs SFT vs Chosen vs SFT vs Chosen w/ Preferences DPOprefsubscriptDPOpref DPO_ prefDPOpref 66.3 65.1 58.0 37.0 EXOprefsubscriptEXOpref EXO_ prefEXOpref 76.4 76.7 73.0 51.0 w/ Reward Model Best-of-N 94.6 98.2 86.0 63.0 PPO 75.0 74.0 66.5 52.0 DPOrwsubscriptDPOrw DPO_ rwDPOrw 79.9 81.3 75.5 49.0 EXOrwsubscriptEXOrw EXO_ rwEXOrw 85.6 87.2 83.5 60.0 Table 2: Win rates against the SFT generated texts and the chosen texts on the Anthropic-H dataset. Best results from the computationally efficient methods are highlighted in boldface. vs DPOvs PPOvs SFT2020202030303030404040405050505060606060Win rate (%)GPT-4 evaluation (a) vs DPOvs PPOvs SFT0010101010202020203030303040404040Human evaluationWinLose (b) Figure 4: Win rates by comparing EXO to various baselines on the instruction-following task judged by GPT-4 and human labelers. 5 Related Work Large language models (Rae et al., 2021; Hoffmann et al., 2022; Touvron et al., 2023) learned to predict the next tokens on large corpora have significantly improved the performance of various natural language processing (NLP) tasks in zero shot (Radford et al., 2019) or few-shot (Brown et al., 2020; Chowdhery et al., 2023) settings. To bridge the mismatch between the training objective and users’ objective of solving specific NLP tasks, instruction-tuning is proposed, which fine-tunes the language models on human curated instruction-response pairs in a fully supervised manner (Sanh et al., 2021; Chung et al., 2022; Longpre et al., 2023; Wang et al., 2023). Despite its performance and generalization to unseen instruction sets (Chung et al., 2022), there have been concerns that the model learned to predict the next token might only capture surficial patterns rather than comprehending the tasks (Kung & Peng, 2023). To address the aforementioned deficiencies, the framework of reinforcement learning from human feedback (RLHF) is proposed (Ziegler et al., 2019), which relies on only relative human preferences on response quality and optimizes the language model by RL algorithms (Williams, 1992), e.g., PPO (Schulman et al., 2017), under the supervision of a reward model which encapsulates the implicit preference of humans. Despite the success of RLHF in various tasks where alignment is strongly emphasized, e.g., translation (Kreutzer et al., 2018), summarization (Stiennon et al., 2020), instruction following (Ouyang et al., 2022), and etc., RL fine-tuning for large language models still faces serious challenges in stability and scalability (Zheng et al., 2023b). Thereby, a new trend of recent works proposed to optimize the language model to follow human preferences without applying RL algorithms (Yuan et al., 2023; Zhao et al., 2023; Rafailov et al., 2023). While most approaches are empirically set to model alignment as a re-ranking problem, DPO proposed a theoretically sound way to realize direct policy optimization from preference data based on the analytic optimal solution of the reward maximization objective in RLHF. Afterwards, there have been several studies that extend DPO in various ways, e.g., altering the preference data by sampling from the optimal policy via rejection sampling (Liu et al., 2023), substituting the point-wise reward with a pair-wise preference function (Azar et al., 2023), extending the preference pairs to rankings of preferences of any size (Song et al., 2023), and etc. However, these approaches are still based on the formation of DPO. In this work, we propose to learn the policy by exactly optimizing the RLHF objective via probability matching that minimizes the reverse KL. In the literature of maximum-entropy RL (Eysenbach & Levine, 2019; Korbak et al., 2022), this equivalent form of probability matching was discussed only for analysis purposes, while we are the first to derive a practical way to optimize it. We also revisit DPO under this perspective and recognize that it actually corresponds to minimizing the forward KL in its general form. To this sense, DPO shares the same spirit of weighted regression (Peters & Schaal, 2007; Peng et al., 2019; Wang et al., 2020), an algorithm that directly utilizes behavioral actions to supervise the policy in offline RL (Peters et al., 2010; Lu et al., 2023). However, this approach is known to be suboptimal when the policy model is limited in distributional expressivity (Yue et al., 2022; Chen et al., 2023). We analyze the characteristics of the probability density learned by DPO with both theoretical insight and empirical experimentations. 6 Limitations and Future Work The alignment framework proposed by Ziegler et al. (2019) relies on the KL regularization to the SFT policy to prevent the optimized policy from greedily maximizing out the reward model which is estimated from the human preference data. Despite the regularization, our experiments still reveal instances of reward over-optimization, a phenomenon possibly due to insufficient focus on the reward model estimation and use. Importantly, ensuring that the reward model accurately reflects the true modes of the oracle distribution of human is more vital for PPO and EXO that optimizes a mode-seeking objective. This opens possible avenues to improve the alignment framework at a broader scope. For instance, rather than relying on a static preference dataset for reward model training, it could be more effective to dynamically improve the reward model with the development of the policy, thereby offering more precise feedback. Upon the current regularization that solely focuses on proximity to the initial policy, one can take into account the uncertainty of the reward model output to avoid over-exploitation of the reward model. It is beneficial to take into account these aspects to develop efficient and effective method towards closer alignment with human preferences. Additionally, while we already evaluated EXO on advanced language models up to 6B on realistic scenarios, scaling EXO to models that are orders of magnitude larger can present profound implications. At the other end of the spectrum, systematically dissecting and comparing PPO, DPO and EXO, e.g., regarding their variance and bias during optimization is essential to broaden our understanding of these methods. 7 Conclusion In this work, we consider the problem of aligning language models with human preferences. Although reinforcement learning (RL) for reward maximization presents the direct and apparent solution, we reframe the problem in a supervised probability matching framework, which underscores the probabilistic interpretation of the alignment procedure. This derives our efficient exact optimization (EXO) of the KL-regularized reward maximization objective of alignment. Formally, we prove the asymptotic equivalence between the EXO objective and the alignment objective. In practice, EXO enables efficient optimization via probability matching between empirical distributions, which avoids the complexities of RL algorithms. We further demonstrate that DPO in its general form actually corresponds to minimizing the forward KL against the optimal policy, which is shown to be less effective in capturing the modes of the optimal policy under realistic model parametrization with both theoretical and empirical justifications. Finally, we demonstrate the effectiveness and scalability of EXO on various text generation tasks with real human preferences. Acknowledgements This work was supported by the NSFC projects (with No. 61936010 and No. 62306160). This work was supported by the National Science Foundation for Distinguished Young Scholars (with No. 62125604). This work was also supported by China National Postdoctoral Program for Innovative Talents (No. BX20230194) and China Postdoctoral Science Foundation (No. 2023M731952). We would also like to thank Zhipu AI for sponsoring the computation resources and annotation cost used in this work. Impact Statement This paper presents a method whose goal is to advance the alignment of language models with human preferences. This endeavor, while technically challenging, carries significant implications for the ethical use and societal impact of artificial intelligence. The goal of alignment aims to mitigate the inherent biases of AI systems and ensure that they reflect diverse human values, goals and intentions that are safe and ethical. It enhances AI’s utility in various sectors to make reliable decisions that are in line with organizational goals and ethical standards. References Anschel et al. (2017) Anschel, O., Baram, N., and Shimkin, N. Averaged-dqn: Variance reduction and stabilization for deep reinforcement learning. In International conference on machine learning, p. 176–185. PMLR, 2017. Azar et al. (2023) Azar, M. G., Rowland, M., Piot, B., Guo, D., Calandriello, D., Valko, M., and Munos, R. A general theoretical paradigm to understand learning from human preferences. CoRR, abs/2310.12036, 2023. doi: 10.48550/ARXIV.2310.12036. URL https://doi.org/10.48550/arXiv.2310.12036. Bai et al. (2022) Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., DasSarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., Joseph, N., Kadavath, S., Kernion, J., Conerly, T., Showk, S. E., Elhage, N., Hatfield-Dodds, Z., Hernandez, D., Hume, T., Johnston, S., Kravec, S., Lovitt, L., Nanda, N., Olsson, C., Amodei, D., Brown, T. B., Clark, J., McCandlish, S., Olah, C., Mann, B., and Kaplan, J. Training a helpful and harmless assistant with reinforcement learning from human feedback. CoRR, abs/2204.05862, 2022. doi: 10.48550/ARXIV.2204.05862. URL https://doi.org/10.48550/arXiv.2204.05862. Bender et al. (2021) Bender, E. M., Gebru, T., McMillan-Major, A., and Shmitchell, S. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT ’21, p. 610–623, New York, NY, USA, 2021. Association for Computing Machinery. ISBN 9781450383097. doi: 10.1145/3442188.3445922. URL https://doi.org/10.1145/3442188.3445922. Biderman et al. (2023) Biderman, S., Schoelkopf, H., Anthony, Q. G., Bradley, H., O’Brien, K., Hallahan, E., Khan, M. A., Purohit, S., Prashanth, U. S., Raff, E., et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, p. 2397–2430. PMLR, 2023. Bishop & Nasrabadi (2006) Bishop, C. M. and Nasrabadi, N. M. Pattern recognition and machine learning, volume 4. Springer, 2006. Brown et al. (2020) Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. Language models are few-shot learners. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. URL https://proceedings.neurips.c/paper/2020/hash/1457c0d6bfcb4967418bfb8ac142f64a-Abstract.html. Chan et al. (2022) Chan, A., Silva, H., Lim, S., Kozuno, T., Mahmood, A. R., and White, M. Greedification operators for policy optimization: Investigating forward and reverse kl divergences. The Journal of Machine Learning Research, 23(1):11474–11552, 2022. Chen et al. (2023) Chen, H., Lu, C., Ying, C., Su, H., and Zhu, J. Offline reinforcement learning via high-fidelity generative behavior modeling. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL https://openreview.net/pdf?id=42zs3qa2kpy. Chowdhery et al. (2023) Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1–113, 2023. Chung et al. (2022) Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, Y., Wang, X., Dehghani, M., Brahma, S., et al. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416, 2022. Du et al. (2022) Du, Z., Qian, Y., Liu, X., Ding, M., Qiu, J., Yang, Z., and Tang, J. GLM: general language model pretraining with autoregressive blank infilling. In Muresan, S., Nakov, P., and Villavicencio, A. (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, p. 320–335. Association for Computational Linguistics, 2022. doi: 10.18653/V1/2022.ACL-LONG.26. URL https://doi.org/10.18653/v1/2022.acl-long.26. Eysenbach & Levine (2019) Eysenbach, B. and Levine, S. If maxent rl is the answer, what is the question? arXiv preprint arXiv:1910.01913, 2019. Gao et al. (2023) Gao, L., Schulman, J., and Hilton, J. Scaling laws for reward model overoptimization. In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J. (eds.), International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, p. 10835–10866. PMLR, 2023. URL https://proceedings.mlr.press/v202/gao23h.html. Hoffmann et al. (2022) Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., de Las Casas, D., Hendricks, L. A., Welbl, J., Clark, A., Hennigan, T., Noland, E., Millican, K., van den Driessche, G., Damoc, B., Guy, A., Osindero, S., Simonyan, K., Elsen, E., Rae, J. W., Vinyals, O., and Sifre, L. Training compute-optimal large language models. CoRR, abs/2203.15556, 2022. doi: 10.48550/ARXIV.2203.15556. URL https://doi.org/10.48550/arXiv.2203.15556. Ji et al. (2023a) Ji, H., Ke, P., Hu, Z., Zhang, R., and Huang, M. Tailoring language generation models under total variation distance. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023a. URL https://openreview.net/pdf?id=VELL0PlWfc. Ji et al. (2023b) Ji, H., Ke, P., Wang, H., and Huang, M. Language model decoding as direct metrics optimization. arXiv preprint arXiv:2310.01041, 2023b. Ji et al. (2023c) Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., and Fung, P. Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12):1–38, 2023c. Korbak et al. (2022) Korbak, T., Perez, E., and Buckley, C. L. Rl with kl penalties is better viewed as bayesian inference. arXiv preprint arXiv:2205.11275, 2022. Kreutzer et al. (2018) Kreutzer, J., Uyheng, J., and Riezler, S. Reliability and learnability of human bandit feedback for sequence-to-sequence reinforcement learning. arXiv preprint arXiv:1805.10627, 2018. Kung & Peng (2023) Kung, P.-N. and Peng, N. Do models really learn to follow instructions? an empirical study of instruction tuning. arXiv preprint arXiv:2305.11383, 2023. Lin et al. (2021) Lin, C.-C., Jaech, A., Li, X., Gormley, M., and Eisner, J. Limitations of autoregressive models and their alternatives. In NAACL, 2021. Liu et al. (2023) Liu, T., Zhao, Y., Joshi, R., Khalman, M., Saleh, M., Liu, P. J., and Liu, J. Statistical rejection sampling improves preference optimization. arXiv preprint arXiv:2309.06657, 2023. Longpre et al. (2023) Longpre, S., Hou, L., Vu, T., Webson, A., Chung, H. W., Tay, Y., Zhou, D., Le, Q. V., Zoph, B., Wei, J., et al. The flan collection: Designing data and methods for effective instruction tuning. arXiv preprint arXiv:2301.13688, 2023. Lu et al. (2023) Lu, C., Chen, H., Chen, J., Su, H., Li, C., and Zhu, J. Contrastive energy prediction for exact energy-guided diffusion sampling in offline reinforcement learning. In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J. (eds.), International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, p. 22825–22855. PMLR, 2023. URL https://proceedings.mlr.press/v202/lu23d.html. Maas et al. (2011) Maas, A. L., Daly, R. E., Pham, P. T., Huang, D., Ng, A. Y., and Potts, C. Learning word vectors for sentiment analysis. In Lin, D., Matsumoto, Y., and Mihalcea, R. (eds.), The 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, Proceedings of the Conference, 19-24 June, 2011, Portland, Oregon, USA, p. 142–150. The Association for Computer Linguistics, 2011. URL https://aclanthology.org/P11-1015/. Ouyang et al. (2022) Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P. F., Leike, J., and Lowe, R. Training language models to follow instructions with human feedback. In NeurIPS, 2022. Papini et al. (2018) Papini, M., Binaghi, D., Canonaco, G., Pirotta, M., and Restelli, M. Stochastic variance-reduced policy gradient. In International conference on machine learning, p. 4026–4035. PMLR, 2018. Peng et al. (2019) Peng, X. B., Kumar, A., Zhang, G., and Levine, S. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning. CoRR, abs/1910.00177, 2019. URL http://arxiv.org/abs/1910.00177. Peters & Schaal (2007) Peters, J. and Schaal, S. Reinforcement learning by reward-weighted regression for operational space control. In Ghahramani, Z. (ed.), Machine Learning, Proceedings of the Twenty-Fourth International Conference (ICML 2007), Corvallis, Oregon, USA, June 20-24, 2007, volume 227 of ACM International Conference Proceeding Series, p. 745–750. ACM, 2007. doi: 10.1145/1273496.1273590. URL https://doi.org/10.1145/1273496.1273590. Peters et al. (2010) Peters, J., Mülling, K., and Altun, Y. Relative entropy policy search. In Fox, M. and Poole, D. (eds.), Proceedings of the Twenty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2010, Atlanta, Georgia, USA, July 11-15, 2010, p. 1607–1612. AAAI Press, 2010. doi: 10.1609/AAAI.V24I1.7727. URL https://doi.org/10.1609/aaai.v24i1.7727. Radford et al. (2019) Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019. Rae et al. (2021) Rae, J. W., Borgeaud, S., Cai, T., Millican, K., Hoffmann, J., Song, H. F., Aslanides, J., Henderson, S., Ring, R., Young, S., Rutherford, E., Hennigan, T., Menick, J., Cassirer, A., Powell, R., van den Driessche, G., Hendricks, L. A., Rauh, M., Huang, P., Glaese, A., Welbl, J., Dathathri, S., Huang, S., Uesato, J., Mellor, J., Higgins, I., Creswell, A., McAleese, N., Wu, A., Elsen, E., Jayakumar, S. M., Buchatskaya, E., Budden, D., Sutherland, E., Simonyan, K., Paganini, M., Sifre, L., Martens, L., Li, X. L., Kuncoro, A., Nematzadeh, A., Gribovskaya, E., Donato, D., Lazaridou, A., Mensch, A., Lespiau, J., Tsimpoukelli, M., Grigorev, N., Fritz, D., Sottiaux, T., Pajarskas, M., Pohlen, T., Gong, Z., Toyama, D., de Masson d’Autume, C., Li, Y., Terzi, T., Mikulik, V., Babuschkin, I., Clark, A., de Las Casas, D., Guy, A., Jones, C., Bradbury, J., Johnson, M. J., Hechtman, B. A., Weidinger, L., Gabriel, I., Isaac, W., Lockhart, E., Osindero, S., Rimell, L., Dyer, C., Vinyals, O., Ayoub, K., Stanway, J., Bennett, L., Hassabis, D., Kavukcuoglu, K., and Irving, G. Scaling language models: Methods, analysis & insights from training gopher. CoRR, abs/2112.11446, 2021. URL https://arxiv.org/abs/2112.11446. Rafailov et al. (2023) Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., and Finn, C. Direct preference optimization: Your language model is secretly a reward model. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://arxiv.org/abs/2305.18290. Sanh et al. (2021) Sanh, V., Webson, A., Raffel, C., Bach, S. H., Sutawika, L., Alyafeai, Z., Chaffin, A., Stiegler, A., Scao, T. L., Raja, A., et al. Multitask prompted training enables zero-shot task generalization. arXiv preprint arXiv:2110.08207, 2021. Schulman et al. (2017) Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. Song et al. (2023) Song, F., Bowen, Y., Li, M., Yu, H., Huang, F., Li, Y., and Wang, H. Preference ranking optimization for human alignment. ArXiv, abs/2306.17492, 2023. URL https://api.semanticscholar.org/CorpusID:259308873. Stiennon et al. (2020) Stiennon, N., Ouyang, L., Wu, J., Ziegler, D. M., Lowe, R., Voss, C., Radford, A., Amodei, D., and Christiano, P. F. Learning to summarize from human feedback. CoRR, abs/2009.01325, 2020. URL https://arxiv.org/abs/2009.01325. Touvron et al. (2023) Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023. Völske et al. (2017) Völske, M., Potthast, M., Syed, S., and Stein, B. Tl;dr: Mining reddit to learn automatic summarization. In Wang, L., Cheung, J. C. K., Carenini, G., and Liu, F. (eds.), Proceedings of the Workshop on New Frontiers in Summarization, NFiS@EMNLP 2017, Copenhagen, Denmark, September 7, 2017, p. 59–63. Association for Computational Linguistics, 2017. doi: 10.18653/V1/W17-4508. URL https://doi.org/10.18653/v1/w17-4508. Wang et al. (2023) Wang, Y., Kordi, Y., Mishra, S., Liu, A., Smith, N. A., Khashabi, D., and Hajishirzi, H. Self-instruct: Aligning language models with self-generated instructions. In Rogers, A., Boyd-Graber, J. L., and Okazaki, N. (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, p. 13484–13508. Association for Computational Linguistics, 2023. doi: 10.18653/V1/2023.ACL-LONG.754. URL https://doi.org/10.18653/v1/2023.acl-long.754. Wang et al. (2020) Wang, Z., Novikov, A., Zolna, K., Merel, J., Springenberg, J. T., Reed, S. E., Shahriari, B., Siegel, N. Y., Gülçehre, Ç., Heess, N., and de Freitas, N. Critic regularized regression. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. URL https://proceedings.neurips.c/paper/2020/hash/588cb956d6bbe67078f29f8de420a13d-Abstract.html. Williams (1992) Williams, R. J. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Mach. Learn., 8:229–256, 1992. doi: 10.1007/BF00992696. URL https://doi.org/10.1007/BF00992696. Yao et al. (2023) Yao, Z., Aminabadi, R. Y., Ruwase, O., Rajbhandari, S., Wu, X., Awan, A. A., Rasley, J., Zhang, M., Li, C., Holmes, C., Zhou, Z., Wyatt, M., Smith, M., Kurilenko, L., Qin, H., Tanaka, M., Che, S., Song, S. L., and He, Y. Deepspeed-chat: Easy, fast and affordable RLHF training of chatgpt-like models at all scales. CoRR, abs/2308.01320, 2023. doi: 10.48550/ARXIV.2308.01320. URL https://doi.org/10.48550/arXiv.2308.01320. Yuan et al. (2023) Yuan, Z., Yuan, H., Tan, C., Wang, W., Huang, S., and Huang, F. Rrhf: Rank responses to align language models with human feedback without tears. arXiv preprint arXiv:2304.05302, 2023. Yue et al. (2022) Yue, Y., Kang, B., Ma, X., Xu, Z., Huang, G., and YAN, S. Boosting offline reinforcement learning via data rebalancing. In 3rd Offline RL Workshop: Offline RL as a ”Launchpad”, 2022. URL https://openreview.net/forum?id=vOC01fqW2T. Zhao et al. (2023) Zhao, Y., Joshi, R., Liu, T., Khalman, M., Saleh, M., and Liu, P. J. Slic-hf: Sequence likelihood calibration with human feedback. arXiv preprint arXiv:2305.10425, 2023. Zheng et al. (2023a) Zheng, L., Chiang, W., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., Zhang, H., Gonzalez, J. E., and Stoica, I. Judging llm-as-a-judge with mt-bench and chatbot arena. CoRR, abs/2306.05685, 2023a. doi: 10.48550/ARXIV.2306.05685. URL https://doi.org/10.48550/arXiv.2306.05685. Zheng et al. (2023b) Zheng, R., Dou, S., Gao, S., Hua, Y., Shen, W., Wang, B., Liu, Y., Jin, S., Liu, Q., Zhou, Y., et al. Secrets of rlhf in large language models part i: Ppo. arXiv preprint arXiv:2307.04964, 2023b. Ziegler et al. (2019) Ziegler, D. M., Stiennon, N., Wu, J., Brown, T. B., Radford, A., Amodei, D., Christiano, P. F., and Irving, G. Fine-tuning language models from human preferences. CoRR, abs/1909.08593, 2019. URL http://arxiv.org/abs/1909.08593. Appendix A Proofs and Derivations A.1 Proof of Theorem 3.1 Proof. We derive the optimal πθ∗subscriptsuperscript _θ^*πitalic_θ∗ that maximizes the generalized alignment objective lhfβr⁢(πθβπ)superscriptsubscriptlhfsubscriptsuperscriptsubscriptsubscriptJ_ lhf _r( _θ _π)Jlhfitalic_βitalic_r ( πitalic_θitalic_βitalic_π ) and show that it equals to the optimal solution πβ∗superscriptsubscript _β^*πitalic_β∗ of lhfβ⁢(πθ)superscriptsubscriptlhfsubscriptJ_ lhf^β( _θ)Jlhfitalic_β ( πitalic_θ ) given unlimited model capacity. First, we restate the formation of lhfβr⁢(πθβπ)superscriptsubscriptlhfsubscriptsuperscriptsubscriptsubscriptJ_ lhf _r( _θ _π)Jlhfitalic_βitalic_r ( πitalic_θitalic_βitalic_π ): lhfβr(πθβπ)=∼pref(πθβπ⁢(|)[rϕ(,)]−βrKL[πθβπ(|)∥πsft(|)]),J_ lhf _r( _θ _π)=E_% x pref (E_ _θ% _π( y| x)[r_φ( x,% y)]- _rD_ KL[ _θ _π% ( y| x)\| _ sft( y|% x)] ),Jlhfitalic_βitalic_r ( πitalic_θitalic_βitalic_π ) = blackboard_Eitalic_x ∼ Dpref ( blackboard_Eπ start_POSTSUBSCRIPT θitalic_βitalic_π ( italic_y | italic_x ) end_POSTSUBSCRIPT [ ritalic_ϕ ( italic_x , italic_y ) ] - βitalic_r blackboard_DKL [ πitalic_θitalic_βitalic_π ( italic_y | italic_x ) ∥ πsft ( italic_y | italic_x ) ] ) , (19) where πθβπsuperscriptsubscriptsubscript _θ _πitalic_θitalic_βitalic_π is defined as: πθβπ⁢(|)∝πθ⁢(|)βπ⁢πsft⁢(|)1−βπ.proportional-tosuperscriptsubscriptsubscriptconditionalsubscriptsuperscriptconditionalsubscriptsubscriptsftsuperscriptconditional1subscript _θ _π( y| x) _θ(% y| x) _π _ sft( % y| x)^1- _π.πitalic_θitalic_βitalic_π ( italic_y | italic_x ) ∝ πitalic_θ ( italic_y | italic_x )βitalic_π πsft ( italic_y | italic_x )1 - βitalic_π . (20) Then we substitute θ with the optimal θ∗superscriptθ^*θ∗ in Eq. (20) where πθ∗subscriptsuperscript _θ^*πitalic_θ∗ maximizes lhfβr⁢(πθβπ)superscriptsubscriptlhfsubscriptsuperscriptsubscriptsubscriptJ_ lhf _r( _θ _π)Jlhfitalic_βitalic_r ( πitalic_θitalic_βitalic_π ): πθ∗βπ⁢(|)∝πθ∗⁢(|)βπ⁢πsft⁢(|)1−βπ.proportional-tosuperscriptsubscriptsuperscriptsubscriptconditionalsubscriptsuperscriptsuperscriptconditionalsubscriptsubscriptsftsuperscriptconditional1subscript _θ^* _π( y| x) _% θ^*( y| x) _π _ sft% ( y| x)^1- _π.πitalic_θ∗italic_βitalic_π ( italic_y | italic_x ) ∝ πitalic_θ∗ ( italic_y | italic_x )βitalic_π πsft ( italic_y | italic_x )1 - βitalic_π . (21) Since πθ∗βπsuperscriptsubscriptsuperscriptsubscript _θ^* _πitalic_θ∗italic_βitalic_π is also the optimal policy that maximizes lhfβr⁢(⋅)superscriptsubscriptlhfsubscript⋅J_ lhf _r(·)Jlhfitalic_βitalic_r ( ⋅ ), it should satisfy Eq. (4) which gives: πθ∗βπ⁢(|)=πβr∗⁢(|)∝πsft⁢(|)⁢e1βr⁢r⁢(,).superscriptsubscriptsuperscriptsubscriptconditionalsuperscriptsubscriptsubscriptconditionalproportional-tosubscriptsftconditionalsuperscript1subscript _θ^* _π( y| x)= _ _r% ^*( y| x) _ sft( y% | x)e 1 _rr( x, y).πitalic_θ∗italic_βitalic_π ( italic_y | italic_x ) = πitalic_β start_POSTSUBSCRIPT r end_POSTSUBSCRIPT∗ ( italic_y | italic_x ) ∝ πsft ( italic_y | italic_x ) edivide start_ARG 1 end_ARG start_ARG βitalic_r end_ARG r ( italic_x , italic_y ) . (22) Together with Eq. (21) and Eq. (22), we obtain the form of πθ∗subscriptsuperscript _θ^*πitalic_θ∗ via some simple algebra: πθ∗⁢(|)subscriptsuperscriptconditional _θ^*( y| x)πitalic_θ∗ ( italic_y | italic_x ) ∝(πβr∗⁢(|)⁢πsft⁢(|)βπ−1)1βπproportional-toabsentsuperscriptsuperscriptsubscriptsubscriptconditionalsubscriptsftsuperscriptconditionalsubscript11subscript ( _ _r^*( y| x) _% sft( y| x) _π-1) 1% _π∝ ( πitalic_β start_POSTSUBSCRIPT r end_POSTSUBSCRIPT∗ ( italic_y | italic_x ) πsft ( italic_y | italic_x )βitalic_π - 1 )divide start_ARG 1 end_ARG start_ARG βitalic_π end_ARG ∝(πsft⁢(|)⁢e1βr⁢r⁢(,))1βπ⁢πsft⁢(|)βπ−1βπproportional-toabsentsuperscriptsubscriptsftconditionalsuperscript1subscript1subscriptsubscriptsftsuperscriptconditionalsubscript1subscript ( _ sft( y| x)e^% 1 _rr( x, y)) 1 _π% _ sft( y| x) _π-1% _π∝ ( πsft ( italic_y | italic_x ) edivide start_ARG 1 end_ARG start_ARG βitalic_r end_ARG r ( italic_x , italic_y ) )divide start_ARG 1 end_ARG start_ARG βitalic_π end_ARG πsft ( italic_y | italic_x )divide start_ARG βitalic_π - 1 end_ARG start_ARG βitalic_π end_ARG ∝πsft⁢(|)1βπ⁢e1βr⁢βπ⁢r⁢(,)⁢πsft⁢(|)βπ−1βπproportional-toabsentsubscriptsftsuperscriptconditional1subscriptsuperscript1subscriptsubscriptsubscriptsftsuperscriptconditionalsubscript1subscript _ sft( y| x) % 1 _πe 1 _r _πr( x,% y) _ sft( y| x) % _π-1 _π∝ πsft ( italic_y | italic_x )divide start_ARG 1 end_ARG start_ARG βitalic_π end_ARG edivide start_ARG 1 end_ARG start_ARG βitalic_r βitalic_π end_ARG r ( italic_x , italic_y ) πsft ( italic_y | italic_x )divide start_ARG βitalic_π - 1 end_ARG start_ARG βitalic_π end_ARG ∝πsft⁢(|)⁢e1βr⁢βπ⁢r⁢(,).proportional-toabsentsubscriptsftconditionalsuperscript1subscriptsubscript _ sft( y| x)e^% 1 _r _πr( x, y).∝ πsft ( italic_y | italic_x ) edivide start_ARG 1 end_ARG start_ARG βitalic_r βitalic_π end_ARG r ( italic_x , italic_y ) . By definition, βr⁢βπ=βsubscriptsubscript _r _π=βitalic_r βitalic_π = β, then πθ∗subscriptsuperscript _θ^*πitalic_θ∗ reduces to the same form of the optimal solution of lhfβ⁢(πθ)superscriptsubscriptlhfsubscriptJ_ lhf^β( _θ)Jlhfitalic_β ( πitalic_θ ) defined in Eq. (4): πθ∗⁢(|)=πβ∗⁢(|)∝πsft⁢(|)⁢e1β⁢r⁢(,),subscriptsuperscriptconditionalsuperscriptsubscriptconditionalproportional-tosubscriptsftconditionalsuperscript1 _θ^*( y| x)= _β^*( y% | x) _ sft( y| x)e^% 1βr( x, y),πitalic_θ∗ ( italic_y | italic_x ) = πitalic_β∗ ( italic_y | italic_x ) ∝ πsft ( italic_y | italic_x ) edivide start_ARG 1 end_ARG start_ARG β end_ARG r ( italic_x , italic_y ) , which completes the proof. ∎ A.2 Derivation of Eq. (3.1) We first start by rearranging lhfβr⁢(πθβπ)superscriptsubscriptlhfsubscriptsuperscriptsubscriptsubscriptJ_ lhf _r( _θ _π)Jlhfitalic_βitalic_r ( πitalic_θitalic_βitalic_π ) into the expectation of a log ratio: lhfβr⁢(πθβπ)superscriptsubscriptlhfsubscriptsuperscriptsubscriptsubscript _ lhf _r( _θ _π)Jlhfitalic_βitalic_r ( πitalic_θitalic_βitalic_π ) =∼pref(πθβπ⁢(|)[rϕ(,)]−βrKL[πθβπ(|)∥πsft(|)]) =E_ x pref (% E_ _θ _π( y| x)[r_% φ( x, y)]- _rD_ KL[ _% θ _π( y| x)\| _ sft(% y| x)] )= blackboard_Eitalic_x ∼ Dpref ( blackboard_Eπ start_POSTSUBSCRIPT θitalic_βitalic_π ( italic_y | italic_x ) end_POSTSUBSCRIPT [ ritalic_ϕ ( italic_x , italic_y ) ] - βitalic_r blackboard_DKL [ πitalic_θitalic_βitalic_π ( italic_y | italic_x ) ∥ πsft ( italic_y | italic_x ) ] ) =∼pref⁢(πθβπ⁢(|)⁢[rϕ⁢(,)]−βr⁢πθβπ⁢(|)⁢[log⁡πθβπ⁢(|)πsft⁢(|)])absentsubscriptsimilar-tosuperscriptprefsubscriptsuperscriptsubscriptsubscriptconditionaldelimited-[]subscriptitalic-ϕsubscriptsubscriptsuperscriptsubscriptsubscriptconditionaldelimited-[]superscriptsubscriptsubscriptconditionalsubscriptsftconditional =E_ x pref (% E_ _θ _π( y| x)[r_% φ( x, y)]- _rE_ _θ^% _π( y| x) [ _θ^% _π( y| x) _ sft( % y| x) ] )= blackboard_Eitalic_x ∼ Dpref ( blackboard_Eπ start_POSTSUBSCRIPT θitalic_βitalic_π ( italic_y | italic_x ) end_POSTSUBSCRIPT [ ritalic_ϕ ( italic_x , italic_y ) ] - βitalic_r blackboard_Eπ start_POSTSUBSCRIPT θitalic_βitalic_π ( italic_y | italic_x ) end_POSTSUBSCRIPT [ log divide start_ARG πitalic_θitalic_βitalic_π ( italic_y | italic_x ) end_ARG start_ARG πsft ( italic_y | italic_x ) end_ARG ] ) =∼pref⁢(βr⁢∼πθβπ⁢(|)⁢[log⁡e1βr⁢rϕ⁢(,)]−βr⁢πθβπ⁢(|)⁢[log⁡πθβπ⁢(|)πsft⁢(|)])absentsubscriptsimilar-tosuperscriptprefsubscriptsubscriptsimilar-tosuperscriptsubscriptsubscriptconditionaldelimited-[]superscript1subscriptsubscriptitalic-ϕsubscriptsubscriptsuperscriptsubscriptsubscriptconditionaldelimited-[]superscriptsubscriptsubscriptconditionalsubscriptsftconditional =E_ x pref (% _rE_ y _θ _π(% y| x) [ e 1 _rr_φ(% x, y) ]- _rE_ _θ^% _π( y| x) [ _θ^% _π( y| x) _ sft( % y| x) ] )= blackboard_Eitalic_x ∼ Dpref ( βitalic_r blackboard_Eitalic_y ∼ π start_POSTSUBSCRIPT θitalic_βitalic_π ( italic_y | italic_x ) end_POSTSUBSCRIPT [ log edivide start_ARG 1 end_ARG start_ARG βitalic_r end_ARG ritalic_ϕ ( italic_x , italic_y ) ] - βitalic_r blackboard_Eπ start_POSTSUBSCRIPT θitalic_βitalic_π ( italic_y | italic_x ) end_POSTSUBSCRIPT [ log divide start_ARG πitalic_θitalic_βitalic_π ( italic_y | italic_x ) end_ARG start_ARG πsft ( italic_y | italic_x ) end_ARG ] ) =∼pref⁢∼πθβπ⁢(|)⁢[βr⁢log⁡πsft⁢(|)⁢e1βr⁢rϕ⁢(,)πθβπ⁢(|)]absentsubscriptsimilar-tosuperscriptprefsubscriptsimilar-tosuperscriptsubscriptsubscriptconditionaldelimited-[]subscriptsubscriptsftconditionalsuperscript1subscriptsubscriptitalic-ϕsuperscriptsubscriptsubscriptconditional =E_ x pref% E_ y _θ _π( y|% x) [ _r _ sft( % y| x)e 1 _rr_φ( x, % y) _θ _π( y| x) ]= blackboard_Eitalic_x ∼ Dpref blackboard_Eitalic_y ∼ π start_POSTSUBSCRIPT θitalic_βitalic_π ( italic_y | italic_x ) end_POSTSUBSCRIPT [ βitalic_r log divide start_ARG πsft ( italic_y | italic_x ) edivide start_ARG 1 end_ARG start_ARG βitalic_r end_ARG ritalic_ϕ ( italic_x , italic_y ) end_ARG start_ARG πitalic_θitalic_βitalic_π ( italic_y | italic_x ) end_ARG ] Notice the analytical form of πβr∗superscriptsubscriptsubscript _ _r^*πitalic_β start_POSTSUBSCRIPT r end_POSTSUBSCRIPT∗: πβr∗⁢(|)=1Zβr⁢()⁢πsft⁢(|)⁢e1βr⁢rϕ⁢(,).superscriptsubscriptsubscriptconditional1subscriptsubscriptsubscriptsftconditionalsuperscript1subscriptsubscriptitalic-ϕ _ _r^*( y| x)= 1Z_ _r(% x) _ sft( y| x)e 1% _rr_φ( x, y).πitalic_β start_POSTSUBSCRIPT r end_POSTSUBSCRIPT∗ ( italic_y | italic_x ) = divide start_ARG 1 end_ARG start_ARG Zitalic_β start_POSTSUBSCRIPT r ( italic_x ) end_POSTSUBSCRIPT end_ARG πsft ( italic_y | italic_x ) edivide start_ARG 1 end_ARG start_ARG βitalic_r end_ARG ritalic_ϕ ( italic_x , italic_y ) . We substitute πsft⁢(|)⁢e1βr⁢rϕ⁢(,)subscriptsftconditionalsuperscript1subscriptsubscriptitalic-ϕ _ sft( y| x)e 1 _rr_% φ( x, y)πsft ( italic_y | italic_x ) edivide start_ARG 1 end_ARG start_ARG βitalic_r end_ARG ritalic_ϕ ( italic_x , italic_y ) into the expression of lhfβr⁢(πθβπ)superscriptsubscriptlhfsubscriptsuperscriptsubscriptsubscriptJ_ lhf _r( _θ _π)Jlhfitalic_βitalic_r ( πitalic_θitalic_βitalic_π ): lhfβr⁢(πθβπ)superscriptsubscriptlhfsubscriptsuperscriptsubscriptsubscript _ lhf _r( _θ _π)Jlhfitalic_βitalic_r ( πitalic_θitalic_βitalic_π ) =∼pref⁢∼πθβπ⁢(|)⁢[βr⁢log⁡Zβr⁢()⁢πβr∗⁢(|)πθβπ⁢(|)]absentsubscriptsimilar-tosuperscriptprefsubscriptsimilar-tosuperscriptsubscriptsubscriptconditionaldelimited-[]subscriptsubscriptsubscriptsuperscriptsubscriptsubscriptconditionalsuperscriptsubscriptsubscriptconditional =E_ x pref% E_ y _θ _π( y|% x) [ _r Z_ _r( x)% _ _r^*( y| x) _θ _π% ( y| x) ]= blackboard_Eitalic_x ∼ Dpref blackboard_Eitalic_y ∼ π start_POSTSUBSCRIPT θitalic_βitalic_π ( italic_y | italic_x ) end_POSTSUBSCRIPT [ βitalic_r log divide start_ARG Zitalic_β start_POSTSUBSCRIPT r ( italic_x ) end_POSTSUBSCRIPT πitalic_β start_POSTSUBSCRIPT r end_POSTSUBSCRIPT∗ ( italic_y | italic_x ) end_ARG start_ARG πitalic_θitalic_βitalic_π ( italic_y | italic_x ) end_ARG ] =βr∈pref[−KL(πθβπ(⋅|)∥πβr∗(⋅|))+logZβr()]. = _rE_ x % pref [-D_ KL( _θ _π(·|% x)\| _ _r^*(·| x))+ Z_ _r% ( x) ].= βitalic_r blackboard_Eitalic_x ∈ Dpref [ - blackboard_DKL ( πitalic_θitalic_βitalic_π ( ⋅ | italic_x ) ∥ πitalic_β start_POSTSUBSCRIPT r end_POSTSUBSCRIPT∗ ( ⋅ | italic_x ) ) + log Zitalic_β start_POSTSUBSCRIPT r end_POSTSUBSCRIPT ( italic_x ) ] . A.3 Proof of Theorem 3.2 Proof. We first restate the definition of ℒexo⁢(πθ)subscriptℒexosubscriptL_ exo( _θ)Lexo ( πitalic_θ ) by substituting Eq. (10), (12) into (3.2): ℒexo⁢(πθ)=∼pref⁢πsft⁢(1:K|)⁢[∑i=1Kefθ⁢(,i)∑j=1Kefθ⁢(,j)⁢(log⁡efθ⁢(,i)∑j=1Kefθ⁢(,j)−log⁡e1βr⁢rϕ⁢(,i)∑j=1Ke1βr⁢rϕ⁢(,j))]subscriptℒexosubscriptsubscriptsimilar-tosuperscriptprefsubscriptsubscriptsftconditionalsubscript:1delimited-[]superscriptsubscript1superscriptsubscriptsubscriptsuperscriptsubscript1superscriptsubscriptsubscriptsuperscriptsubscriptsubscriptsuperscriptsubscript1superscriptsubscriptsubscriptsuperscript1subscriptsubscriptitalic-ϕsubscriptsuperscriptsubscript1superscript1subscriptsubscriptitalic-ϕsubscriptL_ exo( _θ)=E_ x % D prefE_ _ sft( y_1% :K| x) [ _i=1^K e^f_θ( x,% y_i) _j=1^Ke^f_θ( x, y% _j) ( e^f_θ( x, y_i)% _j=1^Ke^f_θ( x, y_j)- e^% 1 _rr_φ( x, y_i) _j=1^% Ke 1 _rr_φ( x, y_j) )% ]Lexo ( πitalic_θ ) = blackboard_Eitalic_x ∼ Dpref blackboard_Eπ start_POSTSUBSCRIPT sft ( italic_y1 : K | italic_x ) end_POSTSUBSCRIPT [ ∑i = 1K divide start_ARG eitalic_fitalic_θ ( italic_x , italic_yitalic_i ) end_ARG start_ARG ∑j = 1K eitalic_fitalic_θ ( italic_x , italic_yitalic_j ) end_ARG ( log divide start_ARG eitalic_fitalic_θ ( italic_x , italic_yitalic_i ) end_ARG start_ARG ∑j = 1K eitalic_fitalic_θ ( italic_x , italic_yitalic_j ) end_ARG - log divide start_ARG edivide start_ARG 1 end_ARG start_ARG βitalic_r end_ARG ritalic_ϕ ( italic_x , italic_yitalic_i ) end_ARG start_ARG ∑j = 1K edivide start_ARG 1 end_ARG start_ARG βitalic_r end_ARG ritalic_ϕ ( italic_x , italic_yitalic_j ) end_ARG ) ] (23) Since fθ⁢(,)=log⁡πθβπ⁢(|)−log⁡πsft⁢(|)subscriptsuperscriptsubscriptsubscriptconditionalsubscriptsftconditionalf_θ( x, y)= _θ _π(% y| x)- _ sft( y|% x)fitalic_θ ( italic_x , italic_y ) = log πitalic_θitalic_βitalic_π ( italic_y | italic_x ) - log πsft ( italic_y | italic_x ), we have that: ℒexo⁢(πθ)subscriptℒexosubscript _ exo( _θ)Lexo ( πitalic_θ ) =∼pref⁢πsft⁢(1:K|)⁢[∑i=1Kelog⁡πθβπ⁢(i|)πsft⁢(i|)∑j=1Kelog⁡πθβπ⁢(j|)πsft⁢(j|)⁢(log⁡elog⁡πθβπ⁢(i|)πsft⁢(i|)∑j=1Kelog⁡πθβπ⁢(j|)πsft⁢(j|)−log⁡e1βr⁢rϕ⁢(,i)∑j=1Ke1βr⁢rϕ⁢(,j))]absentsubscriptsimilar-tosuperscriptprefsubscriptsubscriptsftconditionalsubscript:1delimited-[]superscriptsubscript1superscriptsuperscriptsubscriptsubscriptconditionalsubscriptsubscriptsftconditionalsubscriptsuperscriptsubscript1superscriptsuperscriptsubscriptsubscriptconditionalsubscriptsubscriptsftconditionalsubscriptsuperscriptsuperscriptsubscriptsubscriptconditionalsubscriptsubscriptsftconditionalsubscriptsuperscriptsubscript1superscriptsuperscriptsubscriptsubscriptconditionalsubscriptsubscriptsftconditionalsubscriptsuperscript1subscriptsubscriptitalic-ϕsubscriptsuperscriptsubscript1superscript1subscriptsubscriptitalic-ϕsubscript =E_ x pref% E_ _ sft( y_1:K| x) [% _i=1^K e _θ _π( y_i% | x) _ sft( y_i| x)% _j=1^Ke _θ _π( y_j|% x) _ sft( y_j| x)% ( e _θ _π( y_i|% x) _ sft( y_i| x)% _j=1^Ke _θ _π( y_j|% x) _ sft( y_j| x)-% e 1 _rr_φ( x, y_i)% _j=1^Ke 1 _rr_φ( x, y% _j) ) ]= blackboard_Eitalic_x ∼ Dpref blackboard_Eπ start_POSTSUBSCRIPT sft ( italic_y1 : K | italic_x ) end_POSTSUBSCRIPT [ ∑i = 1K divide start_ARG eroman_log divide start_ARG πitalic_θ start_POSTSUPERSCRIPT βitalic_π ( italic_yitalic_i | italic_x ) end_ARG start_ARG πsft ( italic_yitalic_i | italic_x ) end_ARG end_POSTSUPERSCRIPT end_ARG start_ARG ∑j = 1K eroman_log divide start_ARG πitalic_θ start_POSTSUPERSCRIPT βitalic_π ( italic_yitalic_j | italic_x ) end_ARG start_ARG πsft ( italic_yitalic_j | italic_x ) end_ARG end_POSTSUPERSCRIPT end_ARG ( log divide start_ARG eroman_log divide start_ARG πitalic_θ start_POSTSUPERSCRIPT βitalic_π ( italic_yitalic_i | italic_x ) end_ARG start_ARG πsft ( italic_yitalic_i | italic_x ) end_ARG end_POSTSUPERSCRIPT end_ARG start_ARG ∑j = 1K eroman_log divide start_ARG πitalic_θ start_POSTSUPERSCRIPT βitalic_π ( italic_yitalic_j | italic_x ) end_ARG start_ARG πsft ( italic_yitalic_j | italic_x ) end_ARG end_POSTSUPERSCRIPT end_ARG - log divide start_ARG edivide start_ARG 1 end_ARG start_ARG βitalic_r end_ARG ritalic_ϕ ( italic_x , italic_yitalic_i ) end_ARG start_ARG ∑j = 1K edivide start_ARG 1 end_ARG start_ARG βitalic_r end_ARG ritalic_ϕ ( italic_x , italic_yitalic_j ) end_ARG ) ] =∼pref⁢πsft⁢(1:K|)⁢[∑i=1Kπθβπ⁢(i|)πsft⁢(i|)∑j=1Kπθβπ⁢(j|)πsft⁢(j|)⁢(log⁡πθβπ⁢(i|)πsft⁢(i|)∑j=1Kπθβπ⁢(j|)πsft⁢(j|)−log⁡e1βr⁢rϕ⁢(,i)∑j=1Ke1βr⁢rϕ⁢(,j))].absentsubscriptsimilar-tosuperscriptprefsubscriptsubscriptsftconditionalsubscript:1delimited-[]superscriptsubscript1superscriptsubscriptsubscriptconditionalsubscriptsubscriptsftconditionalsubscriptsuperscriptsubscript1superscriptsubscriptsubscriptconditionalsubscriptsubscriptsftconditionalsubscriptsuperscriptsubscriptsubscriptconditionalsubscriptsubscriptsftconditionalsubscriptsuperscriptsubscript1superscriptsubscriptsubscriptconditionalsubscriptsubscriptsftconditionalsubscriptsuperscript1subscriptsubscriptitalic-ϕsubscriptsuperscriptsubscript1superscript1subscriptsubscriptitalic-ϕsubscript =E_ x pref% E_ _ sft( y_1:K| x) [% _i=1^K _θ _π( y_i|% x) _ sft( y_i| x)% _j=1^K _θ _π( y_j| % x) _ sft( y_j| x) ( % _θ _π( y_i| x) _% sft( y_i| x) _j=1^K _% θ _π( y_j| x) _ sft% ( y_j| x)- e 1 _rr_% φ( x, y_i) _j=1^Ke 1 _r% r_φ( x, y_j) ) ].= blackboard_Eitalic_x ∼ Dpref blackboard_Eπ start_POSTSUBSCRIPT sft ( italic_y1 : K | italic_x ) end_POSTSUBSCRIPT [ ∑i = 1K divide start_ARG divide start_ARG πitalic_θitalic_βitalic_π ( italic_yitalic_i | italic_x ) end_ARG start_ARG πsft ( italic_yitalic_i | italic_x ) end_ARG end_ARG start_ARG ∑j = 1K divide start_ARG πitalic_θitalic_βitalic_π ( italic_yitalic_j | italic_x ) end_ARG start_ARG πsft ( italic_yitalic_j | italic_x ) end_ARG end_ARG ( log divide start_ARG divide start_ARG πitalic_θitalic_βitalic_π ( italic_yitalic_i | italic_x ) end_ARG start_ARG πsft ( italic_yitalic_i | italic_x ) end_ARG end_ARG start_ARG ∑j = 1K divide start_ARG πitalic_θitalic_βitalic_π ( italic_yitalic_j | italic_x ) end_ARG start_ARG πsft ( italic_yitalic_j | italic_x ) end_ARG end_ARG - log divide start_ARG edivide start_ARG 1 end_ARG start_ARG βitalic_r end_ARG ritalic_ϕ ( italic_x , italic_yitalic_i ) end_ARG start_ARG ∑j = 1K edivide start_ARG 1 end_ARG start_ARG βitalic_r end_ARG ritalic_ϕ ( italic_x , italic_yitalic_j ) end_ARG ) ] . (24) Since ii=1Ksuperscriptsubscriptsubscript1\ y_i\_i=1^K italic_yitalic_i i = 1K are sampled from πsft(⋅|) _ sft(·| x)πsft ( ⋅ | italic_x ), when K→∞→K→∞K → ∞, for arbitary function g:→ℝ:→ℝg:Y : Y → blackboard_R, the estimate 1K⁢∑i=1Kg⁢(i)1superscriptsubscript1subscript 1K _i=1^Kg( y_i)divide start_ARG 1 end_ARG start_ARG K end_ARG ∑i = 1K g ( italic_yitalic_i ) is unbiased, i.e., limK→∞1K⁢∑i=1Kg⁢(i)=πsft⁢(|)⁢[g⁢()]subscript→1superscriptsubscript1subscriptsubscriptsubscriptsftconditionaldelimited-[] _K→∞ 1K _i=1^Kg( y_i)=% E_ _ sft( y| x)[g( % y)]limitalic_K → ∞ divide start_ARG 1 end_ARG start_ARG K end_ARG ∑i = 1K g ( italic_yitalic_i ) = blackboard_Eπ start_POSTSUBSCRIPT sft ( italic_y | italic_x ) end_POSTSUBSCRIPT [ g ( italic_y ) ]. We consider the following two instantiations of g⁢(⋅)⋅g(·)g ( ⋅ ). For g⁢()=πθβπ⁢(|)πsft⁢(|)superscriptsubscriptsubscriptconditionalsubscriptsftconditionalg( y)= _θ _π( y| % x) _ sft( y| x)g ( italic_y ) = divide start_ARG πitalic_θitalic_βitalic_π ( italic_y | italic_x ) end_ARG start_ARG πsft ( italic_y | italic_x ) end_ARG, we have: ∑j=1Kπθβπ⁢(j|)πsft⁢(j|)superscriptsubscript1superscriptsubscriptsubscriptconditionalsubscriptsubscriptsftconditionalsubscript _j=1^K _θ _π( y_j% | x) _ sft( y_j| x)∑j = 1K divide start_ARG πitalic_θitalic_βitalic_π ( italic_yitalic_j | italic_x ) end_ARG start_ARG πsft ( italic_yitalic_j | italic_x ) end_ARG =K⁢πsft⁢(|)⁢[πθβπ⁢(|)πsft⁢(|)]absentsubscriptsubscriptsftconditionaldelimited-[]superscriptsubscriptsubscriptconditionalsubscriptsftconditional =KE_ _ sft( y| x)% [ _θ _π( y| x)π% _ sft( y| x) ]= K blackboard_Eπ start_POSTSUBSCRIPT sft ( italic_y | italic_x ) end_POSTSUBSCRIPT [ divide start_ARG πitalic_θitalic_βitalic_π ( italic_y | italic_x ) end_ARG start_ARG πsft ( italic_y | italic_x ) end_ARG ] =K⁢∑∈πsft⁢(|)⁢πθβπ⁢(|)πsft⁢(|)absentsubscriptsubscriptsftconditionalsuperscriptsubscriptsubscriptconditionalsubscriptsftconditional =K _ y _ sft(% y| x) _θ _π( y|% x) _ sft( y| x)= K ∑italic_y ∈ Y πsft ( italic_y | italic_x ) divide start_ARG πitalic_θitalic_βitalic_π ( italic_y | italic_x ) end_ARG start_ARG πsft ( italic_y | italic_x ) end_ARG =K⁢∑∈πθβπ⁢(|)absentsubscriptsuperscriptsubscriptsubscriptconditional =K _ y _θ _π(% y| x)= K ∑italic_y ∈ Y πitalic_θitalic_βitalic_π ( italic_y | italic_x ) =K.absent =K.= K . For g⁢()=e1βr⁢rϕ⁢(,)superscript1subscriptsubscriptitalic-ϕg( y)=e 1 _rr_φ( x, % y)g ( italic_y ) = edivide start_ARG 1 end_ARG start_ARG βitalic_r end_ARG ritalic_ϕ ( italic_x , italic_y ), we have: ∑j=1Ke1βr⁢rϕ⁢(,j)superscriptsubscript1superscript1subscriptsubscriptitalic-ϕsubscript _j=1^Ke 1 _rr_φ( x,% y_j)∑j = 1K edivide start_ARG 1 end_ARG start_ARG βitalic_r end_ARG ritalic_ϕ ( italic_x , italic_yitalic_j ) =K⁢πsft⁢(|)⁢[e1βr⁢rϕ⁢(,)]absentsubscriptsubscriptsftconditionaldelimited-[]superscript1subscriptsubscriptitalic-ϕ =KE_ _ sft( y| x)% [e 1 _rr_φ( x, y) ]= K blackboard_Eπ start_POSTSUBSCRIPT sft ( italic_y | italic_x ) end_POSTSUBSCRIPT [ edivide start_ARG 1 end_ARG start_ARG βitalic_r end_ARG ritalic_ϕ ( italic_x , italic_y ) ] =K⁢∑∈πsft⁢(|)⁢e1βr⁢rϕ⁢(,)absentsubscriptsubscriptsftconditionalsuperscript1subscriptsubscriptitalic-ϕ =K _ y _ sft(% y| x)e 1 _rr_φ( x,% y)= K ∑italic_y ∈ Y πsft ( italic_y | italic_x ) edivide start_ARG 1 end_ARG start_ARG βitalic_r end_ARG ritalic_ϕ ( italic_x , italic_y ) =K⁢Zβr⁢().absentsubscriptsubscript =KZ_ _r( x).= K Zitalic_β start_POSTSUBSCRIPT r end_POSTSUBSCRIPT ( italic_x ) . Then we simplify ℒexosubscriptℒexoL_ exoLexo by substituting the expression of ∑j=1Kπθβπ⁢(j|)πsft⁢(j|)superscriptsubscript1superscriptsubscriptsubscriptconditionalsubscriptsubscriptsftconditionalsubscript _j=1^K _θ _π( y_j| % x) _ sft( y_j| x)∑j = 1K divide start_ARG πitalic_θitalic_βitalic_π ( italic_yitalic_j | italic_x ) end_ARG start_ARG πsft ( italic_yitalic_j | italic_x ) end_ARG and ∑j=1Ke1βr⁢rϕ⁢(,j)superscriptsubscript1superscript1subscriptsubscriptitalic-ϕsubscript _j=1^Ke 1 _rr_φ( x, y_% j)∑j = 1K edivide start_ARG 1 end_ARG start_ARG βitalic_r end_ARG ritalic_ϕ ( italic_x , italic_yitalic_j ) when K→∞→K→∞K → ∞ into Eq. (24). ℒexo⁢(πθ)subscriptℒexosubscript _ exo( _θ)Lexo ( πitalic_θ ) =∼pref⁢πsft⁢(1:K|)⁢[∑i=1Kπθβπ⁢(i|)K⁢πsft⁢(i|)⁢(log⁡πθβπ⁢(i|)K⁢πsft⁢(i|)−log⁡e1βr⁢rϕ⁢(,i)K⁢Zβr⁢())]absentsubscriptsimilar-tosuperscriptprefsubscriptsubscriptsftconditionalsubscript:1delimited-[]superscriptsubscript1superscriptsubscriptsubscriptconditionalsubscriptsubscriptsftconditionalsubscriptsuperscriptsubscriptsubscriptconditionalsubscriptsubscriptsftconditionalsubscriptsuperscript1subscriptsubscriptitalic-ϕsubscriptsubscriptsubscript =E_ x pref% E_ _ sft( y_1:K| x) [% _i=1^K _θ _π( y_i| % x)K _ sft( y_i| x) ( % _θ _π( y_i| x)K _ % sft( y_i| x)- e 1 _r% r_φ( x, y_i)KZ_ _r( x)% ) ]= blackboard_Eitalic_x ∼ Dpref blackboard_Eπ start_POSTSUBSCRIPT sft ( italic_y1 : K | italic_x ) end_POSTSUBSCRIPT [ ∑i = 1K divide start_ARG πitalic_θitalic_βitalic_π ( italic_yitalic_i | italic_x ) end_ARG start_ARG K πsft ( italic_yitalic_i | italic_x ) end_ARG ( log divide start_ARG πitalic_θitalic_βitalic_π ( italic_yitalic_i | italic_x ) end_ARG start_ARG K πsft ( italic_yitalic_i | italic_x ) end_ARG - log divide start_ARG edivide start_ARG 1 end_ARG start_ARG βitalic_r end_ARG ritalic_ϕ ( italic_x , italic_yitalic_i ) end_ARG start_ARG K Zitalic_β start_POSTSUBSCRIPT r end_POSTSUBSCRIPT ( italic_x ) end_ARG ) ] =∼pref⁢πsft⁢(1:K|)⁢[1K⁢∑i=1Kπθβπ⁢(i|)πsft⁢(i|)⁢log⁡πθβπ⁢(i|)πsft⁢(i|)⁢e1βr⁢rϕ⁢(,i)/Zβr⁢()].absentsubscriptsimilar-tosuperscriptprefsubscriptsubscriptsftconditionalsubscript:1delimited-[]1superscriptsubscript1superscriptsubscriptsubscriptconditionalsubscriptsubscriptsftconditionalsubscriptsuperscriptsubscriptsubscriptconditionalsubscriptsubscriptsftconditionalsubscriptsuperscript1subscriptsubscriptitalic-ϕsubscriptsubscriptsubscript =E_ x pref% E_ _ sft( y_1:K| x) [% 1K _i=1^K _θ _π( y_i|% x) _ sft( y_i| x) % _θ _π( y_i| x) _% sft( y_i| x)e 1 _rr_% φ( x, y_i)/Z_ _r( x)% ].= blackboard_Eitalic_x ∼ Dpref blackboard_Eπ start_POSTSUBSCRIPT sft ( italic_y1 : K | italic_x ) end_POSTSUBSCRIPT [ divide start_ARG 1 end_ARG start_ARG K end_ARG ∑i = 1K divide start_ARG πitalic_θitalic_βitalic_π ( italic_yitalic_i | italic_x ) end_ARG start_ARG πsft ( italic_yitalic_i | italic_x ) end_ARG log divide start_ARG πitalic_θitalic_βitalic_π ( italic_yitalic_i | italic_x ) end_ARG start_ARG πsft ( italic_yitalic_i | italic_x ) edivide start_ARG 1 end_ARG start_ARG βitalic_r end_ARG ritalic_ϕ ( italic_x , italic_yitalic_i ) / Zitalic_β start_POSTSUBSCRIPT r end_POSTSUBSCRIPT ( italic_x ) end_ARG ] . Notice the analytic form of πβr∗⁢(|)=1Zβr⁢()⁢πsft⁢(|)⁢e1βr⁢rϕ⁢(,)superscriptsubscriptsubscriptconditional1subscriptsubscriptsubscriptsftconditionalsuperscript1subscriptsubscriptitalic-ϕ _ _r^*( y| x)= 1Z_ _r(% x) _ sft( y| x)e 1% _rr_φ( x, y)πitalic_β start_POSTSUBSCRIPT r end_POSTSUBSCRIPT∗ ( italic_y | italic_x ) = divide start_ARG 1 end_ARG start_ARG Zitalic_β start_POSTSUBSCRIPT r end_POSTSUBSCRIPT ( italic_x ) end_ARG πsft ( italic_y | italic_x ) edivide start_ARG 1 end_ARG start_ARG βitalic_r end_ARG ritalic_ϕ ( italic_x , italic_y ), we substitute πβr∗superscriptsubscriptsubscript _ _r^*πitalic_β start_POSTSUBSCRIPT r end_POSTSUBSCRIPT∗ into the above equation: ℒexo⁢(πθ)subscriptℒexosubscript _ exo( _θ)Lexo ( πitalic_θ ) =∼pref⁢πsft⁢(1:K|)⁢[1K⁢∑i=1Kπθβπ⁢(i|)πsft⁢(i|)⁢log⁡πθβπ⁢(i|)πβr∗⁢(i|)]absentsubscriptsimilar-tosuperscriptprefsubscriptsubscriptsftconditionalsubscript:1delimited-[]1superscriptsubscript1superscriptsubscriptsubscriptconditionalsubscriptsubscriptsftconditionalsubscriptsuperscriptsubscriptsubscriptconditionalsubscriptsuperscriptsubscriptsubscriptconditionalsubscript =E_ x pref% E_ _ sft( y_1:K| x) [% 1K _i=1^K _θ _π( y_i|% x) _ sft( y_i| x) % _θ _π( y_i| x) _% _r^*( y_i| x) ]= blackboard_Eitalic_x ∼ Dpref blackboard_Eπ start_POSTSUBSCRIPT sft ( italic_y1 : K | italic_x ) end_POSTSUBSCRIPT [ divide start_ARG 1 end_ARG start_ARG K end_ARG ∑i = 1K divide start_ARG πitalic_θitalic_βitalic_π ( italic_yitalic_i | italic_x ) end_ARG start_ARG πsft ( italic_yitalic_i | italic_x ) end_ARG log divide start_ARG πitalic_θitalic_βitalic_π ( italic_yitalic_i | italic_x ) end_ARG start_ARG πitalic_β start_POSTSUBSCRIPT r end_POSTSUBSCRIPT∗ ( italic_yitalic_i | italic_x ) end_ARG ] =∼pref⁢[1K⁢∑i=1Kπsft⁢(i|)⁢[πθβπ⁢(i|)πsft⁢(i|)⁢log⁡πθβπ⁢(i|)πβr∗⁢(i|)]]absentsubscriptsimilar-tosuperscriptprefdelimited-[]1superscriptsubscript1subscriptsubscriptsftconditionalsubscriptdelimited-[]superscriptsubscriptsubscriptconditionalsubscriptsubscriptsftconditionalsubscriptsuperscriptsubscriptsubscriptconditionalsubscriptsuperscriptsubscriptsubscriptconditionalsubscript =E_ x pref % [ 1K _i=1^KE_ _ sft( y_i% | x) [ _θ _π( y_i|% x) _ sft( y_i| x) % _θ _π( y_i| x) _% _r^*( y_i| x) ] ]= blackboard_Eitalic_x ∼ Dpref [ divide start_ARG 1 end_ARG start_ARG K end_ARG ∑i = 1K blackboard_Eπ start_POSTSUBSCRIPT sft ( italic_yitalic_i | italic_x ) end_POSTSUBSCRIPT [ divide start_ARG πitalic_θitalic_βitalic_π ( italic_yitalic_i | italic_x ) end_ARG start_ARG πsft ( italic_yitalic_i | italic_x ) end_ARG log divide start_ARG πitalic_θitalic_βitalic_π ( italic_yitalic_i | italic_x ) end_ARG start_ARG πitalic_β start_POSTSUBSCRIPT r end_POSTSUBSCRIPT∗ ( italic_yitalic_i | italic_x ) end_ARG ] ] =∼pref⁢[1K⁢∑i=1K∑i∈πθβπ⁢(i|)⁢log⁡πθβπ⁢(i|)πβr∗⁢(i|)]absentsubscriptsimilar-tosuperscriptprefdelimited-[]1superscriptsubscript1subscriptsubscriptsuperscriptsubscriptsubscriptconditionalsubscriptsuperscriptsubscriptsubscriptconditionalsubscriptsuperscriptsubscriptsubscriptconditionalsubscript =E_ x pref % [ 1K _i=1^K _ y_i _θ% _π( y_i| x) _θ^β% _π( y_i| x) _ _r^*( y% _i| x) ]= blackboard_Eitalic_x ∼ Dpref [ divide start_ARG 1 end_ARG start_ARG K end_ARG ∑i = 1K ∑italic_y start_POSTSUBSCRIPT i ∈ Y end_POSTSUBSCRIPT πitalic_θitalic_βitalic_π ( italic_yitalic_i | italic_x ) log divide start_ARG πitalic_θitalic_βitalic_π ( italic_yitalic_i | italic_x ) end_ARG start_ARG πitalic_β start_POSTSUBSCRIPT r end_POSTSUBSCRIPT∗ ( italic_yitalic_i | italic_x ) end_ARG ] =∼pref⁢[∑∈πθβπ⁢(|)⁢log⁡πθβπ⁢(|)πβr∗⁢(|)]absentsubscriptsimilar-tosuperscriptprefdelimited-[]subscriptsuperscriptsubscriptsubscriptconditionalsuperscriptsubscriptsubscriptconditionalsuperscriptsubscriptsubscriptconditional =E_ x pref % [ _ y _θ _π( y% | x) _θ _π( y|% x) _ _r^*( y| x) ]= blackboard_Eitalic_x ∼ Dpref [ ∑italic_y ∈ Y πitalic_θitalic_βitalic_π ( italic_y | italic_x ) log divide start_ARG πitalic_θitalic_βitalic_π ( italic_y | italic_x ) end_ARG start_ARG πitalic_β start_POSTSUBSCRIPT r end_POSTSUBSCRIPT∗ ( italic_y | italic_x ) end_ARG ] =∼pref[KL(πθβπ(|)∥πβr∗(|))]. =E_ x pref[% D_ KL( _θ _π( y| % x)\| _ _r^*( y| x))].= blackboard_Eitalic_x ∼ Dpref [ blackboard_DKL ( πitalic_θitalic_βitalic_π ( italic_y | italic_x ) ∥ πitalic_β start_POSTSUBSCRIPT r end_POSTSUBSCRIPT∗ ( italic_y | italic_x ) ) ] . Accordingly, by taking the derivative with respect to θ, we complete the proof of Eq. (14). To prove Eq. (15), we utilize Eq. (14) to substitute into Eq. (3.1) to build the connection between lhfβr⁢(πθβπ)superscriptsubscriptlhfsubscriptsuperscriptsubscriptsubscriptJ_ lhf _r( _θ _π)Jlhfitalic_βitalic_r ( πitalic_θitalic_βitalic_π ) and ℒexosubscriptℒexoL_ exoLexo: lhfβr⁢(πθβπ)=−βr⁢ℒexo⁢(πθ)+βr⁢∼pref⁢[log⁡Zβr⁢()].superscriptsubscriptlhfsubscriptsuperscriptsubscriptsubscriptsubscriptsubscriptℒexosubscriptsubscriptsubscriptsimilar-tosuperscriptprefdelimited-[]subscriptsubscript _ lhf _r( _θ _π% )=- _rL_ exo( _θ)+ _rE_% x pref[ Z_ _r(% x)].Jlhfitalic_βitalic_r ( πitalic_θitalic_βitalic_π ) = - βitalic_r Lexo ( πitalic_θ ) + βitalic_r blackboard_Eitalic_x ∼ Dpref [ log Zitalic_β start_POSTSUBSCRIPT r end_POSTSUBSCRIPT ( italic_x ) ] . Then we take the gradient with respect to the parameters θ of the above formulat: ∇θlhfβr⁢(πθβπ)=−βr⁢∇θℒexo⁢(πθ),subscript∇superscriptsubscriptlhfsubscriptsuperscriptsubscriptsubscriptsubscriptsubscript∇subscriptℒexosubscript _θJ_ lhf _r( _θ _π% )=- _r _θL_ exo( _θ),∇θ Jlhfitalic_βitalic_r ( πitalic_θitalic_βitalic_π ) = - βitalic_r ∇θ Lexo ( πitalic_θ ) , which completes the proof of Eq. (15). ∎ A.4 Proof of Theorem 3.3 Proof. We utilize the definition of πθβπ⁢(|)∝πθ⁢(|)βπ⁢πsft⁢(|)1−βπproportional-tosuperscriptsubscriptsubscriptconditionalsubscriptsuperscriptconditionalsubscriptsubscriptsftsuperscriptconditional1subscript _θ _π( y| x) _θ(% y| x) _π _ sft( y% | x)^1- _πitalic_θitalic_βitalic_π ( italic_y | italic_x ) ∝ πitalic_θ ( italic_y | italic_x )βitalic_π πsft ( italic_y | italic_x )1 - βitalic_π in Eq. (8) and divide both sides by πsft⁢(|)subscriptsftconditional _ sft( y| x)πsft ( italic_y | italic_x ): πθβπ⁢(|)πsft⁢(|)∝(πθ⁢(|)πsft⁢(|))βπproportional-tosuperscriptsubscriptsubscriptconditionalsubscriptsftconditionalsuperscriptsubscriptconditionalsubscriptsftconditionalsubscript _θ _π( y| x) _ % sft( y| x) ( _θ(% y| x) _ sft( y|% x) ) _πdivide start_ARG πitalic_θitalic_βitalic_π ( italic_y | italic_x ) end_ARG start_ARG πsft ( italic_y | italic_x ) end_ARG ∝ ( divide start_ARG πitalic_θ ( italic_y | italic_x ) end_ARG start_ARG πsft ( italic_y | italic_x ) end_ARG )βitalic_π Then we substitute the above equation into ℒdpo-rwsubscriptℒdpo-rwL_ dpo-rwLdpo-rw: ℒdpo-rw⁢(πθ)subscriptℒdpo-rwsubscript _ dpo-rw( _θ)Ldpo-rw ( πitalic_θ ) =∼pref⁢πsft⁢(1:K|)⁢[−∑i=1Ke1βr⁢rϕ⁢(,i)∑j=1Ke1βr⁢rϕ⁢(,j)⁢log⁡eβπ⁢log⁡πθ⁢(i|)πsft⁢(i|)∑j=1Keβπ⁢log⁡πθ⁢(j|)πsft⁢(j|)]absentsubscriptsimilar-tosuperscriptprefsubscriptsubscriptsftconditionalsubscript:1delimited-[]superscriptsubscript1superscript1subscriptsubscriptitalic-ϕsubscriptsuperscriptsubscript1superscript1subscriptsubscriptitalic-ϕsubscriptsuperscriptsubscriptsubscriptconditionalsubscriptsubscriptsftconditionalsubscriptsuperscriptsubscript1superscriptsubscriptsubscriptconditionalsubscriptsubscriptsftconditionalsubscript =E_ x pref% E_ _ sft( y_1:K| x) [% - _i=1^K e 1 _rr_φ( x,% y_i) _j=1^Ke 1 _rr_φ(% x, y_j) e _π _% θ( y_i| x) _ sft( y% _i| x) _j=1^Ke _π _θ(% y_j| x) _ sft( y_j|% x) ]= blackboard_Eitalic_x ∼ Dpref blackboard_Eπ start_POSTSUBSCRIPT sft ( italic_y1 : K | italic_x ) end_POSTSUBSCRIPT [ - ∑i = 1K divide start_ARG edivide start_ARG 1 end_ARG start_ARG βitalic_r end_ARG ritalic_ϕ ( italic_x , italic_yitalic_i ) end_ARG start_ARG ∑j = 1K edivide start_ARG 1 end_ARG start_ARG βitalic_r end_ARG ritalic_ϕ ( italic_x , italic_yitalic_j ) end_ARG log divide start_ARG eitalic_βitalic_π log divide start_ARG πitalic_θ ( italic_yitalic_i | italic_x ) end_ARG start_ARG πsft ( italic_yitalic_i | italic_x ) end_ARG end_ARG start_ARG ∑j = 1K eitalic_βitalic_π log divide start_ARG πitalic_θ ( italic_yitalic_j | italic_x ) end_ARG start_ARG πsft ( italic_yitalic_j | italic_x ) end_ARG end_ARG ] =∼pref⁢πsft⁢(1:K|)⁢[−∑i=1Ke1βr⁢rϕ⁢(,i)∑j=1Ke1βr⁢rϕ⁢(,j)⁢log⁡elog⁡πθβπ⁢(i|)πsft⁢(i|)∑j=1Kelog⁡πθβπ⁢(j|)πsft⁢(j|)]absentsubscriptsimilar-tosuperscriptprefsubscriptsubscriptsftconditionalsubscript:1delimited-[]superscriptsubscript1superscript1subscriptsubscriptitalic-ϕsubscriptsuperscriptsubscript1superscript1subscriptsubscriptitalic-ϕsubscriptsuperscriptsuperscriptsubscriptsubscriptconditionalsubscriptsubscriptsftconditionalsubscriptsuperscriptsubscript1superscriptsuperscriptsubscriptsubscriptconditionalsubscriptsubscriptsftconditionalsubscript =E_ x pref% E_ _ sft( y_1:K| x) [% - _i=1^K e 1 _rr_φ( x,% y_i) _j=1^Ke 1 _rr_φ(% x, y_j) e _θ^β% _π( y_i| x) _ sft( y% _i| x) _j=1^Ke _θ _π% ( y_j| x) _ sft( y_j|% x) ]= blackboard_Eitalic_x ∼ Dpref blackboard_Eπ start_POSTSUBSCRIPT sft ( italic_y1 : K | italic_x ) end_POSTSUBSCRIPT [ - ∑i = 1K divide start_ARG edivide start_ARG 1 end_ARG start_ARG βitalic_r end_ARG ritalic_ϕ ( italic_x , italic_yitalic_i ) end_ARG start_ARG ∑j = 1K edivide start_ARG 1 end_ARG start_ARG βitalic_r end_ARG ritalic_ϕ ( italic_x , italic_yitalic_j ) end_ARG log divide start_ARG eroman_log divide start_ARG πitalic_θ start_POSTSUPERSCRIPT βitalic_π ( italic_yitalic_i | italic_x ) end_ARG start_ARG πsft ( italic_yitalic_i | italic_x ) end_ARG end_POSTSUPERSCRIPT end_ARG start_ARG ∑j = 1K eroman_log divide start_ARG πitalic_θ start_POSTSUPERSCRIPT βitalic_π ( italic_yitalic_j | italic_x ) end_ARG start_ARG πsft ( italic_yitalic_j | italic_x ) end_ARG end_POSTSUPERSCRIPT end_ARG ] =∼pref⁢πsft⁢(1:K|)⁢[−∑i=1Ke1βr⁢rϕ⁢(,i)∑j=1Ke1βr⁢rϕ⁢(,j)⁢log⁡πθβπ⁢(i|)πsft⁢(i|)∑j=1Kπθβπ⁢(j|)πsft⁢(j|)]absentsubscriptsimilar-tosuperscriptprefsubscriptsubscriptsftconditionalsubscript:1delimited-[]superscriptsubscript1superscript1subscriptsubscriptitalic-ϕsubscriptsuperscriptsubscript1superscript1subscriptsubscriptitalic-ϕsubscriptsuperscriptsubscriptsubscriptconditionalsubscriptsubscriptsftconditionalsubscriptsuperscriptsubscript1superscriptsubscriptsubscriptconditionalsubscriptsubscriptsftconditionalsubscript =E_ x pref% E_ _ sft( y_1:K| x) [% - _i=1^K e 1 _rr_φ( x,% y_i) _j=1^Ke 1 _rr_φ(% x, y_j) _θ _π% ( y_i| x) _ sft( y_i|% x) _j=1^K _θ _π( y% _j| x) _ sft( y_j| x)% ]= blackboard_Eitalic_x ∼ Dpref blackboard_Eπ start_POSTSUBSCRIPT sft ( italic_y1 : K | italic_x ) end_POSTSUBSCRIPT [ - ∑i = 1K divide start_ARG edivide start_ARG 1 end_ARG start_ARG βitalic_r end_ARG ritalic_ϕ ( italic_x , italic_yitalic_i ) end_ARG start_ARG ∑j = 1K edivide start_ARG 1 end_ARG start_ARG βitalic_r end_ARG ritalic_ϕ ( italic_x , italic_yitalic_j ) end_ARG log divide start_ARG divide start_ARG πitalic_θitalic_βitalic_π ( italic_yitalic_i | italic_x ) end_ARG start_ARG πsft ( italic_yitalic_i | italic_x ) end_ARG end_ARG start_ARG ∑j = 1K divide start_ARG πitalic_θitalic_βitalic_π ( italic_yitalic_j | italic_x ) end_ARG start_ARG πsft ( italic_yitalic_j | italic_x ) end_ARG end_ARG ] (25) Since ii=1Ksuperscriptsubscriptsubscript1\ y_i\_i=1^K italic_yitalic_i i = 1K are sampled from πsft(⋅|) _ sft(·| x)πsft ( ⋅ | italic_x ), when K→∞→K→∞K → ∞, we follow the proof of Theorem 3.2 and directly give the following results: ∑j=1Kπθβπ⁢(j|)πsft⁢(j|)superscriptsubscript1superscriptsubscriptsubscriptconditionalsubscriptsubscriptsftconditionalsubscript _j=1^K _θ _π( y_j% | x) _ sft( y_j| x)∑j = 1K divide start_ARG πitalic_θitalic_βitalic_π ( italic_yitalic_j | italic_x ) end_ARG start_ARG πsft ( italic_yitalic_j | italic_x ) end_ARG =K,absent =K,= K , ∑j=1Ke1βr⁢rϕ⁢(,j)superscriptsubscript1superscript1subscriptsubscriptitalic-ϕsubscript _j=1^Ke 1 _rr_φ( x,% y_j)∑j = 1K edivide start_ARG 1 end_ARG start_ARG βitalic_r end_ARG ritalic_ϕ ( italic_x , italic_yitalic_j ) =K⁢Zβr⁢().absentsubscriptsubscript =KZ_ _r( x).= K Zitalic_β start_POSTSUBSCRIPT r end_POSTSUBSCRIPT ( italic_x ) . Then we simplify ℒdpo-rwsubscriptℒdpo-rwL_ dpo-rwLdpo-rw by substituting the above results of ∑j=1Kπθβπ⁢(j|)πsft⁢(j|)superscriptsubscript1superscriptsubscriptsubscriptconditionalsubscriptsubscriptsftconditionalsubscript _j=1^K _θ _π( y_j| % x) _ sft( y_j| x)∑j = 1K divide start_ARG πitalic_θitalic_βitalic_π ( italic_yitalic_j | italic_x ) end_ARG start_ARG πsft ( italic_yitalic_j | italic_x ) end_ARG and ∑j=1Ke1βr⁢rϕ⁢(,j)superscriptsubscript1superscript1subscriptsubscriptitalic-ϕsubscript _j=1^Ke 1 _rr_φ( x, y_% j)∑j = 1K edivide start_ARG 1 end_ARG start_ARG βitalic_r end_ARG ritalic_ϕ ( italic_x , italic_yitalic_j ) when K→∞→K→∞K → ∞ into Eq. (25): ℒdpo-rw⁢(πθ)=∼pref⁢πsft⁢(1:K|)⁢[−∑i=1Ke1βr⁢rϕ⁢(,i)K⁢Zβr⁢()⁢log⁡πθβπ⁢(i|)K⁢πsft⁢(i|)]subscriptℒdpo-rwsubscriptsubscriptsimilar-tosuperscriptprefsubscriptsubscriptsftconditionalsubscript:1delimited-[]superscriptsubscript1superscript1subscriptsubscriptitalic-ϕsubscriptsubscriptsubscriptsuperscriptsubscriptsubscriptconditionalsubscriptsubscriptsftconditionalsubscript _ dpo-rw( _θ)=E_% x prefE_ _ sft% ( y_1:K| x) [- _i=1^K e 1% _rr_φ( x, y_i)KZ_ _r(% x) _θ _π( y_i|% x)K _ sft( y_i| x)% ]Ldpo-rw ( πitalic_θ ) = blackboard_Eitalic_x ∼ Dpref blackboard_Eπ start_POSTSUBSCRIPT sft ( italic_y1 : K | italic_x ) end_POSTSUBSCRIPT [ - ∑i = 1K divide start_ARG edivide start_ARG 1 end_ARG start_ARG βitalic_r end_ARG ritalic_ϕ ( italic_x , italic_yitalic_i ) end_ARG start_ARG K Zitalic_β start_POSTSUBSCRIPT r end_POSTSUBSCRIPT ( italic_x ) end_ARG log divide start_ARG πitalic_θitalic_βitalic_π ( italic_yitalic_i | italic_x ) end_ARG start_ARG K πsft ( italic_yitalic_i | italic_x ) end_ARG ] Notice the analytic form of πβr∗⁢(|)=1Zβr⁢()⁢πsft⁢(|)⁢e1βr⁢rϕ⁢(,)superscriptsubscriptsubscriptconditional1subscriptsubscriptsubscriptsftconditionalsuperscript1subscriptsubscriptitalic-ϕ _ _r^*( y| x)= 1Z_ _r(% x) _ sft( y| x)e 1% _rr_φ( x, y)πitalic_β start_POSTSUBSCRIPT r end_POSTSUBSCRIPT∗ ( italic_y | italic_x ) = divide start_ARG 1 end_ARG start_ARG Zitalic_β start_POSTSUBSCRIPT r end_POSTSUBSCRIPT ( italic_x ) end_ARG πsft ( italic_y | italic_x ) edivide start_ARG 1 end_ARG start_ARG βitalic_r end_ARG ritalic_ϕ ( italic_x , italic_y ), we rearrange πβr∗superscriptsubscriptsubscript _ _r^*πitalic_β start_POSTSUBSCRIPT r end_POSTSUBSCRIPT∗ and substitute πsft⁢(|)=πβr∗⁢(|)⁢Zβr⁢()⁢e−1βr⁢rϕ⁢(,)subscriptsftconditionalsuperscriptsubscriptsubscriptconditionalsubscriptsubscriptsuperscript1subscriptsubscriptitalic-ϕ _ sft( y| x)= _ _r^*(% y| x)Z_ _r( x)e^- 1β% _rr_φ( x, y)πsft ( italic_y | italic_x ) = πitalic_β start_POSTSUBSCRIPT r end_POSTSUBSCRIPT∗ ( italic_y | italic_x ) Zitalic_β start_POSTSUBSCRIPT r end_POSTSUBSCRIPT ( italic_x ) e- divide start_ARG 1 end_ARG start_ARG βitalic_r end_ARG ritalic_ϕ ( italic_x , italic_y ) into the above equation to simplify it: ℒdpo-rw⁢(πθ)subscriptℒdpo-rwsubscript _ dpo-rw( _θ)Ldpo-rw ( πitalic_θ ) =∼pref⁢πsft⁢(1:K|)⁢[−∑i=1Ke1βr⁢rϕ⁢(,i)K⁢Zβr⁢()⁢log⁡(πθβπ⁢(i|)πβr∗⁢(i|)⋅e1βr⁢rϕ⁢(,i)K⁢Zβr⁢())]absentsubscriptsimilar-tosuperscriptprefsubscriptsubscriptsftconditionalsubscript:1delimited-[]superscriptsubscript1superscript1subscriptsubscriptitalic-ϕsubscriptsubscriptsubscript⋅superscriptsubscriptsubscriptconditionalsubscriptsuperscriptsubscriptsubscriptconditionalsubscriptsuperscript1subscriptsubscriptitalic-ϕsubscriptsubscriptsubscript =E_ x pref% E_ _ sft( y_1:K| x) [% - _i=1^K e 1 _rr_φ( x,% y_i)KZ_ _r( x) ( _% θ _π( y_i| x) _ _r^*(% y_i| x)· e 1 _rr_φ% ( x, y_i)KZ_ _r( x) )% ]= blackboard_Eitalic_x ∼ Dpref blackboard_Eπ start_POSTSUBSCRIPT sft ( italic_y1 : K | italic_x ) end_POSTSUBSCRIPT [ - ∑i = 1K divide start_ARG edivide start_ARG 1 end_ARG start_ARG βitalic_r end_ARG ritalic_ϕ ( italic_x , italic_yitalic_i ) end_ARG start_ARG K Zitalic_β start_POSTSUBSCRIPT r end_POSTSUBSCRIPT ( italic_x ) end_ARG log ( divide start_ARG πitalic_θitalic_βitalic_π ( italic_yitalic_i | italic_x ) end_ARG start_ARG πitalic_β start_POSTSUBSCRIPT r end_POSTSUBSCRIPT∗ ( italic_yitalic_i | italic_x ) end_ARG ⋅ divide start_ARG edivide start_ARG 1 end_ARG start_ARG βitalic_r end_ARG ritalic_ϕ ( italic_x , italic_yitalic_i ) end_ARG start_ARG K Zitalic_β start_POSTSUBSCRIPT r end_POSTSUBSCRIPT ( italic_x ) end_ARG ) ] =∼pref⁢πsft⁢(1:K|)⁢[−∑i=1Ke1βr⁢rϕ⁢(,i)K⁢Zβr⁢()⁢log⁡πθβπ⁢(i|)πβr∗⁢(i|)−∑i=1Ke1βr⁢rϕ⁢(,i)K⁢Zβr⁢()⁢log⁡e1βr⁢rϕ⁢(,i)K⁢Zβr⁢()]absentsubscriptsimilar-tosuperscriptprefsubscriptsubscriptsftconditionalsubscript:1delimited-[]superscriptsubscript1superscript1subscriptsubscriptitalic-ϕsubscriptsubscriptsubscriptsuperscriptsubscriptsubscriptconditionalsubscriptsuperscriptsubscriptsubscriptconditionalsubscriptsuperscriptsubscript1superscript1subscriptsubscriptitalic-ϕsubscriptsubscriptsubscriptsuperscript1subscriptsubscriptitalic-ϕsubscriptsubscriptsubscript =E_ x pref% E_ _ sft( y_1:K| x) [% - _i=1^K e 1 _rr_φ( x,% y_i)KZ_ _r( x) _θ^% _π( y_i| x) _ _r^*(% y_i| x)- _i=1^K e 1 _r% r_φ( x, y_i)KZ_ _r( x)% e 1 _rr_φ( x, y_i)% KZ_ _r( x) ]= blackboard_Eitalic_x ∼ Dpref blackboard_Eπ start_POSTSUBSCRIPT sft ( italic_y1 : K | italic_x ) end_POSTSUBSCRIPT [ - ∑i = 1K divide start_ARG edivide start_ARG 1 end_ARG start_ARG βitalic_r end_ARG ritalic_ϕ ( italic_x , italic_yitalic_i ) end_ARG start_ARG K Zitalic_β start_POSTSUBSCRIPT r end_POSTSUBSCRIPT ( italic_x ) end_ARG log divide start_ARG πitalic_θitalic_βitalic_π ( italic_yitalic_i | italic_x ) end_ARG start_ARG πitalic_β start_POSTSUBSCRIPT r end_POSTSUBSCRIPT∗ ( italic_yitalic_i | italic_x ) end_ARG - ∑i = 1K divide start_ARG edivide start_ARG 1 end_ARG start_ARG βitalic_r end_ARG ritalic_ϕ ( italic_x , italic_yitalic_i ) end_ARG start_ARG K Zitalic_β start_POSTSUBSCRIPT r end_POSTSUBSCRIPT ( italic_x ) end_ARG log divide start_ARG edivide start_ARG 1 end_ARG start_ARG βitalic_r end_ARG ritalic_ϕ ( italic_x , italic_yitalic_i ) end_ARG start_ARG K Zitalic_β start_POSTSUBSCRIPT r end_POSTSUBSCRIPT ( italic_x ) end_ARG ] The second term of the final equality remains constant in relation to θ, and thus can be omitted when computing the derivative with respect to θ. Then we further consider the gradient of ℒdpo-rwsubscriptℒdpo-rwL_ dpo-rwLdpo-rw: ∇θℒdpo-rw⁢(πθ)subscript∇subscriptℒdpo-rwsubscript _θL_ dpo-rw( _θ)∇θ Ldpo-rw ( πitalic_θ ) =∇θ∼pref⁢[−1K⁢∑i=1Kπsft⁢(i|)⁢[e1βr⁢rϕ⁢(,i)Zβr⁢()⁢log⁡πθβπ⁢(i|)πβr∗⁢(i|)]]absentsubscript∇subscriptsimilar-tosuperscriptprefdelimited-[]1superscriptsubscript1subscriptsubscriptsftconditionalsubscriptdelimited-[]superscript1subscriptsubscriptitalic-ϕsubscriptsubscriptsubscriptsuperscriptsubscriptsubscriptconditionalsubscriptsuperscriptsubscriptsubscriptconditionalsubscript = _θE_ x ^% pref [- 1K _i=1^KE_ _ sft% ( y_i| x) [ e 1 _rr% _φ( x, y_i)Z_ _r( x)% _θ _π( y_i| x) _% _r^*( y_i| x) ] ]= ∇θ blackboard_Eitalic_x ∼ Dpref [ - divide start_ARG 1 end_ARG start_ARG K end_ARG ∑i = 1K blackboard_Eπ start_POSTSUBSCRIPT sft ( italic_yitalic_i | italic_x ) end_POSTSUBSCRIPT [ divide start_ARG edivide start_ARG 1 end_ARG start_ARG βitalic_r end_ARG ritalic_ϕ ( italic_x , italic_yitalic_i ) end_ARG start_ARG Zitalic_β start_POSTSUBSCRIPT r end_POSTSUBSCRIPT ( italic_x ) end_ARG log divide start_ARG πitalic_θitalic_βitalic_π ( italic_yitalic_i | italic_x ) end_ARG start_ARG πitalic_β start_POSTSUBSCRIPT r end_POSTSUBSCRIPT∗ ( italic_yitalic_i | italic_x ) end_ARG ] ] =∇θ∼pref⁢[−πsft⁢(|)⁢[e1βr⁢rϕ⁢(,)Zβr⁢()⁢log⁡πθβπ⁢(|)πβr∗⁢(|)]]absentsubscript∇subscriptsimilar-tosuperscriptprefdelimited-[]subscriptsubscriptsftconditionaldelimited-[]superscript1subscriptsubscriptitalic-ϕsubscriptsubscriptsuperscriptsubscriptsubscriptconditionalsuperscriptsubscriptsubscriptconditional = _θE_ x ^% pref [-E_ _ sft( y|% x) [ e 1 _rr_φ( x,% y)Z_ _r( x) _θ _% π( y| x) _ _r^*( y|% x) ] ]= ∇θ blackboard_Eitalic_x ∼ Dpref [ - blackboard_Eπ start_POSTSUBSCRIPT sft ( italic_y | italic_x ) end_POSTSUBSCRIPT [ divide start_ARG edivide start_ARG 1 end_ARG start_ARG βitalic_r end_ARG ritalic_ϕ ( italic_x , italic_y ) end_ARG start_ARG Zitalic_β start_POSTSUBSCRIPT r end_POSTSUBSCRIPT ( italic_x ) end_ARG log divide start_ARG πitalic_θitalic_βitalic_π ( italic_y | italic_x ) end_ARG start_ARG πitalic_β start_POSTSUBSCRIPT r end_POSTSUBSCRIPT∗ ( italic_y | italic_x ) end_ARG ] ] =∇θ∼pref⁢[−∑∈πsft⁢(|)⁢e1βr⁢rϕ⁢(,)Zβr⁢()⁢log⁡πθβπ⁢(|)πβr∗⁢(|)]absentsubscript∇subscriptsimilar-tosuperscriptprefdelimited-[]subscriptsubscriptsftconditionalsuperscript1subscriptsubscriptitalic-ϕsubscriptsubscriptsuperscriptsubscriptsubscriptconditionalsuperscriptsubscriptsubscriptconditional = _θE_ x ^% pref [- _ y _ sft% ( y| x) e 1 _rr_φ(% x, y)Z_ _r( x) _% θ _π( y| x) _ _r^*(% y| x) ]= ∇θ blackboard_Eitalic_x ∼ Dpref [ - ∑italic_y ∈ Y πsft ( italic_y | italic_x ) divide start_ARG edivide start_ARG 1 end_ARG start_ARG βitalic_r end_ARG ritalic_ϕ ( italic_x , italic_y ) end_ARG start_ARG Zitalic_β start_POSTSUBSCRIPT r end_POSTSUBSCRIPT ( italic_x ) end_ARG log divide start_ARG πitalic_θitalic_βitalic_π ( italic_y | italic_x ) end_ARG start_ARG πitalic_β start_POSTSUBSCRIPT r end_POSTSUBSCRIPT∗ ( italic_y | italic_x ) end_ARG ] =∇θ∼pref⁢[−∑∈πβr∗⁢(|)⁢log⁡πθβπ⁢(|)πβr∗⁢(|)]absentsubscript∇subscriptsimilar-tosuperscriptprefdelimited-[]subscriptsuperscriptsubscriptsubscriptconditionalsuperscriptsubscriptsubscriptconditionalsuperscriptsubscriptsubscriptconditional = _θE_ x ^% pref [- _ y _ _r^*% ( y| x) _θ _π(% y| x) _ _r^*( y| % x) ]= ∇θ blackboard_Eitalic_x ∼ Dpref [ - ∑italic_y ∈ Y πitalic_β start_POSTSUBSCRIPT r end_POSTSUBSCRIPT∗ ( italic_y | italic_x ) log divide start_ARG πitalic_θitalic_βitalic_π ( italic_y | italic_x ) end_ARG start_ARG πitalic_β start_POSTSUBSCRIPT r end_POSTSUBSCRIPT∗ ( italic_y | italic_x ) end_ARG ] =∇θ∼pref[KL(πβr∗(⋅|)∥πθβπ(⋅|))], = _θE_ x ^% pref [D_ KL( _ _r^*(·|% x)\| _θ _π(·| x)) ],= ∇θ blackboard_Eitalic_x ∼ Dpref [ blackboard_DKL ( πitalic_β start_POSTSUBSCRIPT r end_POSTSUBSCRIPT∗ ( ⋅ | italic_x ) ∥ πitalic_θitalic_βitalic_π ( ⋅ | italic_x ) ) ] , which completes the proof of Theorem 3.3. ∎ A.5 Mechanistic Understanding of ∇θℒexo⁢(πθ)subscript∇subscriptℒexosubscript _θL_ exo( _θ)∇θ Lexo ( πitalic_θ ) We present the gradient of ℒexo⁢(πθ)subscriptℒexosubscriptL_ exo( _θ)Lexo ( πitalic_θ ) defined in Eq. (3.2): ∇θℒexo⁢(πθ)=−∼pref⁢πsft⁢(1:K|)⁢[∑k=1Kpfθ⁢(k)⁢(KL⁢(pfθ∥prϕ)⏟distributional gap−log⁡pfθ⁢(k)prϕ⁢(k)⏟point-wise correction)⁢∇θlog⁡πθ⁢(k|)],subscript∇subscriptℒexosubscriptsubscriptsimilar-tosuperscriptprefsubscriptsubscriptsftconditionalsubscript:1delimited-[]superscriptsubscript1subscriptsubscriptsubscript⏟subscriptKLconditionalsubscriptsubscriptsubscriptsubscriptitalic-ϕdistributional gapsubscript⏟subscriptsubscriptsubscriptsubscriptitalic-ϕpoint-wise correctionsubscript∇subscriptconditionalsubscript _θL_ exo( _θ)=-E_% x prefE_ _ sft% ( y_1:K| x) [ _k=1^Kp_f_θ(k)% ( D_ KL(p_f_θ\|p_r_φ)_% distributional gap- p_f_θ(k)p_r_% φ(k)_point-wise correction ) _θ _% θ( y_k| x) ],∇θ Lexo ( πitalic_θ ) = - blackboard_Eitalic_x ∼ Dpref blackboard_Eπ start_POSTSUBSCRIPT sft ( italic_y1 : K | italic_x ) end_POSTSUBSCRIPT [ ∑k = 1K pitalic_f start_POSTSUBSCRIPT θ end_POSTSUBSCRIPT ( k ) ( under⏟ start_ARG blackboard_DKL ( pitalic_f start_POSTSUBSCRIPT θ end_POSTSUBSCRIPT ∥ pitalic_r start_POSTSUBSCRIPT ϕ end_POSTSUBSCRIPT ) end_ARGdistributional gap - under⏟ start_ARG log divide start_ARG pitalic_f start_POSTSUBSCRIPT θ end_POSTSUBSCRIPT ( k ) end_ARG start_ARG pitalic_r start_POSTSUBSCRIPT ϕ end_POSTSUBSCRIPT ( k ) end_ARG end_ARGpoint-wise correction ) ∇θ log πitalic_θ ( italic_yitalic_k | italic_x ) ] , (26) where pfθ⁢(k)subscriptsubscriptp_f_θ(k)pitalic_f start_POSTSUBSCRIPT θ end_POSTSUBSCRIPT ( k ) and prϕ⁢(k)subscriptsubscriptitalic-ϕp_r_φ(k)pitalic_r start_POSTSUBSCRIPT ϕ end_POSTSUBSCRIPT ( k ) are short for pfθ⁢(k|1:K,)subscriptsubscriptconditionalsubscript:1p_f_θ(k| y_1:K, x)pitalic_f start_POSTSUBSCRIPT θ end_POSTSUBSCRIPT ( k | italic_y1 : K , italic_x ) and prϕ⁢(k|1:K,)subscriptsubscriptitalic-ϕconditionalsubscript:1p_r_φ(k| y_1:K, x)pitalic_r start_POSTSUBSCRIPT ϕ end_POSTSUBSCRIPT ( k | italic_y1 : K , italic_x ), which are defined in Eq. (11) and Eq. (12) respectively. Next, we provide a mechanistic understanding of ∇θℒexo⁢(πθ)subscript∇subscriptℒexosubscript _θL_ exo( _θ)∇θ Lexo ( πitalic_θ ), which is the expected weighted sum of the gradients of the log likelihood on samples 1:Ksubscript:1 y_1:Kitalic_y1 : K drawn from πsft(⋅|) _ sft(·| x)πsft ( ⋅ | italic_x ). The weight is proportional to the difference between the log probability ratio log⁡pfθ⁢(k)prϕ⁢(k)subscriptsubscriptsubscriptsubscriptitalic-ϕ p_f_θ(k)p_r_φ(k)log divide start_ARG pitalic_f start_POSTSUBSCRIPT θ end_POSTSUBSCRIPT ( k ) end_ARG start_ARG pitalic_r start_POSTSUBSCRIPT ϕ end_POSTSUBSCRIPT ( k ) end_ARG and the KL divergence DKL⁢(pfθ∥prϕ)subscriptKLconditionalsubscriptsubscriptsubscriptsubscriptitalic-ϕD_KL(p_f_θ\|p_r_φ)DKL ( pitalic_f start_POSTSUBSCRIPT θ end_POSTSUBSCRIPT ∥ pitalic_r start_POSTSUBSCRIPT ϕ end_POSTSUBSCRIPT ). Intuitively, if the policy has already correctly weighted the sample yksubscripty_kyitalic_k according to the reward model, i.e., prϕ⁢(k)=pfθ⁢(k)subscriptsubscriptitalic-ϕsubscriptsubscriptp_r_φ(k)=p_f_θ(k)pitalic_r start_POSTSUBSCRIPT ϕ end_POSTSUBSCRIPT ( k ) = pitalic_f start_POSTSUBSCRIPT θ end_POSTSUBSCRIPT ( k ), then log⁡pfθ⁢(k)prϕ⁢(k)=0subscriptsubscriptsubscriptsubscriptitalic-ϕ0 p_f_θ(k)p_r_φ(k)=0log divide start_ARG pitalic_f start_POSTSUBSCRIPT θ end_POSTSUBSCRIPT ( k ) end_ARG start_ARG pitalic_r start_POSTSUBSCRIPT ϕ end_POSTSUBSCRIPT ( k ) end_ARG = 0 and the weight suggests that it only needs to minimize the overall KL divergence between pfθsubscriptsubscriptp_f_θpitalic_f start_POSTSUBSCRIPT θ end_POSTSUBSCRIPT and prϕsubscriptsubscriptitalic-ϕp_r_φpitalic_r start_POSTSUBSCRIPT ϕ end_POSTSUBSCRIPT on the distribution level. If the policy πθsubscript _θπitalic_θ overestimates or underestimates the sample yksubscripty_kyitalic_k, i.e., log⁡pfθ⁢(k)prϕ⁢(k)>0subscriptsubscriptsubscriptsubscriptitalic-ϕ0 p_f_θ(k)p_r_φ(k)>0log divide start_ARG pitalic_f start_POSTSUBSCRIPT θ end_POSTSUBSCRIPT ( k ) end_ARG start_ARG pitalic_r start_POSTSUBSCRIPT ϕ end_POSTSUBSCRIPT ( k ) end_ARG > 0 or log⁡pfθ⁢(k)prϕ⁢(k)<0subscriptsubscriptsubscriptsubscriptitalic-ϕ0 p_f_θ(k)p_r_φ(k)<0log divide start_ARG pitalic_f start_POSTSUBSCRIPT θ end_POSTSUBSCRIPT ( k ) end_ARG start_ARG pitalic_r start_POSTSUBSCRIPT ϕ end_POSTSUBSCRIPT ( k ) end_ARG < 0, this log-ratio will be used to calibrate the KL divergence to penalize or encourage the policy to update towards increasing the likelihood of this sample at a faster rate. Finally, this gradient form offers us the insight of when the optimization should stop: the gradient ∇θℒexo⁢(πθ)subscript∇subscriptℒexosubscript _θL_exo( _θ)∇θ Lexo ( πitalic_θ ) becomes 0 when the two distribution pfθsubscriptsubscriptp_f_θpitalic_f start_POSTSUBSCRIPT θ end_POSTSUBSCRIPT and prϕsubscriptsubscriptitalic-ϕp_r_φpitalic_r start_POSTSUBSCRIPT ϕ end_POSTSUBSCRIPT are identical. A.6 Illustrating the Relationship among the Objectives in §3 We illustrate the relationship among the objectives lhfβ⁢(πθ)superscriptsubscriptlhfsubscriptJ_ lhf^β( _θ)Jlhfitalic_β ( πitalic_θ ), lhfβr⁢(πθβπ)superscriptsubscriptlhfsubscriptsuperscriptsubscriptsubscriptJ_ lhf _r( _θ _π)Jlhfitalic_βitalic_r ( πitalic_θitalic_βitalic_π ), ℒdpo⁢(πθ)subscriptℒdposubscriptL_ dpo( _θ)Ldpo ( πitalic_θ ), ℒdpo-rw⁢(πθ)subscriptℒdpo-rwsubscriptL_ dpo-rw( _θ)Ldpo-rw ( πitalic_θ ) and ℒexo⁢(πθ)subscriptℒexosubscriptL_ exo( _θ)Lexo ( πitalic_θ ) in Figure 5. Figure 5: Illustration of the relationship among the different objectives discussed in §3. 1: lhfβr⁢(πθβπ)superscriptsubscriptlhfsubscriptsuperscriptsubscriptsubscriptJ_ lhf _r( _θ _π)Jlhfitalic_βitalic_r ( πitalic_θitalic_βitalic_π ) is a generalized version of lhfβ⁢(πθ)superscriptsubscriptlhfsubscriptJ_ lhf^β( _θ)Jlhfitalic_β ( πitalic_θ ) by distributing the KL regularization to both the learned policy πθsubscript _θπitalic_θ and the reward model rϕsubscriptitalic-ϕr_φritalic_ϕ (§3.1). 2: ℒdpo⁢(πθ)subscriptℒdposubscriptL_ dpo( _θ)Ldpo ( πitalic_θ ) is derived based on the optimal policy of lhfβ⁢(πθ)superscriptsubscriptlhfsubscriptJ_ lhf^β( _θ)Jlhfitalic_β ( πitalic_θ ) (§2.3). 3: ℒexo⁢(πθ)subscriptℒexosubscriptL_ exo( _θ)Lexo ( πitalic_θ ) is equivalent to lhfβr⁢(πθβπ)superscriptsubscriptlhfsubscriptsuperscriptsubscriptsubscriptJ_ lhf _r( _θ _π)Jlhfitalic_βitalic_r ( πitalic_θitalic_βitalic_π ) in terms of their optimization directions (§3.2). 4: ℒdpo-rwsubscriptℒdpo-rwL_ dpo-rwLdpo-rw is the generalized version of ℒdposubscriptℒdpoL_ dpoLdpo by subsituting the pariwise loss with softmax loss over K responses.(§3.3). The optimal policy, denoted by a dotted line, assumes unlimited model capacity. The solution, shown with a solid line, is the practically achievable policy within the realistic constraints of model capacity. Appendix B Training and Evaluation Details Training. In the controlled text generation task, we use βπ=0.1subscript0.1 _π=0.1βitalic_π = 0.1 for EXOprefsubscriptEXOpref EXO_ prefEXOpref and DPOprefsubscriptDPOpref DPO_ prefDPOpref, and additionally use βr=0.1subscript0.1 _r=0.1βitalic_r = 0.1 and K∈4,848K∈\4,8\K ∈ 4 , 8 for EXOrwsubscriptEXOrw EXO_ rwEXOrw and DPOrwsubscriptDPOrw DPO_ rwDPOrw. For the tasks of summarization, dialogue generation, and instruction following, we use βπ=0.5subscript0.5 _π=0.5βitalic_π = 0.5 for EXOprefsubscriptEXOpref EXO_ prefEXOpref and DPOprefsubscriptDPOpref DPO_ prefDPOpref, and additionally use βr=0.1subscript0.1 _r=0.1βitalic_r = 0.1 and K=44K=4K = 4 for EXOrwsubscriptEXOrw EXO_ rwEXOrw and DPOrwsubscriptDPOrw DPO_ rwDPOrw. We provide additional results of tuning βrsubscript _rβitalic_r and βπsubscript _πβitalic_π in Appendix C.1 to justify our choice of hyperparameters. In all experiments, we set the label smoothing hyperparameter ε ε in EXOprefsubscriptEXOpref EXO_ prefEXOpref to 1e-3. For DPO and EXO, we use the Adam optimizer with a universal learning rate of 1e-6 and a batch size of 64 and train for one epoch on each dataset, although both methods converge within one epoch. All the hyperparameters are set to be the same for DPO and EXO for a fair comparison. For PPO, we modify based on the implementation of DeepSpeed-Chat (Yao et al., 2023), which sets γ=11γ=1γ = 1 and β=0.10.1β=0.1β = 0.1 by default. We pretrain the critic model for the first few steps while freezing the actor model and find it to improve convergence. We empirically tune the number of actor-freezing steps, total training steps, learning rate for actor and critic model, and the batch size for PPO on each dataset, as PPO is sensitive to these hyperparameters. Specifically, we conduct 15 trials of hyperparameter search on the IMDB dataset and 10 trials on the real human preference datasets in total. In the experiment, we report the PPO performance with the best hyperparameters obtained under constrained number of hyperparameter search trials. We conduct the experiments except for instruction following on 8 V100 GPUs. For instruction following task, we train the models on 8 A100 GPUs. Evaluation. At inference time, we sample 4 completions from the learned policy for each prompt and consider 512 prompts from the test set for all datasets. Except for the instruction following task, we sample from the policy with the same temperature τ=0.80.8τ=0.8τ = 0.8 that is set during training for consistency. For the instruction following task, we use top-p sampling and empirically set p=0.90.9p=0.9p = 0.9 and temperature τ=0.950.95τ=0.95τ = 0.95 given its performance. To calculate the win rate evaluated by the reward model, we consider all combinations of pairs between the completions generated by the learned policy and the base completions (either generated by the SFT policy or the chosen completion in the dataset) and then compare the scores from the reward model on the pairs of generations. For the evaluations using GPT-4, we sample 100 prompts and 1 completion for each prompt under each policy. To mitigate the position bias of GPT-4, we evaluate one pair of generations twice by swapping the order of responses in each pair. To evaluate the quality of the summaries, we use the concise prompt of Rafailov et al. (2023) as shown in Table 3. To evaluate the helpfulness of the generated dialogues, we use the prompt shown in Table 4, which is modified based on the prompt of Rafailov et al. (2023) for single-turn dialogue to accommodate the general multi-turn setting. For the instruction-following task, we use the prompt modified from the prompt for reference-guided pairwise comparison provided in Zheng et al. (2023a). Human Assessment. We conduct human assessment to evaluate the instruction following task more thoroughly. Specifically, we select three matchups that pair the generated outputs of EXO with those produced by DPO, PPO and the SFT policy. Given 100 randomly sampled test instructions, each model generates 100 responses, which results in a total of 300 pairs of comparisons. We assign 3 human labelers to each comparison, producing 900 judgements in total. Given the instruction, each human annotator is provided with two generated answers by two systems respectively together with a high-quality reference answer. The annotator is then asked to make a preference among win, tie or lose by comparing the generated answers with the reference answer, considering the criteria including adherence to instruction, correctness, fluency, safety and helpfulness. Specifically, adherence to instruction encapsulates the model’s comprehension and following of the prompt’s intention. Correctness involves the identification of inaccurate knowledge or logical inconsistencies within the generated responses. Fluency assesses the linguistic coherence, encompassing an examination of sentence completeness, grammatical accuracy, and the presence of a consistent language structure. Safety refers to the inspection for potentially harmful content. Lastly, helpfulness indicates whether the responses provide the information required by the prompt or contribute to problem resolution. Which of the following summaries does a better job of summarizing the most important points in the given forum post, without including unimportant or irrelevant details? A good summary is both precise and concise. Post: <post> Summary A: <Summary A> Summary B: <Summary B> FIRST provide a one-sentence comparison of the two summaries, explaining which you prefer and why. SECOND, on a new line, state only "A" or "B" to indicate your choice. Your response should use the format: Comparison: <one-sentence comparison and explanation> Preferred: <"A" or "B"> Table 3: Prompt for GPT-4 evaluation on the summarization task. Texts in blue are placeholders to be substituted by the real data. For the following dialogue history to a chatbot, which response is more helpful? Dialogue history: <dialogue history> Response A: <Response A> Response B: <Response B> FIRST provide a one-sentence comparison of the two responses and explain which you feel is more helpful. SECOND, on a new line, state only "A" or "B" to indicate which response is more helpful. Your response should use the format: Comparison: <one-sentence comparison and explanation> More helpful: <"A" or "B"> Table 4: Prompt for GPT-4 evaluation on the dialogue generation task. Texts in blue are placeholders to be substituted by the real data. Appendix C Additional Experiment Results C.1 Ablation Study of βrsubscript _rβitalic_r and βπsubscript _πβitalic_π We present an ablation study to investigate the performance of EXOrwsubscriptEXOrw EXO_ rwEXOrw on the dialogue generation task by varying βrsubscript _rβitalic_r and βπsubscript _πβitalic_π respectively. We execute multiple runs bifurcated into two series. We set βπ=0.5subscript0.5 _π=0.5βitalic_π = 0.5 as the default value and vary βr∈0.1,0.25,0.5,0.75,1.0subscript0.10.250.50.751.0 _r∈\0.1,0.25,0.5,0.75,1.0\βitalic_r ∈ 0.1 , 0.25 , 0.5 , 0.75 , 1.0 . Subsequently, the process is reversed whereby we fix βr=0.1subscript0.1 _r=0.1βitalic_r = 0.1, and alter βπ∈0.1,0.25,0.5,0.75,1.0subscript0.10.250.50.751.0 _π∈\0.1,0.25,0.5,0.75,1.0\βitalic_π ∈ 0.1 , 0.25 , 0.5 , 0.75 , 1.0 . We present the results in Figure 6. (a) Reward model win rate. (b) GPT-4 win rate. Figure 6: Win rate performance of EXOrwsubscriptEXOrw EXO_ rwEXOrw evaluated by the reward model and the GPT-4 by varying βrsubscript _rβitalic_r and βπsubscript _πβitalic_π. From Figure 6 (a), we observe that reducing either βrsubscript _rβitalic_r and βπsubscript _πβitalic_π increase the reward model win rate, which is expected as a small βr×βπsubscriptsubscript _r× _πβitalic_r × βitalic_π encourages the policy to optimize the reward model while neglecting the KL regularization. In Figure 6 (b), the GPT-4 win rate starts to decline when βπ<0.5subscript0.5 _π<0.5βitalic_π < 0.5, which suggests where the reward model starts to be over-optimized. The different effects of tuning βπsubscript _πβitalic_π and βrsubscript _rβitalic_r on the performance could be attributed to their different roles in our algorithm, i.e., βπsubscript _πβitalic_π scales the log probability in the parametrized policy while βrsubscript _rβitalic_r scales the reward. Based on the results, we recommend adopting a moderate value for βπsubscript _πβitalic_π and a lower value for βrsubscript _rβitalic_r, for instance, βπ=0.5subscript0.5 _π=0.5βitalic_π = 0.5 and βr=0.1subscript0.1 _r=0.1βitalic_r = 0.1. C.2 Effect of βrsubscript _rβitalic_r and βπsubscript _πβitalic_π beyond the Product β=βr⁢βπsubscriptsubscriptβ= _r _πβ = βitalic_r βitalic_π To further demonstrate the effect of βrsubscript _rβitalic_r and βπsubscript _πβitalic_π beyond the impact on their product β=βπ⁢βrsubscriptsubscriptβ= _π _rβ = βitalic_π βitalic_r, we fix β=0.050.05β=0.05β = 0.05 while tuning βπ∈1,0.5,0.1,0.05subscript10.50.10.05 _π∈\1,0.5,0.1,0.05\βitalic_π ∈ 1 , 0.5 , 0.1 , 0.05 and βr∈0.05,0.1,0.5,1subscript0.050.10.51 _r∈\0.05,0.1,0.5,1\βitalic_r ∈ 0.05 , 0.1 , 0.5 , 1 accordingly. The result is shown in Table 5. According to the evaluation by GPT-4, the best performnce is obtained at βπ=0.5,βr=0.1formulae-sequencesubscript0.5subscript0.1 _π=0.5, _r=0.1βitalic_π = 0.5 , βitalic_r = 0.1, while neither the hyper-parameter choice of βπ=1,βr=0.05formulae-sequencesubscript1subscript0.05 _π=1, _r=0.05βitalic_π = 1 , βitalic_r = 0.05 (the configuration of PPO) nor the choice of βπ=0.05,βr=1formulae-sequencesubscript0.05subscript1 _π=0.05, _r=1βitalic_π = 0.05 , βitalic_r = 1 (the configuration of DPO) yield the highest win rate evaluated by GPT-4. While using a small βπsubscript _πβitalic_π is more likely to trigger over-optimization of the reward model, which indicates the asymmetric effect of the two hyperparameters βrsubscript _rβitalic_r and βπsubscript _πβitalic_π on the optimization process. βrsubscript _rβitalic_r βπsubscript _πβitalic_π Reward Model (%) GPT-4 (%) vs SFT vs Chosen vs SFT vs Chosen 1.0 0.05 87.1 88.4 81.0 57.5 0.5 0.1 87.1 88.0 77.0 59.5 0.1 0.5 85.6 87.2 83.5 60.0 0.05 1.0 84.1 84.8 76.5 52.0 Table 5: Effect of βrsubscript _rβitalic_r and βπsubscript _πβitalic_π beyond β by tuning βπsubscript _πβitalic_π and βrsubscript _rβitalic_r while keeping their product β=βr⁢βπsubscriptsubscriptβ= _r _πβ = βitalic_r βitalic_π fixed. C.3 Frontier of Classifier Accuarcy against KL Divergence We additionally calculate the accuracy of the sentiment classifier by taking the sigmoid of the oracle reward and plot the frontier of the accuracy vs reverse KL in Figure 7. Figure 7: The frontier of classifier accuracy vs reverse KL to the SFT policy of different methods in the controlled experiment. C.4 Optimization Efficiency To demonstrate the efficency of optimizing the oracle reward, we plot the curve of the oracle reward with standard deviation on the test set of different training steps. Specifically, the standard deviation is calculated on the 4 samples generated given the same prompt and then averaged across 512 prompts. Figure 8: The curve of the oracle reward with standard deviation on the test set versus the training steps of different methods in the controlled experiment. C.5 Additional Visualization Results We additionally sample more prompts from the test set of the IMDb dataset, and plot the estimated density ratio of the policy learned by EXO and DPO and the optimal policy in Figure 9. Similar results were obtained among these diverse set of prompts: the density ratio under the EXO policy better matches the optimal policy at the high probability region against πsftsubscriptsft _ sftπsft than the DPO policy. And very appearently, the DPO policy tends to overestimate the long tail in πsftsubscriptsft _ sftπsft and thus often loses resolution in capturing the modes of the optimal policy. C.6 Sample Outputs with GPT-4 Judgments We present the sample outputs generated by EXO and DPO and the judgments made by GPT-4 when compared to the chosen response in the preference dataset. Table 6 and Table 7 show examples on the task of dialogue generation and summarization respectively. (a) Estimated density ratio of the EXO, DPO and optimal policy given the prompt “Is this supposed to be serious? I hope not”. (b) Estimated density ratio of the EXO, DPO and optimal policy given the prompt “Great book, great movie, great soundtrack. Frank”. (c) Estimated density ratio of the EXO, DPO and optimal policy given the prompt “This is indeed the film that popularized kung”. (d) Estimated density ratio of the EXO, DPO and optimal policy given the prompt “This movie is about a group of people who are”. (e) Estimated density ratio of the EXO, DPO and optimal policy given the prompt “What we have here the standard Disney direct to DVD”. (f) Estimated density ratio of the EXO, DPO and optimal policy given the prompt “Once the slow beginning gets underway, the film kicks”. Figure 9: Visualization of the estimated density ratio of the policy learned by EXO and DPO, and the optimal policy given 10 test prompts randomly sampled from the IMDb dataset. Dialogue history Human: How do I make a fishtail braid? Assistant: Hey human, just walk down the left and then down the right side, then at the end, you pull through. Are you looking to just braid, or a tight ponytail? Human: Yes. Can you tell me the steps to do it? Chosen response Assistant: Sure, here’s how to do a side braid: 1) Pull hair up to the back of your head, leaving it out from the rest of your hair so you can see it. 2) Put one hair on the side where you want to make your braid. 3) Hold the braid with your thumb and forefinger 4) Bring your braid through the opposite side (putting the hair to the other side of the braid. 5) Repeat steps 2, 3, and 4 as needed until you’ve completed your braid. EXO response Assistant: Sure! I’l walk you through it step by step: 1. Stand with your left hand on the back of your head. Grab a left side section of hair, using the right hand to pull it through. 2. Take a right side section of hair, and pull it through 3. Grab a left side section of hair, and pull it through 4. Grab a right side section of hair, and pull it through 5. Repeat steps 2-4 for the last section of hair 6. Wrap the hair around your head and secure it at the nape of your neck. GPT-4 Judgment (A = EXO, B = Chosen) Response A provides a more detailed and step-by-step guide on how to make a fishtail braid, while Response B seems to describe a different type of braid and is less clear. GPT-4 Judgment (A = Chosen, B = EXO) Response B is more helpful because it provides a clearer, step-by-step guide on how to make a fishtail braid, whereas Response A seems to describe a different type of braid and is less clear. DPO response Assistant: Sure, let’s walk through it. You’l want to start with a braid about two and a half inches long, then you want to braid it together and pull through, which means you’l somehow thread it through your own hair and tie it off at the other end. Then you’re going to braid it again, but this time it will be a fishtail, since you’l pull it through. You’l keep doing this, braiding the fishtail. GPT-4 Judgment (A = DPO, B = Chosen) Response B is more helpful as it provides a clear, step-by-step guide on how to make a fishtail braid, while Response A is confusing and lacks clarity. GPT-4 Judgment (A = Chosen, B = DPO) Response A provides a clear, step-by-step guide on how to make a side braid, while Response B gives a vague description of how to make a fishtail braid without clear steps. Table 6: Responses generated by EXO and DPO given the dialogue history sampled from the Anthropic-H test set. GPT-4 consistently prefers EXO and disprefers DPO over the chosen response regardless of the order in which the evaluated pairs are presented. Post SUBREDDIT: r/Dogtraining TITLE: Best walks when on a choke chain: dog isn’t being choked, but is it bringing up bad memories? POST: My 3 year old GSD x Husky rescue stayed with some relatives for a few days while my boyfriend and I moved house, and the relatives found an old choke chain to walk him with, instead of his Mekuti harness. We have had real problems with him pulling on the lead, and the Mekuti harness has worked wonders, but didn’t eliminate the problem. He hasn’t pulled when on the choke chain at all, even when it was first put on apparently, so we think that is what he was trained on (and damage from that process is probably why his barks and whines sound a bit odd). We now think the people who had him before us were violent towards him (if you mime kicking he runs across the room terrified), so, my question is: he is best behaved on walks when wearing a choke chain, and it isn’t hurting him at all, but will the harm chokers and the people using them caused him in the past affect him in any way now? I 100% DO NOT want to use a choke chain on him and I don’t think anyone should ever, ever train a dog that way, but this chain is the only lead we’ve used that has completely solved the problem with him pulling, and he hasn’t seemed stressed out by it. If he ever started pulling on the lead so that the choker was tight, I would stop using it straight away. The last thing I want is my dog being hurt. If I’d known these relatives were going to put one on him, I wouldn’t have let them look after him. Still, it works and doesn’t hurt him, my only worry is it’l do psychological damage somehow. Chosen summary TL;DR: 3yr old husky rescue pulling on choke chain w/ no damage, but using it on walks occasionally causes him stress. Can this damage actually cause harm in the future? EXO summary TL;DR: Best walks are on a choke chain but dog isn’t pulling on lead, has had no problems with pull before, not sure if past violent treatment will affect him now, does it cause harm to dogs being on choke chains/when being walked on them? GPT-4 Judgment (A = EXO, B = Chosen) Summary A provides a more comprehensive and accurate portrayal of the post’s points, despite a few misplaced details, whereas Summary B incorrectly states the dog is showing stress and pulling on the choke chain. GPT-4 Judgment (A = Chosen, B = EXO) Summary B is preferred because it more accurately summarizes the main points of the user’s post by mentioning the choke chain’s effect on walks, mentioning the absence of negative effects of pull so far, and raising concerns about past violent treatment potentially affecting the dog’s behavior. DPO summary TL;DR: If you took a dog and had it wear chokes it’s self-choked, will it be traumatized in any way when you put it on and no longer train it to be choked? GPT-4 Judgment (A = DPO, B = Chosen) Summary B is preferred as it captures the key details more accurately, including the dog’s breed, the issue of pulling, and the potential future harm, while Summary A is ambiguous and loss a lot of context. GPT-4 Judgment (A = Chosen, B = DPO) Summary A is more accurate and detailed because it correctly describes the specific situation of the dog and the concerns associated with using a choke chain, while Summary B is vague and confusing in its wording. Table 7: Summaries generated by EXO and DPO given the post sampled from the TL;DR test set. GPT-4 consistently prefers EXO and disprefers DPO over the chosen summary regardless of the order in which the evaluated pairs are presented.

Instant research discovery