Paper deep dive
Learning a Pessimistic Reward Model in RLHF
Yinglun Xu, Hangoo Kang, Tarun Suresh, Yuxuan Wan, Gagandeep Singh
Models: Pythia-1B
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 95%
Last extracted: 3/12/2026, 6:02:29 PM
Summary
The paper introduces 'PET' (Pessimistic Reward Fine-Tuning), a method to learn a pessimistic reward model for offline RLHF. By training the reward model adversarially against a rejection sampling policy, PET mitigates reward hacking without requiring KL regularization, enabling the agent to greedily optimize policies that achieve high performance and high KL divergence from the dataset distribution.
Entities (5)
Relation Signals (3)
PET → mitigates → Reward Hacking
confidence 98% · A reward model fine-tuned by PET is pessimistic and robust against reward hacking.
PET → appliedto → TL;DR summarization dataset
confidence 95% · We test our methods on the standard TL;DR summarization dataset.
PPO-PET → optimizes → Policy
confidence 95% · We further use the PPO algorithm for policy optimization, which we call ‘PPO-PET’.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:This work proposes `PET', a novel pessimistic reward fine-tuning method, to learn a pessimistic reward model robust against reward hacking in offline reinforcement learning from human feedback (RLHF). Traditional reward modeling techniques in RLHF train an imperfect reward model, on which a KL regularization plays a pivotal role in mitigating reward hacking when optimizing a policy. Such an intuition-based method still suffers from reward hacking, and the policies with large KL divergence from the dataset distribution are excluded during learning. In contrast, we show that when optimizing a policy on a pessimistic reward model fine-tuned through PET, reward hacking can be prevented without relying on any regularization. We test our methods on the standard TL;DR summarization dataset. We find that one can learn a high-quality policy on our pessimistic reward without using any regularization. Such a policy has a high KL divergence from the dataset distribution while having high performance in practice. In summary, our work shows the feasibility of learning a pessimistic reward model against reward hacking. The agent can greedily search for the policy with a high pessimistic reward without suffering from reward hacking.
Tags
Links
- Source: https://arxiv.org/abs/2505.20556
- Canonical: https://arxiv.org/abs/2505.20556
Trouble viewing inline? Open PDF directly →
Full Text
127,023 characters extracted from source content.
Expand or collapse full text
Learning a Pessimistic Reward Model in RLHF Yinglun Xu, Hangoo Kang11footnotemark: 1, Tarun Suresh, Yuxuan Wan, Gagandeep Singh University of Illinois Urbana-Champaign yinglun6, hangook2, tsuresh3, yuxuanw8, ggnds@illinois.edu Equal contribution Abstract This work proposes ‘PET’, a novel pessimistic reward fine-tuning method, to learn a pessimistic reward model robust against reward hacking in offline reinforcement learning from human feedback (RLHF). Traditional reward modeling techniques in RLHF train an imperfect reward model, on which a KL regularization plays a pivotal role in mitigating reward hacking when optimizing a policy. Such an intuition-based method still suffers from reward hacking, and the policies with large KL divergence from the dataset distribution are excluded during learning. In contrast, we show that when optimizing a policy on a pessimistic reward model fine-tuned through PET, reward hacking can be prevented without relying on any regularization. We test our methods on the standard TL;DR summarization dataset. We find that one can learn a high-quality policy on our pessimistic reward without using any regularization. Such a policy has a high KL divergence from the dataset distribution while having high performance in practice. In summary, our work shows the feasibility of learning a pessimistic reward model against reward hacking. The agent can greedily search for the policy with a high pessimistic reward without suffering from reward hacking. 1 Introduction Reinforcement learning from human feedback (RLHF) (Christiano et al., 2017) has become crucial in aligning large language models (LLMs), making LLMs more helpful, truthful, and harmless (Stiennon et al., 2020; Bai et al., 2022a; Ouyang et al., 2022; Rafailov et al., 2023). In a typical RLHF training framework (Ouyang et al., 2022), an agent first learns a reward model as a proxy for human preference that best interprets the preference training data. Then, the agent optimizes a policy on the proxy reward model to achieve a high reward. In the real world, such a standard training framework faces the critical problem of ‘reward hacking,’ also known as ‘reward over-optimization’ (Eisenstein et al., 2023; Tien et al., 2022; Gao et al., 2023). Due to the dataset’s size limitation, the proxy reward model is not always accurate. Given a prompt (task input), there could exist a response (task output) that is not favored by human preference, but the learned proxy reward model overestimates the response and gives it a high reward. A greedy agent that searches for the policy with the highest proxy reward can learn to output such low-quality and overestimated responses. This is a typical instance of reward hacking. Due to the over-estimations in the proxy reward model, reward hacking can happen during policy optimization and decrease the learning efficiency of an algorithm (Gao et al., 2023), making it one of the most urgent challenges in RLHF. Challenges in preventing reward hacking: To avoid reward hacking, the idea of ‘pessimism’ is necessary for the learning agent (Levine et al., 2020). At a high level, the agent must be pessimistic when evaluating the policy’s performance based on the dataset. For example, in the framework described above, to avoid over-estimation, the agent should expect the case where the true performance of a policy is less than its performance evaluated based on the proxy reward. The key challenge here is controlling the degree of pessimism, that is, how low the true performance of a policy can be based on the preference dataset. While sufficient pessimism can avoid over-estimation and mitigate reward hacking, over-pessimism results in an under-estimation of high-quality policies, causing the agent to exclude these good policies during training. KL regularization plays a pivotal role in current RLHF methods to mitigate the risk of reward hacking Ouyang et al. (2022). Intuitively, the KL divergence between the response distributions of a policy and the dataset can indicate the degree of uncertainty in evaluating the policy. A policy with higher uncertainty in evaluation requires the agent to be more pessimistic about it. Current algorithms explicitly (such as PPO Schulman et al. (2017)) or implicitly (such as rejection sampling Touvron et al. (2023)) rely on the KL divergence during policy optimization to mitigate the over-estimations in the proxy reward. However, such intuition-based methods are not always efficient in controlling pessimism. Reward hacking is still observed when using such methods Gao et al. (2023); Rafailov et al. (2024), indicating that they are not pessimistic enough for some policies. In the meantime, the methods may be overly pessimistic for some policies with a large KL divergence from the dataset distribution. This is verified by our empirical observations in Section 4. Recent studies use the adversarial training technique Zhang et al. (2024); Xie et al. (2024); Liu et al. (2024b); Cen et al. (2024); Ji et al. (2024b) to achieve provable robustness against reward hacking. However, these methods are based on the direct policy optimization (DPO) technique Rafailov et al. (2023) that requires KL regularization, which again unnecessarily induces over-pessimism. While KL regularization has almost always been considered necessary in practice, it is unlikely to be the most efficient approach to control pessimism. Therefore, a research question arises: Can efficient pessimism be achieved by learning a pessimistic reward model whose prediction is trustworthy? Can an agent greedily optimize a policy on the pessimistic reward without using regularization to prevent reward hacking? Figure 1: A three-step reward-based learning framework. The first step is the traditional reward modeling that trains a reward model with minimal loss on predicting the dataset preference. The second step fine-tunes the learned reward model to make it pessimistic. Particularly, the reward model is adversarially trained against a policy model induced by the rejection sampling process. The reward model should still induce minimal prediction loss on the dataset. In the last step, the framework optimizes a policy on the pessimistic reward and outputs the learned policy. Rejection sampling with adversarial training: In this work, we provide positive answers to the above questions. Our method is based on the adversarial training technique (Cheng et al., 2022; Bhardwaj et al., 2023; Zhan et al., 2023; Gupta et al., 2025). In the framework, a policy model is trained to achieve a high reward on a reward model, while the reward model is trained to give a low reward to the policy model. Here, the main challenge comes from the instability of adversarial training. Intuitively, the policy model is optimized on different reward models during training, which complicates training convergence. Current works, such as Liu et al. (2024b), mitigate the challenge by using the DPO technique to simplify the training process. However, as discussed earlier, such a method requires KL regularization, which induces over-pessimism. To mitigate the challenge without using regularization, we consider using the rejection sampling technique (Touvron et al., 2023; Beirami et al., 2024) to simplify the adversarial training process. Given an initial policy model and a reward model, the rejection sampling process samples several responses from the policy model and outputs the response with the highest reward as described in Alg 4 in the appendix. Rejection sampling performs policy optimization at inference time, making it easy to implement and compute. Note that the traditional rejection sampling process suffers from reward hacking on the proxy reward Touvron et al. (2023). Fortunately, the combination of rejection sampling with adversarial training simultaneously solves the problems of reward hacking in rejection sampling and the difficulty of policy optimization in adversarial training. Based on the idea of adversarial training against rejection sampling, we develop a novel reward fine-tuning method and a corresponding three-step RLHF algorithm. More specifically, our contributions are as follows: 1. We develop a novel reward fine-tuning method called ‘PET’. A reward model fine-tuned by PET is pessimistic and robust against reward hacking. Under standard assumptions Liu et al. (2024b), we theoretically prove that the rejection sampling process on the pessimistic reward fine-tuned by PET has a performance comparable to the rejection sampling process on any reward, as long as the corresponding policy induced by the process is covered by the dataset. 2. Based on ‘PET’, we develop a three-step RLHF framework. In Fig. 1 we intuitively show how the framework works. The first step is the standard reward modeling that learns a proxy reward model with minimal prediction loss on the preference dataset. The second step is PET, which fine-tunes the proxy reward to make it pessimistic. The last step is policy optimization on the learned pessimistic reward model. In principle, in the last step, the agent can greedily optimize a policy to achieve the highest reward on the pessimistic reward model without using any regularization. 3. We test our method on the TL;DR summarization dataset Völske et al. (2017b) and the IMDB dataset Maas et al. (2011). On the summarization dataset, we observe that the rejection sampling process on a pessimistic reward fine-tuned by PET significantly outperforms the traditional sampling process on the proxy reward. The win rate of the policy response against human response is increased from 32.0%percent32.032.0\%32.0 % to 39.2%percent39.239.2\%39.2 %. We further use the PPO algorithm for policy optimization, which we call ‘PPO-PET’. We observe that the policy learned by PPO-PET has a high performance and a high KL divergence from the dataset distribution. This contradicts the traditional impression that policies with high KL divergence are vulnerable to reward hacking. On the summarization dataset, our policy in general has a comparable or higher performance compared to current state-of-the-art RLHF algorithms, including DPO Rafailov et al. (2023), RPO Liu et al. (2024b), and χPO Huang et al. (2024a). On the IMDB dataset, the ground truth reward is available, and the policy learned by PPO-PET achieves a reward significantly higher than the baselines. 2 Preliminaries 2.1 Offline RLHF In this work, we consider a standard RLHF problem in the offline learning setting. For a given task, let XX be the input space (e.g., prompt), and AA be the output space (e.g., response). There exists a preference model (⋅|⋅,⋅,⋅):×→Δ(≻,≺)P(·|·,·,·):X×A×% A→ (\ , \)P ( ⋅ | ⋅ , ⋅ , ⋅ ) : X × A × A → Δ ( ≻ , ≺ ) that takes a prompt x and a pair of responses a1,a2subscript1subscript2a_1,a_2a1 , a2 as input and stochastically outputs a preference signal σ∼(⋅|x,a1,a2)σ (·|x,a_1,a_2)σ ∼ P ( ⋅ | x , a1 , a2 ), representing its preference on the two responses. For example, σ being ≻succeeds ≻ means the preference is a1≻a2succeedssubscript1subscript2a_1 a_2a1 ≻ a2 given the prompt x. In addition, we assume the preference model is a Bradley-Terry model Bradley and Terry (1952) associated with a reward model r:×→ℝ:→ℝr:X×A : X × A → blackboard_R as r(≻|x,a1,a2)=sigmoid(r(x,a1)−r(x,a2)),P_r( |x,a_1,a_2)=sigmoid(r(x,a_1)-r(x,a_2)),Pitalic_r ( ≻ | x , a1 , a2 ) = sigmoid ( r ( x , a1 ) - r ( x , a2 ) ) , where sigmoid(y):=11−exp(−y)assignsigmoid11sigmoid(y):= 11- (-y)sigmoid ( y ) := divide start_ARG 1 end_ARG start_ARG 1 - exp ( - y ) end_ARG is the sigmoid function. Here, the reward model r represents the quality of a response. Given a prompt, a response with a higher reward is more likely to be preferred. We say r is the true reward behind the preference model rsubscriptP_rPitalic_r. A policy π(⋅|⋅):→Δ()π(·|·):X→ (A)π ( ⋅ | ⋅ ) : X → Δ ( A ) takes a prompt x as an input and stochastically outputs a response a∼π(⋅|x)a π(·|x)a ∼ π ( ⋅ | x ). Let μ be a probability distribution over the prompt space, the quality of the policy on the reward r over the distribution μ is represented by Vμr(π)=x∼μ,a∼π(⋅|x)[r(x,a)].V^r_μ(π)=E_x μ,a π(·|x)[r(x,a)].Vitalic_ritalic_μ ( π ) = blackboard_Ex ∼ μ , a ∼ π ( ⋅ | x ) [ r ( x , a ) ] . In the offline RLHF setting, there exists an offline preference dataset consisting of N tuples =(xi,a1i,a2i,σi)i=1N.superscriptsubscriptsuperscriptsuperscriptsubscript1superscriptsubscript2superscript1D=\(x^i,a_1^i,a_2^i,σ^i)\_i=1^N.D = ( xitalic_i , a1italic_i , a2italic_i , σitalic_i ) i = 1N . The prompt and responses in each tuple is sampled from an i.i.d. distribution (xi,a1i,a2i)∼μsimilar-tosuperscriptsuperscriptsubscript1superscriptsubscript2subscript(x^i,a_1^i,a_2^i) _D( xitalic_i , a1italic_i , a2italic_i ) ∼ μcaligraphic_D, and the preference signal is sampled from the preference model σi∼(⋅|xi,a1i,a2i).σ^i (·|x^i,a_1^i,a_2^i).σitalic_i ∼ P ( ⋅ | xitalic_i , a1italic_i , a2italic_i ) . We call μDsubscript _Dμitalic_D the dataset distribution and denote μ as the distribution of the prompts in μsubscript _Dμcaligraphic_D. The agent has access to the offline dataset DD, the prompt space XX, the response space AA, and the prompt distribution μ. Let r∗superscriptr^*r∗ be the true reward model behind the preference model that generates the preferences in the dataset. The general goal of the agent is to align a policy with the reward model reliably, that is, to find a policy of high performance Vr∗μ(π)subscriptsuperscriptsuperscriptV^μ_r^*(π)Vitalic_μitalic_r∗ ( π ) with a high probability. 2.2 Reward Modeling Reward modeling is a process to learn a reward model that can well explain the preference signals in the dataset. Given a reward model r^ rover start_ARG r end_ARG, its prediction loss on the preference dataset DD is defined as ℒ(r^)=∑i=1N−logr^(σi|xi,a1i,a2i).subscriptℒ^superscriptsubscript1subscript^conditionalsuperscriptsuperscriptsuperscriptsubscript1superscriptsubscript2L_D( r)= _i=1^N- _ r(% σ^i|x^i,a_1^i,a_2^i).Lcaligraphic_D ( over start_ARG r end_ARG ) = ∑i = 1N - log Pover start_ARG r end_ARG ( σitalic_i | xitalic_i , a1italic_i , a2italic_i ) . The prediction loss of a reward model represents its quality in interpreting the preference signals in the dataset. The process of traditional reward modeling is to find the reward model from a model class ℛRR (e.g., an LLM) that minimizes the prediction loss over the offline dataset: r^∈argminr∈ℛℒ(r).^subscriptℛsubscriptℒ r∈ _r L_D(r).over start_ARG r end_ARG ∈ arg minitalic_r ∈ R Lcaligraphic_D ( r ) . For convenience, we say a reward model is a ‘proxy reward model’ if it has low prediction loss on the dataset. 2.3 Policy Optimization After learning a reward model, the next step is to optimize a policy on the reward model to achieve a high reward. In the appendix, we provide a detailed description of two typical policy optimization techniques most related to our study: KL regularized proximal policy optimization (KL-PPO) and rejection sampling (RS). Here, we briefly go over the process of rejection sampling. Rejection sampling, also known as best-of-N sampling, is an inference-time policy optimization method. Given a base policy model π0subscript0 _0π0, a reward model r^ rover start_ARG r end_ARG, and a positive integer n, the process of rejection sampling is defined in Alg 4 in the appendix. In practice, the reward model r^ rover start_ARG r end_ARG is trained by minimizing the prediction loss, and the base policy π0subscript0 _0π0 is usually set as a proxy policy for the dataset. The proxy policy is trained to reproduce the response in the dataset. This is often achieved by a standard supervised fine-tuning process on the prompts and responses from the dataset DD (Touvron et al., 2023). The rejection sampling process is effectively a policy as it takes a prompt as input and stochastically outputs a response. We denote πRS(π0,r^,n)subscriptRSsubscript0 _RS( _0, r,n)πRS ( π0 , over start_ARG r end_ARG , n ) as the policy to represent the rejection sampling process, which we call the ‘RS policy’. Particularly, we highlight an intuitive yet important property of a rejection sampling policy in Proposition 2.1. The proof is in the appendix. Later, we show how this property helps us implement our method. Proposition 2.1. For any prompt distribution μ, base policy π0subscript0 _0π0, number of sampling n, and reward model r0subscript0r_0r0, the rejection sampling policies satisfy: Vr0μ(πRS(π0,r0,n))=maxrVr0μ(πRS(π0,r,n)).subscriptsuperscriptsubscript0subscriptRSsubscript0subscript0subscriptsubscriptsuperscriptsubscript0subscriptRSsubscript0V^μ_r_0( _RS( _0,r_0,n))= _rV^μ_r_0( _% RS( _0,r,n)).Vitalic_μitalic_r start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ( πRS ( π0 , r0 , n ) ) = maxitalic_r Vitalic_μitalic_r start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT ( πRS ( π0 , r , n ) ) . 3 Robust RLHF with Pessimistic Reward Finetuning (PET) Robust rejection sampling as a minimax problem: We achieve robustness against reward hacking by training a pessimistic reward model. Specifically, we aim to find a reward model with a low prediction loss on the dataset that gives a minimal relative score to its corresponding RS policy. The relative score is compared to the score of a fixed reference policy. In practice, the reference policy is usually the proxy policy of the dataset. We focus on the relative score because the preference model only depends on the relative score between two responses. We consider RS policies because they are easy to acquire and represent near-optimal policies on a reward model. In addition, later we will show how the property in Proposition 2.1 reduces the difficulty of implementing our algorithm. Formally, to find such a pessimistic reward, we consider solving a minimax problem where a critic and an actor are trained against each other. The actor controls a policy model, and the critic controls a reward model. The minimax problem is as follows. minr∈ℛmaxπ∈ΠRSn,π0(Vrμ(π)−Vrμ(πref))+β⋅ℒ(r).subscriptℛsubscriptsubscriptsuperscriptΠsubscript0RSsubscriptsuperscriptsubscriptsuperscriptsubscriptref⋅subscriptℒ _r _π∈ ^n, _0_RS (V^μ_% r(π)-V^μ_r( _ref) )+β·L_ % D(r).minitalic_r ∈ R maxitalic_π ∈ Πitalic_n , π start_POSTSUBSCRIPT 0RS end_POSTSUBSCRIPT ( Vitalic_μitalic_r ( π ) - Vitalic_μitalic_r ( πref ) ) + β ⋅ Lcaligraphic_D ( r ) . (1) Here, ℛRR is a reward model class, ΠRSn,π0=πRS(r,n,π0):r∈ℛsuperscriptsubscriptΠRSsubscript0conditional-setsubscriptRSsubscript0ℛ _RS^n, _0=\ _RS(r,n, _0):r \ΠRSitalic_n , π0 = πRS ( r , n , π0 ) : r ∈ R is the class of the RS policies with base policy π0subscript0 _0π0, n samples, and any reward model r∈ℛr ∈ R. The minimax goal f(π,r):=(Vrμ(π)−Vrμ(πref))+β⋅ℒ(r)assignsubscriptsuperscriptsubscriptsuperscriptsubscriptref⋅subscriptℒf(π,r):= (V^μ_r(π)-V^μ_r( _ref) )+β% ·L_D(r)f ( π , r ) := ( Vitalic_μitalic_r ( π ) - Vitalic_μitalic_r ( πref ) ) + β ⋅ Lcaligraphic_D ( r ) consists of two terms. The first term (Vrμ(π)−Vrμ(πref))subscriptsuperscriptsubscriptsuperscriptsubscriptref (V^μ_r(π)-V^μ_r( _ref) )( Vitalic_μitalic_r ( π ) - Vitalic_μitalic_r ( πref ) ) is a relative score between a policy π and a reference policy πrefsubscriptref _refπref evaluated by the reward model r over a prompt distribution μ. The actor aims to find a policy that achieves a high score on the reward model, while the reward model aims to give a low relative score to the policy. The second term β⋅ℒ(r)⋅subscriptℒβ·L_D(r)β ⋅ Lcaligraphic_D ( r ) is the prediction loss on the preference dataset DD multiplied by a positive weight β>00β>0β > 0, which is only determined by the reward model. This term constrains the reward model in that it must be a high-quality reward model that well explains the dataset. Such a constraint prevents the critic from being overly pessimistic. Intuitively, we want the critic to find a tight lower bound on the true performance of a policy. Remark 3.1. Liu et al. (2024b) proposes to solve a similar minimax problem as ours. Here, we highlight two critical differences between the formulations. The first difference is that the KL regularization is included in (Liu et al., 2024b) but not in ours. Our insight is that the idea of pessimism is already included in formulating a minimax game between the policy and the reward models, so it is not necessary to add KL regularization for more pessimism. In the appendix, we show that in the theoretical analysis, the KL regularization results in over-pessimism and decreases the learning efficiency, which is undesired. Our formulation has no regularization and achieves a tighter theoretical guarantee on learning efficiency. Second difference is that we focus on the policies based on rejection sampling. Using the rejection sampling process allows us to avoid KL regularization in problem formulation. In addition, the rejection sampling process is a convenient policy optimization technique and makes the implementation easy for adversarial training, as we will show next. PET and three-step RLHF: Based on the result from Proposition 2.1, we have πRS(π0,n,r)∈maxπ∈ΠRSn,π0Vrμ(π).subscriptRSsubscript0subscriptsubscriptsuperscriptΠsubscript0RSsubscriptsuperscript _RS( _0,n,r)∈ _π∈ ^n, _0_RSV^μ% _r(π).πRS ( π0 , n , r ) ∈ maxitalic_π ∈ Πitalic_n , π start_POSTSUBSCRIPT 0RS end_POSTSUBSCRIPT Vitalic_μitalic_r ( π ) . Therefore, the minimax problem in Eq 1 can be simplified to a minimization problem as shown in Eq. 2. minr∈ℛ(Vrμ(πRS(π0,n,r))−Vrμ(πref))+β⋅ℒ(r).subscriptℛsubscriptsuperscriptsubscriptRSsubscript0subscriptsuperscriptsubscriptref⋅subscriptℒ _r (V^μ_r( _RS( _0,n,r))-V^μ% _r( _ref) )+β·L_D(r).minitalic_r ∈ R ( Vitalic_μitalic_r ( πRS ( π0 , n , r ) ) - Vitalic_μitalic_r ( πref ) ) + β ⋅ Lcaligraphic_D ( r ) . (2) To solve the minimization above, we propose a novel algorithm called ‘pessimistic reward fine-tuning’ (PET). In the Appendix, we explain in detail that PET is essentially the process of stochastic gradient descent on the minimization goal in Eq 2. We show PET in Algorithm 1. In PET, the policy model and the reward model are iteratively updated. In each iteration, the policy model is updated through the rejection sampling process on the current reward model. The reward loss is computed as the minimization goal in Eq. 2 with the updated policy model on a batch of prompts sampled from the dataset. Then, the reward model performs a gradient descent step on the reward loss. Algorithm 1 Pessimistic Reward Fine-Tuning (PET) with Adversarially Trained Rejection Sampling 1:Input: Initial reward model r^ rover start_ARG r end_ARG, Dataset DD, base policy π0subscript0 _0π0, reference policy πrefsubscriptref _refπref, number of samples n, pessimistic coefficient β, learning rate α 2:Initialize: r1←r^←superscript1^r^1← rr1 ← over start_ARG r end_ARG 3:for t=1,…,T1…t=1,…,Tt = 1 , … , T do 4: Update πt←πRS(π0,n,rt)←superscriptsubscriptRSsubscript0superscriptπ^t← _RS( _0,n,r^t)πitalic_t ← πRS ( π0 , n , ritalic_t ) 5: Sample mini batch t=xi,ai1,ai2,σii=1M,(xi,ai1,ai2,σi)∼i.i.dsubscriptsuperscriptsubscriptsubscriptsubscriptsuperscript1subscriptsuperscript2subscript1subscriptsubscriptsuperscript1subscriptsuperscript2subscripti.i.dsimilar-toD_t=\x_i,a^1_i,a^2_i, _i\_i=1^M,(x_i,a^1% _i,a^2_i, _i) i.i.d DDitalic_t = xitalic_i , a1italic_i , a2italic_i , σitalic_i i = 1M , ( xitalic_i , a1italic_i , a2italic_i , σitalic_i ) overi.i.d start_ARG ∼ end_ARG D 6: Sample responses ai∼πt(⋅|xi),aref,i∼πref(⋅|xi),∀i∈[M]a_i π^t(·|x_i),a_ref,i _ref(·|x_i% ),∀ i∈[M]aitalic_i ∼ πitalic_t ( ⋅ | xitalic_i ) , aref , i ∼ πref ( ⋅ | xitalic_i ) , ∀ i ∈ [ M ] 7: Compute pessimistic loss lt=∑i∈[M][rt(xi,πt(xi,ai))−rt(xi,πt(xi,aref,i))+β⋅ℒt(rt)]superscriptsubscriptdelimited-[]delimited-[]superscriptsubscriptsuperscriptsuperscriptsubscriptsuperscriptsubscriptsuperscriptsuperscriptsubscriptref⋅subscriptℒsubscriptsuperscriptl^t= _i∈[M][r^t(x_i,π^t(x^i,a_i))-r^t(x_i,π^t(x^% i,a_ref,i))+β·L_D_t(r^t)]litalic_t = ∑i ∈ [ M ] [ ritalic_t ( xitalic_i , πitalic_t ( xitalic_i , aitalic_i ) ) - ritalic_t ( xitalic_i , πitalic_t ( xitalic_i , aref , i ) ) + β ⋅ Lcaligraphic_D start_POSTSUBSCRIPT t end_POSTSUBSCRIPT ( ritalic_t ) ] 8: Update rt+1←rt−α⋅∇lt←superscript1superscript⋅∇superscriptr^t+1← r^t-α·∇ l^tritalic_t + 1 ← ritalic_t - α ⋅ ∇ litalic_t 9:end for 10:Return: Reward model rT+1superscript1r^T+1ritalic_T + 1 Alg 1 proposes a pessimistic reward fine-tuning process to make a reward pessimistic. Based on PET, we develop a three-step RLHF learning framework as shown in Alg 2. In this framework, the first step learns a proxy reward model that minimizes the prediction loss. The second step fine-tunes the proxy reward to make it pessimistic. In this setup, the second step is initialized from the proxy reward with a low prediction loss already. This intuitively reduces the complexity of pessimistic reward fine-tuning. The last step optimizes a policy model on the pessimistic reward. In principle, it is unnecessary in the last step to use additional pessimistic learning tricks such as regularization during policy optimization, as the reward model is already pessimistic. The agent can greedily optimize a policy to achieve a high pessimistic reward without worrying about the risk of reward hacking. For example, in the standard KL-PPO algorithm, the optimization goal of the PPO algorithm is the reward of a policy regularized by the KL divergence between the policy and the proxy dataset policy. In our learning framework, the empirical results in Section 4 show that the PPO algorithm can learn a high-quality policy in reality on the pessimistic reward directly without using any regularization. This empirically verifies that the pessimistic reward fine-tuned by PET is trustworthy. Algorithm 2 Three-Step Pessimistic RLHF Framework 1:Input: Dataset DD, Base policy π0subscript0 _0π0, Reference policy πrefsubscriptref _refπref, number of samples n 2:Step 1: Reward modeling r^1←minr∈ℛℒ(r)←subscript^1subscriptℛsubscriptℒ r_1← _r L_D(r)over start_ARG r end_ARG1 ← minitalic_r ∈ R Lcaligraphic_D ( r ) 3:Step 2: Pessimistic reward fine-tuning r^2←PET(r^1)←subscript^2PETsubscript^1 r_2 ( r_1)over start_ARG r end_ARG2 ← PET ( over start_ARG r end_ARG1 ) 4:Step 3: Policy optimization π^←maxπ∈ΠVr^2μ(π)←^subscriptΠsubscriptsuperscriptsubscript^2 π← _π∈ V^μ_ r_2(π)over start_ARG π end_ARG ← maxitalic_π ∈ Π Vitalic_μover start_ARG r end_ARG start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT ( π ) 5:Return: Policy model π^ πover start_ARG π end_ARG Theoretical guarantees: Here we provide theoretical guarantees on the optimality of the solution to Eq 1. The proof technique follows the one in Liu et al. (2024b). We take a standard definition of the dataset coverage on a policy in Definition 3.2 to characterize how well a dataset covers a policy π. Definition 3.2. For a policy π, given a reference policy πrefsubscriptref _refπref, a dataset distribution μsubscript _Dμcaligraphic_D, and a reward model class ℛRR, the coverage coefficient is defined as Zhan et al. (2023) μ(ℛ,π,πref):=assignsubscriptsubscriptℛsubscriptrefabsentC_ _D(R,π, _ref):=Citalic_μ start_POSTSUBSCRIPT D end_POSTSUBSCRIPT ( R , π , πref ) := max0,supr∈ℛx∼μ,a1∼π(⋅|x),a2∼πref(⋅|x)[(r∗(x,a1)−r∗(x,a2))−(r(x,a1)−r(x,a2))]x,a1,a2∼μ[|(r∗(x,a1)−r∗(x,a2))−(r(x,a1)−r(x,a2))|2], \0, _r E_x μ,a_1 π(·|x)% ,a_2 _ref(·|x)[ (r^*(x,a_1)-r^*(x,a_2)% )- (r(x,a_1)-r(x,a_2) )]E_x,a_1,a_2 % _D[| (r^*(x,a_1)-r^*(x,a_2) )- (r(x,% a_1)-r(x,a_2) )|^2]\,max 0 , supitalic_r ∈ R divide start_ARG blackboard_Ex ∼ μ , a start_POSTSUBSCRIPT 1 ∼ π ( ⋅ | x ) , a2 ∼ πref ( ⋅ | x ) end_POSTSUBSCRIPT [ ( r∗ ( x , a1 ) - r∗ ( x , a2 ) ) - ( r ( x , a1 ) - r ( x , a2 ) ) ] end_ARG start_ARG blackboard_Ex , a start_POSTSUBSCRIPT 1 , a2 ∼ μcaligraphic_D end_POSTSUBSCRIPT [ | ( r∗ ( x , a1 ) - r∗ ( x , a2 ) ) - ( r ( x , a1 ) - r ( x , a2 ) ) |2 ] end_ARG , where μ is the prompt distribution in μsubscript _Dμcaligraphic_D. Formally, in Theorem 3.3, we show a theoretical guarantee on the performance of the policy in the solution to the minimax problem in Eq 1. The proof for Theorem 3.3 can be found in the appendix. Theorem 3.3. Consider a bounded reward model class ℛRR such that ∀r∈ℛ,r(⋅,⋅)∈[−R,R]formulae-sequencefor-allℛ⋅∀ r ,r(·,·)∈[-R,R]∀ r ∈ R , r ( ⋅ , ⋅ ) ∈ [ - R , R ] .Assume the true reward is included in the reward model class r∗∈ℛsuperscriptℛr^* ∗ ∈ R. Let π^=πRS(π0,r^,n)^subscriptRSsubscript0 π= _RS( _0, r,n)over start_ARG π end_ARG = πRS ( π0 , over start_ARG r end_ARG , n ), where r^ rover start_ARG r end_ARG is the reward model solution to the minimax problem in Eq 1. For any RS policy π∈ΠRSn,π0subscriptsuperscriptΠsubscript0RSπ∈ ^n, _0_RSπ ∈ Πitalic_n , π0RS covered by the dataset μ(π,πref,ℛ)<+∞subscriptsubscriptrefℛC_μ(π, _ref,R)<+∞Citalic_μ ( π , πref , R ) < + ∞, by setting β=N(1+exp(R))226⋅log(Nϵ(ℛ,∥⋅∥∞)δ)β= N(1+ (R))^22 6· ( N_% ε(R,\|·\|_∞)δ)β = divide start_ARG square-root start_ARG N end_ARG ( 1 + exp ( R ) )2 end_ARG start_ARG 2 square-root start_ARG 6 end_ARG ⋅ square-root start_ARG log ( divide start_ARG Nitalic_ϵ ( R , ∥ ⋅ ∥∞ ) end_ARG start_ARG δ end_ARG ) end_ARG end_ARG, with probability at least 1−δ11- 1 - δ the performance gap between π^ πover start_ARG π end_ARG and π is bounded by Vr∗μ(π)−Vr∗μ(π^)≤(1+exp(R))2⋅(μ(ℛ,π,πref)2+1)⋅6log(Nϵ(ℛ,∥⋅∥∞)δ)4N,V^μ_r^*(π)-V^μ_r^*( π)≤ (1+ (R))^2·(% C_ _D(R,π, _ref)^2+1)·% 6 ( N_ε(R,\|·\|_∞)δ)4% N,Vitalic_μitalic_r∗ ( π ) - Vitalic_μitalic_r∗ ( over start_ARG π end_ARG ) ≤ divide start_ARG ( 1 + exp ( R ) )2 ⋅ ( Citalic_μ start_POSTSUBSCRIPT D end_POSTSUBSCRIPT ( R , π , πref )2 + 1 ) ⋅ square-root start_ARG 6 log ( divide start_ARG Nitalic_ϵ ( R , ∥ ⋅ ∥∞ ) end_ARG start_ARG δ end_ARG ) end_ARG end_ARG start_ARG 4 square-root start_ARG N end_ARG end_ARG , where Nϵ(ℛ,∥⋅∥∞)N_ε(R,\|·\|_∞)Nitalic_ϵ ( R , ∥ ⋅ ∥∞ ) is the ϵitalic-ϵεϵ-covering number of the reward model class Cheng et al. (2022). Remark 3.4. It is standard in the related literature Liu et al. (2024b); Zhan et al. (2023) to assume that the true reward is included in the reward model class. Theorem 3.3 shows that the performance of π^ πover start_ARG π end_ARG is comparable to any RS policy π∈ΠRSπ0,nsuperscriptsubscriptΠRSsubscript0π∈ _RS _0,nπ ∈ ΠRSitalic_π0 , n if the policy is covered by the dataset. As long as a high-quality policy is well covered by the dataset, the solution π^ πover start_ARG π end_ARG will also have high quality, which indicates that reward hacking will not happen. This is an ideal result one can hope for in the offline learning setting Huang et al. (2024a). 4 Experiments 4.1 Experiment Setup Dataset and model: Our experiments focus on the standard TL;DR summarization datasets (Völske et al., 2017a; Stiennon et al., 2020) and use the Pythia-1b model (Biderman et al., 2023) for all policy and reward models during training. Here, the task is to generate a summary for a given piece of text. We consider the IMDB dataset (Maas et al., 2011) as an ablation study. Implementation: For the reference policy πrefsubscriptref _refπref, it is trained by the standard SFT process in (Huang et al., 2024b). The base policy π0subscript0 _0π0 for the rejection sampling process uses the same SFT model. During inference, the base policy π0subscript0 _0π0 uses a higher temperature (T=0.70.7T=0.7T = 0.7) than the reference policy (T=0.10.1T=0.1T = 0.1) to increase the randomness in the generation and boost the efficiency of rejection sampling. Due to limitations on our computational resources, we set n=6464n=64n = 64 for rejection sampling during PET, while PET in principle can work with arbitrarily large values of n. The proxy reward model is trained by the standard reward modeling process in (Huang et al., 2024b). For fair comparison, all RLHF methods share the same reference policy and the proxy reward, if needed. We set the pessimistic coefficient 1/β=0.110.11/β=0.11 / β = 0.1 following the setups in (Rafailov et al., 2023). Evaluation: We evaluate the policy models learned by the RLHF algorithms on the test split of the TL;DR SFT dataset, which consists of (query, human summary) pairs. First, we used the policy model to generate a summary for each query in the dataset. Then, we used a judge LLM (Qwen-2.5-32B-Instruct) to evaluate the quality of the generated summaries against the human responses in the dataset. The win rate of the model against the human response is computed by the percentage of instances where the judge LLM preferred the model’s summary over the human summary. Baselines: We consider multiple state-of-the-art offline RLHF algorithms as baselines, including the traditional PPO-KL Bai et al. (2022a), traditional rejection sampling/best of N sampling Touvron et al. (2023) (on the proxy reward), DPO Rafailov et al. (2023), RPO (Liu et al., 2024b), and χ-PO (Huang et al., 2024a). Our implementations of DPO and RPO are directly from the source code provided in the original papers. We re-implement the χ-PO algorithm as the source code is unavailable. We use the same hyperparameter setup mentioned in the paper to ensure fair comparison. We try to re-implement another related RLHF baseline, ‘InferenceTimePessimism’ (Huang et al., 2025). The algorithm is based on the rejection sampling process, and the number of samples is set as n=213superscript213n=2^13n = 213 in the paper. Due to our computational resource limitation, we set the number of samples as n=6464n=64n = 64. This setup probably underestimates the method’s efficiency, and its learned policy has a low win rate. Therefore, we skip this baseline. 4.2 Pessimistic Reward Finetuning Improves Rejection Sampling Here, we evaluate our three-step learning framework with the rejection sampling process for policy optimization. We test the number of samplings to be n=16,32,64,128163264128n=\16,32,64,128\n = 16 , 32 , 64 , 128 during PET and policy optimization, and the results are shown in Table 1. We observe that the rejection sampling policy based on the pessimistic reward model significantly outperforms the one based on the proxy reward model. The finding empirically validates that the rejection sampling process greatly benefits from the adversarial training process against a reward model. n=1616n=16n = 16 n=3232n=32n = 32 n=6464n=64n = 64 n=128128n=128n = 128 RS-PET 38.238.238.238.2 36.636.636.636.6 39.239.239.239.2 36.836.836.836.8 RS-Proxy 28.428.428.428.4 32.032.032.032.0 32.032.032.032.0 34.234.234.234.2 Table 1: Win rate against human responses for the rejection sampling process on the pessimistic or proxy reward with different numbers n of samplings on the IMDB dataset. ‘RS-PET’ means the rejection sampling process is performed on the pessimistic reward fine-tuned by PET. Similarly, ‘RS-Proxy’ means the reward model is the proxy reward. 4.3 KL Regularization Is Unnecessary For Policy Optimization On Pessimistic Reward regularized unregularized win rate KL win rate KL PPO-Pessimistic 36.036.036.036.0 11.611.611.611.6 40.840.840.840.8 114.0114.0114.0114.0 PPO-Proxy 40.240.240.240.2 9.29.29.29.2 7.27.27.27.2 192.6192.6192.6192.6 Table 2: The policies here are learned by the PPO algorithm on the pessimistic reward or proxy reward, with (‘regularized’) or without KL (‘unregularized’) regularization. We show the win rate against human responses of the policies and their KL divergence to the reference policy. As introduced earlier, the proxy reward model may overestimate the rewards for out-of-distribution prompt responses, so KL regularization is always considered necessary during policy optimization to prevent reward hacking. In this section, we show that our pessimistic reward model, which is fine-tuned on the proxy reward, is robust against reward hacking such that no regularization is necessary when optimizing a policy on the pessimistic reward. Here, we use the PPO algorithm for the policy optimization step in the three-step learning framework. We compare the two cases where KL regularization or no regularization is applied during policy optimization, and we train the algorithms on the proxy reward and the pessimistic reward. Without regularization, the agent would greedily search for the policy with the highest reward. Such an agent is usually considered vulnerable to reward hacking. The results are shown in Table 2. We observe that for the proxy reward model, the policy learned under the KL regularization performs much better than one with no regularization. The results confirm that the proxy reward model is not pessimistic, and greedily optimizing a policy on proxy reward without KL regularization is vulnerable to reward hacking. In contrast, for the pessimistic reward model, the policy learned under no regularization performs better than the one with regularization. The results intuitively show that the pessimistic fine-tuning successfully makes the reward model pessimistic, and a greedy agent is robust against reward hacking when learning from the pessimistic reward model without using any regularization. We notice another important finding from Table 2. Traditionally, the experience is that policies with high KL divergence from the reference policy have a high risk of reward hacking, so they should be excluded during learning through regularization. This agrees with the case of optimizing a policy on the proxy reward with no regularization, as the learned policy has low performance and high KL divergence. However, such an experience does not hold for the case when the reward model is a pessimistic reward. We find that the policy learned on the pessimistic reward fine-tuned by PET has a high performance in reality while having a large KL divergence. This is a sign of over-pessimism when using KL regularization for pessimism, as such good policies are excluded. Our empirical results prove that it is possible to find a high-quality policy on the dataset with a large KL divergence from the reference policy. To include such policies during learning without causing reward hacking, one can perform policy optimization on the pessimistic reward fine-tuned by PET. 4.4 Comparison between Different Algorithms on Summarization and IMDB Datasets Here, we compare our method against the RLHF baselines. We consider using the PPO algorithm for policy optimization with no KL regularization in our three-step learning framework, and we call it ‘PET-PPO’. Since the RLHF methods we consider generally apply to other NLP tasks in principle, we evaluate them on the IMDB dataset as well to make the evaluation more comprehensive. In this task, the model is prompted with a prefix to a movie review, and it must generate a continuation of that review with positive sentiment. Our data construction and evaluation metric exactly follows the setups in Rafailov et al. (2023). To make the task more challenging, the size of our dataset is about 25%percent2525\%25 % of the dataset’s size described in Rafailov et al. (2023). Training with less data can better highlight the sampling efficiency of an algorithm. The evaluation metric is the likelihood of positive sentiment for the completions generated by the model judged by an LLM. The same LLM is used to generate the preference labels in the dataset, so the likelihood is the ground truth in the task. SFT RS-proxy KL-PPO DPO RPO χPO PET-PPO Summarization 25.625.625.625.6 32.032.032.032.0 40.240.240.240.2 38.838.838.838.8 34.034.034.034.0 39.239.239.239.2 40.840.840.840.8 IMDB 89.689.689.689.6 90.790.790.790.7 91.891.891.891.8 95.095.095.095.0 93.593.593.593.5 92.892.892.892.8 98.398.398.398.3 Table 3: Performance of different RLHF methods learning on the summarization and IMDB dataset. The metric on the summarization dataset is the win rate against human responses. On the IMDB dataset, the metric is the likelihood of the responses to have positive sentiment. In Table 3, we show the standard evaluations on the performance of the policies learned by different algorithms. On the IMDB dataset, our method significantly outperforms the baselines by a clear margin. Note that in the IMDB dataset, a policy model is strictly better than another model if it achieves a higher evaluation score, as the score is given by the ground truth. On the summarization dataset, our method outperforms other methods in general. Here, the win rate is compared against human response, so a model does not necessarily beat another model if it has a higher win rate. Therefore, we directly compare policy models learned by the baselines against our policy model. In Table 4, we show the win rate of our PPO-PET method against other methods. We observe that, in general, the policy model learned by PPO-PET has a >50%absentpercent50>50\%> 50 % win rate against other models. The empirical results show that the performance of our PET-PPO method is very competitive compared to current SOTA RLHF methods. SFT RS-proxy KL-PPO DPO RPO χPO PET-PPO 72.072.072.072.0 61.661.661.661.6 57.257.257.257.2 50.150.150.150.1 58.458.458.458.4 52.852.852.852.8 Table 4: Direct comparison between our PET-PPO method against other RLHF baselines on the summarization dataset. 5 Related Work 5.1 RLHF based on explicit reward modeling It is typical in RLHF to train an explicit reward model first and then optimize a policy on the learned reward model. A popular choice for policy optimization is the rejection sampling technique (Bai et al., 2022b; Touvron et al., 2023). Theoretical understandings of the properties of the rejection sampling process have been developed to explain the efficiency of the method (Beirami et al., 2024; Huang et al., 2025). Empirically, different variants of rejection sampling have been proposed to improve the learning efficiency and robustness against reward hacking (Huang et al., 2025; Jinnai et al., 2024; Liu et al., 2023; Xiong et al., 2025; Khaki et al., 2024). Another popular choice for policy optimization is to use the PPO algorithm (Schulman et al., 2017). In this case, it is prevalent in current methods to add a KL regularization in the policy optimization goal to avoid reward hacking (Stiennon et al., 2020; Bai et al., 2022a; Ouyang et al., 2022; Christiano et al., 2017). To improve the learning efficiency of KL-PPO, methods have been proposed to improve the quality of the reward model (Liu et al., 2024a; Sun et al., 2025; Shen et al., 2024; Ramé et al., 2024) or further constrain the agent from uncovered policies (Dai et al., 2025) 5.2 RLHF based on direct preference alignment Another typical RLHF technique skips the step of explicit reward modeling through direct preference alignment, also known as direct preference optimization (DPO) (Rafailov et al., 2023). In DPO, a reward model is implicitly represented by a policy model, so the agent can directly train a policy model to minimize the prediction loss. Numerous direct preference alignment techniques have been proposed to achieve a higher learning efficiency (Azar et al., 2024; Xiong et al., 2023b; Tang et al., 2024; Ji et al., 2024a; Liang et al., 2024; Rosset et al., 2024). Provably robust algorithms against reward hacking based on DPO have also been developed (Fisch et al., 2024; Zhang et al., 2024; Xie et al., 2024; Liu et al., 2024b; Cen et al., 2024; Ji et al., 2024b). Note that KL regularization plays a critical role in DPO, as the regularization term makes it possible to represent a reward model by the optimal policy on it under the regularization. 5.3 RLHF with online preference collection This work focuses on RLHF in the offline setting, where the learning agent only has access to an offline preference dataset. Another important setting of RLHF is the online setting, where an agent can collect preference data online. The online setting can break the limit of the offline setting in that the agent can control the data distribution to cover high-quality policies actively. The online setting is also more expensive than the offline setting, as it needs to collect customized data during training. Theoretical studies have developed efficient online exploration algorithms to improve the quality of online data collection Qi et al. (2025); Wu and Sun (2023); Zhao et al. (2025). Practical methods and learning frameworks have also been developed and have achieved promising empirical results Dong et al. (2024); Xiong et al. (2023a); Bai et al. (2022a); Touvron et al. (2023). 6 Conclusions and Limitations This work develops PET, a reward fine-tuning method. A pessimistic reward model can be learned by fine-tuning a proxy reward with PET. When optimizing a policy on a pessimistic reward, a greedy agent can learn a high-performing policy with no regularization. We develop an RLHF method called PET-PPO that uses the PPO algorithm for policy optimization on a pessimistic reward fine-tuned by PET. Our empirical results show that PET-PPO achieves comparable or stronger performance on the IMDB and the TL;DR summarization dataset than multiple current SOTA RLHF baselines. The scope of this work is limited to RLHF in the offline setting. Due to limited computational resources, the evaluation is performed on a subset of RLHF tasks based on LLMs with relatively small size. References Azar et al. (2024) Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm to understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics, pages 4447–4455. PMLR, 2024. Bai et al. (2022a) Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022a. Bai et al. (2022b) Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073, 2022b. Beirami et al. (2024) Ahmad Beirami, Alekh Agarwal, Jonathan Berant, Alexander D’Amour, Jacob Eisenstein, Chirag Nagpal, and Ananda Theertha Suresh. Theoretical guarantees on the best-of-n alignment policy. arXiv preprint arXiv:2401.01879, 2024. Bhardwaj et al. (2023) Mohak Bhardwaj, Tengyang Xie, Byron Boots, Nan Jiang, and Ching-An Cheng. Adversarial model for offline reinforcement learning. Advances in Neural Information Processing Systems, 36:1245–1269, 2023. Biderman et al. (2023) Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. Pythia: A suite for analyzing large language models across training and scaling, 2023. URL https://arxiv.org/abs/2304.01373. Bradley and Terry (1952) Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324–345, 1952. Cen et al. (2024) Shicong Cen, Jincheng Mei, Katayoon Goshvadi, Hanjun Dai, Tong Yang, Sherry Yang, Dale Schuurmans, Yuejie Chi, and Bo Dai. Value-incentivized preference optimization: A unified approach to online and offline rlhf. arXiv preprint arXiv:2405.19320, 2024. Cheng et al. (2022) Ching-An Cheng, Tengyang Xie, Nan Jiang, and Alekh Agarwal. Adversarially trained actor critic for offline reinforcement learning. In International Conference on Machine Learning, pages 3852–3878. PMLR, 2022. Christiano et al. (2017) Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017. Dai et al. (2025) Juntao Dai, Taiye Chen, Yaodong Yang, Qian Zheng, and Gang Pan. Mitigating reward over-optimization in rlhf via behavior-supported regularization. arXiv preprint arXiv:2503.18130, 2025. Dong et al. (2024) Hanze Dong, Wei Xiong, Bo Pang, Haoxiang Wang, Han Zhao, Yingbo Zhou, Nan Jiang, Doyen Sahoo, Caiming Xiong, and Tong Zhang. Rlhf workflow: From reward modeling to online rlhf. arXiv preprint arXiv:2405.07863, 2024. Eisenstein et al. (2023) Jacob Eisenstein, Chirag Nagpal, Alekh Agarwal, Ahmad Beirami, Alex D’Amour, DJ Dvijotham, Adam Fisch, Katherine Heller, Stephen Pfohl, Deepak Ramachandran, et al. Helping or herding? reward model ensembles mitigate but do not eliminate reward hacking. arXiv preprint arXiv:2312.09244, 2023. Fisch et al. (2024) Adam Fisch, Jacob Eisenstein, Vicky Zayats, Alekh Agarwal, Ahmad Beirami, Chirag Nagpal, Pete Shaw, and Jonathan Berant. Robust preference optimization through reward model distillation. arXiv preprint arXiv:2405.19316, 2024. Gao et al. (2023) Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, pages 10835–10866. PMLR, 2023. Gupta et al. (2025) Dhawal Gupta, Adam Fisch, Christoph Dann, and Alekh Agarwal. Mitigating preference hacking in policy optimization with pessimism. arXiv preprint arXiv:2503.06810, 2025. Huang et al. (2024a) Audrey Huang, Wenhao Zhan, Tengyang Xie, Jason D Lee, Wen Sun, Akshay Krishnamurthy, and Dylan J Foster. Correcting the mythos of kl-regularization: Direct alignment without overoptimization via chi-squared preference optimization. arXiv preprint arXiv:2407.13399, 2024a. Huang et al. (2025) Audrey Huang, Adam Block, Qinghua Liu, Nan Jiang, Dylan J Foster, and Akshay Krishnamurthy. Is best-of-n the best of them? coverage, scaling, and optimality in inference-time alignment. arXiv preprint arXiv:2503.21878, 2025. Huang et al. (2024b) Shengyi Huang, Michael Noukhovitch, Arian Hosseini, Kashif Rasul, Weixun Wang, and Lewis Tunstall. The n+ implementation details of rlhf with ppo: A case study on tl;dr summarization, 2024b. URL https://arxiv.org/abs/2403.17031. Ji et al. (2024a) Haozhe Ji, Cheng Lu, Yilin Niu, Pei Ke, Hongning Wang, Jun Zhu, Jie Tang, and Minlie Huang. Towards efficient exact optimization of language model alignment. arXiv preprint arXiv:2402.00856, 2024a. Ji et al. (2024b) Xiang Ji, Sanjeev Kulkarni, Mengdi Wang, and Tengyang Xie. Self-play with adversarial critic: Provable and scalable offline alignment for language models. arXiv preprint arXiv:2406.04274, 2024b. Jinnai et al. (2024) Yuu Jinnai, Tetsuro Morimura, Kaito Ariu, and Kenshi Abe. Regularized best-of-n sampling to mitigate reward hacking for language model alignment. In ICML 2024 Workshop on Models of Human Feedback for AI Alignment, 2024. Khaki et al. (2024) Saeed Khaki, JinJin Li, Lan Ma, Liu Yang, and Prathap Ramachandra. Rs-dpo: A hybrid rejection sampling and direct preference optimization method for alignment of large language models. arXiv preprint arXiv:2402.10038, 2024. Levine et al. (2020) Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643, 2020. Liang et al. (2024) Xize Liang, Chao Chen, Jie Wang, Yue Wu, Zhihang Fu, Zhihao Shi, Feng Wu, and Jieping Ye. Robust preference optimization with provable noise tolerance for llms. arXiv e-prints, pages arXiv–2404, 2024. Liu et al. (2023) Tianqi Liu, Yao Zhao, Rishabh Joshi, Misha Khalman, Mohammad Saleh, Peter J Liu, and Jialu Liu. Statistical rejection sampling improves preference optimization. arXiv preprint arXiv:2309.06657, 2023. Liu et al. (2024a) Tianqi Liu, Wei Xiong, Jie Ren, Lichang Chen, Junru Wu, Rishabh Joshi, Yang Gao, Jiaming Shen, Zhen Qin, Tianhe Yu, et al. Rrm: Robust reward model training mitigates reward hacking. arXiv preprint arXiv:2409.13156, 2024a. Liu et al. (2024b) Zhihan Liu, Miao Lu, Shenao Zhang, Boyi Liu, Hongyi Guo, Yingxiang Yang, Jose Blanchet, and Zhaoran Wang. Provably mitigating overoptimization in rlhf: Your sft loss is implicitly an adversarial regularizer. arXiv preprint arXiv:2405.16436, 2024b. Maas et al. (2011) Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Dekang Lin, Yuji Matsumoto, and Rada Mihalcea, editors, Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 142–150, Portland, Oregon, USA, June 2011. Association for Computational Linguistics. URL https://aclanthology.org/P11-1015/. Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744, 2022. Qi et al. (2025) Han Qi, Haochen Yang, Qiaosheng Zhang, and Zhuoran Yang. Sample-efficient reinforcement learning from human feedback via information-directed sampling. arXiv preprint arXiv:2502.05434, 2025. Rafailov et al. (2023) Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36:53728–53741, 2023. Rafailov et al. (2024) Rafael Rafailov, Yaswanth Chittepu, Ryan Park, Harshit Sushil Sikchi, Joey Hejna, Brad Knox, Chelsea Finn, and Scott Niekum. Scaling laws for reward model overoptimization in direct alignment algorithms. Advances in Neural Information Processing Systems, 37:126207–126242, 2024. Ramé et al. (2024) Alexandre Ramé, Nino Vieillard, Léonard Hussenot, Robert Dadashi, Geoffrey Cideron, Olivier Bachem, and Johan Ferret. Warm: On the benefits of weight averaged reward models. arXiv preprint arXiv:2401.12187, 2024. Rosset et al. (2024) Corby Rosset, Ching-An Cheng, Arindam Mitra, Michael Santacroce, Ahmed Awadallah, and Tengyang Xie. Direct nash optimization: Teaching language models to self-improve with general preferences. arXiv preprint arXiv:2404.03715, 2024. Schulman et al. (2017) John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. Shen et al. (2024) Jiaming Shen, Ran Xu, Yennie Jun, Zhen Qin, Tianqi Liu, Carl Yang, Yi Liang, Simon Baumgartner, and Michael Bendersky. Boosting reward model with preference-conditional multi-aspect synthetic data generation. arXiv preprint arXiv:2407.16008, 2024. Stiennon et al. (2020) Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in neural information processing systems, 33:3008–3021, 2020. Sun et al. (2025) Hao Sun, Yunyi Shen, and Jean-Francois Ton. Rethinking reward modeling in preference-based large language model alignment. In The Thirteenth International Conference on Learning Representations, 2025. Tang et al. (2024) Yunhao Tang, Zhaohan Daniel Guo, Zeyu Zheng, Daniele Calandriello, Rémi Munos, Mark Rowland, Pierre Harvey Richemond, Michal Valko, Bernardo Ávila Pires, and Bilal Piot. Generalized preference optimization: A unified approach to offline alignment. arXiv preprint arXiv:2402.05749, 2024. Tien et al. (2022) Jeremy Tien, Jerry Zhi-Yang He, Zackory Erickson, Anca D Dragan, and Daniel S Brown. Causal confusion and reward misidentification in preference-based reward learning. arXiv preprint arXiv:2204.06601, 2022. Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023. Völske et al. (2017a) Michael Völske, Martin Potthast, Shahbaz Syed, and Benno Stein. TL;DR: Mining Reddit to learn automatic summarization. In Lu Wang, Jackie Chi Kit Cheung, Giuseppe Carenini, and Fei Liu, editors, Proceedings of the Workshop on New Frontiers in Summarization, pages 59–63, Copenhagen, Denmark, September 2017a. Association for Computational Linguistics. doi: 10.18653/v1/W17-4508. URL https://aclanthology.org/W17-4508/. Völske et al. (2017b) Michael Völske, Martin Potthast, Shahbaz Syed, and Benno Stein. Tl; dr: Mining reddit to learn automatic summarization. In Proceedings of the Workshop on New Frontiers in Summarization, pages 59–63, 2017b. Wu and Sun (2023) Runzhe Wu and Wen Sun. Making rl with preference-based feedback efficient via randomization. arXiv preprint arXiv:2310.14554, 2023. Xie et al. (2024) Tengyang Xie, Dylan J Foster, Akshay Krishnamurthy, Corby Rosset, Ahmed Awadallah, and Alexander Rakhlin. Exploratory preference optimization: Harnessing implicit q*-approximation for sample-efficient rlhf. arXiv preprint arXiv:2405.21046, 2024. Xiong et al. (2023a) Wei Xiong, Hanze Dong, Chenlu Ye, Ziqi Wang, Han Zhong, Heng Ji, Nan Jiang, and Tong Zhang. Iterative preference learning from human feedback: Bridging theory and practice for rlhf under kl-constraint. arXiv preprint arXiv:2312.11456, 2023a. Xiong et al. (2023b) Wei Xiong, Hanze Dong, Chenlu Ye, Han Zhong, Nan Jiang, and Tong Zhang. Gibbs sampling from human feedback: A provable kl-constrained framework for rlhf. CoRR, 2023b. Xiong et al. (2025) Wei Xiong, Jiarui Yao, Yuhui Xu, Bo Pang, Lei Wang, Doyen Sahoo, Junnan Li, Nan Jiang, Tong Zhang, Caiming Xiong, et al. A minimalist approach to llm reasoning: from rejection sampling to reinforce. arXiv preprint arXiv:2504.11343, 2025. Zhan et al. (2023) Wenhao Zhan, Masatoshi Uehara, Nathan Kallus, Jason D Lee, and Wen Sun. Provable offline preference-based reinforcement learning. arXiv preprint arXiv:2305.14816, 2023. Zhang et al. (2024) Shenao Zhang, Donghan Yu, Hiteshi Sharma, Han Zhong, Zhihan Liu, Ziyi Yang, Shuohang Wang, Hany Hassan, and Zhaoran Wang. Self-exploring language models: Active preference elicitation for online alignment. arXiv preprint arXiv:2405.19332, 2024. Zhao et al. (2025) Heyang Zhao, Chenlu Ye, Wei Xiong, Quanquan Gu, and Tong Zhang. Logarithmic regret for online kl-regularized reinforcement learning. arXiv preprint arXiv:2502.07460, 2025. Appendix A Detailed Theoretical Analysis A.1 Theoretical understanding on solving the minimax problem in Eq 1 with PET Here, we explain in detail how the PET algorithm in Alg 1 solves the theoretical problem in Eq. 1. Recall that Eq. 1 is a minimax problem based on rejection sampling as follows: minr∈ℛmaxπ∈ΠRSn,π0(Vrμ(π)−Vrμ(πref))+β⋅ℒ(r).subscriptℛsubscriptsubscriptsuperscriptΠsubscript0RSsubscriptsuperscriptsubscriptsuperscriptsubscriptref⋅subscriptℒ _r _π∈ ^n, _0_RS (V^μ_% r(π)-V^μ_r( _ref) )+β·L_ % D(r).minitalic_r ∈ R maxitalic_π ∈ Πitalic_n , π start_POSTSUBSCRIPT 0RS end_POSTSUBSCRIPT ( Vitalic_μitalic_r ( π ) - Vitalic_μitalic_r ( πref ) ) + β ⋅ Lcaligraphic_D ( r ) . By Proposition 2.1, we have πRS(π0,n,rt)∈argmaxπ∈ΠRSn,π0Vrμ(π).subscriptRSsubscript0superscriptsubscriptsubscriptsuperscriptΠsubscript0RSsubscriptsuperscript _RS( _0,n,r^t)∈ _π∈ ^n, _0_RS% V^μ_r(π).πRS ( π0 , n , ritalic_t ) ∈ arg maxitalic_π ∈ Πitalic_n , π start_POSTSUBSCRIPT 0RS end_POSTSUBSCRIPT Vitalic_μitalic_r ( π ) . Therefore, solving the minimax problem in Eq 1 is equivalent to solving the minimization problem as follows: minr∈ℛ(Vrμ(πRS(π0,n,r))−Vrμ(πref))+β⋅ℒ(r).subscriptℛsubscriptsuperscriptsubscriptRSsubscript0subscriptsuperscriptsubscriptref⋅subscriptℒ _r (V^μ_r( _RS( _0,n,r))-V^μ% _r( _ref) )+β·L_D(r).minitalic_r ∈ R ( Vitalic_μitalic_r ( πRS ( π0 , n , r ) ) - Vitalic_μitalic_r ( πref ) ) + β ⋅ Lcaligraphic_D ( r ) . Denote f(r,π):=(Vrμ(π)−Vrμ(πref))+β⋅ℒ(r)assignsubscriptsuperscriptsubscriptsuperscriptsubscriptref⋅subscriptℒf(r,π):= (V^μ_r(π)-V^μ_r( _ref) )+β% ·L_D(r)f ( r , π ) := ( Vitalic_μitalic_r ( π ) - Vitalic_μitalic_r ( πref ) ) + β ⋅ Lcaligraphic_D ( r ) and h(r):=(Vrμ(πRS(π0,n,rt))−Vrμ(πref))+β⋅ℒ(r).assignℎsubscriptsuperscriptsubscriptRSsubscript0superscriptsubscriptsuperscriptsubscriptref⋅subscriptℒh(r):= (V^μ_r( _RS( _0,n,r^t))-V^μ_r( _% ref) )+β·L_D(r).h ( r ) := ( Vitalic_μitalic_r ( πRS ( π0 , n , ritalic_t ) ) - Vitalic_μitalic_r ( πref ) ) + β ⋅ Lcaligraphic_D ( r ) . Then, our goal is to solve the problem minr∈ℛh(r)subscriptℛℎ _r h(r)minitalic_r ∈ R h ( r ). For any reward model r0∈ℛsubscript0ℛr_0 0 ∈ R, we have ∇h(r)|r=r0evaluated-at∇ℎsubscript0 ∇ h(r)|_r=r_0∇ h ( r ) |r = r start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT =∇rf(r,π)|r=r0,π=πRS(π0,n,r0)+∇πf(r,π)|r=r0,π=πRS(π0,n,r0)⋅∇rπRS(π0,n,r0)|r=r0absentevaluated-atsubscript∇formulae-sequencesubscript0subscriptRSsubscript0subscript0evaluated-at⋅evaluated-atsubscript∇formulae-sequencesubscript0subscriptRSsubscript0subscript0subscript∇subscriptRSsubscript0subscript0subscript0 = _rf(r,π)|_r=r_0,π= _RS( _0,n,r_0% )+ _πf(r,π)|_r=r_0,π= _RS( _0,n,r_0)·% _r _RS( _0,n,r_0)|_r=r_0= ∇r f ( r , π ) |r = r start_POSTSUBSCRIPT 0 , π = πRS ( π0 , n , r0 ) end_POSTSUBSCRIPT + ∇π f ( r , π ) |r = r start_POSTSUBSCRIPT 0 , π = πRS ( π0 , n , r0 ) end_POSTSUBSCRIPT ⋅ ∇r πRS ( π0 , n , r0 ) |r = r start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT =∇rf(r,π)|r=r0,π=πRS(π0,n,r0)absentevaluated-atsubscript∇formulae-sequencesubscript0subscriptRSsubscript0subscript0 = _rf(r,π)|_r=r_0,π= _RS( _0,n,r_0)= ∇r f ( r , π ) |r = r start_POSTSUBSCRIPT 0 , π = πRS ( π0 , n , r0 ) end_POSTSUBSCRIPT The second equality holds because π=πRS(π0,n,r)subscriptRSsubscript0π= _RS( _0,n,r)π = πRS ( π0 , n , r ) is the optimizer for maxπ∈ΠRSn,π0f(r,π),subscriptsubscriptsuperscriptΠsubscript0RS _π∈ ^n, _0_RSf(r,π),maxitalic_π ∈ Πitalic_n , π start_POSTSUBSCRIPT 0RS end_POSTSUBSCRIPT f ( r , π ) , so we have ∇πf(r,π)|r=r0,π=πRS(π0,n,r0)≡0,∀r0∈ℛformulae-sequenceevaluated-atsubscript∇formulae-sequencesubscript0subscriptRSsubscript0subscript00for-allsubscript0ℛ _πf(r,π)|_r=r_0,π= _RS( _0,n,r_0)≡ 0,% ∀ r_0 ∇π f ( r , π ) |r = r start_POSTSUBSCRIPT 0 , π = πRS ( π0 , n , r0 ) end_POSTSUBSCRIPT ≡ 0 , ∀ r0 ∈ R. This is an important result as it shows that to compute ∇h(r)|r=r0evaluated-at∇ℎsubscript0∇ h(r)|_r=r_0∇ h ( r ) |r = r start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT, we don’t need to compute ∇rπRS(π0,n,r)|r=r0.evaluated-atsubscript∇subscriptRSsubscript0subscript0 _r _RS( _0,n,r)|_r=r_0.∇r πRS ( π0 , n , r ) |r = r start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT . The latter represents how the rejection sampling would change if the reward function changes, which is hard to approximate in practice. A standard approach to find the minimizer of minrh(r)subscriptℎ _rh(r)minitalic_r h ( r ) is to perform stochastic gradient descent on h(r).ℎh(r).h ( r ) . By the previous results, we can implement this process by approximating the value of ∇h(r)|r=r0=∇rf(r,π)|π=πRS(π0,n,rt)).∇ h(r)|_r=r_0= _rf(r,π)|_π= _RS( _0,n,r^t% )).∇ h ( r ) |r = r start_POSTSUBSCRIPT 0 end_POSTSUBSCRIPT = ∇r f ( r , π ) |π = π start_POSTSUBSCRIPT RS ( π0 , n , ritalic_t ) ) end_POSTSUBSCRIPT . Next, we show that PET in Alg 1 is essentially the process of stochastic gradient descent to solve the problem minr∈ℛh(r)subscriptℛℎ _r h(r)minitalic_r ∈ R h ( r ). Recall Vμr(π)=x∼μ,a∼π(⋅|x)[r(x,a)].V^r_μ(π)=E_x μ,a π(·|x)[r(x,a)].Vitalic_ritalic_μ ( π ) = blackboard_Ex ∼ μ , a ∼ π ( ⋅ | x ) [ r ( x , a ) ] . At line 5555, the prompts xisubscriptx_ixitalic_i and the data minibatch tsubscriptD_tDitalic_t are sampled from the dataset DD. At line 6666, the responses ai,aref,isubscriptsubscriptrefa_i,a_ref,iaitalic_i , aref , i are stochastically sampled from the current policy πtsuperscriptπ^tπitalic_t and the reference policy πref.subscriptref _ref.πref . Therefore, the expectation of the average loss lt/Msuperscriptl^t/Mlitalic_t / M equals to the value of h(rt)ℎsuperscripth(r^t)h ( ritalic_t ): [∑i∈[M][rt(xi,πt(xi,ai))−rt(xi,πt(xi,aref,i))+β⋅ℒt(rt)]]/M=(Vrμ(πt)−Vrμ(πref))+β⋅ℒ(r)delimited-[]subscriptdelimited-[]delimited-[]superscriptsubscriptsuperscriptsuperscriptsubscriptsuperscriptsubscriptsuperscriptsuperscriptsubscriptref⋅subscriptℒsubscriptsuperscriptsubscriptsuperscriptsuperscriptsubscriptsuperscriptsubscriptref⋅subscriptℒE[ _i∈[M][r^t(x_i,π^t(x^i,a_i))-r^t(x_i,π^t% (x^i,a_ref,i))+β·L_D_t(r^t)]]/M=% (V^μ_r(π^t)-V^μ_r( _ref) )+β·% L_D(r)blackboard_E [ ∑i ∈ [ M ] [ ritalic_t ( xitalic_i , πitalic_t ( xitalic_i , aitalic_i ) ) - ritalic_t ( xitalic_i , πitalic_t ( xitalic_i , aref , i ) ) + β ⋅ Lcaligraphic_D start_POSTSUBSCRIPT t end_POSTSUBSCRIPT ( ritalic_t ) ] ] / M = ( Vitalic_μitalic_r ( πitalic_t ) - Vitalic_μitalic_r ( πref ) ) + β ⋅ Lcaligraphic_D ( r ) At line 8888, the algorithm computes the gradient of ltsuperscriptl^tlitalic_t, so Alg 1 is essentially a standard process of stochastic gradient descent to solve the minimization problem minr∈ℛh(r).subscriptℛℎ _r h(r).minitalic_r ∈ R h ( r ) . Therefore, PET is a convenient implementation to solve the minimax problem based on rejection sampling in Eq. 1. Remark A.1. Here we argue that PET approximates the most pessimistic low-prediction-loss reward model against the rejection sampling process. Denote r^ rover start_ARG r end_ARG the solution to the minimization problem r^∈minr∈ℛ(Vrμ(π)−Vrμ(πref))+β⋅ℒ(r).^subscriptℛsubscriptsuperscriptsubscriptsuperscriptsubscriptref⋅subscriptℒ r∈ _r (V^μ_r(π)-V^μ_r( _% ref) )+β·L_D(r).over start_ARG r end_ARG ∈ minitalic_r ∈ R ( Vitalic_μitalic_r ( π ) - Vitalic_μitalic_r ( πref ) ) + β ⋅ Lcaligraphic_D ( r ) . For any reward model r∈ℛr ∈ R that has the same or lower prediction loss on the dataset ℒ(r)≤ℒ(r^)subscriptℒsubscriptℒ^L_D(r) _D( r)Lcaligraphic_D ( r ) ≤ Lcaligraphic_D ( over start_ARG r end_ARG ), we have (Vr^μ(π)−Vr^μ(πref))≤(Vrμ(π)−Vrμ(πref)).subscriptsuperscript^subscriptsuperscript^subscriptrefsubscriptsuperscriptsubscriptsuperscriptsubscriptref (V^μ_ r(π)-V^μ_ r( _ref) )≤% (V^μ_r(π)-V^μ_r( _ref) ).( Vitalic_μover start_ARG r end_ARG ( π ) - Vitalic_μover start_ARG r end_ARG ( πref ) ) ≤ ( Vitalic_μitalic_r ( π ) - Vitalic_μitalic_r ( πref ) ) . This is because (Vr^μ(π)−Vr^μ(πref))+β⋅ℒ(r)≤(Vrμ(π)−Vrμ(πref))+β⋅ℒ(r)subscriptsuperscript^subscriptsuperscript^subscriptref⋅subscriptℒsubscriptsuperscriptsubscriptsuperscriptsubscriptref⋅subscriptℒ (V^μ_ r(π)-V^μ_ r( _ref% ) )+β·L_D(r)≤ (V^μ_r(π)-V% ^μ_r( _ref) )+β·L_D(r)( Vitalic_μover start_ARG r end_ARG ( π ) - Vitalic_μover start_ARG r end_ARG ( πref ) ) + β ⋅ Lcaligraphic_D ( r ) ≤ ( Vitalic_μitalic_r ( π ) - Vitalic_μitalic_r ( πref ) ) + β ⋅ Lcaligraphic_D ( r ) ⇒ ⇒ (Vr^μ(π)−Vr^μ(πref))−(Vrμ(π)−Vrμ(πref))≤β⋅(ℒ(r)−ℒ(r^))subscriptsuperscript^subscriptsuperscript^subscriptrefsubscriptsuperscriptsubscriptsuperscriptsubscriptref⋅subscriptℒsubscriptℒ (V^μ_ r(π)-V^μ_ r( _ref% ) )- (V^μ_r(π)-V^μ_r( _ref) )≤% β·(L_D(r)-L_D( r))( Vitalic_μover start_ARG r end_ARG ( π ) - Vitalic_μover start_ARG r end_ARG ( πref ) ) - ( Vitalic_μitalic_r ( π ) - Vitalic_μitalic_r ( πref ) ) ≤ β ⋅ ( Lcaligraphic_D ( r ) - Lcaligraphic_D ( over start_ARG r end_ARG ) ) ⇒ ⇒ (Vr^μ(π)−Vr^μ(πref))−(Vrμ(π)−Vrμ(πref))≤0subscriptsuperscript^subscriptsuperscript^subscriptrefsubscriptsuperscriptsubscriptsuperscriptsubscriptref0 (V^μ_ r(π)-V^μ_ r( _ref% ) )- (V^μ_r(π)-V^μ_r( _ref) )≤ 0( Vitalic_μover start_ARG r end_ARG ( π ) - Vitalic_μover start_ARG r end_ARG ( πref ) ) - ( Vitalic_μitalic_r ( π ) - Vitalic_μitalic_r ( πref ) ) ≤ 0 ⇒ ⇒ (Vr^μ(π)−Vr^μ(πref))≤(Vrμ(π)−Vrμ(πref))subscriptsuperscript^subscriptsuperscript^subscriptrefsubscriptsuperscriptsubscriptsuperscriptsubscriptref (V^μ_ r(π)-V^μ_ r( _ref% ) )≤ (V^μ_r(π)-V^μ_r( _ref) )( Vitalic_μover start_ARG r end_ARG ( π ) - Vitalic_μover start_ARG r end_ARG ( πref ) ) ≤ ( Vitalic_μitalic_r ( π ) - Vitalic_μitalic_r ( πref ) ) In practice, we use the PET algorithm to approximate the solution π0subscript0 _0π0 and find that the prediction loss of the learned reward model is as low as that of the proxy reward, which is specially trained to minimize the prediction loss. This implies that the reward model r^ rover start_ARG r end_ARG is also the most pessimistic reward model that gives minimal relative score to the rejection sampling process among the reward models with low values of prediction loss. This observation also supports our argument that the reward model learned from PET is pessimistic. A.2 Proof for Proposition 2.1 Proposition 2.1 describes an intuitive property of rejection sampling. It is equivalent to the statement that when optimizing a policy on a reward model r1subscript1r_1r1 through the rejection sampling process, the RS process that sets its reward model as r=r1subscript1r=r_1r = r1 achieves the highest reward on r1subscript1r_1r1. For any rejection sampling process with the same base policy and number of samples, the distribution of the sampled responses during the process is always the same. Therefore, for any outcome of sampled responses, the RS process that sets r=r1subscript1r=r_1r = r1 can always output the response with the highest reward on r1subscript1r_1r1. Formally, the proof for Proposition 2.1 is as follows. Proof. Given a base policy π0subscript0 _0π0, a positive integer n, and a prompt x, consider a stochastic process where n responses are i.i.d drawn from the policy at the prompt ai∼i.i.dπ0(⋅|x),i∈[n].a_i i.i.d _0(·|x),i∈[n].aitalic_i start_OVERACCENT i . i . d end_OVERACCENT start_ARG ∼ end_ARG π0 ( ⋅ | x ) , i ∈ [ n ] . Let E(x)E(x)E ( x ) be the space of all possible outcomes. For any outcome e∈E(x)e∈ E(x)e ∈ E ( x ), let a1,…,ansubscript1…subscript\a_1,…,a_n\ a1 , … , aitalic_n be the sampled responses. For any two different reward models r1≠r2subscript1subscript2r_1≠ r_2r1 ≠ r2, denote i1∗∈argmaxi∈[n]r1(x,ai)subscriptsuperscript1subscriptdelimited-[]subscript1subscripti^*_1∈ _i∈[n]r_1(x,a_i)i∗1 ∈ arg maxitalic_i ∈ [ n ] r1 ( x , aitalic_i ) and i2∗∈argmaxi∈[n]r2(x,ai)subscriptsuperscript2subscriptdelimited-[]subscript2subscripti^*_2∈ _i∈[n]r_2(x,a_i)i∗2 ∈ arg maxitalic_i ∈ [ n ] r2 ( x , aitalic_i ) as the optimal index on the two rewards. Denote v1(e,r):=r(x,ai1∗)assignsubscript1subscriptsubscriptsuperscript1v_1(e,r):=r(x,a_i^*_1)v1 ( e , r ) := r ( x , aitalic_i∗ start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT ) and v2(e,r):=r(x,ai2∗)assignsubscript2subscriptsubscriptsuperscript2v_2(e,r):=r(x,a_i^*_2)v2 ( e , r ) := r ( x , aitalic_i∗ start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT ) where r is any reward model, then by definition, we have v1(e,r1)≥v2(e,r1).subscript1subscript1subscript2subscript1v_1(e,r_1)≥ v_2(e,r_1).v1 ( e , r1 ) ≥ v2 ( e , r1 ) . Consider two rejection sampling policies π1=πRS(π0,r1,n),π2=πRS(π0,r2,n)formulae-sequencesubscript1subscriptRSsubscript0subscript1subscript2subscriptRSsubscript0subscript2 _1= _RS( _0,r_1,n), _2= _RS( _0,r_2% ,n)π1 = πRS ( π0 , r1 , n ) , π2 = πRS ( π0 , r2 , n ) defined on the base policy π0subscript0 _0π0, sampling number n, and the reward models r1,r2subscript1subscript2r_1,r_2r1 , r2. Their performance on the reward model r1subscript1r_1r1 for any prompt distribution μ satisfies Vr1μ(π1)=x∼μ,e∼E(x)[v1(e,r1)]subscriptsuperscriptsubscript1subscript1subscriptformulae-sequencesimilar-tosimilar-todelimited-[]subscript1subscript1V^μ_r_1( _1)=E_x μ,e E(x)[v_1(e,r_1)]Vitalic_μitalic_r start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT ( π1 ) = blackboard_Ex ∼ μ , e ∼ E ( x ) [ v1 ( e , r1 ) ], Vr1μ(π2)=x∼μ,e∼E(x)[v2(e,r1)]subscriptsuperscriptsubscript1subscript2subscriptformulae-sequencesimilar-tosimilar-todelimited-[]subscript2subscript1V^μ_r_1( _2)=E_x μ,e E(x)[v_2(e,r_1)]Vitalic_μitalic_r start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT ( π2 ) = blackboard_Ex ∼ μ , e ∼ E ( x ) [ v2 ( e , r1 ) ]. By the previous result v1(e,r1)≥v2(e,r1),subscript1subscript1subscript2subscript1v_1(e,r_1)≥ v_2(e,r_1),v1 ( e , r1 ) ≥ v2 ( e , r1 ) , we have Vr1μ(π1)≥Vr1μ(π2),subscriptsuperscriptsubscript1subscript1subscriptsuperscriptsubscript1subscript2V^μ_r_1( _1)≥ V^μ_r_1( _2),Vitalic_μitalic_r start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT ( π1 ) ≥ Vitalic_μitalic_r start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT ( π2 ) , which conclude the proof. ∎ A.3 Proof for Theorem 3.3 Proof. Our proof generally follows the proof technique in Liu et al. [2024b]. Let r^ rover start_ARG r end_ARG be the reward solution to r^∈argminr∈ℛ^subscriptℛ r∈ _r over start_ARG r end_ARG ∈ arg minitalic_r ∈ R Vr∗μ(π)−Vr∗μ(π^)subscriptsuperscriptsuperscriptsubscriptsuperscriptsuperscript V^μ_r^*(π)-V^μ_r^*( π)Vitalic_μitalic_r∗ ( π ) - Vitalic_μitalic_r∗ ( over start_ARG π end_ARG ) =(Vr∗μ(π)−Vr^μ(π))+(Vr^μ(π)−Vr^μ(π^))+(Vr^μ(π^)−Vr∗μ(π^))absentsubscriptsuperscriptsuperscriptsubscriptsuperscript^subscriptsuperscript^subscriptsuperscript^^subscriptsuperscript^^subscriptsuperscriptsuperscript = (V^μ_r^*(π)-V^μ_ r(π) )+% (V^μ_ r(π)-V^μ_ r( π) )+ (V^% μ_ r( π)-V^μ_r^*( π) )= ( Vitalic_μitalic_r∗ ( π ) - Vitalic_μover start_ARG r end_ARG ( π ) ) + ( Vitalic_μover start_ARG r end_ARG ( π ) - Vitalic_μover start_ARG r end_ARG ( over start_ARG π end_ARG ) ) + ( Vitalic_μover start_ARG r end_ARG ( over start_ARG π end_ARG ) - Vitalic_μitalic_r∗ ( over start_ARG π end_ARG ) ) ≤(Vr∗μ(π)−Vr^μ(π))+(Vr^μ(π^)−Vr∗μ(π^))absentsubscriptsuperscriptsuperscriptsubscriptsuperscript^subscriptsuperscript^^subscriptsuperscriptsuperscript ≤ (V^μ_r^*(π)-V^μ_ r(π) )+% (V^μ_ r( π)-V^μ_r^*( π) )≤ ( Vitalic_μitalic_r∗ ( π ) - Vitalic_μover start_ARG r end_ARG ( π ) ) + ( Vitalic_μover start_ARG r end_ARG ( over start_ARG π end_ARG ) - Vitalic_μitalic_r∗ ( over start_ARG π end_ARG ) ) =(Vr∗μ(π)−Vr^μ(π))+(Vr^μ(π^)−Vr^μ(πref)+β⋅ℒ(r^)−(Vr∗μ(π^)−Vr^μ(πref)+β⋅ℒ(r∗)))+absentsubscriptsuperscriptsuperscriptsubscriptsuperscript^limit-fromsubscriptsuperscript^^subscriptsuperscript^subscriptref⋅subscriptℒ^subscriptsuperscriptsuperscript^subscriptsuperscript^subscriptref⋅subscriptℒsuperscript = (V^μ_r^*(π)-V^μ_ r(π) )+% (V^μ_ r( π)-V^μ_ r( _ref)+β% ·L_D( r)-(V^μ_r^*( π)-V^μ_% r( _ref)+β·L_D(r^*)) )+= ( Vitalic_μitalic_r∗ ( π ) - Vitalic_μover start_ARG r end_ARG ( π ) ) + ( Vitalic_μover start_ARG r end_ARG ( over start_ARG π end_ARG ) - Vitalic_μover start_ARG r end_ARG ( πref ) + β ⋅ Lcaligraphic_D ( over start_ARG r end_ARG ) - ( Vitalic_μitalic_r∗ ( over start_ARG π end_ARG ) - Vitalic_μover start_ARG r end_ARG ( πref ) + β ⋅ Lcaligraphic_D ( r∗ ) ) ) + β⋅(ℒ(r∗)−ℒ(r^))⋅subscriptℒsuperscriptsubscriptℒ β·(L_D(r^*)-L_ % D( r))β ⋅ ( Lcaligraphic_D ( r∗ ) - Lcaligraphic_D ( over start_ARG r end_ARG ) ) ≤(Vr∗μ(π)−Vr^μ(π))+β⋅(ℒ(r∗)−ℒ(r^))absentsubscriptsuperscriptsuperscriptsubscriptsuperscript^⋅subscriptℒsuperscriptsubscriptℒ ≤ (V^μ_r^*(π)-V^μ_ r(π) )+% β·(L_D(r^*)-L_D( r))≤ ( Vitalic_μitalic_r∗ ( π ) - Vitalic_μover start_ARG r end_ARG ( π ) ) + β ⋅ ( Lcaligraphic_D ( r∗ ) - Lcaligraphic_D ( over start_ARG r end_ARG ) ) The first equality utilizes the optimality of the policy solution π^∈argmaxπ∈ΠRSVr^μ(π)^subscriptsubscriptΠRSsubscriptsuperscript π∈ _π∈ _RSV^μ_ r(π)over start_ARG π end_ARG ∈ arg maxitalic_π ∈ Π start_POSTSUBSCRIPT RS end_POSTSUBSCRIPT Vitalic_μover start_ARG r end_ARG ( π ). The second equality utilizes the optimality of the reward solution r^∈argminr∈ℛ(Vr^μ(π)−Vr^μ(πref)+β⋅ℒ(r))^subscriptℛsubscriptsuperscript^subscriptsuperscript^subscriptref⋅subscriptℒ r∈ _r (V^μ_ r(π)-V^μ_% r( _ref)+β·L_D(r) )over start_ARG r end_ARG ∈ arg minitalic_r ∈ R ( Vitalic_μover start_ARG r end_ARG ( π ) - Vitalic_μover start_ARG r end_ARG ( πref ) + β ⋅ Lcaligraphic_D ( r ) ). Intuitively, the formulation in the last line is bounded for any reward function r^ rover start_ARG r end_ARG. If the reward function is close to the true reward r∗superscriptr^*r∗, then both differences Vr∗μ(π)−Vr^μ(π)subscriptsuperscriptsuperscriptsubscriptsuperscript^V^μ_r^*(π)-V^μ_ r(π)Vitalic_μitalic_r∗ ( π ) - Vitalic_μover start_ARG r end_ARG ( π ) and ℒ(r∗)−ℒ(r^)subscriptℒsuperscriptsubscriptℒ^L_D(r^*)-L_D( r)Lcaligraphic_D ( r∗ ) - Lcaligraphic_D ( over start_ARG r end_ARG ) should be small. If r^ rover start_ARG r end_ARG is very different from r∗superscriptr^*r∗, then its prediction loss on the dataset ℒ(r^)subscriptℒ^L_D( r)Lcaligraphic_D ( over start_ARG r end_ARG ) would also be high, so that the performance gap can still be bounded. This indicates the importance of adding the prediction loss as a constraint on the reward model in Eq. 1. Formally, based on the results of Theorem 5.3 in Liu et al. [2024b], with probability at least 1−δ11- 1 - δ, the last term can be bound by Vr∗μ(π)−Vr∗μ(π^)≤μ(ℛ,π,πref)28κ2⋅β+3βNlog(Nϵ(ℛ,∥⋅∥∞)δ).V^μ_r^*(π)-V^μ_r^*( π)≤ C_ _% D(R,π, _ref)^28κ^2·β+% 3βN ( N_ε(R,\|·\|_∞)% δ).Vitalic_μitalic_r∗ ( π ) - Vitalic_μitalic_r∗ ( over start_ARG π end_ARG ) ≤ divide start_ARG Citalic_μ start_POSTSUBSCRIPT D end_POSTSUBSCRIPT ( R , π , πref )2 end_ARG start_ARG 8 κ2 ⋅ β end_ARG + divide start_ARG 3 β end_ARG start_ARG N end_ARG log ( divide start_ARG Nitalic_ϵ ( R , ∥ ⋅ ∥∞ ) end_ARG start_ARG δ end_ARG ) . Here, κ=1(1+exp(R))21superscript12κ= 1(1+ (R))^2κ = divide start_ARG 1 end_ARG start_ARG ( 1 + exp ( R ) )2 end_ARG is a constant, and Nϵ(ℛ,∥⋅∥∞)N_ε(R,\|·\|_∞)Nitalic_ϵ ( R , ∥ ⋅ ∥∞ ) is the ϵitalic-ϵεϵ-covering number for the reward model class Cheng et al. [2022]. Setting β=N2κ⋅6log(Nϵ(ℛ,∥⋅∥∞)δ)β= N2κ· 6 ( N_ε(R,% \|·\|_∞)δ)β = divide start_ARG square-root start_ARG N end_ARG end_ARG start_ARG 2 κ ⋅ square-root start_ARG 6 log ( divide start_ARG Nitalic_ϵ ( R , ∥ ⋅ ∥∞ ) end_ARG start_ARG δ end_ARG ) end_ARG end_ARG concludes the proof. Note that the bound here is tighter than the bound on Liu et al. [2024b], because there is no KL regularization in our formulation at Eq. 1. Adding a KL regularization multiplied by any positive coefficient to Eq. 1 will increase the bound on the performance gap analysis, which is undesired for sampling efficiency. ∎ Appendix B Policy Optimization Methods KL regularized proximal policy optimization (KL-PPO): For a prompt distribution μ, the ‘KL divergence’ between two policies can be defined as KLμ(π1,π2):=x∼μ[KL(π1(⋅|x)∥π2(⋅|x))]KL_μ( _1, _2):=E_x μ[KL( _1(% ·|x)\| _2(·|x))]KLμ ( π1 , π2 ) := blackboard_Ex ∼ μ [ KL ( π1 ( ⋅ | x ) ∥ π2 ( ⋅ | x ) ) ], where KL(π1(⋅|x)∥π2(⋅|x)):=∑a∈π1(a|x)⋅logπ1(a|x)π2(a|x)KL( _1(·|x)\| _2(·|x)):= _a _1(a% |x)· _1(a|x) _2(a|x)KL ( π1 ( ⋅ | x ) ∥ π2 ( ⋅ | x ) ) := ∑a ∈ A π1 ( a | x ) ⋅ log divide start_ARG π1 ( a | x ) end_ARG start_ARG π2 ( a | x ) end_ARG is the standard KL divergence between two distributions. Consider a proxy policy πrefsubscriptref _refπref of the dataset. The proxy policy is trained to generate a response distribution that is similar to the dataset response distribution. This is often achieved by a standard supervised fine-tuning process on the prompts and responses from the dataset DD [Touvron et al., 2023]. The KL divergence between a policy π and πrefsubscriptref _refπref intuitively indicates how well it is covered by the dataset. Then, the learning goal of the agent becomes π^←argmaxπ∈ΠVr^μ(π)+η⋅KLμ(π,πref),←^subscriptΠsubscriptsuperscript^⋅subscriptKLsubscriptref π← _π∈ V^μ_ r(π)+η·% KL_μ(π, _ref),over start_ARG π end_ARG ← arg maxitalic_π ∈ Π Vitalic_μover start_ARG r end_ARG ( π ) + η ⋅ KLμ ( π , πref ) , where Π Π is a model family, η>00η>0η > 0 is the weight of the KL regularization in the optimization target. The most popular way to solve this optimization problem is by using the PPO algorithm [Ouyang et al., 2022]. Combining the reward modeling step and the policy optimization process gives the popular RLHF algorithm ‘PPO-KL’ as shown in Alg 3 Rejection sampling process (RS): Rejection sampling, also known as best-of-N sampling, is an inference-time policy optimization method. Given a base policy model π0subscript0 _0π0, a reward model r^ rover start_ARG r end_ARG, and a positive integer n, the process of rejection sampling is defined in Alg 4. In practice, the reward model r^ rover start_ARG r end_ARG is trained by minimizing the prediction loss, and the base policy π0subscript0 _0π0 is usually set as a proxy policy for the dataset. The rejection sampling process is effectively a policy as it takes a prompt as input and stochastically outputs a response. An RS policy can be directly implemented on a policy model and a reward model without any training. It also achieves a higher reward on the reward model r^ rover start_ARG r end_ARG compared to the base policy π0subscript0 _0π0. The pessimism in rejection sampling implicitly relies on the notion of KL regularization. When the value of n is small, the RS policies will have a limited KL regularization compared to the base policy π0subscript0 _0π0, which intuitively reduces the risk of reward hacking Beirami et al. [2024]. One can achieve a higher performance on the reward model with rejection sampling by increasing the value of n, but in this case, the risk of reward hacking also increases Huang et al. [2025]. In practice, reward hacking has been observed in RS with a relatively high value of n Touvron et al. [2023], Gao et al. [2023]. Therefore, it is important to develop principled methods to free RS policies from reward hacking. Algorithm 3 KL-PPO 1:Input: Reference policy πrefsubscriptref _refπref, Dataset DD, KL weight η 2:Step 1: Reward modeling r^←minr∈ℛℒ(r)←^subscriptℛsubscriptℒ r← _r L_D(r)over start_ARG r end_ARG ← minitalic_r ∈ R Lcaligraphic_D ( r ) 3:Step 2: Policy optimization with PPO π^←maxπ∈ΠVr^μ(π)−η⋅KLμ(π,πref)←^subscriptΠsubscriptsuperscript^⋅subscriptKLsubscriptref π← _π∈ V^μ_ r(π)-η·KL_% μ(π, _ref)over start_ARG π end_ARG ← maxitalic_π ∈ Π Vitalic_μover start_ARG r end_ARG ( π ) - η ⋅ KLμ ( π , πref ) 4:Return: Policy model π^ πover start_ARG π end_ARG Algorithm 4 Rejection sampling process 1:Input: Base policy π0subscript0 _0π0, reward model r^ rover start_ARG r end_ARG, number of samples n, prompt x 2:Draw responses ai∼i.i.d.π0(⋅|x),∀i∈[n]a_i i.i.d. _0(·|x),∀ i∈[n]aitalic_i start_OVERACCENT i . i . d . end_OVERACCENT start_ARG ∼ end_ARG π0 ( ⋅ | x ) , ∀ i ∈ [ n ] 3:Generate rewards ri=r^(x,ai),∀i∈[n]formulae-sequencesubscript^subscriptfor-alldelimited-[]r_i= r(x,a_i),∀ i∈[n]ritalic_i = over start_ARG r end_ARG ( x , aitalic_i ) , ∀ i ∈ [ n ] 4:Return: Response ai∗subscriptsuperscripta_i^*aitalic_i∗, where i∗∈argmaxi∈[n]risuperscriptsubscriptdelimited-[]subscripti^*∈ _i∈[n]r_ii∗ ∈ arg maxitalic_i ∈ [ n ] ritalic_i Appendix C Additional Experiment Details Hyper-parameter setups: Configuration PET PPO learning rate 3e−8383e-83 e - 8 3e−6363e-63 e - 6 learning scheduler type cosine cosine batch size 128128128128 128128128128 gradient accumulation steps 16 16 training epoch 1111 1111 pessimistic coefficient β 10101010 / KL regularization weight η / 0.050.050.050.05 optimizer adamw torch adamw torch precision bfloat16 bfloat16 Table 5: Training configurations for PET and PPO. We list detailed training configurations for PET and PPO in Table 5. For standard SFT model and proxy reward model training, we follow the same setup as in [Huang et al., 2024b]. For all baseline methods, we follow the original authors’ implementations or descriptions. Specifically, for RPO, we remove the default chat template provided by its source code, as it is unsuitable for our summarization and IMDB tasks. This is confirmed by our observation that including the template degrades RPO’s performance on our tasks, so the template is not included in our RPO training. Statistical significance: To verify that the methods we test with have stable output, we test our method PET-PPO and other baselines, including KL-PPO, DPO, RPO, and χPO, on both TL;DR summarization and IMDB datasets with a different random seed. In Table. 6 and Table. 7, we show the empirical mean and confidence intervals of the evaluation results on the RLHF methods from the two random seeds. KL-PPO DPO RPO χPO PET-PPO Summarization 40.5±0.3plus-or-minus40.50.340.5± 0.340.5 ± 0.3 37.5±1.3plus-or-minus37.51.337.5± 1.337.5 ± 1.3 33.6±0.4plus-or-minus33.60.433.6± 0.433.6 ± 0.4 38.2±1.0plus-or-minus38.21.038.2± 1.038.2 ± 1.0 41.4±0.6plus-or-minus41.40.641.4± 0.641.4 ± 0.6 IMDB 92.0±0.2plus-or-minus92.00.292.0± 0.292.0 ± 0.2 95.0±0.0plus-or-minus95.00.095.0± 0.095.0 ± 0.0 93.1±0.4plus-or-minus93.10.493.1± 0.493.1 ± 0.4 93.0±0.2plus-or-minus93.00.293.0± 0.293.0 ± 0.2 96.1±2.2plus-or-minus96.12.296.1± 2.296.1 ± 2.2 Table 6: Empirical mean and standard errors of the evaluation results for the RLHF methods on the summarization and IMDB dataset KL-PPO DPO RPO χPO PET-PPO 57.2±0.0plus-or-minus57.20.057.2± 0.057.2 ± 0.0 51.2±1.2plus-or-minus51.21.251.2± 1.251.2 ± 1.2 62.0±3.6plus-or-minus62.03.662.0± 3.662.0 ± 3.6 55.0±2.2plus-or-minus55.02.255.0± 2.255.0 ± 2.2 Table 7: Empirical mean and standard errors for the win rate of the PET-PPO method against other RLHF baselines on the summarization dataset. Computational resource: We use 2 NVIDIA A100-PCIE GPUs, each with 40GB VRAM and 2 48-core230 Intel Xeon Silver 4214R CPU.