Paper deep dive
RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs
Shreyas Chaudhari, Pranjal Aggarwal, Vishvak Murahari, Tanmay Rajpurohit, Ashwin Kalyan, Karthik Narasimhan, Ameet Deshpande, Bruno Castro da Silva
Models: BERT, GPT-3, LLaMA, OPT, PaLM, T5
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 95%
Last extracted: 3/12/2026, 7:26:46 PM
Summary
This paper provides a critical analysis of Reinforcement Learning from Human Feedback (RLHF) for Large Language Models (LLMs). It examines the fundamental components of RLHF, specifically focusing on the reward model, modeling choices, and function approximation caveats. The authors identify key limitations such as incorrect generalization, model misspecification, and feedback sparsity, while providing a Bayesian interpretation of the RLHF framework and a comprehensive literature review.
Entities (5)
Relation Signals (4)
RLHF â appliesto â LLM
confidence 100% ¡ It has since been adopted for fine-tuning LLMs using human feedback.
PPO â isa â Reinforcement Learning Algorithm
confidence 95% ¡ Proximal Policy Optimization (PPO) Schulman et al. [2017] is elected as the reinforcement learning algorithm.
RLHF â utilizes â Reward Model
confidence 95% ¡ The reward function forms the central cog of the RLHF procedure
DPO â alternativeto â RLHF
confidence 90% ¡ the recently proposed direct preference optimization (DPO) approach Rafailov et al. [2023] replaces reinforcement learning
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:State-of-the-art large language models (LLMs) have become indispensable tools for various tasks. However, training LLMs to serve as effective assistants for humans requires careful consideration. A promising approach is reinforcement learning from human feedback (RLHF), which leverages human feedback to update the model in accordance with human preferences and mitigate issues like toxicity and hallucinations. Yet, an understanding of RLHF for LLMs is largely entangled with initial design choices that popularized the method and current research focuses on augmenting those choices rather than fundamentally improving the framework. In this paper, we analyze RLHF through the lens of reinforcement learning principles to develop an understanding of its fundamentals, dedicating substantial focus to the core component of RLHF -- the reward model. Our study investigates modeling choices, caveats of function approximation, and their implications on RLHF training algorithms, highlighting the underlying assumptions made about the expressivity of reward. Our analysis improves the understanding of the role of reward models and methods for their training, concurrently revealing limitations of the current methodology. We characterize these limitations, including incorrect generalization, model misspecification, and the sparsity of feedback, along with their impact on the performance of a language model. The discussion and analysis are substantiated by a categorical review of current literature, serving as a reference for researchers and practitioners to understand the challenges of RLHF and build upon existing efforts.
Tags
Links
- Source: https://arxiv.org/abs/2404.08555
- Canonical: https://arxiv.org/abs/2404.08555
Trouble viewing inline? Open PDF directly â
Full Text
217,648 characters extracted from source content.
Expand or collapse full text
RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs Shreyas Chaudhari*1 &Pranjal Aggarwal*2 &Vishvak Murahari3 &Tanmay Rajpurohit4 &Ashwin Kalyan5 Karthik Narasimhan3 Ameet Deshpande3 Bruno Castro da Silva1 1University of Massachusetts Amherst 2Department of Computer Science, Indian Institute of Technology, Delhi 3Department of Computer Science, Princeton University 4Georgia Tech 5Independent Researcher *Equal Contribution schaudhari@cs.umass.edu, pranjal2041@gmail.com Abstract State-of-the-art large language models (LLMs) have become indispensable tools for various tasks. However, training LLMs to serve as effective assistants for humans requires careful consideration. A promising approach is reinforcement learning from human feedback (RLHF), which leverages human feedback to update the model in accordance with human preferences and mitigate issues like toxicity and hallucinations. Yet, an understanding of RLHF for LLMs is largely entangled with initial design choices that popularized the method and current research focuses on augmenting those choices rather than fundamentally improving the framework. In this paper, we analyze RLHF through the lens of reinforcement learning principles to develop an understanding of its fundamentals, dedicating substantial focus to the core component of RLHFâthe reward model. Our study investigates modeling choices, caveats of function approximation, and their implications on RLHF training algorithms, highlighting the underlying assumptions made about the expressivity of reward. Our analysis improves the understanding of the role of reward models and methods for their training, concurrently revealing limitations of the current methodology. We characterize these limitations, including incorrect generalization, model misspecification, and the sparsity of feedback, along with their impact on the performance of a language model. The discussion and analysis are substantiated by a categorical review of current literature, serving as a reference for researchers and practitioners to understand the challenges of RLHF and build upon existing efforts. Contents 1 Introduction 2 Motivation: Eliminating Objective Mismatch in Pre-Trained Language Models 2.1 Bayesian Interpretation of RLHF 3 Formulation: Text Generation as Sequential Decision-Making 4 The Role of Reward 4.1 Oracular Reward and the Role of Human Feedback 4.2 Reward Modeling 4.3 Measuring Alignment 5 Inferring the Reward from Human Feedback 5.1 Limitations of the Reward Model 6 Reinforcement Learning with Imperfect Rewards 6.1 Policy Gradient Algorithms 6.2 Misalignment due to Imperfect Rewards 6.3 Why use Reinforcement Learning Algorithms? 7 Review of Reinforcement Learning from Human Feedback for Language Models 7.1 Language Model Pre-Training: Foundation for Large Language Models 7.2 Reinforcement Learning from Human Feedback (RLHF): Overview and Motivation 7.3 Human Feedback 7.3.1 Nature and Objectives of Human Feedback 7.3.2 Types of Human Feedback 7.3.3 Collection of High-Quality Human Feedback 7.4 Supervised Fine-Tuning: Limitations and Role 7.5 Reward Modeling 7.6 RLHF Finetuning of Language Models 7.6.1 Training Procedure 7.6.2 Training Algorithms 7.6.3 Improving Training Stability 7.6.4 Iterated RLHF 7.7 Limitations of RLHF Models 7.8 Enriching Reward Signals in Reinforcement Learning 7.9 Moving Beyond RL Training 7.9.1 Alternatives to RL using Reward Model 7.9.2 Alternatives to RL without Explicit Reward Models 8 Discussion and Conclusion 1 Introduction Figure 1: Overview of the RLHF procedure, illustrating the challenges encountered at each step. The paper conducts a detailed examination of these challenges, providing valuable insights into each stage of the procedure. Large Language Models (LLMs) demonstrate remarkable capabilities that extend beyond basic language tasks, leading to their widespread adoption across various industries. The remarkable utility of these models holds the potential to transform established workflows in critical sectors such as technology, healthcare, finance, and education Singhal et al. [2022]; Wu et al. [2023a]; Yan et al. [2023]. As they become integral to these domains, itâs crucial to ensure that the behavior of LLMs is predictable, safe, and trustworthyââmeeting the expectations set for a human performing the same tasks. This challenge of making LLMs exhibit human-like qualities, known as alignment with human objectives, is central to making these models suitable for diverse tasks. An effective method for addressing this challenge is reinforcement learning from human feedback (RLHF). RLHF first gained popularity due to its ability to solve reinforcement learning (RL) problems like simulated robotic locomotion and playing Atari games Christiano et al. [2017] without access to a reward function, by simply leveraging human feedback about preferences on demonstrated behaviors. It has since been adopted for fine-tuning LLMs using human feedback. This leads to a natural inquiry: How can a method designed to master games be effectively used to align LLMs with human objectives? The method has proven to be immensely successful OpenAI [2022], but not without well-documented limitations Casper et al. [2023]. A comprehensive understanding of why it achieves its success remains largely elusive. Consequently, research efforts on the topic are stuck in a local minima, with variants focused on augmenting the components of the methodâincluding the training algorithm Ramamurthy et al. [2022], reward model Wu et al. [2023c], and even RL-free approaches Rafailov et al. [2023]. However, some fundamental limitations of the approach remain obscured due to the overarching goal of recent work to refine the initial design choices. In this work, we develop a comprehensive understanding of RLHF by analyzing the core components of the method. We begin the study by motivating the necessity for RLHF by highlighting the problem of objective mismatch in pre-trained LMs (Section 2). To formulate foundational questions about the framework, we adopt a Bayesian perspective of RLHF. It serves to highlight the significance of the reward function in particular (Section 4). The reward function forms the central cog of the RLHF procedure, and the design choices used to model it form a major focus of our study. The current formulation of RLHF relies on a set of assumptions to model the reward function (Section 4.1, 4.2). Following the delineation of these assumptions, an analysis of the reward model independent of specific modeling choices follows. The analysis, in a principled manner, provides an understanding of issues such as: 1. The impractical requirement for extensive amounts of feedback data for training accurate reward models. 2. The combination of very limited feedback data and the use of function approximation results in misgeneralization, wherein inaccurate reward values are assigned to inputs not seen during training. These imperfections of the reward model, along with challenges such as reward sparsity and reward model misspecification, are highlighted in the paper (Section 5.1). Their impact on the performance of a language model is explored in detail (Section 6.2). The course of the analysis leads to the formalization of concepts such as an oracular reward that serve as the theoretical golden standard for future efforts (Section 4.1). An overview of the RLHF procedure along with the various challenges studied in this work is provided in Figure 1. The discussion is followed by an extensive survey of an expanding body of literature related to the topic. The survey is organized into sections that outline the framework of RLHF. Starting with a high-level overview of Large Language Models (LLMs), the survey systematically covers various aspects: ⢠Different types of human (and non-human) feedback (Section 7.3), ⢠The training methods in RLHF (Section 7.6), ⢠Alternative approaches that do not rely on RL or reward models (Section 7.9). This structure aims to provide a comprehensive overview of the extensive landscape of works that have contributed to the remarkable success of RLHF. 2 Motivation: Eliminating Objective Mismatch in Pre-Trained Language Models Large pre-trained language models (PLMs) are massive neural networks that are trained on a huge corpus of texts using a self-supervised learning objective. Originally utilized for representation learning Devlin et al. [2019]; Liu et al. [2019] with encoder-only models, recent research, particularly influenced by Brown et al. [2020], has shifted its focus towards training PLMs to directly generate answers for textual problems. State-of-the-art PLMs typically employ an auto-regressive transformer architecture Vaswani et al. [2017] and are trained with a causal language modeling objective. These models implicitly capture a conditional probability distribution Ďθsubscript _θĎitalic_θ, reflecting the likelihood of sampling the next token after observing a sequence of previous tokens. The probability of a text sequence x:=(x1,âŚ,xT)assignsubscript1âŚsubscriptx:=(x_1,âŚ,x_T)x := ( x1 , ⌠, xitalic_T ), under this model is denoted as PrâĄ(x;Ďθ)=ât=1Tâ1Ďθâ˘(xt+1âŁxt,âŚ,x1)Prsubscriptsuperscriptsubscriptproduct11subscriptconditionalsubscript1subscriptâŚsubscript1 (x; _θ)= _t=1^T-1 _θ(x_t+1 x_t,âŚ,x_% 1)Pr ( x ; Ďitalic_θ ) = ât = 1T - 1 Ďitalic_θ ( xitalic_t + 1 ⣠xitalic_t , ⌠, x1 ). The model is trained to estimate the pre-training data generating probability distribution over text sequences by minimizing the (forward) KL divergence between the modelâs data-generating distribution and the pre-training data distribution, denoted by Ppre-trainâ˘(â )subscriptpre-trainâ P_pre-train(¡)Ppre-train ( â ). minθDKL(Ppre-train(x)âŁPr(x;Ďθ))=minθxâźPpre-train[logPpre-train(x)]âxâźPpre-train[logPr(x;Ďθ)]. _θD_ KL(P_pre-train(x) (x% ; _θ))= _θE_x P_pre-train[ P_% pre-train(x)]-E_x P_pre-train[ (x; _% θ)].minitalic_θ DKL ( Ppre-train ( x ) ⣠⣠Pr ( x ; Ďitalic_θ ) ) = minitalic_θ blackboard_Ex âź P start_POSTSUBSCRIPT pre-train end_POSTSUBSCRIPT [ log Ppre-train ( x ) ] - blackboard_Ex âź P start_POSTSUBSCRIPT pre-train end_POSTSUBSCRIPT [ log Pr ( x ; Ďitalic_θ ) ] . (1) The first term, representing the entropy of Ppre-trainsubscriptpre-trainP_pre-trainPpre-train, is independent of θ and can be disregarded during optimization. Consequently, the objective simplifies to the following cross-entropy minimization form: minθâxâźPpre-trainâ˘[logâĄPrâĄ(x;Ďθ)].subscriptsubscriptsimilar-tosubscriptpre-traindelimited-[]Prsubscript _θ-E_x P_pre-train[ (x;% _θ)].minitalic_θ - blackboard_Ex âź P start_POSTSUBSCRIPT pre-train end_POSTSUBSCRIPT [ log Pr ( x ; Ďitalic_θ ) ] . (2) The expectation is approximated using samples from an unsupervised pretraining text corpus DD, which comprises text sequences sampled from Ppre-trainsubscriptpre-trainP_pre-trainPpre-train. This leads us to the following objective: minθâ1||â˘âxâât=1Tâ1logâĄĎθâ˘(xt+1âŁxt,âŚ,x1).subscript1subscriptsuperscriptsubscript11subscriptconditionalsubscript1subscriptâŚsubscript1 _θ- 1|D| _x _% t=1^T-1 _θ(x_t+1 x_t,âŚ,x_1).minitalic_θ - divide start_ARG 1 end_ARG start_ARG | D | end_ARG âx â D ât = 1T - 1 log Ďitalic_θ ( xitalic_t + 1 ⣠xitalic_t , ⌠, x1 ) . (3) The remarkable property about PLMs lies in the contrast between the simplicity of the training recipe and the remarkable results that they deliver Brown et al. [2020]. Simply capturing language statistics along with scaling up the number of trainable parameters, endows PLMs with robust semantic representations, vast commonsense knowledge, and strong pattern-following capabilities. However, for adopting PLMs to assist humans with tasks that require an understanding of human intentions and the ability to follow instructions, the simple training recipe of PLMs is insufficient. These models demonstrate a shallow understanding of human intentions, often generating undesirable outputs, including incorrect facts or conveying biased and toxic opinions. Fundamentally, PLMs suffer from an objective mismatch problem: the training-time objective of capturing language statistics does not necessarily align with the deployment-time objective of fulfilling a human userâs specific goals. Eliminating this mismatch at first glance seems feasible: just train PLMs to optimize for the user objective. Unfortunately, for many tasks, it is impossible to express the user objective as an optimization target. For example, when a userâs objective pertains to eliciting humorous responses, establishing specific criteria for objectively evaluating the humor in a generated response becomes an inherently challenging task. There are currently two primary ways to deal with the problem: the behaviorist approach and the cognition-driven approach. The behaviorist approach, implemented by supervised fine-tuning (SFT), aims to replicate observable behaviors that humans perceive as desirable without explicit consideration of the underlying user objective. For instance, if a user desires good summaries of articles, this approach trains a model to imitate examples of good summaries without explicitly defining the criteria for a good summary. In contrast, the cognition-driven approach, implemented by reinforcement learning from human feedback (RLHF), aims to uncover the underlying user objective that governs the observed behaviors. It then updates the model by optimizing the uncovered objective. This approach relies on certain assumptionsâwhich in the case of RLHF are: (i) the user objective can bear the form of a reward function, which can assign a numerical score to behaviors of the model, and (i) this function can be approximated by a machine learning model (e.g., a neural network). RLHF estimates this reward function and updates the PLM via reinforcement learning to optimize for rewards. Regardless of the approach, the process of addressing the objective mismatch problem is commonly referred to as the fine-tuning or alignment process. Presently, state-of-the-art language models typically initiate this process with the behaviorist approach, followed by the cognition-driven approach. 2.1 Bayesian Interpretation of RLHF RLHF relies on observing human feedback to deduce the (latent) user reward function. Human feedback is provided on the outputs from a language model. RLHF assumes that there exists an underlying human reward function that governs the feedback they provide in a particular manner, i.e., there exists some mapping from reward to actions of a human. Suppose the reward function is being inferred by a model RĎsubscriptitalic-ĎR_ĎRitalic_Ď parameterized by Ďitalic-ĎĎĎ. Adopting a Bayesian inference perspective Korbak et al. [2022], the parameters Ďitalic-ĎĎĎ can be viewed as a hypothesis with the dataset of human feedback HFsubscriptHFD_HFDHF as the evidence for this hypothesis. Given a prior distribution over the hypothesis PrâĄ(Ď)Pritalic-Ď (Ď)Pr ( Ď ), we can apply Bayesâ rule to derive the posterior distribution over the hypotheses after observing the evidence as: PrâĄ(ĎâŁHF)âPrâĄ(HFâŁRĎ)â˘PrâĄ(Ď)proportional-toPrconditionalitalic-ĎsubscriptHFPrconditionalsubscriptHFsubscriptitalic-ĎPritalic-Ď (Ď _HF) (D_ % HF R_Ď) (Ď)Pr ( Ď âŁ DHF ) â Pr ( DHF ⣠Ritalic_Ď ) Pr ( Ď ) (4) Reward modeling in RLHF can be seen as computing the maximum a posteriori (MAP) estimate of the parameters of a reward model, ĎMAP=argâĄmaxĎâĄPrâĄ(ĎâŁHF)subscriptitalic-ĎMAPsubscriptitalic-ĎPrconditionalitalic-ĎsubscriptHF _ MAP= _Ď~ (Ď _% HF)ĎMAP = arg maxitalic_Ď Pr ( Ď âŁ DHF ) =argâĄmaxĎâĄPrâĄ(HFâŁRĎ)â(a)â˘PrâĄ(Ď)â(b)absentsubscriptitalic-ĎsubscriptâPrconditionalsubscriptHFsubscriptitalic-Ď(a)subscriptâPritalic-Ď(b) = _Ď~ (D_HF R_% Ď)_(a)~ (Ď)_(b)= arg maxitalic_Ď underâ start_ARG Pr ( DHF ⣠Ritalic_Ď ) end_ARG(a) underâ start_ARG Pr ( Ď ) end_ARG(b) (5) The first term (a) is the log-likelihood of the feedback dataset, specifying how a humanâs internal objective (reward function) governs their feedback. The second term (b) represents constraints on the hypothesis space, which is enforced through explicit and implicit regularization techniques in neural-network training. The presented framework raises two major questions: 1. What is the form of the likelihood function PrâĄ(HFâŁRĎ)PrconditionalsubscriptHFsubscriptitalic-Ď (D_HF R_Ď)Pr ( DHF ⣠Ritalic_Ď )? In other words, how do we mathematically model the influence of a humanâs latent objective on their observable feedback? 2. What is the reinforcement learning algorithm used for optimizing the reward model? In other words, how do we ensure the model acts consistently with its objective? A set of answers to these questions forms the basis for an RLHF algorithm. The RLHF methodology, popularized by Christiano et al. [2017], employs pairwise ranking feedback and uses the Bradley-Terry model Bradley and Terry [1952] as the likelihood function. Proximal Policy Optimization (PPO) Schulman et al. [2017] is elected as the reinforcement learning algorithm. Before we move into the analysis of this method, we urge the readers to take a moment to reflect on the choices and assumptions we have made so far to derive the general recipe of RLHF. Are there alternative choices? Can the assumptions be relaxed or improved? Thinking critically about these foundational decisions is the key to understanding the strengths and weaknesses of RLHF algorithms and innovating them. For example, the recently proposed direct preference optimization (DPO) approach Rafailov et al. [2023] replaces reinforcement learning with a reformulation of the objective. Next, we formalize the problem setup of text generation as an agent interacting with a sequential decision process, laying the foundation for the analysis of RLHF. We refer the reader to Section 7.6 for a detailed outline of the RLHF procedure, and Figure 5 for a summarized overview. 3 Formulation: Text Generation as Sequential Decision-Making Figure 2: Text generation from LLMs modeled as a Markov decision process. The generation process is auto-regressive, utilizing the token output (action) from the previous time step and the context (state) as input to produce the next token through the language model (policy). Given a context c, the language model produces the token o1subscript1o_1o1 at the first timestep. A concatenation of the two [c,o1]subscript1[c,o_1][ c , o1 ] forms the input to the policy at the next timestep (Table 1). A reward function scores the generated output for a given context. In this section, we formulate the text generation procedure from a language model as a sequential decision-making process. This formulation is essential for constructing reinforcement learning algorithms. Markov decision process. A common framework for modeling sequential decision-making processes is Markov Decision Process (MDP) Markov [1954]. An MDP is defined as a tuple (,,p,R,Ď)(S,A,p,R,Ď)( S , A , p , R , Ď ) where SS is the set of states, AA is the set of actions, p:ĂâÎâ˘():âÎp:SĂAâ (S)p : S Ă A â Î ( S ) is the transition function, R:Ăââ:ââR:SĂA : S Ă A â blackboard_R is the reward function, and Ď:âÎâ˘():âÎĎ:Sâ (S)Ď : S â Î ( S ) is the initial state distribution. Each sequential time step of the process is denoted by t, and st,at,rtsubscriptsubscriptsubscripts_t,a_t,r_tsitalic_t , aitalic_t , ritalic_t denote the values of the state, action, and reward at time step t. A discounting factor Îłâ(0,1]01Îłâ(0,1]Îł â ( 0 , 1 ] is defined for discounting rewards over time, particularly useful for modeling an MDP with an infinite number of time steps (i.e., an infinite-horizon MDP). However, the outputs of language models are truncated after a finite number of steps. We use T to denote the maximum time step. An agent acts in an MDP using a policy Ď:âÎâ˘():âÎĎ:Sâ (A)Ď : S â Î ( A ). The agent starts in state s1âźĎâ˘(â )similar-tosubscript1â s_1 Ď(¡)s1 âź Ď ( â ). At time step t, it chooses an action atâźĎ(â âŁst)a_t Ď(~¡ s_t)aitalic_t âź Ď ( â ⣠sitalic_t ), executes the action, transitions to a new state st+1âźp(â âŁst,at)s_t+1 p(~¡ s_t,a_t)sitalic_t + 1 âź p ( â ⣠sitalic_t , aitalic_t ), and receives a reward rt=Râ˘(st,at)subscriptsubscriptsubscriptr_t=R(s_t,a_t)ritalic_t = R ( sitalic_t , aitalic_t ). The term âMarkovâ in MDP refers to the Markov property, in that the distribution over the next state st+1subscript1s_t+1sitalic_t + 1 depends on only the current state stsubscripts_tsitalic_t and action atsubscripta_taitalic_t. Language models as agents in MDP. For simplicity, we consider text generation tasks that include only one turn of interaction between the user and the model. We make a distinction between the text that a user inputs into the model, denoted by c and referred to as the context or the prompt, and the text that the model generates by itself to the context, denoted by o and referred to as the output or simply the generated text. Let V be the set of all tokens that the model can generate (the vocabulary), CC the set of all possible contexts, and OO the set of all possible outputs. Given a context câc â C as input, the model generates an output oâo â O token by token. Specifically, let otsubscripto_toitalic_t be the t-th token in generated output o, then the model parameterized by θ first outputs token o1âźĎθ(â âŁc)o_1 _θ(~¡ c)o1 âź Ďitalic_θ ( â ⣠c ), and then conditioned on the concatenation of o1subscript1o_1o1 and c it generates o2âźĎθ(â âŁ[c,o1])o_2 _θ(~¡ [c,o_1])o2 âź Ďitalic_θ ( â ⣠[ c , o1 ] ), and so on. We can see that this generation process resembles an agent traversing in an MDP (Figure 2). The model acts according to a policy Ďθsubscript _θĎitalic_θ. The start-state distribution Ď is the distribution over user-provided contexts. The action space is the vocabulary V. The action atsubscripta_taitalic_t is the generated token otsubscripto_toitalic_t. The state stsubscripts_tsitalic_t is the concatenation of the context c and all the tokens the model has generated up to time step tâ11t-1t - 1. The transition function p(â âŁst,at)=δ([st,at])p(~¡ s_t,a_t)=δ([s_t,a_t])p ( â ⣠sitalic_t , aitalic_t ) = δ ( [ sitalic_t , aitalic_t ] ) is a delta distribution, i.e., the next state is deterministic given the current state and action. Reward rtsubscriptr_tritalic_t given at time step t is computed by the reward model as RĎâ˘(st,at)subscriptitalic-ĎsubscriptsubscriptR_Ď(s_t,a_t)Ritalic_Ď ( sitalic_t , aitalic_t ) which is either a human or a function learned from human feedback. MDP element Description Text generation equivalence s1subscript1s_1s1 Initial state c stsubscripts_tsitalic_t State at time step t [c,o1:tâ1]=[s1,a1:tâ1]=[stâ1,atâ1]subscript:11subscript1subscript:11subscript1subscript1[c,o_1:t-1]=[s_1,a_1:t-1]=[s_t-1,a_t-1][ c , o1 : t - 1 ] = [ s1 , a1 : t - 1 ] = [ sitalic_t - 1 , aitalic_t - 1 ] atsubscripta_taitalic_t Action taken at time step t otsubscripto_toitalic_t Ďâ˘(atâŁst)conditionalsubscriptsubscriptĎ(a_t s_t)Ď ( aitalic_t ⣠sitalic_t ) Policy Ďθâ˘(otâŁ[c,o1:tâ1])subscriptconditionalsubscriptsubscript:11 _θ(o_t [c,o_1:t-1])Ďitalic_θ ( oitalic_t ⣠[ c , o1 : t - 1 ] ) rtsubscriptr_tritalic_t Reward at time step t RĎâ˘([c,o1:tâ1])subscriptitalic-Ďsubscript:11R_Ď([c,o_1:t-1])Ritalic_Ď ( [ c , o1 : t - 1 ] ) Ďâ˘(s1)subscript1Ď(s_1)Ď ( s1 ) Initial state distribution PrâĄ(c)Pr (c)Pr ( c ) pâ˘(st+1âŁst,at)conditionalsubscript1subscriptsubscriptp(s_t+1 s_t,a_t)p ( sitalic_t + 1 ⣠sitalic_t , aitalic_t ) Transition function δâ˘([st,at])subscriptsubscriptδ([s_t,a_t])δ ( [ sitalic_t , aitalic_t ] ) Table 1: Mapping from text generation to MDP The text generation MDP has several special properties: 1. The action space is extremely large. For example, the LLaMa model Touvron et al. [2023a, b] employs a vocabulary of size 32K. Having a gigantic action space blows up the search space for reinforcement learning algorithms. 2. The structure of the state space is complex, as a state is essentially a text sequence. Pre-training on large amounts of texts is necessary to learn an initially good representation of this space. 3. The initial state distribution has an enormous support. All conceivable contexts lie in the support, thus strongly testing the ability of the policy to generalize to out-of-distribution states. 4. The reward function used for training can differ from the evaluation reward function. This is because the humans providing rewards during evaluation may be different from the humans involved in trainnig the reward model. Analogous to transfer learning in RL, the agent must then adapt to the new reward function. 5. The transition function is deterministic. Algorithmic and analysis tools tailored for deterministic MDPs can be applied. Thus, solving a text generation MDP requires specialized treatment that takes advantage of its properties and overcomes its inherent challenges. Reinforcement learning Sutton and Barto [2018]; Bertsekas and Tsitsiklis [1996] provides solutions for optimally solving an MDP, i.e., learning a policy that maximizes the accumulated reward. Consequently, RLHF updates the language model to generate more rewarding outputs. Naturally, the reward function plays a critical role in the process of fine-tuning model outputs, determining practical and fundamental limits Casper et al. [2023] of the efficacy of RLHF. 4 The Role of Reward The goal of reward learning in RLHF is to convert human feedback into an optimizable reward function. The reward serves a dual purpose: it encodes the task information (for example, identical input-output pairs would receive distinct rewards depending on whether the task involved summarization or text expansion) 111Unless the task is specified in the input prompt itself, in which case the inputs differ. as well as preferences over those outputs (a condescending summary is rewarded less than a neutral summary). The reward thus encodes relevant information for measuring (Section 4.3) as well as inducing alignment with human objectives. By setting the reward function of the sequential decision process to the one estimated from human feedback RĎsubscriptitalic-ĎR_ĎRitalic_Ď, reinforcement learning algorithms can be used to learn a language model policy that maximizes the cumulative reward, resulting in an aligned language model. 4.1 Oracular Reward and the Role of Human Feedback An implicit assumption made in RLHF is that a humanâs feedback behavior is governed by and can be represented as an oracular reward function Râ:Ăââ:superscriptâââR :CĂO â : C Ă O â blackboard_R. We assume that this function is deterministic in line with the current methodology. The function takes as input a context c and an output o, and outputs a scalar number reflecting the preference on o as a continuation of c. Because the [c,o][c,o][ c , o ] is essentially a state in the MDP formulation, the reward function is essentially defined over states of the MDP. The language model that maximizes the oracular reward accurately reflects the goals and preferences inherent in the human feedback, and maximization of this reward consequently aligns the model with the human preferences. The oracular reward may not be accessible or learnable, but under the reward hypothesis Sutton [2004]; Silver et al. [2021], the mere existence of such a reward may be assumedâthough this may be challenged Knox et al. [2022]. The oracular reward forms the golden standard for training as well as evaluating any language model. In general, humans can give a variety of feedback. RLHF operates with feedback that discloses information about the oracular reward function. Most methods focus on two types of feedback: point-wise numerical feedback (or rating), and pairwise ranking feedback (or preferences). Providing ratings is the most straightforward way to communicate the reward function. Given a pair (c,o)(c,o)( c , o ), the rating is a scalar r=Rââ˘(c,o)superscriptâr=R (c,o)r = Râ ( c , o ). While ratings can be fed directly into a reinforcement learning algorithm, learning a reward model takes advantage of the generalizability of the reward model on unseen outputs and contexts. Preference feedback compares two outputs generated for the same context. Given two outputs o and oⲠgenerated for context c, a human denoted a preference oâťoâ˛succeedssuperscriptâ˛o o o âť oⲠif the first input is preferred and oâ˛âťosucceedssuperscriptâ˛o oⲠ❠o otherwise. Preferences in their raw form are not compatible learning signals for reinforcement learning algorithms. Hence, a reward model must be learned for this type of feedback. To do so, an assumption must be made about the relationship between preferences and RâsuperscriptâR Râ. We will discuss this in more detail in the next section. A discussion about the various methodologies used for encoding preferences can be found in Section 7.5. An alternative approach for ranking outputs on the basis of preferences is provided by the learning-to-rank paradigm Liu et al. [2009]. Using preference feedback offers several advantages compared to using ratings. Firstly, we get more training data for the reward model. In practice, people collect a ranking of N outputs and create preference pairs Ouyang et al. [2022]. Collecting N ratings for N outputs provides we get N training points. Ranking N outputs provided Nâ˘(Nâ1)/212N(N-1)/2N ( N - 1 ) / 2 pairwise comparisons. Second, preferences require assigning a only relative order rather than an absolute precise score to an output; the latter task could take significantly more cognitive effort and is more prone to inconsistency. Finally, a preference is presumably easier to provide because it offers a âbaselineâ for comparison (the worse output). In contrast, when giving a rating, a human can rely on only the evaluation guidelines. A note on stochastic rewards: The reward function is considered to be a deterministic mapping from text to a scalar value. This amounts to averaging the preferences of all humans that provided human feedback. Moreover, it assumes that a human must always rate an input-output pair with the same score, discounting the inherent variability of human preferences. There are numerous scenariosâlike personalization, in-context adaptation to ongoing dialogue, and diverse output generationâwhere a deterministic mapping is limiting. The rewards are more appropriately modeled as being stochastic, wherein each input-output pair is scored by a distribution over scalar rewards, say râźRhuman(â âŁc,o)r R_human(¡ c,o)r âź Rhuman ( â ⣠c , o ). This modeling accounts for the two sources of uncertainty: (i) uncertainty over the specific human from a group of humans who provide feedback, and (i) variability in a humanâs preferences due to changes in unobserved factors Nguyen et al. [2017]. Some work in reinforcement learning aims to address this by learning Bayesian preferences, primarily for uncertainty quantification and safety analysis Ramachandran and Amir [2007]; Brown and Niekum [2019], and can be adapted to model a distribution of preferences over text. Some recent efforts along these lines Barnett et al. [2023] have proven to be effective. We focus on deterministic rewards for the analysis that follows. 4.2 Reward Modeling Learning a reward model serves two purposes: (i) to convert RLHF into a canonical reinforcement learning problem, and (i) to reduce the cost of online feedback-collection. Reinforcement learning algorithms define their objective in terms of a reward function. To apply these algorithms, we need to infer a reward function from a feedback dataset, collecting which is notoriously expensive. Currently, large language models require thousands to millions of feedback data points. To gather that amount, many human evaluators need to be recruited to work in parallel. To ensure the assumptions regarding the oracular reward function hold, the evaluators must be trained to agree with one another on the evaluation criteria. This process is continual: multiple rounds of feedback collections need to be conducted to iteratively improve the model. The premise of approaches that learn a reward model is that the generalization error of the reward model is expected to decrease faster than that of the policy as a function of the number of labeled data points, arising from the notion that supervised learning is often considered a simpler problem than generative modeling. Following the previous section, we denote the reward model by RĎâ˘(c,o)subscriptitalic-ĎR_Ď(c,o)Ritalic_Ď ( c , o ) and the feedback dataset by HFsubscriptHFD_HFDHF. Our goal is to decide a likelihood function PrâĄ(HFâŁĎ)PrconditionalsubscriptHFitalic-Ď (D_HF Ď)Pr ( DHF âŁ Ď ) and find Ďitalic-ĎĎĎ that maximizes this function: maxĎâĄPrâĄ(HFâŁĎ)subscriptitalic-ĎPrconditionalsubscriptHFitalic-Ď _Ď~ (D_HF Ď)maxitalic_Ď Pr ( DHF âŁ Ď ) (6) With rating feedback, the reward-modeling problem can be formulated as a prediction problem with continuous output. A common objective for this type of problem is the minimization of the mean squared error (MSE): minĎâ˘â(c,o,r)âHF(RĎâ˘(c,o)âr)2subscriptitalic-ĎsubscriptsubscriptHFsuperscriptsubscriptitalic-Ď2 _Ď _(c,o,r) _HF(R_Ď(c,o)% -r)^2minitalic_Ď â( c , o , r ) â D start_POSTSUBSCRIPT HF end_POSTSUBSCRIPT ( Ritalic_Ď ( c , o ) - r )2 (7) To incorporate preference feedback, we need to choose the form of the likelihood function denoting each preference, i.e., PrâĄ((oâťoâ˛,c)âŁĎ)Prconditionalsucceedssuperscriptâ˛italic-Ď ((o o ,c) Ď)Pr ( ( o âť oⲠ, c ) âŁ Ď ). The RLHF method of Ouyang et al. [2022] employs the Bradley-Terry model to represent the likelihood of a data point: PrâĄ((oâťoâ˛,c)âŁĎ)Prconditionalsucceedssuperscriptâ˛italic-Ď ((o o ,c) Ď)Pr ( ( o âť oⲠ, c ) âŁ Ď ) =expâĄ(RĎâ˘(c,o))expâĄ(RĎâ˘(c,o))+expâĄ(RĎâ˘(c,oâ˛))absentsubscriptitalic-Ďsubscriptitalic-Ďsubscriptitalic-ĎsuperscriptⲠ= (R_Ď(c,o)) (R_Ď(c,o))+ (R_Ď(c,o% ))= divide start_ARG exp ( Ritalic_Ď ( c , o ) ) end_ARG start_ARG exp ( Ritalic_Ď ( c , o ) ) + exp ( Ritalic_Ď ( c , oⲠ) ) end_ARG (8) =11+expâĄ(RĎâ˘(c,oâ˛)âRĎâ˘(c,o))absent11subscriptitalic-Ďsuperscriptâ˛subscriptitalic-Ď = 11+ (R_Ď(c,o )-R_Ď(c,o))= divide start_ARG 1 end_ARG start_ARG 1 + exp ( Ritalic_Ď ( c , oⲠ) - Ritalic_Ď ( c , o ) ) end_ARG (9) =Ďâ˘[RĎâ˘(c,o)âRĎâ˘(c,oâ˛)]absentdelimited-[]subscriptitalic-Ďsubscriptitalic-ĎsuperscriptⲠ=Ď[R_Ď(c,o)-R_Ď(c,o )]= Ď [ Ritalic_Ď ( c , o ) - Ritalic_Ď ( c , oⲠ) ] (10) where Ďâ˘(x)=11+eâx11superscriptĎ(x)= 11+e^-xĎ ( x ) = divide start_ARG 1 end_ARG start_ARG 1 + e- x end_ARG is the sigmoid function. The learning objective for maximizing the log-likelihood of the dataset HFsubscriptHFD_HFDHF is, maxĎâ˘â(c,o,oâ˛)âHFlogâĄPrâĄ((oâťoâ˛,c)âŁĎ)=maxĎâ˘â(c,o,oâ˛)âHFlogâĄĎâ˘[RĎâ˘(c,o)âRĎâ˘(c,oâ˛)].subscriptitalic-Ďsubscriptsuperscriptâ˛subscriptHFPrconditionalsucceedssuperscriptâ˛italic-Ďsubscriptitalic-Ďsubscriptsuperscriptâ˛subscriptHFdelimited-[]subscriptitalic-Ďsubscriptitalic-ĎsuperscriptⲠ_Ď _(c,o,o ) _HF % ((o o ,c) Ď)= _Ď _(c,o,o )â% D_HF Ď[R_Ď(c,o)-R_Ď(c,o )]~.maxitalic_Ď â( c , o , oⲠ) â D start_POSTSUBSCRIPT HF end_POSTSUBSCRIPT log Pr ( ( o âť oⲠ, c ) âŁ Ď ) = maxitalic_Ď â( c , o , oⲠ) â D start_POSTSUBSCRIPT HF end_POSTSUBSCRIPT log Ď [ Ritalic_Ď ( c , o ) - Ritalic_Ď ( c , oⲠ) ] . (11) In Section 5, we further generalize the form of feedback and the likelihood function to conduct an analysis independent of the specifics of particular design choices. 4.3 Measuring Alignment Evaluation of natural language tasks is a difficult problem, and the study of evaluation metrics is an active area of research. Of particular importance, and difficulty, is to measure the alignment of a language model to a humanâs objectives, which in practice is evaluated along the axes of helpfulness, harmlessness, and honesty. The oracular reward that governs a humanâs preferences serves as a yardstick for measuring the degree of alignment. The task of alignment is then reformulated as encoding the preferences demonstrated by a human into a reward function, and updating the parameters of the language model to produce output that maximizes this reward. A reward provides an analytical metric to measure the overall performance of a language model Ď, where the performance captures the degree of alignment with human preferences along with the degree of satisfaction of the task itself Ngo et al. [2022]. The performance of a model Ď, for distribution over contexts dCâ˘(â )subscriptâ d_C(¡)ditalic_C ( â ), can be measured by averaging the rewards for the outputs generated by Ď given the contexts. Let the performance be denoted by Jâ˘(Ď)J(Ď)J ( Ď ): Jâ˘(Ď):=âcâodCâ˘(c)â˘Ďâ˘(oâŁc)â˘Rââ˘(c,o)=câźdCâ˘(â )â˘[OâźĎ(â |c)â˘[Rââ˘(c,O)|C=c]]J(Ď):= _c _od_C(c)Ď(o c)R (c,o)=E_c d% _C(¡) [E_O Ď(¡|c) [R (c,O)|C=c % ] ]J ( Ď ) := âc âo ditalic_C ( c ) Ď ( o ⣠c ) Râ ( c , o ) = blackboard_Ec âź d start_POSTSUBSCRIPT C ( â ) end_POSTSUBSCRIPT [ blackboard_EO âź Ď ( â | c ) [ Râ ( c , O ) | C = c ] ] (12) The context distribution dCâ˘(â )subscriptâ d_C(¡)ditalic_C ( â ) can be the distribution of contexts in the training data, test data, or a held-out validation dataset, depending on the data on which the performance of the model is being evaluated. The sequential nature of the output generation equivalently allows us to express Jâ˘(Ď)J(Ď)J ( Ď ) as: Jâ˘(Ď):=Ďâ˘[âtRââ˘(st,at)]=Ďâ˘[âtRââ˘(c,o1:tâ1,ot)]=âtĎâ˘[Rââ˘(c,o1:tâ1,ot)]assignsubscriptdelimited-[]subscriptsuperscriptâsubscriptsubscriptsubscriptdelimited-[]subscriptsuperscriptâsubscript:11subscriptsubscriptsubscriptdelimited-[]superscriptâsubscript:11subscriptJ(Ď):=E_Ď [ _tR (s_t,a_t) ]=E% _Ď [ _tR (c,o_1:t-1,o_t) ]= _tE_% Ď [R (c,o_1:t-1,o_t) ]J ( Ď ) := blackboard_EĎ [ ât Râ ( sitalic_t , aitalic_t ) ] = blackboard_EĎ [ ât Râ ( c , o1 : t - 1 , oitalic_t ) ] = ât blackboard_EĎ [ Râ ( c , o1 : t - 1 , oitalic_t ) ] (13) In practice, most current reward models only provide a reward after the complete output has been generated and Equation 13 reduces to Equation 12. The definition of Jâ˘(Ď)J(Ď)J ( Ď ) uses the oracular reward that is not accessible in practice. An estimate of the performance can obtained from the estimated reward RĎsubscriptitalic-ĎR_ĎRitalic_Ď, by plugging it into Equation 12: J^â˘(Ď):=câźdCâ˘(â )â˘[OâźĎ(â |c)â˘[RĎâ˘(c,O)|C=c]] J(Ď):=E_c d_C(¡) [E_O Ď(% ¡|c) [R_Ď(c,O)|C=c ] ]over start_ARG J end_ARG ( Ď ) := blackboard_Ec âź d start_POSTSUBSCRIPT C ( â ) end_POSTSUBSCRIPT [ blackboard_EO âź Ď ( â | c ) [ Ritalic_Ď ( c , O ) | C = c ] ] (14) The pre-trained language model is denoted by Ďpre:âÎâ˘():subscriptpreâÎ _pre:Câ (O)Ďpre : C â Î ( O ) and the model updated using RLHF by Ďrlhf:âÎâ˘():subscriptrlhfâÎ _rlhf:Câ (O)Ďrlhf : C â Î ( O ). The goal of RLHF is to update the parameters of Ďrlhfsubscriptrlhf _rlhfĎrlhf such that Jâ˘(Ďrlhf)âĽJâ˘(Ďpre)subscriptrlhfsubscriptpreJ( _rlhf)⼠J( _pre)J ( Ďrlhf ) ⼠J ( Ďpre ), i.e., as evaluated using the oracular reward. In practice, it is only possible to verify that J^â˘(Ďrlhf)âĽJ^â˘(Ďpre)^subscriptrlhf^subscriptpre J( _rlhf)⼠J( _pre)over start_ARG J end_ARG ( Ďrlhf ) ⼠over start_ARG J end_ARG ( Ďpre ), which may be non-informative when the estimated reward model RĎsubscriptitalic-ĎR_ĎRitalic_Ď has inaccuracies for the context-output pairs being evaluated (Section 6.2). 5 Inferring the Reward from Human Feedback In the following sections, we study the properties of the reward estimated from human feedback. As reviewed in Section 7.5, various procedures exist for encoding human feedback into a reward model. Both the form of human feedback and the encoding mechanism continue to be studied further, with the procedures continually evolving and improving. Currently, the most common form of human feedback is pair-wise preference feedback that is encoded into a reward according to the Bradley-Terry model (Section 4.2). To perform an analysis agnostic to specifics of a particular reward learning method, 1. Let feedback denote a general form of sufficiently informative feedback. 2. Let Ί Ί denote the model of human behavior, or the encoding mechanism, that maps the feedback and the text to a reward value. The generality of this formulation allows the following analysis to cover all existing RLHF-style approaches (for example, RLAIF Bai et al. [2022b]) as well as future methods for fine-tuning LLMs, that employ a reward model. Let :=(c,o):câ,oâassignconditional-setformulae-sequenceD:=\(c,o):c ,o \D := ( c , o ) : c â C , o â O denote a hypothetical dataset of all possible contexts and outputs that a language model can encounter, i.e., a humongous dataset of size ||Ă||=|V|Tsuperscript|C|Ă|O|=|V|^T| C | Ă | O | = | V |T. This dataset cannot be realized in practice and is invoked to shed light on the practical limitations of the existing methodology. Denote the dataset of collected human feedback by HF:=(c,o,feedback):câHF,oâHFassignsubscriptHFconditional-setfeedbackformulae-sequencesubscriptHFsubscriptHFD_HF:=\(c,o,feedback):c _ HF% ,o _ HF\DHF := ( c , o , feedback ) : c â CHF , o â OHF where HFâ,HFâformulae-sequencesubscriptHFsubscriptHFC_ HF ,O_ HFâ% OCHF â C , OHF â O are the subsets of context-output pairs (human-)annotated with feedback.222The subsets are significantly smaller than CC and OO. Additionally, the feedback can be of any form: ratings, pair-wise feedback, or language feedback (Section 7.3). The reward encoding mechanism that maps context-output pairs along human feedback to rewards (for instance, the Bradley-Terry model) is denoted by Ί:(c,o,feedback)ââ:Ίâfeedbackâ :(c,o,feedback) Ί : ( c , o , feedback ) â blackboard_R.333feedback is overloaded to capture additional mechanism-specific meta-data. For instance, for pair-wise preference, feedback can store the preference relation and the (c,o)(c,o)( c , o ) pair compared against. To uncover RâsuperscriptâR Râ, it is assumed that Ί Ί accurately maps back human feedback to the oracular reward, i.e., for sufficiently informative feedback, we have Ίâ˘(c,o,feedback)=Rââ˘(c,o),âc,oâHF,HF.formulae-sequenceΊfeedbacksuperscriptâfor-allsubscriptHFsubscriptHF (c,o, feedback)=R (c,o),~â~c,o _% HF,O_ HF.Ί ( c , o , feedback ) = Râ ( c , o ) , â c , o â CHF , OHF . Under that assumption, Ί Ί can operate on HFsubscriptHFD_HFDHF to create a dataset of context-output-reward tuples, rew=(c,o,r):câHF,oâHFsubscriptrewconditional-setformulae-sequencesubscriptHFsubscriptHFD_rew=\(c,o,r):c _ HF,o _% HF\Drew = ( c , o , r ) : c â CHF , o â OHF where r=Rââ˘(c,o)superscriptâr=R (c,o)r = Râ ( c , o ). With rewsubscriptrewD_rewDrew, learning the reward model RĎsubscriptitalic-ĎR_ĎRitalic_Ď reduces to a regression problem employing a function approximator. The regression problem is however underdetermined Bishop [2006], and consequently multiple RĎsubscriptitalic-ĎR_ĎRitalic_Ď functions can perfectly fit the training data rewsubscriptrewD_rewDrew. However, almost all of these functions fail to accurately represent the oracular reward (Figure 3). Due to the cost of human annotation, practically human feedback can be collected on a very small subset of context and output pairs, i.e., HF,HFâ,formulae-sequencesubscriptHFsubscriptHFC_ HF,O_ HF ,OCHF , OHF â C , O. The size of the reward and feedback datasets relative to the hypothetical dataset of all possible inputs and outputs DD can be measured by: 1. Context coverage: Îş:=|HF|||assignsubscriptHFÎş:= |C_ HF||C|Îş := divide start_ARG | CHF | end_ARG start_ARG | C | end_ARG 2. Output coverage: Ď:=|HF||â˛|assignsubscriptHFsuperscriptâ˛Ď:= |O_HF||O |Ď := divide start_ARG | OHF | end_ARG start_ARG | OⲠ| end_ARG, where â˛=o:(c,o)â,âcâHFsuperscriptâ˛conditional-setformulae-sequencefor-allsubscriptHFO =\o:(c,o) ,~â c _ % HF\OⲠ= o : ( c , o ) â D , â c â CHF Well-understood results in supervised learning suggest that the ratios Ď and Îş along with the generalization capabilities of the function approximator Nakkiran et al. [2019]; Schaeffer et al. [2023]; Bailly et al. [2022] determine the generalization performance of the reward model for (c,o)â,(c,o) ,O( c , o ) â C , O. In practice, the values of Ď and Îş are extremely small and consquently the reward model often incorrectly generalizes on unseen (out-of-distribution) context-output pairs, assigning incorrect rewards to such inputs. In the following sections, we study practical limitations of estimating reward models. 5.1 Limitations of the Reward Model The reward model parameterized RĎ:Ăââ:subscriptitalic-ĎââR_Ď:CĂO _Ď : C Ă O â blackboard_R is trained on DrewsubscriptrewD_rewDrew using a sufficiently representative function approximator, to perfectly fit the training data, that is RĎâ˘(c,o)=Rââ˘(c,o),âo,câDrewformulae-sequencesubscriptitalic-Ďsuperscriptâfor-allsubscriptrewR_Ď(c,o)=R (c,o),~â~o,câ D_rewRitalic_Ď ( c , o ) = Râ ( c , o ) , â o , c â Drew. The limitations of the resultant reward model may be studied under the following categories: Figure 3: The reward model tends to misgeneralize for inputs not found in its training data, i.e., for (c,o)ârewsubscriptrew(c,o) _rew( c , o ) â Drew. This occurs in two ways: 1) when the context is not sampled by the prompting distribution for generating output and receiving feedback on (represented by Îş), and 2) when the support of the output generating distributionâthe language modelâfor a context does not span all possible outputs (represented by Ď). The latter is depicted in this figure. Misgeneralization: Human feedback is obtained on a very small subset of all possible context-output pairs. This partial coverage over contexts and outputs in DrewsubscriptrewD_rewDrew combined with the use of function approximators for learning the reward model results in the reward model RĎâ˘(c,o)subscriptitalic-ĎR_Ď(c,o)Ritalic_Ď ( c , o ) incorrectly generalizing to data points that are out-of-distribution relative to DrewsubscriptrewD_rewDrew. We have assumed a sufficiently representative function approximator that perfectly fits the training data, c,oâźrewâ˘[(Rââ˘(c,o)âRĎâ˘(c,o))2]=0subscriptsimilar-tosubscriptrewdelimited-[]superscriptsuperscriptâsubscriptitalic-Ď20E_c,o _rew [(R (c,o)-R_Ď(c,o% ))^2 ]=0blackboard_Ec , o âź D start_POSTSUBSCRIPT rew end_POSTSUBSCRIPT [ ( Râ ( c , o ) - Ritalic_Ď ( c , o ) )2 ] = 0. However, it cannot be ensured that c,oârewâ˘[(Rââ˘(c,o)âRĎâ˘(c,o))2]subscriptsubscriptrewdelimited-[]superscriptsuperscriptâsubscriptitalic-Ď2E_c,o _rew [(R (c,o)-R_Ď(c% ,o))^2 ]blackboard_Ec , o â D start_POSTSUBSCRIPT rew end_POSTSUBSCRIPT [ ( Râ ( c , o ) - Ritalic_Ď ( c , o ) )2 ] will be zero. It would require a function approximator to perfectly generalize outside the training data distribution, which is not generally attainable, especially when the ratios Ď,ÎşĎ,ÎşĎ , Îş are minuscule. The benefits of reinforcement learning algorithms over other methods for finetuning are contingent on access to an accurate reward function (Section 6.3), necessitating accurate out-of-distribution generalization of the reward model. The inaccurate extrapolation out-of-distribution results in an âimperfectâ reward model that provides feedback on context-output pairs in a manner that when optimized, arbitrarily misaligns with human feedback (and resultant preferences) for those context-output pairs. The output distribution of Ďrlhfsubscriptrlhf _rlhfĎrlhf trained on this inaccurate feedback and can only be as good (or bad) as the reward signal provided by the reward model. This inaccurate generalization in the reward model is one of the primary causes of phenomena like âreward hackingâ and âhallucinationsâ Kalai and Vempala [2023], observed in practice. Delayed feedback and Reward Sparsity: Reinforcement learning algorithms benefit from dense rewards as they serve to quickly guide the agent to rewarding states, providing informative feedback to intermediate actions along the trajectory. In RLHF, the feedback from human annotators is obtained for complete output generations. Consequently, the reward model is trained to provide reward feedback only at the end of the generated output for a given context. This delayed feedback increases the difficulty of optimization with RL algorithms, increasing their sample complexity. Sparse feedback is a constraint inherent to dealing with text and language Sokolov et al. [2016], as it is often unlikely for a human to provide feedback on incomplete sentences. Methods in RL developed to deal with sparse feedback, for instance by stitching together information from partial trajectories Andrychowicz et al. [2017], cannot be applied directly to textual output due to the semantic constraints of dealing with partial sentences. Denser rewards and corresponding feedback result in faster training, improved sample efficiency Wu et al. [2023d], and potentially better generalization. Insights from linguistics may be employed to obtain feedback on partial output generations and in turn denser rewards. Marginalization over preferences: The reward model averages over the preferences of all human annotators (and other sources of feedback) to output a deterministic scalar reward for a given context-output pair. The expectation is that averaging over the preferences of multiple sources would be representative of the preferences of an average human persona Deshpande et al. [2023b]. The results in rewards that are inconsistent with any single humanâs preferences. Such preferences are more appropriately denoted by an distribution of rewards for a context-output pair. A deterministic model, in addition to discounting the uncertainty and variability of human preferences, cannot model such a distribution, highlighting a case of model misspecification. The reward model forms the core component of RLHF and dictates the performance of a language model. The aforementioned shortcomings of the reward model highlight the need for safety measures that must be employed while using a language model fine-tuned using RLHF. 6 Reinforcement Learning with Imperfect Rewards Reinforcement learning algorithms can be broadly categorized into value-based methods and policy-gradient methods Sutton and Barto [2018]. Value-based methods aim to learn the value of states (or state-action pairs) as measured by the expected cumulative future reward from that state under a policy. These values serve as guides for picking the most rewarding actions from each state, and the policy can be inferred from the value functions. Policy-gradient methods directly train a parameterized policy using reward feedback to perform gradient ascent over the policy parameters and maximize the expected cumulative reward. A benefit of policy gradient methods for language tasks is that they naturally permit the optimization of stochastic policies, making them amendable for optimizing language models with stochastic decoding algorithms. Below we provide a brief overview of policy gradient algorithms and refer interested readers to Sutton et al. [1999]; Williams [1992]; Weng [2018] for a more rigorous treatment of the topic. 6.1 Policy Gradient Algorithms Policy gradient algorithms update the parameters of an agentâs policy using reward feedback. Being gradient-based algorithms, their update rule is of the form: θâľÎ¸+Îąâ˘âθJâ˘(Ďθ)âľsubscriptâsubscriptθ θ+Îą _θJ( _θ)θ ⾠θ + Îą âθ J ( Ďitalic_θ ) (15) where Jâ˘(Ďθ)subscriptJ( _θ)J ( Ďitalic_θ ) is the performance (Equation (12)) of the policy parameterized by θ. The gradient of the performance of a policy âθJâ˘(Ďθ)subscriptâsubscript _θJ( _θ)âθ J ( Ďitalic_θ ) can be estimated from samples in numerous ways, each affording varying degrees of variance and estimation error. In sparse rewards settings, the gradient estimation variance is a common issue that baselines Mei et al. [2022] help address. A class of methods called actor-critic methods update the policy by leveraging estimated value functions, called critics, to reduce gradient estimation variance. The algorithm used for training most current state-of-the-art large language models, Proximal Policy Optimization (PPO) Schulman et al. [2017] is an actor-critic algorithm with improvements over vanilla actor-critic to ensure stability during training. The improvements restrict parameter updates at each iteration to prevent the policy distribution from drastically changing. The training loss objective for PPO (PPO-Clip) takes the form: âppo-clip â˘(θ)=â˘[minâĄ(Ďθâ˘(atâŁst)Ďθold â˘(atâŁst)â˘A^â˘(st,at),clipâĄ(Ďθâ˘(atâŁst)Ďθold â˘(atâŁst),1âĎľ,1+Ďľ)â˘A^â˘(st,at))]subscriptâppo-clip delimited-[]subscriptconditionalsubscriptsubscriptsubscriptsubscriptold conditionalsubscriptsubscript^subscriptsubscriptclipsubscriptconditionalsubscriptsubscriptsubscriptsubscriptold conditionalsubscriptsubscript1italic-Ďľ1italic-Ďľ^subscriptsubscriptL_ppo-clip (θ)=E [ ( _% θ (a_t s_t ) _ _old (a_t s% _t ) A(s_t,a_t),clip ( _θ% (a_t s_t ) _ _old (a_t s_t% ),1-Îľ,1+Îľ ) A(s_t,a_t) ) ]Lppo-clip ( θ ) = blackboard_E [ min ( divide start_ARG Ďitalic_θ ( aitalic_t ⣠sitalic_t ) end_ARG start_ARG Ďitalic_θ start_POSTSUBSCRIPT old end_POSTSUBSCRIPT ( aitalic_t ⣠sitalic_t ) end_ARG over start_ARG A end_ARG ( sitalic_t , aitalic_t ) , clip ( divide start_ARG Ďitalic_θ ( aitalic_t ⣠sitalic_t ) end_ARG start_ARG Ďitalic_θ start_POSTSUBSCRIPT old end_POSTSUBSCRIPT ( aitalic_t ⣠sitalic_t ) end_ARG , 1 - Ďľ , 1 + Ďľ ) over start_ARG A end_ARG ( sitalic_t , aitalic_t ) ) ] (16) where A^â˘(st,at)^subscriptsubscript A(s_t,a_t)over start_ARG A end_ARG ( sitalic_t , aitalic_t ) is the estimate of the advantage function Aâ˘(st,at):=Qâ˘(st,at)âVâ˘(st)assignsubscriptsubscriptsubscriptsubscriptsubscriptA(s_t,a_t):=Q(s_t,a_t)-V(s_t)A ( sitalic_t , aitalic_t ) := Q ( sitalic_t , aitalic_t ) - V ( sitalic_t ) that captures the advantage obtained in terms of cumulative reward by taking an action atsubscripta_taitalic_t from state stsubscripts_tsitalic_t and then following the current policy, relative to following the policy starting from state stsubscripts_tsitalic_t. While this background suffices for the discussion in this paper, we urge the reader to refer to Weng [2018] for a more in-depth explanation of the topic. 6.2 Misalignment due to Imperfect Rewards In practice, a KL penalty DKL(Ďθ||Ďpre)D_KL ( _θ|| _pre )DKL ( Ďitalic_θ | | Ďpre ) with some weight β is added to the PPO training objective. This can be interpreted either as a regularizer or a prior which helps prevent overoptimization of an imperfect reward model. Using a reward model RĎsubscriptitalic-ĎR_ĎRitalic_Ď, the policy at convergence learnt by training with the updated PPO objective can expressed directly as a function of the reward Scheurer et al. [2023]; Rafailov et al. [2023] as, Ďrlhfâ˘(oâŁc)âĎpreâ˘(oâŁc)â˘expâĄ(1βâ˘RĎâ˘(c,o))proportional-tosubscriptrlhfconditionalsubscriptpreconditional1subscriptitalic-Ď _rlhf(o c) _pre(o c) ( 1% βR_Ď(c,o) )Ďrlhf ( o ⣠c ) â Ďpre ( o ⣠c ) exp ( divide start_ARG 1 end_ARG start_ARG β end_ARG Ritalic_Ď ( c , o ) ) (17) where β is the weight on the KL penalty. Let rewâsubscriptrewC_rew â C be the set of contexts in rewsubscriptrewD_rewDrew.444Note that rewsubscriptrewC_rewCrew is the same as HFsubscriptHFC_HFCHF. The subscript is used for clarity under the current context. After training, Ďrlhfsubscriptrlhf _rlhfĎrlhf must generate desirable (most rewarding) outputs when prompted with cârewsubscriptrewc _rewc â Crew. But for out-of-distribution contexts, where the reward estimation may be erroneous, the output distribution of Ďrlhfsubscriptrlhf _rlhfĎrlhf may be arbitrarily misaligned with human preferences and generate undesirable output. This misalignment can be quantified by comparing against the policy trained with the oracular reward. The set of contexts on which the performance of Ďrlhfsubscriptrlhf _rlhfĎrlhf is evaluated is denoted by evalsubscriptevalC_evalCeval with devalsubscriptsubscriptevald_C_evaldcaligraphic_C start_POSTSUBSCRIPT eval end_POSTSUBSCRIPT being the distribution over those contexts. Let Câ˛=eval/rewsuperscriptâ˛subscriptevalsubscriptrewC =C_eval/C_rewCⲠ= Ceval / Crew be the set of contexts in the evaluation set that are not present in rewsubscriptrewC_rewCrew. The performance of Ďrlhfsubscriptrlhf _rlhfĎrlhf is given by: Jâ˘(Ďrlhf)=câźdeval(â ),oâźĎrlhf(â âŁc)â˘[Rââ˘(c,o)]=âcârew,oâdevalâ˘(c)â˘Ďrlhfâ˘(oâŁc)â˘Rââ˘(c,o)+âcâCâ˛,oâdevalâ˘(c)â˘Ďrlhfâ˘(oâŁc)â˘Rââ˘(c,o)=(a)â˘âcârew,oâdevalâ˘(c)â˘Ďpreâ˘(oâŁc)â˘[Ďrlhfâ˘(o|c)Ďpreâ˘(o|c)â˘Rââ˘(c,o)]+âcâCâ˛,oâdevalâ˘(c)â˘Ďpreâ˘(oâŁc)â˘[Ďrlhfâ˘(o|c)Ďpreâ˘(o|c)â˘Rââ˘(c,o)]â(b)â˘âcârew,oâdevalâ˘(c)â˘Ďpreâ˘(oâŁc)â˘[expâĄ(1βâ˘Rââ˘(c,o))â˘Rââ˘(c,o)]+âcâCâ˛,oâdevalâ˘(c)â˘Ďpreâ˘(oâŁc)â˘[expâĄ(1βâ˘RĎâ˘(c,o))â˘Rââ˘(c,o)]âout-of-distribution splitJ( _rlhf)&=E_c d_C_% eval(¡),~o _rlhf(¡ c) [R (c,o)% ]\\ &= _ subarrayc [rgb]0,.5,.5 [named]% pgfstrokecolorrgb0,.5,.5c _rew,\\ o subarrayd_C_eval(c) _rlhf% (o c)R (c,o)+ _ subarrayc [rgb]1,0,0% [named]pgfstrokecolorrgb1,0,0câ C ,\\ ~o subarrayd_C_eval(c) _% rlhf(o c)R (c,o)\\ & (a)= _ subarrayc [rgb]0,.5,.5 [% named]pgfstrokecolorrgb0,.5,.5c _rew,\\ o subarrayd_C_eval(c) _pre% (o c) [ _rlhf(o|c) _pre(o|c)R (c% ,o) ]+ _ subarrayc [rgb]1,0,0 [named]% pgfstrokecolorrgb1,0,0câ C ,\\ ~o subarrayd_C_eval(c) _% pre(o c) [ _rlhf(o|c) _pre(o|c)R^% (c,o) ]\\ & (b) _ subarrayc [rgb]0,.5,.5% [named]pgfstrokecolorrgb0,.5,.5c _rew% ,\\ o subarrayd_C_eval(c) _pre% (o c) [ ( 1β [rgb]0,0,1 [named]% pgfstrokecolorrgb0,0,1R (c,o) )R (c,o) ]+% _ subarrayc [rgb]1,0,0 [named]% pgfstrokecolorrgb1,0,0câ C ,\\ ~o subarrayd_C_eval(c) _% pre(o c) [ ( 1β [rgb]1,.5,0 [% named]pgfstrokecolorrgb1,.5,0R_Ď(c,o) )R (c,o) ]% _out-of-distribution splitstart_ROW start_CELL J ( Ďrlhf ) end_CELL start_CELL = blackboard_Ec âź d start_POSTSUBSCRIPT C start_POSTSUBSCRIPT eval end_POSTSUBSCRIPT ( â ) , o âź Ďrlhf ( â ⣠c ) end_POSTSUBSCRIPT [ Râ ( c , o ) ] end_CELL end_ROW start_ROW start_CELL end_CELL start_CELL = âstart_ARG start_ROW start_CELL c â C start_POSTSUBSCRIPT rew , end_CELL end_ROW start_ROW start_CELL o â O end_CELL end_ROW end_ARG end_POSTSUBSCRIPT dcaligraphic_C start_POSTSUBSCRIPT eval end_POSTSUBSCRIPT ( c ) Ďrlhf ( o ⣠c ) Râ ( c , o ) + âstart_ARG start_ROW start_CELL c â CⲠ, end_CELL end_ROW start_ROW start_CELL o â O end_CELL end_ROW end_ARG dcaligraphic_C start_POSTSUBSCRIPT eval end_POSTSUBSCRIPT ( c ) Ďrlhf ( o ⣠c ) Râ ( c , o ) end_CELL end_ROW start_ROW start_CELL end_CELL start_CELL start_OVERACCENT ( a ) end_OVERACCENT start_ARG = end_ARG âstart_ARG start_ROW start_CELL c â C start_POSTSUBSCRIPT rew , end_CELL end_ROW start_ROW start_CELL o â O end_CELL end_ROW end_ARG end_POSTSUBSCRIPT dcaligraphic_C start_POSTSUBSCRIPT eval end_POSTSUBSCRIPT ( c ) Ďpre ( o ⣠c ) [ divide start_ARG Ďrlhf ( o | c ) end_ARG start_ARG Ďpre ( o | c ) end_ARG Râ ( c , o ) ] + âstart_ARG start_ROW start_CELL c â CⲠ, end_CELL end_ROW start_ROW start_CELL o â O end_CELL end_ROW end_ARG dcaligraphic_C start_POSTSUBSCRIPT eval end_POSTSUBSCRIPT ( c ) Ďpre ( o ⣠c ) [ divide start_ARG Ďrlhf ( o | c ) end_ARG start_ARG Ďpre ( o | c ) end_ARG Râ ( c , o ) ] end_CELL end_ROW start_ROW start_CELL end_CELL start_CELL start_OVERACCENT ( b ) end_OVERACCENT start_ARG â end_ARG âstart_ARG start_ROW start_CELL c â C start_POSTSUBSCRIPT rew , end_CELL end_ROW start_ROW start_CELL o â O end_CELL end_ROW end_ARG end_POSTSUBSCRIPT dcaligraphic_C start_POSTSUBSCRIPT eval end_POSTSUBSCRIPT ( c ) Ďpre ( o ⣠c ) [ exp ( divide start_ARG 1 end_ARG start_ARG β end_ARG Râ ( c , o ) ) Râ ( c , o ) ] + underâ start_ARG âstart_ARG start_ROW start_CELL c â CⲠ, end_CELL end_ROW start_ROW start_CELL o â O end_CELL end_ROW end_ARG dcaligraphic_C start_POSTSUBSCRIPT eval end_POSTSUBSCRIPT ( c ) Ďpre ( o ⣠c ) [ exp ( divide start_ARG 1 end_ARG start_ARG β end_ARG Ritalic_Ď ( c , o ) ) Râ ( c , o ) ] end_ARGout-of-distribution end_CELL end_ROW where (a) is permitted by the following: âo,c:Ďrlhfâ˘(o|c)>0,Ďpreâ˘(o|c)>0:for-allformulae-sequencesubscriptrlhfconditional0subscriptpreconditional0â o,c: _rlhf(o|c)>0, _pre(o|c)>0â o , c : Ďrlhf ( o | c ) > 0 , Ďpre ( o | c ) > 0 and (b) follows from Equation (17). Let ĎâsuperscriptĎ^*Ďâ be the policy trained using the oracular reward with RLHF. It can be expressed as: Ďrlhfââ˘(oâŁc)âĎpreâ˘(oâŁc)â˘expâĄ(1βâ˘Rââ˘(c,o))proportional-tosubscriptsuperscriptrlhfconditionalsubscriptpreconditional1superscriptâĎ^*_rlhf(o c) _pre(o c) ( % 1βR (c,o) )Ďârlhf ( o ⣠c ) â Ďpre ( o ⣠c ) exp ( divide start_ARG 1 end_ARG start_ARG β end_ARG Râ ( c , o ) ) The performance of ĎrlhfâsubscriptsuperscriptrlhfĎ^*_rlhfĎârlhf can be written as: Jâ˘(Ďrlhfâ)=câźdeval(â ),oâźĎrlhfâ(â âŁc)â˘[Rââ˘(c,o)]ââcârew,oâdevalâ˘(c)â˘Ďpreâ˘(oâŁc)â˘[expâĄ(1βâ˘Rââ˘(c,o))â˘Rââ˘(c,o)]+âcâCâ˛,oâdevalâ˘(c)â˘Ďpreâ˘(oâŁc)â˘[expâĄ(1βâ˘Rââ˘(c,o))â˘Rââ˘(c,o)]âout-of-distribution splitJ(Ď^*_rlhf)&=E_c d_C_ % eval(¡),~o Ď^*_rlhf(¡ c) [R (c,o% ) ]\\ & _ subarrayc [rgb]0,.5,.5 [named]% pgfstrokecolorrgb0,.5,.5c _rew,\\ o subarrayd_C_eval(c) _pre% (o c) [ ( 1β [rgb]0,0,1 [named% ]pgfstrokecolorrgb0,0,1R (c,o) )R (c,o) ]+% _ subarrayc [rgb]1,0,0 [named]% pgfstrokecolorrgb1,0,0câ C ,\\ ~o subarrayd_C_eval(c) _% pre(o c) [ ( 1β [rgb]0,0,1 [% named]pgfstrokecolorrgb0,0,1R (c,o) )R (c,o) ]% _out-of-distribution\\ splitstart_ROW start_CELL J ( Ďârlhf ) end_CELL start_CELL = blackboard_Ec âź d start_POSTSUBSCRIPT C start_POSTSUBSCRIPT eval end_POSTSUBSCRIPT ( â ) , o âź Ďârlhf ( â ⣠c ) end_POSTSUBSCRIPT [ Râ ( c , o ) ] end_CELL end_ROW start_ROW start_CELL end_CELL start_CELL â âstart_ARG start_ROW start_CELL c â C start_POSTSUBSCRIPT rew , end_CELL end_ROW start_ROW start_CELL o â O end_CELL end_ROW end_ARG end_POSTSUBSCRIPT dcaligraphic_C start_POSTSUBSCRIPT eval end_POSTSUBSCRIPT ( c ) Ďpre ( o ⣠c ) [ exp ( divide start_ARG 1 end_ARG start_ARG β end_ARG Râ ( c , o ) ) Râ ( c , o ) ] + underâ start_ARG âstart_ARG start_ROW start_CELL c â CⲠ, end_CELL end_ROW start_ROW start_CELL o â O end_CELL end_ROW end_ARG dcaligraphic_C start_POSTSUBSCRIPT eval end_POSTSUBSCRIPT ( c ) Ďpre ( o ⣠c ) [ exp ( divide start_ARG 1 end_ARG start_ARG β end_ARG Râ ( c , o ) ) Râ ( c , o ) ] end_ARGout-of-distribution end_CELL end_ROW The performance gap Îâ˘J:=|Jâ˘(Ďrlhfâ)âJâ˘(Ďrlhf)|assignÎsubscriptsuperscriptrlhfsubscriptrlhf J:=|J(Ď^*_rlhf)-J( _rlhf)|Î J := | J ( Ďârlhf ) - J ( Ďrlhf ) | caused by the imperfections in the reward model can be quantified as, Îâ˘JââcâCâ˛,oâdevalâ˘(c)â˘Ďpreâ˘(oâŁc)â˘[|(expâĄ(1βâ˘Rââ˘(c,o))âexpâĄ(1βâ˘RĎâ˘(c,o)))â˘Rââ˘(c,o)|]proportional-toÎsubscriptformulae-sequencesuperscriptâ˛subscriptsubscriptevalsubscriptpreconditionaldelimited-[]1superscriptâ1subscriptitalic-Ďsuperscriptâ J _ [rgb]1,0,0 [named]pgfstrokecolor% rgb1,0,0câ C ,~o d_C_eval(c% ) _pre(o c) [ ( ( 1β% [rgb]0,0,1 [named]pgfstrokecolorrgb0,0,1R (c,o% ) )- ( 1β [rgb]1,.5,0 [named]% pgfstrokecolorrgb1,.5,0R_Ď(c,o) ) )R (c,o) % ]Î J â âc â CⲠ, o â O dcaligraphic_C start_POSTSUBSCRIPT eval end_POSTSUBSCRIPT ( c ) Ďpre ( o ⣠c ) [ | ( exp ( divide start_ARG 1 end_ARG start_ARG β end_ARG Râ ( c , o ) ) - exp ( divide start_ARG 1 end_ARG start_ARG β end_ARG Ritalic_Ď ( c , o ) ) ) Râ ( c , o ) | ] For out-of-distribution contexts and outputs, the reward model is known to misgeneralize. The performance gap increases with increasing discrepancy from the oracular reward, and the discrepancy is further weighted by the likelihood of that (c,o)(c,o)( c , o ) pair and its oracular reward value. Some observations from the above analysis: ⢠Ďrlhfsubscriptrlhf _rlhfĎrlhf assigns high probability to highly rewarding outputs (Equation (17)), which is beneficial in-distribution contexts but can be harmful for out-of-distribution contexts when the reward model is erroneous. ⢠The deviation of the estimated reward from the oracular reward on unseen contexts exacerbates misalignment, which can be mitigated by increasing the weight on the KL penalty due to the 1/β11/ 1 / β dependence in the exponent. ⢠However, there is a trade-off. Increasing the value of β results in ĎrlhfâsubscriptsuperscriptrlhfĎ^*_rlhfĎârlhf and Ďrlhfsubscriptrlhf _rlhfĎrlhf being closer to Ďpresubscriptpre _preĎpre and have a lowered performanceâdue to increased weight in the KL penalty. 6.3 Why use Reinforcement Learning Algorithms? The efficacy of RLHF heavily relies on the quality of the reward model, and thus a large fraction of future research must focus on improving the reward model. Before allocating resources to that effort, it is essential to evaluate the merits and downsides of employing reinforcement learning as the fine-tuning paradigm. In comparison to supervised learning as an alternative approach, examining the gradient updates of a (vanilla) policy gradient algorithm alongside those of a supervised learning algorithm (such as supervised fine-tuning) offers some insights. Comparing Update Rules of Supervised Fine-Tuning and RLHF: In supervised fine-tuning (SFT), supervision is provided with positive samples, and the language model is updated to increase the likelihood of those samples under the model. Notably, there is no supervision provided for neutral or undesirable outputs, although it is a feasible option. Given the optimal policy ĎâsuperscriptĎ^*Ďâ (which may be a human expert), the objective of SFT is, maxθâĄcâźdC,owâźĎâ(â |c)â˘[lnâĄĎθâ˘(ow|c)] _θE_c d_C, [rgb]1,0,0 [named]% pgfstrokecolorrgb1,0,0o_w Ď^*(¡|c) [ _θ(% [rgb]1,0,0 [named]pgfstrokecolorrgb1,0,0o_w|c) % ]maxitalic_θ blackboard_Ec âź d start_POSTSUBSCRIPT C , oitalic_w âź Ďâ ( â | c ) end_POSTSUBSCRIPT [ ln Ďitalic_θ ( oitalic_w | c ) ] and thus the gradients used to update the parameters of the language model are of the form: âθ:=câźdC,owâźĎâ(â |c)â˘[âθlnâĄĎθâ˘(ow|c)]. _θ:=E_c d_C, [rgb]1,0,0 [named% ]pgfstrokecolorrgb1,0,0o_w Ď^*(¡|c) [ _θ% _θ( [rgb]1,0,0 [named]pgfstrokecolorrgb% 1,0,0o_w|c) ].âθ := blackboard_Ec âź d start_POSTSUBSCRIPT C , oitalic_w âź Ďâ ( â | c ) end_POSTSUBSCRIPT [ âθ ln Ďitalic_θ ( oitalic_w | c ) ] . This is analogous to behavior cloning in RL Pomerleau [1988] which is known to struggle when faced with out-of-distribution inputs. The primary benefit that reinforcement learning algorithms provide is that they allow the language model to explore the output space. Through its decoding algorithm, the language model exercises control over the distribution of outputs on which feedback is acquired. This facilitates learning from both positive as well as negative feedback, i.e., maxθâĄcâźdC,oâźĎθ(â |c)â˘[Rââ˘(c,o)] _θE_c d_C, [rgb]0,1,1 [named]% pgfstrokecolorrgb0,1,1 @color@cmyk@stroke1000% @color@cmyk@fill1000o _θ(¡|c) [R % (c, [rgb]0,1,1 [named]pgfstrokecolorrgb0,1,1% @color@cmyk@stroke1000 @color@cmyk@fill1000o)% ]maxitalic_θ blackboard_Ec âź d start_POSTSUBSCRIPT C , o âź Ďitalic_θ ( â | c ) end_POSTSUBSCRIPT [ Râ ( c , o ) ] and the (vanilla) policy gradient update is: âθ:=câźdC,oâźĎθ(â |c)â˘[Rââ˘(c,o)â˘âθlnâĄĎθâ˘(o|c)]. _θ:=E_c d_C, [rgb]0,1,1 [named% ]pgfstrokecolorrgb0,1,1 @color@cmyk@stroke1000% @color@cmyk@fill1000o _θ(¡|c) [R % (c, [rgb]0,1,1 [named]pgfstrokecolorrgb0,1,1% @color@cmyk@stroke1000 @color@cmyk@fill1000o)% _θ _θ( [rgb]0,1,1 [named]% pgfstrokecolorrgb0,1,1 @color@cmyk@stroke1000% @color@cmyk@fill1000o|c) ]~.âθ := blackboard_Ec âź d start_POSTSUBSCRIPT C , o âź Ďitalic_θ ( â | c ) end_POSTSUBSCRIPT [ Râ ( c , o ) âθ ln Ďitalic_θ ( o | c ) ] . As highlighted in color, in SFT, the gradient is estimated only from the positive samples, while in RL, it is computed for all samples (positive, negative, or neutral) weighted by their corresponding rewards. The gradient updates in RL are more informative, leading to better generalization for the language model and improved sample efficiency. Beyond exploration and richer gradients, the field of inverse reinforcement learning provides a natural formulation for training a language model with human feedback Arora and Doshi [2021]. In the following sections, we present a review of works that lead up to and are being rapidly added to this active area of research. This review provides context for the first half of this work and also serves as a comprehensive introduction for readers interested in getting started and understanding the topic of RLHF for language models. 7 Review of Reinforcement Learning from Human Feedback for Language Models 7.1 Language Model Pre-Training: Foundation for Large Language Models Language Models (LMs) have gained significant attention in recent years due to their impressive abilities to model language and retain textual knowledge. The Transformer architecture, characterized by its use of self-attention mechanisms, has become the standard for LMs Vaswani et al. [2017]. It is employed in a range of models, including BERT, T5, LLaMA, GPT-3, PALM, GLaM Devlin et al. [2019]; Raffel et al. [2019]; Touvron et al. [2023a]; Brown et al. [2020]; Chowdhery et al. [2022]; Du et al. [2021]. Pre-training has played an important role in the development of Large Language Models (LLMs), significantly contributing to their remarkable performance across a myriad of downstream tasks Brown et al. [2020]; Chowdhery et al. [2022]; Zhang et al. [2022]. This process involves training models with an unsupervised training objective on extensive datasets, often comprised of a diverse mix of web content, literary works, scientific documents, and code repositories Rae et al. [2021]; Xie et al. [2023]. The scale of these datasets is critical, with studies highlighting the superior performance of smaller models trained on larger datasets Kaplan et al. [2020]; Hoffmann et al. [2022]; Touvron et al. [2023a]. In addition to scale, the quality of training data, ensured through deduplication and filtering of low-quality content, is a key determinant of model performance Rae et al. [2021]; Du et al. [2021]; Hernandez et al. [2022]; Lee et al. [2021]. Masked Language Modeling (MLM) Devlin et al. [2019] and Causal Language Modeling Radford and Narasimhan [2018] are the most common objectives used for pretraining, with latter showing notable success in recent Large Language Model series such as GPT, PaLM, OPT Anil et al. [2023]; OpenAI [2023]; Zhang et al. [2022]. Studies demonstrate that pre-training by itself is responsible for the bulk of the observed capabilities even in downstream tasks Brown et al. [2020]; Raffel et al. [2019]. The simple pre-training objective of next, or masked, token prediction imbibes the LMs with a range of capabilities. They are few-task learners, without the need for fine-tuning. This applies to a variety of tasks from text generation, reasoning, question answering, summarization, and translation to name a few. However, though scaling pretrained language models (PLMs) exhibit remarkable performance across a variety of tasks, they suffer from several limitations, such as the inability to follow human instructions Ouyang et al. [2022]. This is because PLMs suffer from objective mismatch problems (See Section 2), as they are trained on generic internet data. As a result, PLMs need to learn to mimic the conflicting behavior of billions of humans. Further, the Maximum Likelihood Estimate on the next token prediction for such data doesnât explicitly penalize the model for hallucinating concepts, i.e., generating concepts not encapsulated within its internal representation, and even important & unimportant errors are given equal weightage. Moreover, pretrained models often show unintended behavior such as generating harmful, biased, untruthful, and low-quality content Perez et al. [2022]. Supervised-Finetuning To address the shortcomings faced by PLMs, a straightforward approach is to fine-tune them on a set of high-quality downstream datasets that are indicative of the intended task and behavior. For example, for instruction-following, human annotations can be collected on a set of input prompts, or input instances of existing public datasets can be re-formatted for instruction-following format. The model is then simply fine-tuned on these human demonstrations, often with the same pretraining objective. This increases the likelihood of generating desirable text and makes the model less biased and harmful. Nonetheless, in order to generate high-quality text, it is crucial to note that the task of distinguishing between high and low-quality text is inherently subjective and challenging, with end users being humans. Thus, quality assessment rests on human judgment and varies significantly based on the individual evaluatorâs perspective Yi et al. [2019]; Fan et al. [2022]; Ziegler et al. [2019]. Incorporating human feedback into such a process can be challenging, and collecting high-quality human demonstrations can be expensive and not scalable. forest forked edges, for tree= grow=east, reversed=true, anchor=base west, parent anchor=east, child anchor=west, base=left, font=, rectangle, draw=hidden-draw, rounded corners, align=left, minimum width=4em, edge+=darkgray, line width=1pt, s sep=3pt, inner xsep=2pt, inner ysep=3pt, ver/.style=rotate=90, child anchor=north, parent anchor=south, anchor=center, , where level=1text width=3em,font=,, where level=2text width=5.6em,font=,, where level=3text width=5.5em,font=,, where level=4text width=6.6em,font=,, [ Reinforcement Learning from Human Feedback in Language Models, ver [ Pretrained Language Models (§7.1) [ GPT-3 Brown et al. [2020], PALM Chowdhery et al. [2022], OPT Zhang et al. [2022], LLaMA Touvron et al. [2023a] , leaf, text width=25em ] ] [ Human Feedback (§7.3) [ Preference Based [ Kreutzer et al. [2018b], InstructGPT Ouyang et al. [2022], Bai et al. [2022a], Sparrow Glaese et al. [2022] , leaf, text width=25em ] ] [ Rating Based [ Kreutzer et al. [2018a], Liu et al. [2018], Fan et al. [2022] , leaf, text width=25em ] ] [ Language Based [ Li et al. [2016], Scheurer et al. [2023], Nguyen et al. [2021], leaf, text width=25em ] ] [ Miscellaneous Feedback [ Sparrow Glaese et al. [2022], Uesato et al. [2022], Wu et al. [2023c], , leaf, text width=25em ] ] ] [ Supervised Fine-Tuning (§7.4) [ Wei et al. [2021], Zhou et al. [2023], Chiang et al. [2023] , leaf, text width=25em ] ] [ Reward Models [ RL-Training (§7.6) [ Algorithm ] [ Actor-critic Bahdanau et al. [2016] Nguyen et al. [2017] [ Sparrow Glaese et al. [2022], GopherCite Menick et al. [2022], Perez et al. [2022] , leaf, text width=25em ] ] [ Others [ Ramamurthy et al. [2022], Scheurer et al. [2023], Munos et al. [2023] , leaf, text width=25em ] ] ] [ Task [ Translation [ Nguyen et al. [2017], Kreutzer et al. [2018a], Kiegeland and Kreutzer [2021] , leaf, text width=25em ] ] [ Summarization [ Stiennon et al. [2020], Nguyen et al. [2022], Ziegler et al. [2019], , leaf, text width=25em ] ] [ Dialogue [ InstructGPT Ouyang et al. [2022], Bai et al. [2022a], Nano Fan et al. [2022], , leaf, text width=25em ] ] [ Citing Answers [ Menick et al. [2022], Nakano et al. [2021] , leaf, text width=25em ] ] ] ] [ Non-RL Training [ Dong et al. [2023], Yuan et al. [2023], Scheurer et al. [2022] , leaf, text width=25em ] ] ] [ Reward Alternatives (§7.9) [ Zhao et al. [2023], Liu et al. [2023a], Bai et al. [2022b], , leaf, text width=25em ] ] ] Figure 4: Categorization of different components in the RLHF and example representative works from literature. Figure 5: Workflow of RLHF. A pretraining phase, and optionally supervised finetuning (SFT) on human demonstrations, is followed by all RLHF workflows for training language models. This is followed by an iterative loop starting with collecting human feedback on model-generated outputs, training a reward model, and updating the language model using a suitable RL algorithm. 7.2 Reinforcement Learning from Human Feedback (RLHF): Overview and Motivation The Importance of Human Feedback in Language Models The alignment of a model with the userâs intentions and preferences is critical, and incorporating human feedback in model training is a key step towards achieving this (Section 2). However, the process of obtaining high-quality human feedback, particularly in the form of human demonstrations, can be a resource-intensive process, both in terms of time and cost. A more efficient approach is to collect feedback on the outputs generated by the model and train the language model to incorporate this feedback. However, collecting such a large amount of feedback is also costly and impractical for real-time/online collection during training. The Role of RLHF in Language Models Reinforcement Learning from Human Feedback (RLHF) offers a solution to these challenges. In RLHF, human feedback is collected offline and used to train a reward model. This reward model then acts as a surrogate for human feedback during training, providing reward signals to the Language Model. Reinforcement learning algorithms form the natural candidates for training a model from scalar evaluative feedback, as provided by the reward model. This forms the essence of Reinforcement Learning from Human Feedback (RLHF) Christiano et al. [2017] as used to train Language Models. This approach is more sample-efficient and has shown more promising results compared to supervised fine-tuning alone Ouyang et al. [2022]. Applications of RLHF in Language Models In early works, RL has been used in training Language models across various domains such as dialogue generation Yi et al. [2019]; Li et al. [2016]; Jaques et al. [2019], machine translation Kreutzer et al. [2018a]; Nguyen et al. [2017]; Fernandes et al. [2022]; Sokolov et al. [2016], text generation Li et al. [2017]; Shi et al. [2018]; Zhou and Xu [2020]; Ziebart et al. [2008], semantic parsing Lawrence and Riezler [2018], summarization Stiennon et al. [2020]; Ziegler et al. [2019]; Wu et al. [2021]. More commonly, these methods were trained using non-differentiable automated evaluation metrics such as BLEU, ROUGE Ranzato et al. [2015]; Shen et al. [2015]; Keneshloo et al. [2018], or simulated feedback Nguyen et al. [2017]. However, while the combination of RL and human feedback has been extensively studied Knox and Stone [2008]; Christiano et al. [2017], it is only recently that RLHF with LLMs has achieved significant success in sequence-to-sequence tasks such as Summarization Stiennon et al. [2020]; Ziegler et al. [2019]; Wu et al. [2021], providing reliable answers with citations to queries Nakano et al. [2021]; Glaese et al. [2022], creating Helpful, Harmless and Honest dialogue agents aligned with broad human values Ouyang et al. [2022]; Bai et al. [2022a]. Formulating Language Modeling as a RL Problem Reinforcement Learning (RL) is a learning paradigm for a setting where an agent must make a sequence of decisions while interacting with an environment and obtaining evaluative feedback in the form of rewards. The agentâs objective is to maximize the total reward it receives over time. In the context of language models, the agent is the language model itself, and its actions consist of generating tokens from its vocabulary. The agentâs policy, which maps states to actions, is represented by the language modelâs parameters. The agent receives rewards from the environment, which in this case is a reward function that forms a surrogate from human feedback (Section 4). The agentâs objective is to optimize its actions (by updating its policy) to maximize the cumulative reward. A thorough mathematical formulation can be found Section 3, and has been summarized in Table 2 and Figure 2. While these details are sufficient for further discussion in the paper, we refer interested readers to Arulkumaran et al. [2017]; Sutton and Barto [2018] for more details about reinforcement learning. RL Component Language Model Equivalent Agent Language Model State Input prompt + currently generated text Action Predicted next token from vocabulary Policy Output Distribution of language model Reward Evaluative Feedback from the reward model Trajectory Input + completely generated text Table 2: Mapping of terms used in RL literature to training of language models through RLHF. See Table 1 for mathematical formulation. The Workflow of RLHF RLHF, as first popularized by Christiano et al. [2017] for mastering Atari Games consists of three crucial stages. An overview of standard RLHF workflow is highlighted in Figure 5. The first stage involves the collection of human feedback on a set of <input, output> pairs. These pairs can be sourced from existing datasets or generated by the pre-trained model for a given set of input prompts. The second stage involves learning a reward model from the collected human feedback. The reward model is trained to output a scalar reward for a given <input, output> pair, indicating the favorability of the pair. In essence, the reward model is trained to mimic human feedback, such that for a given input, desirable outputs are scored higher than undesirable outputs. The final stage involves the RLHF training of the language model, where the reward model provides reward signals on model outputs, usually in the form of scalar reward. The parameters of the language model are then updated based on these reward signals using an appropriate policy-gradient RL algorithm, updating the model to produce more rewarding outputs. These stages can be performed iteratively, with the intermediately trained model generating more prompts to collect additional human feedback. This feedback is then used to train the reward model, and the process is repeated multiple times Stiennon et al. [2020]; Bai et al. [2022a]; Menick et al. [2022]. In the following sections, we discuss each of these stages in detail. We start with Human Feedback Collection (Section 7.3), followed by training the Initial Policy (Section 7.4), Reward Model Training (Section 7.5), and finally RLHF Training (Section 7.6). Finally, we discuss the properties of RLHF-trained models and their limitations in Section 7.7. 7.3 Human Feedback In this section, we discuss the nature, objectives, and different types of human feedback, followed by the challenges and strategies associated with collecting high-quality feedback. 7.3.1 Nature and Objectives of Human Feedback Tasks such as summarization and providing helpful answers are inherently ambiguous and require human judgment to evaluate the quality of the generated text. Automated metrics like BLEU and ROUGE Lin and Och [2004] often do not correlate with human judgment Liu et al. [2016]; Schluter [2017]; Sellam et al. [2020]; Stiennon et al. [2020], making them unreliable for evaluation and training. Thus, acquiring high-quality human feedback to align the model with human behavior becomes crucial. Feedback is typically provided on the outputs generated by the model (or input-output pairs from the dataset), and subsequently, the model is trained to learn from this feedback. However, capturing diverse human preferences is a challenging task. One approach to encapsulate subjective human preferences is to approximate them using âmodels of human behaviorâ. This concept of human behavior models has roots in diverse fields such as econometrics McFadden [1981], psychology OâConnor [1989], and inverse reinforcement learning. A notable example is the Bradley-Terry model Bradley and Terry [1952], a probabilistic model that encodes the preference of one output over another in pairwise competitions. In the context of RLHF, reward models that form surrogates for human preferences serve as such models of human behavior. The type of feedback collected depends on the intended objective to be displayed by the fine-tuned language model. Askell et al. [2021] proposes three objectives for an aligned Language Model: Helpfulness, Honesty, and Harmlessness (H). These objectives can be broadly defined as follows: - Helpful: A Language Model is considered helpful if it can efficiently complete tasks or answer questions (while being harmless), ask relevant follow-up questions when necessary, and appropriately redirect ill-informed requests. Helpfulness includes context-dependent aspects such as informativeness, coherence, relevance, creativity, and specificity. - Honest: Honesty in a Language Model implies providing accurate information, expressing appropriate levels of uncertainty, and honestly conveying its capabilities, knowledge, and internal state. Language Models are particularly susceptible to hallucination Khandelwal et al. [2019]; Maynez et al. [2020], making it essential to penalize such behavior. Unlike helpfulness, honesty is more objectively evaluated. - Harmless: A harmless Language Model should avoid offensive or biased behavior, refuse to aid in dangerous acts, recognize disguised nefarious attempts, and act with modesty and care when providing advice with potentially sensitive or consequential impacts. These broad objectives, as mentioned above, encompass specific objectives, which can be considered subcategories. For example, in the case of summarization, the summary should be helpful to the reader and should not contain any false or harmful information. Similarly, the goal of reducing bias in a dialogue agentâs responses can be considered a subset of the Harmless objective. At the same time, coherence and creativity in the generated text are aspects of being helpful. These objectives are not mutually exclusive and are context and task-dependent. Even human labelers and researchers have shown disagreements in annotation Kreutzer et al. [2018b]. 7.3.2 Types of Human Feedback Human Feedback is usually collected on model-generated outputs. Good feedback should incorporate information on where the model output is lacking and how to improve it. A simple process is to let human labelers provide feedback on a set of model outputs generated from a dataset of prompts or inputs. Alternatively, existing datasets can be repurposed to incorporate implicit feedback, such as rating different user choices Kreutzer et al. [2018a]. Regardless of the process, human feedback can be collected in various forms, such as binary responses, preference ranking, language feedback, etc. While the choice of feedback type depends on the downstream task, it is essential to note that the feedback should be collected in a way that is easy for humans (labelers) to provide; there is high agreement among the labelers, and it is also informative. In this section, we classify the feedback into four different categories: rating feedback, ranking feedback, language feedback, and miscellaneous feedback. Rating Feedback The simplest form of rating feedback is binary feedback, where the labeler is asked to provide a binary response (yes/no) to a given input Li et al. [2016]; Scheurer et al. [2023]. Binary feedback is easy to collect and interpret. Some works have used binary responses to get feedback on multiple questions (such as if the generated text is coherent) Yi et al. [2019]. A richer form of feedback is to ask labelers to provide a rating on a scale. The scale can be continuous Graham et al. [2013], or be similar to Likert Scale Likert [1932] (where user rate using an integer from 1 to k) Kreutzer et al. [2018a]; Jaques et al. [2019]. A different variant of rating feedback is to provide categorical feedback such as âincorrectâ, âpartially-correctâ, and âcorrectâ Gao et al. [2023]. While rating feedback is easy to specify, often inter-annotator agreement is low because of the subjective nature of the task Kreutzer et al. [2018b]. Further, the order of examples presented to the annotator may bias the results Yannakakis and Hallam [2011]. Moreover, it is challenging to differentiate between data points with outputs of similar quality since feedback is provided individually to each output without comparison. Ranking or Preference Feedback Ranking feedback or Preference-based feedback has been extensively used in the recent development of AI assistants and found to be both convenient to collect and performative. Specifically, the labeler is offered with binary Stiennon et al. [2020] or multiple choice options Ziegler et al. [2019], and asked to select the most appropriate response based on a certain set of instructions (directions). Recently, Zhu et al. [2023a] has shown convergence guarantees for reward models trained using this feedback form. Moreover, given an input prompt, it is common to ask labelers to rank k (> 2) generated responses, which are then repurposed as pairwise comparisons for the reward model Ouyang et al. [2022]. However, collecting pairwise feedback might still be difficult for near similar responses and may result in much time spent by the labelers even on single input Scheurer et al. [2023]. Additionally, preference-based feedback provides a very sparse signal, conveying limited information about the reasoning behind the provided feedback. Moreover, it is provided only on the complete text generated by the model (trajectory) and not on specific parts of the text (particular state) Pang et al. [2022]; Lewis et al. [2017]. Moreover, preference-based feedback provides no further improvement in terms of inter-annotator agreement when compared to rating feedback Kreutzer et al. [2018b]. Language Feedback A more informative way to provide feedback is in free-form language. This provides a dense reward signal, specifying more precisely where the model goes wrong or needs improvement. For example, consider the case where the output generated by the model is âA humorous story about a specific profession involving person A and person B.â The previous feedback forms would provide only sparse signals, such as indicating that the output is inappropriate. However, this feedback alone will not help the model identify the cause of inappropriateness, and the single example alone can imply that the text is inappropriate because: âit is wrong to create humor in general,â âit is wrong to create humor about specific professionsâ or âit is wrong to involve individuals in humorous storiesâ and so on. On the other hand, free-form feedback can provide more precise feedback, such as âIt is inappropriate to create humor that targets specific professions.â This enables the model to understand the issue from a single example better and generalize to similar cases without learning from more examples. Language Feedback has been extensively used in various domains such as Dialogue models Li et al. [2016]; Hancock et al. [2019], Summarization Scheurer et al. [2023], Question-Answering Li et al. [2022], Code generation Chen [2023]. Recently, Scheurer et al. [2023] has shown that language feedback is more effective than preference-based feedback in the context of summarization systems. Also, as Hancock et al. [2019] discusses, getting preference-based feedback is plausible for paid labelers but not for real users using real deployed systems. Real users interact with the system through free-form language; hence, getting human feedback in the free-form language is more natural. Although task-dependent, Scheurer et al. [2023] further find that labelers take only 3x times to provide language feedback compared to preference-based feedback, despite providing much granular information. However, incorporating language feedback in the RLHF pipeline is not straightforward, and there has been limited work in this direction. Miscellaneous Feedback Apart from providing single feedback, methods have experimented with using a combination of feedback types or altogether different types. For example, Glaese et al. [2022] uses a combination of rule violation feedback (binary), preference-based feedback, and rating of evidence. Uesato et al. [2022]; Korbak et al. [2023] provide segment-level feedback instead of the whole text, and Wu et al. [2023c] provide feedback at the token level. Moreover, some studies employ indirect methods for collecting feedback. For example, Kreutzer et al. [2018a] uses human interactions on translated eBay titles to find more preferred translations. Further, it is also possible to provide computational feedback, for example, from automated metrics Bahdanau et al. [2016], forms of synthetic feedback Kim et al. [2023]; Black et al. [2023], web descriptions Hanjie et al. [2022]; Aggarwal et al. [2023], LLM generated feeedback Shinn et al. [2023]; Madaan et al. [2023]; Yang et al. [2022a], which might, in turn, be generated based on certain human requisites or instructions Bai et al. [2022b]; Sun et al. [2023]; Kundu et al. [2023]. However, these methods still use little to no human feedback and may have several unexplored limitations such as instability and lack of robustness Shumailov et al. [2023]; Alemohammad et al. [2023]; Gudibande et al. [2023] and are not the focus of this survey. We refer readers to Fernandes et al. [2023] for discussion on different type of feedback used in Natural Language Generation. 7.3.3 Collection of High-Quality Human Feedback Collecting high-quality human feedback is a challenging task that has been the focus of extensive research. The quality of feedback is pivotal; subpar or noisy feedback can significantly hamper the performance of the final trained model. For example, for summarization tasks, Ziegler et al. [2019] discovered that their model predominantly extracted verbatim lines from the document. This was later attributed to low-quality feedback by Stiennon et al. [2020]. Similarly, the size of the feedback is also crucial. For example, despite employing similar methodologies, Lightman et al. [2023] identified a need for a âgreater amount of feedbackâ for the methods in Uesato et al. [2022] to be effective, as the intended objective was not even observed in the latter work. The provision of clear and unambiguous instructions to the labelers is a fundamental requirement Ziegler et al. [2019]; Nakano et al. [2021]. Failure to do so can not only result in low-quality feedback but also introduce systematic bias in the collected feedback and, consequently, the model Parmar et al. [2022]. Typically, labelers are provided with a comprehensive set of instructions, including guidelines for handling edge cases Bai et al. [2022a]. Glaese et al. [2022] even provides a tutorial to the selected few labelers. Researchers typically screen labelers to ensure they possess the necessary skills to provide feedback. For instance, in the case of translation tasks, bilingual labelers with native proficiency in both languages are preferred Kreutzer et al. [2018b]. Additionally, a minimum educational qualification is generally preferred. For example, Stiennon et al. [2020] requires labelers to have at least a high-school degree, whereas Nakano et al. [2021], Glaese et al. [2022] and Bai et al. [2022a] require a minimum undergraduate and masterâs degree respectively. The end goal also influences the selection of labelers. For instance, creating a harmless and helpful chatbot necessitates a diverse group of labelers with varying backgrounds and demographics Ouyang et al. [2022]; Bai et al. [2022a] as otherwise this may result in implicit biases in the model Peng et al. [2022]. For instance, currently deployed language models have been shown to reflect views more aligned with western audiences Durmus et al. [2023] and may have systematic political biases Santurkar et al. [2023], partly owing to the lack of annotators from diverse demographic groups. However, despite screening, there may be low agreement among the annotators themselves, or even between researchers and annotators Kreutzer et al. [2018a]. The labelers are further screened based on two standard criteria 1.) inter-annotator agreement, i.e., the agreement between different annotators on the same example, and 2.) expert-annotator agreement, i.e., the agreement between annotators and experts Kreutzer et al. [2018b]. Specifically, the former metric ensures that the labelers are consistent in their feedback, and the latter metric is used to keep only those labelers that have a high agreement with experts. Menick et al. [2022] creates a group of super-raters who have a high agreement with experts, and the group is expanded upon iteratively. Even after filtering, some methods ensure a hands-on relationship with labellers Stiennon et al. [2020] and have also created Slack groups for discussing any bugs, issues, or edge cases Bai et al. [2022a]. 7.4 Supervised Fine-Tuning: Limitations and Role Upon the collection of high-quality feedback, the subsequent step is to assimilate this feedback to train the model. The most direct method to achieve this is to perform supervised fine-tuning of the language model based on the collected feedback. Specifically, human feedback is gathered in the form of expert outputs on input prompts, also referred to as human demonstrations. These human demonstrations can be perceived as positive example outputs to prompts that should be generated by the language model. The model is then fine-tuned on these demonstrations using the same pretraining objective, and this process in RL terminology is often termed behavior cloning Nakano et al. [2021]. Additionally, when dealing with preference data, the model can be directly fine-tuned on preferred feedback. However, this approach exhibits limitations by not accounting for negative feedbackâoutputs that the model should avoid generating. This is crucial for training robust models that can handle adversarial situations and identify and rectify errors. To tackle this limitation, alternative methods that incorporate both positive and negative feedback have been developed, as discussed in Section 7.9. In addition to human demonstrations, existing public instances from NLP datasets can be used as instruction tuning demonstrations Wei et al. [2021]. This usually involves creating new instruction-tuning datasets by adding task instructions to existing examples from the dataset Ajith et al. [2023]. In another field of work, prompts from the initial iterations of GPT-3 Brown et al. [2020] served to real customers through Web API were used to fine-tune the model on expert (human) demonstrations provided by contracted labelers Ouyang et al. [2022]. Limitations of Supervised Finetuning While finetuning on supervised data enhances the model beyond its pretrained version in following instructions and intended tasks, it suffers from numerous limitations. For instance, it does not penalize the model for hallucinating or permit it to learn from neutral or negative feedback. This can lead to harmful and unintended behavior, making it easier to prompt such models to elicit them Ganguli et al. [2022]; Perez et al. [2022]. Furthermore, behavior cloning is likely to perform poorly in out-of-distribution prompts Pomerleau [1988]. These limitations may stem from the fact that during behavior cloning, the model is not allowed to explore the vast space of possible actions, i.e., the model is not allowed to generate outputs that are not present in the demonstrations and, in turn, get feedback for them. We refer readers to Section 6.3 for theoretical discussion on the limitations of SFT. SFT as Initial Policy in RLHF models Despite its caveats, supervised fine-tuning plays a pivotal role in RLHF as it provides a robust initial policy, which allows RLHF methods to work well. From an RL perspective, learning algorithms such as the widely used Proximal Policy Optimization (PPO) in training sequence-to-sequence models, struggle to improve from poor initializations, especially when the action space is large, as in the case of text generation. This is because these methods use model-based exploration, which is ineffective when the transition probabilities over many actions are similar Nguyen et al. [2017] i.e., different text outputs have similar probabilities of generation. Furthermore, as we discuss in Section 7.6, usually a KL penalty is applied to ensure the output text generated by our RL-tuned model is close to the initial model. Thus, during RL training, it is preferable to start with an initial model that already generates decent-quality text. Empirical studies have demonstrated that starting with fine-tuning on high-quality human demonstrations results in significant improvements over starting with pretrained language models Stiennon et al. [2020]; Ouyang et al. [2022]. For instance, InstructGPT collects API customer and labeler written prompts and outputs to fine-tune their model before initiating with the RLHF training Ouyang et al. [2022]. Glaese et al. [2022] has also shown that starting with a prompted model (dialogue) instead of fine-tuning on label demonstrations is possible. However, they start with a large model (70B parameters) and do not perform a comparative study starting with fine-tuning on human demonstrations. Thus, it cannot be definitively concluded that starting with a prompted model is equivalent to fine-tuning on human demonstrations. Moreover, prompting has the limitation of using up a major portion of the context length of the model, which, apart from the computational burden, can also be crucial for some tasks because of limited context length. Askell et al. [2021] propose using context distillation by training the model to generate output similar to its prompted counterpart using KL divergence loss. They find similar performance to the prompted model, and the method has been used in their subsequent works Bai et al. [2022a]. In conclusion, while supervised fine-tuning can be utilized independently of the RLHF pipeline, it still suffers from several significant limitations. However, it still serves as an integral step in the RLHF pipeline, providing a robust initial policy crucial for subsequent RL training. 7.5 Reward Modeling Reward as a Proxy for Human Feedback After the collection of human feedback, the next challenge is training the language model effectively. Although supervised fine-tuning offers a straightforward method, its effectiveness is limited by the volume of human feedback. In contrast, RLHF introduces a reward model to emulate human feedback, thereby acting as a stand-in for the true reward function, i.e., the actual human feedback. This reward model, usually much smaller than the language model, facilitates fine-tuning the language model using feedback generated by it on new model outputs, avoiding the need for additional costly human annotation. In practice, using a reward model over supervised fine-tuning has been found more data-efficient Ramamurthy et al. [2022]. Training a Reward Model The reward model is a fine-tuned language model that assigns a scalar reward score to an input-output pair. The last embedding layer is replaced with a single projection layer that outputs this scalar reward. While the reward model can learn from various types of feedback, recent studies highlight the simplicity and effectiveness of preference-based feedback Ouyang et al. [2022]; Bai et al. [2022a]. This approach involves fine-tuning the initialized reward model to predict the preference between two trajectories (output text) given the same input prompt or context. The reward is typically modeled as a Bradley-Terry-Luce (BTL) model Bradley and Terry [1952], where the probability of preferring one trajectory over another is a function of the difference in their reward scores. Mathematically, this can be represented as: PrâĄ((oâťoâ˛,c)âŁĎ)=Ďâ˘[RĎâ˘(c,o)âRĎâ˘(c,oâ˛)]Prconditionalsucceedssuperscriptâ˛italic-Ďdelimited-[]subscriptitalic-Ďsubscriptitalic-ĎsuperscriptⲠ((o o ,c) Ď)=Ď[R_Ď(c,o)-R_Ď(% c,o )]Pr ( ( o âť oⲠ, c ) âŁ Ď ) = Ď [ Ritalic_Ď ( c , o ) - Ritalic_Ď ( c , oⲠ) ] (18) where Ďâ˘(x)=11+eâx11superscriptĎ(x)= 11+e^-xĎ ( x ) = divide start_ARG 1 end_ARG start_ARG 1 + e- x end_ARG is the sigmoid function, o and oâ represent the two trajectories, and their rewards are represented as RĎâ˘(c,o)subscriptitalic-ĎR_Ď(c,o)Ritalic_Ď ( c , o ) and RĎâ˘(c,oâ˛)subscriptitalic-Ďsuperscriptâ˛R_Ď(c,o )Ritalic_Ď ( c , oⲠ) respectively. This form of reward modeling has been found to provide smoother rewards and is less noisy Christiano et al. [2017]. A similar method can then be used for ranking between k trajectories (k > 2), where the reward is modeled as a Plackett-Luce (PL) model Plackett [1975]; Luce [1979]. Moreover, Zhu et al. [2023a] provides theoretical proof of convergence guarantees under the Maximum Likelihood estimate of both BTL and PL models. The size and initialization of the reward model are critical determinants of its performance. While smaller reward models are easier to train, scaling laws suggest that larger models yield better agreement with actual human preferences Askell et al. [2021]. However, Ouyang et al. [2022] found that training very large reward models can be unstable and result in overfitting. Instead, they report good performance even when using a reward model that is 30 times smaller than the policy model. Regarding initialization, multiple methods have been proposed. While Ziegler et al. [2019] fine-tunes a pretrained language model on preference data collected on model-generated outputs, Ouyang et al. [2022] trains a GPT-3 based reward model on publicly available datasets. However, only a slight advantage was found over using pretrained language models or supervised-fine-tuned models. Leveraging publicly available preference datasets (such as ranked answers from StackOverflow), as suggested by Askell et al. [2021], notably enhances reward model performance, especially for smaller models and datasets. Challenges in Reward Modeling The reward model is initially trained on a selected set of input prompts and corresponding initial model outputs. As the model training progresses, it is crucial for the reward model to generalize to new model outputs and potentially new input prompts. We refer readers to Section 5.1 for a deeper theoretical exploration of this aspect. Regarding the generalization capabilities of reward models, Ouyang et al. [2022] presents findings that demonstrate high generalization to held-out test labelers. This capability is of paramount importance since a majority of the inputs encountered during language model training would be out-of-distribution w.r.t. the reward model training phase. Generalization capability depends on various factors such as the datasetâs size, the amount of noise in the feedback dataset, and the characteristics of the pretrained reward model. Moreover, the robustness and calibration of the reward models with respect to actual human preferences are essential for their effectiveness. A well-calibrated reward model should accurately predict the probability of a human preferring one output over another. Bai et al. [2022a] discovered that when training solely on a helpfulness feedback dataset, their model exhibits strong calibration. However, when trained on a mixture of helpfulness and harmlessness datasets, the model is underconfident in its predictions. To assess robustness, a common practice involves evaluating the policy model trained using the reward model. To assess robustness, a common practice involves evaluating the policy model trained using the reward model. Interestingly, Bai et al. [2022a] discerned that smaller reward models and higher rewards correlate with decreased robustness. This phenomenon arises from the reward modelâs initial training on model outputs with naturally low rewards. To address this distribution shift, an approach involving iterated training of the reward model is proposed (see Section 7.6). In summation, the discussion underscores that the trained reward model on preferences is an imperfect proxy of human feedback, especially in out-of-domain cases. Moving Beyond Scalar Rewards Apart from providing a single scalar reward at the end of a trajectory (complete text output), several methods model a more fine-grained approach. Uesato et al. [2022]; Korbak et al. [2023] provides a segment-level reward during training, a method also known as process supervision. Interestingly, while Uesato et al. [2022] did not find any major downstream performance improvement with their method, Lightman et al. [2023] used similar methodology but instead trained larger models on a larger feedback dataset coupled with evaluation on a more difficult task found segment-level feedback to be significantly more useful. Scheurer et al. [2023] uses language feedback from another LLM that implicitly acts like a reward model for the training of the policy model. While ideally, as discussed in Section 4, the reward model provides a dual-purpose reward taking into account both the task information (eg, summarization task) and the task-specific evaluation ((a condescending summary is rewarded less than a neutral summary). However, diversifying the approach, some strategies involve the use of multiple reward models, each specializing in distinct characteristics or specific tasks. Wu et al. [2023c]; RamĂŠ et al. [2023] demonstrate the efficacy of training separate reward models for specific attributes such as coherency and factuality. Similarly, Glaese et al. [2022] introduces two reward modelsâone for preference and another for rule violation in dialogue generation. They found using two models over one to be more effective, likely because of a smaller feedback dataset. Further, since the preference-based reward model provides a delayed reward (reward is provided only at the end of the whole trajectory), the A2C algorithm, when used for sequence modeling Bahdanau et al. [2016] proposes potential-based reward shaping, where intermediate generations (states) are also rewarded. In conclusion, the reward modeling process is a critical component of RLHF which involves the training of a model to emulate human feedback, thereby acting as a surrogate for the true reward function. The size, initialization, and generalization capabilities of the reward model are all crucial factors that influence its performance. The reward model must be robust, well-calibrated, and additionally can provide more fine-grained feedback to the policy model training. 7.6 RLHF Finetuning of Language Models The trained reward model is utilized for finetuning the language model. Framing the task as reinforcement learning, with the language model as the policy, algorithms such as Proximal Policy Optimization (PPO) and Advantage Actor-Critic (A2C) Schulman et al. [2017]; Bahdanau et al. [2016] are used to update the parameters of the language model such that the generated outputs maximize the obtained reward. These are gradient-based methods, called policy-gradient algorithms, that directly update the parameters of the policy using the evaluative reward feedback The following sections primarily focus on the widely used Proximal Policy Optimization (PPO) Algorithm, while the same concepts are applicable to other candidate algorithms Ouyang et al. [2022]. 7.6.1 Training Procedure The pre-trained/SFT language model is prompted with contexts/prompts from a prompting dataset. The prompting dataset may or may not be identical to the one used for collecting human demonstrations in the SFT phase Ouyang et al. [2022]. The model outputs, along with the inputs, are passed to the reward model that generates a scalar output indicating the reward for this input-output pair. The reward is used as evaluative feedback to update the parameters of the language model using suitable RL algorithms that result in increasing the likelihood of the generation of more rewarding outputs. We next discuss a few commonly used RL algorithms for the process. 7.6.2 Training Algorithms The commonly used policy-gradient algorithms for aligning LLMs using RLHF are PPO and A2C Schulman et al. [2017]; Bahdanau et al. [2016]. Both fall under the category of actor-critic algorithms. These algorithms consist of two main components: the critic learns the expected cumulative reward for an input-output pair, called the value function, and the actor is the LLM policy that gets updated based on the cumulative reward estimates obtained from the critic. The reward values are obtained from the previously trained reward model, which is kept frozen during the RL training. As the LLM encounters more interactions and collects more reward feedback, it uses the data to update the value function and the LLM policy parameters. The training objective (Equation (15)) aims to update the parameters of the policy to increase the expected cumulative reward of the LLM policy. A2C and A3C Mnih et al. [2016] use an estimate of the advantage of taking an action instead of the action-value function for that action as a way of incurring lesser variance in policy gradient estimation. PPO additionally constrains the policy update at each iteration from straying too far by using a clipped objective Schulman et al. [2017]. This helps provide additional stability to the training. Training LLMs at a large scale requires an immense engineering effort, and practical implementations of these algorithms require domain-specific variations. While major progress has been made towards efficient training and inference of LLMs Xia et al. ; Lagunas et al. [2021]; Xia et al. [2022]; Su et al. [2023]; Murahari et al. [2023]; Yang et al. [2022b]; Hinton et al. [2015]; Yin et al. [2021]; Murahari et al. [2022], there is still a lot of scope for improvement in the sample efficiency and stability of training algorithms for RLHF. Recent work has addressed these challenges with different variants of these algorithms tackling different aspects ranging from practical implementation issues such as high memory usage Santacroce et al. [2023], changes specific for NLP Ramamurthy et al. [2022]; Wu et al. [2023b], training instability Zheng et al. [2023]. We refer readers to Weng [2018] for a comprehensive survey of policy gradient algorithms. 7.6.3 Improving Training Stability Imperfections in the reward model reduce the effectivity of the training algorithms, as the value functions learned, and in turn thus the gradient updates, become inaccurate. Thus, using the aforementioned algorithms with the learned reward model may lead the language model to exploit the imperfections and generate nonsensical text, often called âreward overoptimizationâ. This can be mitigated with appropriate regularization during training. As the pre-trained or SFT model (policy) is already a highly capable LLM, Jaques et al. [2019] propose using a copy of the initial model to regularize training. The aim is to ensure that even as the policy parameters are updated to maximize reward, the outputs of the updated policy do not stray too far from the initial policy. In particular, an additional regularization term of the Kullback-Leibler (KL) divergence between the policy being trained and the initial policy is added to the RL training objective in the form of a reward penalty, commonly called the KL penalty. Theoretically, the addition of this KL penalty has been shown to be similar to performing Bayesian inference Korbak et al. [2022] on the model. A hyperparameter β controls the weight of this KL penalty regularization during training. Further, it is common to compare different variants of RL algorithms at a fixed KL distance from the initial model, with the aim of maximizing the reward with the lowest possible KL divergence. 7.6.4 Iterated RLHF As training progresses, the reward model can become miscalibrated with human preferences at higher rewards Bai et al. [2022a]. This is because the reward model was trained on outputs from the initial model, which inherently have low-valued rewards. Consequently, several methods Stiennon et al. [2020]; Bai et al. [2022a] have employed an iterative training approach, where new outputs are generated by the updated policy, which are then annotated by humans for feedback. The reward model is then retrained based on this new human feedback, followed by training of the policy model. This process, referred to as Iterated-RLHF or Online-RLHF, is repeated for several iterations. Although effective, this procedure is naturally expensive and time-consuming. Limitations in Current RLHF practices Despite the impressive results achieved by RLHF in practice, it is an unstable training process Choshen et al. [2019]. Moreover, it is highly sensitive to hyperparameters, necessitating a significant amount of hyperparameter tuning Rafailov et al. [2023]; Yuan et al. [2023]. Furthermore, the generalization capabilities of RLHF and other issues, such as underperformance on metrics not captured by the reward model, warrant further investigation. A comprehensive examination of these aspects is discussed in Section 7.7. 7.7 Limitations of RLHF Models Fine-tuning models using Reinforcement Learning from Human Feedback (RLHF) showcase a remarkable ability to align with human preferences and generalize to new scenarios and is more sample-efficient than supervised fine-tuning. Nonetheless, these models exhibit characteristics and behaviors that warrant careful consideration, prompting the need for further exploration and refinement. Alignment Capabilities One intriguing property, referred to as the Alignment Tax, was identified by Ouyang et al. [2022]. The phenomenon reveals that RLHF-trained chat models sometimes perform poorly compared to initial policy in downstream tasks, suggesting a cost linked to aligning human preferences. To mitigate this, they propose incorporating the pre-training objective into RLHF-finetuning, which substantially reduces the Alignment Tax. Moreover, Bai et al. [2022a] indicates that larger models tend to exhibit lower alignment tax. Bai et al. [2022a] also observed that RLHF models better align with human preferences as the scales of both the reward model and policy model increase. It is noteworthy, however, that a similar scaling effect could be seen in instruction-finetuned SFT models. A comprehensive comparison of the scaling effects on RLHF versus SFT models is currently lacking in the literature and would make for an intriguing future study. Generalization Capabilities RLHF models have exhibited impressive generalization capabilities beyond their training data, including generalization on new prompts and human feedback. For instance, Ouyang et al. [2022] demonstrates RLHF-tuned models answering coding questions and following instructions in multiple languages despite being finetuned only in English and with limited code-related prompts. This suggests that the majority of a language modelâs capabilities are acquired during pre-training, and RLHF merely aligns these capabilities to elicit desired behavior. However, this generalization can be a double-edged sword, potentially leading to undesirable outcomes, especially when the feedback signal is sparse. For instance, the initial LLaMA2 Chat Model555https://huggingface.co/meta-llama/Llama-2-70b-chat-hf, when prompted "How to kill a process?" refused to answer, drawing ethical concerns, though the intended answer was about terminating a computer process. This behavior likely stems from the modelâs extended generalization from examples that trained it to reject violent queries. The example further highlights the problems of imperfect rewards leading to misgeneralization, as discussed in Section 6.2. Further, a distributional shift between prompts used for reward model finetuning and RLHF training can result in the policy model misaligning with human preferences Bai et al. [2022a]. Further, during RL training, outputs are sampled from the language model, which is evaluated using the reward model. However, deviations in parameters used for sampling outputs from the model during inference from those in training can yield poor results Ramamurthy et al. [2022]. Diversity & Biases of RLHF model outputs Another characteristic of RLHF models is their low entropy in output distribution Bai et al. [2022a], which challenges generating diverse responses Kirk et al. [2023]. This holds true for both seen and unseen datasets. To address this, entropy regularization techniques are introduced Jaques et al. [2019]; Li et al. [2016] to amplify diversity in the action space, albeit not always resolving the issue Raichuk et al. [2021]. While not conclusive, Bai et al. [2022a] found that while RLHF models exhibit better sentiment towards all classes, they display similar biases to underlying LLMs when sampling with temperature < 1 (i.e., with low diversity samples). This could be attributed to their lower entropy. Furthermore, while pre-trained models often generate probabilities that are well-calibrated, RLHF models may lose this calibration. For instance, OpenAI [2023] found that for pre-trained GPT-4, the probability of generating an answer is often directly proportional to the probability of it being correct. However, in the case of RLHF models, the distribution is skewed towards more likely answers. Objective Misalignment While RLHF aims to align language models with human preferences and intentions, reward model misalignment is frequently possible. For instance, Singhal et al. [2023] finds that reward models provide higher rewards to longer outputs. Further, Robustness and Safety It is imperative to note that the reward model is merely an imperfect proxy for real human preferences/feedback. Due to the lack of calibration and robustness of reward models Bai et al. [2022a], over-optimizing against the reward model can render it an ineffective measure (Goodhartâs Law). This phenomenon, known as Reward Overoptimization, has been studied in the context of language models by Gao et al. [2022]; Coste et al. [2023]. Further, training RLHF models in practice is very difficult for practitioners owing to unstable training Choshen et al. [2019], hyperparameter sensitivity Yuan et al. [2023]; Rafailov et al. [2023], loading multiple models leading to high memory usage Santacroce et al. [2023]. As a result, there have been significant efforts to simplify the training process by learning directly from the available feedback using simpler supervised finetuning objectives, as we discuss in Section 7.9. In conclusion, while RLHF substantially enhances the performance of LLMs and aligns them with human preferences, it is not without its limitations. These include, but are not limited to, issues such as text hallucination McKenna et al. [2023], bias and toxicity Deshpande et al. [2023a]; Ferrara [2023]; Gupta et al. [2023], and the generation of harmful text when probed Perez et al. [2022]; Wei et al. [2023]. Despite significant improvements, these models are not fully aligned with human preferences, underscoring the need for continued research and development in this field. 7.8 Enriching Reward Signals in Reinforcement Learning Challenges with Sparse Rewards in Traditional RL Reinforcement learning (RL) has conventionally employed delayed and sparse rewards, where agents receive scalar feedback at the end of a trajectory or episode Sutton and Barto [2005]. While this approach is straightforward to implement and aligns with the task objective, it is not without its drawbacks. Sparse rewards can lead to sample-inefficient learning due to extensive exploration requirements Bellemare et al. [2016]. Additionally, they may result in reward hacking, where agents exploit unintended strategies to maximize rewards without solving the intended task Ibarz et al. [2018]. Underspecified rewards, which do not fully capture the desired behavior, can also yield suboptimal or degenerate solutions Hadfield-Menell et al. [2017]. Enriching Reward Signals To mitigate the limitations of sparse rewards, researchers have explored various methods for providing richer feedback in environments with inherently sparse rewards. These approaches include reward shaping, where the original reward signal is augmented with additional feedback Ng et al. [1999]; Grzes [2017]; intrinsic motivation, which encourages exploration and learning through internal rewards based on novelty, curiosity, or learning progress Oudeyer et al. [2007]; Bellemare et al. [2016]; Pathak et al. [2017]; and multi-objective optimization with multiple reward signals Roijers et al. [2013]; Roijers [2016]. Hierarchical RL, which decomposes complex tasks into simpler subtasks with their own reward structures, has also been investigated Dietterich [1999]; Barto and Mahadevan [2003]. Moreover, richer forms of feedback, such as learning from corrections Jain et al. [2015]; Bajcsy et al. [2017], demonstrations Rengarajan et al. [2022], and language feedback Matuszek et al. [2012]; Fried et al. [2017], have proven beneficial. Implications for RLHF in LLMs Current RLHF pipelines for LLMs primarily rely on sparse rewards provided at the end of an episode, with reward models trained using sparse preference-based feedback. Similar challenges observed in traditional RL have also been identified in RLHF-tuned LLMs. Some progress has been made in learning from feedback for multi-objective optimization RamĂŠ et al. [2023], language feedback Scheurer et al. [2022], corrective feedback Madaan et al. [2023]; Shinn et al. [2023], and denser rewards Wu et al. [2023c]. Future research should explore the integration of these techniques to address the unique challenges in training LLMs with RLHF, potentially improving generalization and robustness. 7.9 Moving Beyond RL Training While RLHF has been very successful, it still results in unstable training Choshen et al. [2019], is hyperparameter sensitive Yuan et al. [2023]; Rafailov et al. [2023], has high memory usage Santacroce et al. [2023] making it difficult for practitioners to actually use it. As a result, there have been significant efforts to simplify the training process by learning directly from the available feedback using simpler supervised finetuning objectives. 7.9.1 Alternatives to RL using Reward Model Once, a reward model is trained, it is not necessary to perform the RLHF-based training. Instead, an alternate approach during inference is to sample multiple outputs from the LLM and rank them using the reward model Nakano et al. [2021]; Cobbe et al. [2021]. This is also called best-on-n sampling or rejection sampling. If sampling multiple outputs, it is important to ensure diversity of outputs by adjusting the sampling parameters (such as higher temperature). This approach is often considered as either a baseline or augmented with RLHF-trained models for better inference-time results. Further, various works Dong et al. [2023]; Yuan et al. [2023]; Song et al. [2023] use the trained reward model to rank multiple responses and use the signal from the ranked responses to train the policy model, without using an elaborate RL algorithm. In another line of work, RAD Deng and Raffel [2023] uses weighted-decoding of tokens at inference, based on a separately trained reward model. 7.9.2 Alternatives to RL without Explicit Reward Models In this section, we discuss alternative methods to align language models with human feedback that do not rely on reward models. While RLHF-PPO has shown promising results, it suffers from sensitivity to hyperparameters, the need for training additional models, and potential misalignment of the reward model Wu et al. [2023c]; Rafailov et al. [2023]; Pang et al. [2022]; Zhu et al. [2023b]; Singhal et al. [2023]. To address these issues, recent research has explored various techniques that directly incorporate human feedback into the training process, without relying on additional reward models. A straightforward approach is supervised fine-tuning on positive demonstrations from human feedback, such as instruction-finetuned models Ouyang et al. [2022]; Chiang et al. [2023]; Zhou et al. [2023]. However, this method does not utilize negative feedback, which is crucial for training robust models that can handle adversarial situations and identify and correct errors. Recent works, such as Liu et al. [2023a]; Zhang et al. [2023], provide both positive and negative demonstrations/feedback and maximize the likelihood of generating positive/preferred output. These methods have shown better performance than RLHF methods on summarization and dialogue tasks. Zhao et al. [2023] demonstrate that Sequence Likelihood calibration (SLiC) Zhao et al. [2022] can be used to train models on off-policy offline data collected for different models, resulting in better performance than RLHF-based methods on summarization tasks. SLiC uses a ranking calibration loss that contrasts positive and negative sequences while motivating the model to predict the positive class. Further, RSO Liu et al. [2023c] improves policy learning in SLiC by using statistical rejection sampling from the policy. Rafailov et al. [2023]; Azar et al. [2023] further reformulate the objective encoded in the RLHF PPO algorithm, and train the model directly on the new objective, without the need for a separate reward model. This follows the intuition, that the policy model can be implicitly used as a reward model for training itself based on the collected feedback. However, the results are preliminary, and extending to out-of-distribution prompts may not be possible without the introduction of an explicit reward model. Another line of research focuses on refining model-generated responses using human-encoded principles or feedback. Bai et al. [2022b]; Kundu et al. [2023] propose a framework where a list of human-encoded principles (Constitution) guide the model to critique its generations and self-refine the responses. The model is then fine-tuned on the refined responses. Self-Align Sun et al. [2023] follows a similar procedure but further removes the need to start with an RLHF-finetuned model. They fine-tune the pretrained LLaMA Touvron et al. [2023a] base model using less than 300 lines of human feedback (in the form of constitutional principles) and achieve performance comparable to state-of-the-art models in terms of helpfulness and harmlessness. Another direction of work learns to generate or select good feedback for model outputs and apply it to refine language model outputs. Scheurer et al. [2022] takes a similar refinement approach but utilizes available summarization feedback. The initial model is conditioned on input, feedback, and output, generating multiple refinements. The model is then fine-tuned on refinements with the highest similarity to human feedback. Liu et al. [2023b] aligns human moral values by modeling DP (dynamic-programming) based edits from unaligned source text to target aligned text. The model is then fine-tuned on the refinements generated by the edits, using RL for the second part of the process. Xu et al. [2022] fine-tune a dialogue model using multi-modal feedback with the DIRECTOR method Arora et al. [2022], which models both negative and positive sequence labeling directly in the language model head. In summary, these alternative methods generate new data based on feedback or guidelines and then use it to fine-tune the model. These approaches reduce the reliance on reward models and have shown promising results in some tasks, making them a viable alternative to RLHF-PPO. While these models are easier to train and help in alleviating many drawbacks of RLHF, the evaluation performed has been performed only on specific domains, and constrained settings. Moreover, other in-depth analysis such as sample efficiency and properties exhibited by these models, especially on out-of-distribution data needs to be explored further. 8 Discussion and Conclusion In this work, we explore the fundamental aspects of reinforcement learning from human feedback (RLHF), aiming to clarify its mechanisms and limitations. We highlight the underlying assumptions necessary for RLHF and examine the impact of different implementation choices, shedding light on the workings of this approach. Our analysis naturally focuses on the reward models, which constitute the core component of RLHF. We introduce the concept of oracular rewards, which represent the ideal reward signals that reward models should approximate. The challenges encountered in learning these reward functions highlight both the practical and fundamental limitations of RLHF, as thoroughly analyzed by Casper et al. [2023]. Our comprehensive review of the existing literature traces the development of RLHF from its inception to the recent advancements. We cover various aspects: the types of feedback, the details and variations of training algorithms, and alternative methods for achieving alignment without using reinforcement learning. In related work, Kaufmann et al. [2023] extensively surveys RLHF, highlighting its evolution from preference-based learning. Despite the numerous variations of RLHF, the core principle of learning from evaluative feedback remains unchanged. This form of learning is naturally suited to reinforcement learning, while the specifics of agent formulation, the nature of reward feedback, and environment definition continue to evolve. We anticipate the reduction of reliance on human (or AI) feedback by using existing knowledge sources to construct rewards, which is one of the most promising directions for future efforts to enhance the impact of RLHF. Additionally, improving reward encoding mechanisms to better reflect the diversity of human preferences is an important area for further research. As RLHF continues to advance and reach its full potential, supported by research in these areas, the use of LLMs is also expanding. Until we fully understand the implications of RLHF, it is crucial to develop robust methods for quantifying uncertainty in the outputs generated by an LLM. Such techniques would enable us to identify and address low confidence outputs, which is especially important in safety-critical applications. Ultimately, understanding its implications becomes paramount as advancements in RLHF increasingly influence industries and economies. Thus, research in this field is critically important in shaping the future of large-scale language modeling and its societal impact. Acknowledgements We thank Khanh Nguyen for extensive and insightful feedback on earlier versions of the draft. We also thank Wenlong Zhao, Tuhina Tripathi, and Abhiman Neelakanteswara for their help with improving the clarity of the manuscript. References Aggarwal et al. [2023] Pranjal Aggarwal, A. Deshpande, and Karthik Narasimhan. Semsup-xc: Semantic supervision for zero and few-shot extreme classification. In International Conference on Machine Learning, 2023. URL https://api.semanticscholar.org/CorpusID:256274863. Ajith et al. [2023] Anirudh Ajith, Chris Pan, Mengzhou Xia, A. Deshpande, and Karthik Narasimhan. Instructeval: Systematic evaluation of instruction selection methods. ArXiv, abs/2307.00259, 2023. URL https://api.semanticscholar.org/CorpusID:259316853. Alemohammad et al. [2023] Sina Alemohammad, Josue Casco-Rodriguez, Lorenzo Luzi, Ahmed Imtiaz Humayun, Hossein Babaei, Daniel LeJeune, Ali Siahkoohi, and Richard G. Baraniuk. Self-consuming generative models go mad, 2023. Andrychowicz et al. [2017] Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAI Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay. Advances in neural information processing systems, 30, 2017. Anil et al. [2023] Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Tachard Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Z. Chen, Eric Chu, J. Clark, Laurent El Shafey, Yanping Huang, Kathleen S. Meier-Hellstern, Gaurav Mishra, Erica Moreira, Mark Omernick, Kevin Robinson, Sebastian Ruder, Yi Tay, Kefan Xiao, Yuanzhong Xu, Yujing Zhang, Gustavo Hernâandez âAbrego, Junwhan Ahn, Jacob Austin, Paul Barham, Jan A. Botha, James Bradbury, Siddhartha Brahma, Kevin Michael Brooks, Michele Catasta, Yongzhou Cheng, Colin Cherry, Christopher A. Choquette-Choo, Aakanksha Chowdhery, C CrĂŠpy, Shachi Dave, Mostafa Dehghani, Sunipa Dev, Jacob Devlin, M. C. Dâiaz, Nan Du, Ethan Dyer, Vladimir Feinberg, Fan Feng, Vlad Fienber, Markus Freitag, Xavier GarcĂa, Sebastian Gehrmann, Lucas GonzĂĄlez, Guy Gur-Ari, Steven Hand, Hadi Hashemi, Le Hou, Joshua Howland, An Ren Hu, Jeffrey Hui, Jeremy Hurwitz, Michael Isard, Abe Ittycheriah, Matthew Jagielski, Wen Hao Jia, Kathleen Kenealy, Maxim Krikun, Sneha Kudugunta, Chang Lan, Katherine Lee, Benjamin Lee, Eric Li, Mu-Li Li, Wei Li, Yaguang Li, Jian Li, Hyeontaek Lim, Han Lin, Zhong-Zhong Liu, Frederick Liu, Marcello Maggioni, Aroma Mahendru, Joshua Maynez, Vedant Misra, Maysam Moussalem, Zachary Nado, John Nham, Eric Ni, Andrew Nystrom, Alicia Parrish, Marie Pellat, Martin Polacek, Alex Polozov, Reiner Pope, Siyuan Qiao, Emily Reif, Bryan Richter, Parker Riley, Alexandra Ros, Aurko Roy, Brennan Saeta, Rajkumar Samuel, Renee Marie Shelby, Ambrose Slone, Daniel Smilkov, David R. So, Daniela Sohn, Simon Tokumine, Dasha Valter, Vijay Vasudevan, Kiran Vodrahalli, Xuezhi Wang, Pidong Wang, Zirui Wang, Tao Wang, John Wieting, Yuhuai Wu, Ke Xu, Yu Xu, Lin Wu Xue, Pengcheng Yin, Jiahui Yu, Qiaoling Zhang, Steven Zheng, Ce Zheng, Wei Zhou, Denny Zhou, Slav Petrov, and Yonghui Wu. Palm 2 technical report. 2023. Arora et al. [2022] Kushal Arora, Kurt Shuster, Sainbayar Sukhbaatar, and Jason Weston. Director: Generator-classifiers for supervised language modeling. In AACL, 2022. Arora and Doshi [2021] Saurabh Arora and Prashant Doshi. A survey of inverse reinforcement learning: Challenges, methods and progress. Artificial Intelligence, 297:103500, 2021. Arulkumaran et al. [2017] Kai Arulkumaran, Marc Peter Deisenroth, Miles Brundage, and Anil Anthony Bharath. Deep reinforcement learning: A brief survey. IEEE Signal Processing Magazine, 34:26â38, 2017. URL https://api.semanticscholar.org/CorpusID:4884302. Askell et al. [2021] Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, T. J. Henighan, Andy Jones, Nicholas Joseph, Benjamin Mann, Nova DasSarma, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, John Kernion, Kamal Ndousse, Catherine Olsson, Dario Amodei, Tom B. Brown, Jack Clark, Sam McCandlish, Christopher Olah, and Jared Kaplan. A general language assistant as a laboratory for alignment. ArXiv, abs/2112.00861, 2021. Azar et al. [2023] Mohammad Gheshlaghi Azar, Mark Rowland, Bilal Piot, Daniel Guo, Daniele Calandriello, Michal Valko, and RĂŠmi Munos. A general theoretical paradigm to understand learning from human preferences, 2023. Bahdanau et al. [2016] Dzmitry Bahdanau, Philemon Brakel, Kelvin Xu, Anirudh Goyal, Ryan Lowe, Joelle Pineau, Aaron C. Courville, and Yoshua Bengio. An actor-critic algorithm for sequence prediction. ArXiv, abs/1607.07086, 2016. Bai et al. [2022a] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, T. J. Henighan, Nicholas Joseph, Saurav Kadavath, John Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom B. Brown, Jack Clark, Sam McCandlish, Christopher Olah, Benjamin Mann, and Jared Kaplan. Training a helpful and harmless assistant with reinforcement learning from human feedback. ArXiv, abs/2204.05862, 2022a. Bai et al. [2022b] Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, John Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, E Perez, Jamie Kerr, Jared Mueller, Jeff Ladish, J Landau, Kamal Ndousse, KamilÄ LukosiĹŤtÄ, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemâi Mercado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Lawton, Tom Conerly, T. J. Henighan, Tristan Hume, Sam Bowman, Zac Hatfield-Dodds, Benjamin Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom B. Brown, and Jared Kaplan. Constitutional ai: Harmlessness from ai feedback. ArXiv, abs/2212.08073, 2022b. Bailly et al. [2022] Alexandre Bailly, Corentin Blanc, Ălie Francis, Thierry Guillotin, Fadi Jamal, BĂŠchara Wakim, and Pascal Roy. Effects of dataset size and interactions on the prediction performance of logistic regression and deep learning models. Computer Methods and Programs in Biomedicine, 213:106504, 2022. Bajcsy et al. [2017] Andrea V. Bajcsy, Dylan P. Losey, Marcia Kilchenman OâMalley, and Anca D. Dragan. Learning robot objectives from physical human interaction. In Conference on Robot Learning, 2017. URL https://api.semanticscholar.org/CorpusID:28406224. Barnett et al. [2023] Peter Barnett, Rachel Freedman, Justin Svegliato, and Stuart Russell. Active reward learning from multiple teachers. arXiv preprint arXiv:2303.00894, 2023. Barto and Mahadevan [2003] Andrew G. Barto and Sridhar Mahadevan. Recent advances in hierarchical reinforcement learning. Discrete Event Dynamic Systems, 13:41â77, 2003. URL https://api.semanticscholar.org/CorpusID:386824. Bellemare et al. [2016] Marc G. Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and RĂŠmi Munos. Unifying count-based exploration and intrinsic motivation. In NIPS, 2016. URL https://api.semanticscholar.org/CorpusID:8310565. Bertsekas and Tsitsiklis [1996] Dimitri Bertsekas and John N Tsitsiklis. Neuro-dynamic programming. Athena Scientific, 1996. Bishop [2006] Christopher Bishop. Pattern recognition and machine learning. Springer google schola, 2:531â537, 2006. Black et al. [2023] Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. Training diffusion models with reinforcement learning. ArXiv, abs/2305.13301, 2023. Bradley and Terry [1952] Ralph Allan Bradley and Milton E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39:324, 1952. Brown and Niekum [2019] Daniel S Brown and Scott Niekum. Deep bayesian reward learning from preferences. arXiv preprint arXiv:1912.04472, 2019. Brown et al. [2020] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, T. J. Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeff Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. ArXiv, abs/2005.14165, 2020. Casper et al. [2023] Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, JĂŠrĂŠmy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, Tony Wang, Samuel Marks, Charbel-RaphaĂŤl Segerie, Micah Carroll, Andi Peng, Phillip Christoffersen, Mehul Damani, Stewart Slocum, Usman Anwar, Anand Siththaranjan, Max Nadeau, Eric J. Michaud, Jacob Pfau, Dmitrii Krasheninnikov, Xin Chen, Lauro Langosco, Peter Hase, Erdem BÄąyÄąk, Anca Dragan, David Krueger, Dorsa Sadigh, and Dylan Hadfield-Menell. Open problems and fundamental limitations of reinforcement learning from human feedback, 2023. Chen [2023] Angelica Chen. Improving code generation by training with natural language feedback. ArXiv, abs/2303.16749, 2023. URL https://api.semanticscholar.org/CorpusID:257804798. Chiang et al. [2023] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL https://lmsys.org/blog/2023-03-30-vicuna/. Choshen et al. [2019] Leshem Choshen, Lior Fox, Zohar Aizenbud, and Omri Abend. On the weaknesses of reinforcement learning for neural machine translation. ArXiv, abs/1907.01752, 2019. Chowdhery et al. [2022] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam M. Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Benton C. Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier GarcĂa, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark DĂaz, Orhan Firat, Michele Catasta, Jason Wei, Kathleen S. Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling language modeling with pathways. ArXiv, abs/2204.02311, 2022. Christiano et al. [2017] Paul Francis Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. ArXiv, abs/1706.03741, 2017. Cobbe et al. [2021] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. ArXiv, abs/2110.14168, 2021. Coste et al. [2023] Thomas Coste, Usman Anwar, Robert Kirk, and David Krueger. Reward model ensembles help mitigate overoptimization, 2023. Deng and Raffel [2023] Haikang Deng and Colin Raffel. Reward-augmented decoding: Efficient controlled text generation with a unidirectional reward model, 2023. Deshpande et al. [2023a] A. Deshpande, Vishvak Murahari, Tanmay Rajpurohit, A. Kalyan, and Karthik Narasimhan. Toxicity in chatgpt: Analyzing persona-assigned language models. ArXiv, abs/2304.05335, 2023a. URL https://api.semanticscholar.org/CorpusID:258060002. Deshpande et al. [2023b] Ameet Deshpande, Tanmay Rajpurohit, Karthik Narasimhan, and Ashwin Kalyan. Anthropomorphization of ai: Opportunities and risks. arXiv preprint arXiv:2305.14784, 2023b. Devlin et al. [2019] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. ArXiv, abs/1810.04805, 2019. Dietterich [1999] Thomas G. Dietterich. Hierarchical reinforcement learning with the maxq value function decomposition. ArXiv, cs.LG/9905014, 1999. URL https://api.semanticscholar.org/CorpusID:57341. Dong et al. [2023] Hanze Dong, Wei Xiong, Deepanshu Goyal, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and T. Zhang. Raft: Reward ranked finetuning for generative foundation model alignment. ArXiv, abs/2304.06767, 2023. Du et al. [2021] Nan Du, Yanping Huang, Andrew M. Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, Barret Zoph, Liam Fedus, Maarten Bosma, Zongwei Zhou, Tao Wang, Yu Emma Wang, Kellie Webster, Marie Pellat, Kevin Robinson, Kathleen S. Meier-Hellstern, Toju Duke, Lucas Dixon, Kun Zhang, Quoc V. Le, Yonghui Wu, Z. Chen, and Claire Cui. Glam: Efficient scaling of language models with mixture-of-experts. ArXiv, abs/2112.06905, 2021. Durmus et al. [2023] Esin Durmus, Karina Nyugen, Thomas Liao, Nicholas Schiefer, Amanda Askell, Anton Bakhtin, Carol Chen, Zac Hatfield-Dodds, Danny Hernandez, Nicholas Joseph, Liane Lovitt, Sam McCandlish, Orowa Sikder, Alex Tamkin, Janel Thamkul, Jared Kaplan, Jack Clark, and Deep Ganguli. Towards measuring the representation of subjective global opinions in language models. ArXiv, abs/2306.16388, 2023. URL https://api.semanticscholar.org/CorpusID:259275051. Fan et al. [2022] Xiang Fan, Yiwei Lyu, Paul Pu Liang, Ruslan Salakhutdinov, and Louis-Philippe Morency. Nano: Nested human-in-the-loop reward learning for few-shot language model control. ArXiv, abs/2211.05750, 2022. Fernandes et al. [2022] Patrick Fernandes, AntĂłnio Farinhas, Ricardo Rei, JosĂŠ G. C. de Souza, Perez Ogayo, Graham Neubig, and AndrĂŠ F. T. Martins. Quality-aware decoding for neural machine translation. ArXiv, abs/2205.00978, 2022. Fernandes et al. [2023] Patrick Fernandes, Aman Madaan, Emmy Liu, AntĂłnio Farinhas, Pedro Henrique Martins, Amanda Bertsch, JosĂŠ G. C. de Souza, Shuyan Zhou, Tongshuang Sherry Wu, Graham Neubig, and AndrĂŠ F. T. Martins. Bridging the gap: A survey on integrating (human) feedback for natural language generation. ArXiv, abs/2305.00955, 2023. URL https://api.semanticscholar.org/CorpusID:258426970. Ferrara [2023] Emilio Ferrara. Should chatgpt be biased? challenges and risks of bias in large language models. ArXiv, abs/2304.03738, 2023. URL https://api.semanticscholar.org/CorpusID:258041203. Fried et al. [2017] Daniel Fried, Jacob Andreas, and Dan Klein. Unified pragmatic models for generating and following instructions. In North American Chapter of the Association for Computational Linguistics, 2017. URL https://api.semanticscholar.org/CorpusID:21015570. Ganguli et al. [2022] Deep Ganguli, Liane Lovitt, John Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Benjamin Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, Andy Jones, Sam Bowman, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Nelson Elhage, Sheer El-Showk, Stanislav Fort, Zachary Dodds, T. J. Henighan, Danny Hernandez, Tristan Hume, Josh Jacobson, Scurl = https://lilianweng.github.io/posts/2018-04-08-policy-gradient/nd Jared Kaplan, and Jack Clark. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned. ArXiv, abs/2209.07858, 2022. URL https://api.semanticscholar.org/CorpusID:252355458. Gao et al. [2023] Ge Gao, Hung-Ting Chen, Yoav Artzi, and Eunsol Choi. Continually improving extractive qa via human feedback. ArXiv, abs/2305.12473, 2023. Gao et al. [2022] Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. ArXiv, abs/2210.10760, 2022. Glaese et al. [2022] Amelia Glaese, Nathan McAleese, Maja Trkebacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Maribeth Rauh, Laura Weidinger, Martin Chadwick, Phoebe Thacker, Lucy Campbell-Gillingham, Jonathan Uesato, Po-Sen Huang, Ramona Comanescu, Fan Yang, A. See, Sumanth Dathathri, Rory Greig, Charlie Chen, Doug Fritz, Jaume Sanchez Elias, Richard Green, Sovna Mokrâa, Nicholas Fernando, Boxi Wu, Rachel Foley, Susannah Young, Iason Gabriel, William S. Isaac, John F. J. Mellor, Demis Hassabis, Koray Kavukcuoglu, Lisa Anne Hendricks, and Geoffrey Irving. Improving alignment of dialogue agents via targeted human judgements. ArXiv, abs/2209.14375, 2022. Graham et al. [2013] Yvette Graham, Timothy Baldwin, Alistair Moffat, and Justin Zobel. Continuous measurement scales in human evaluation of machine translation. In LAWACL, 2013. Grzes [2017] Marek Grzes. Reward shaping in episodic reinforcement learning. In Adaptive Agents and Multi-Agent Systems, 2017. URL https://api.semanticscholar.org/CorpusID:2093019. Gudibande et al. [2023] Arnav Gudibande, Eric Wallace, Charlie Snell, Xinyang Geng, Hao Liu, Pieter Abbeel, Sergey Levine, and Dawn Song. The false promise of imitating proprietary llms, 2023. Gupta et al. [2023] Shashank Gupta, Vaishnavi Shrivastava, Ameet Deshpande, Ashwin Kalyan, Peter Clark, Ashish Sabharwal, and Tushar Khot. Bias runs deep: Implicit reasoning biases in persona-assigned llms. arXiv preprint arXiv:2311.04892, 2023. Hadfield-Menell et al. [2017] Dylan Hadfield-Menell, Smitha Milli, P. Abbeel, Stuart J. Russell, and Anca D. Dragan. Inverse reward design. ArXiv, abs/1711.02827, 2017. URL https://api.semanticscholar.org/CorpusID:3805733. Hancock et al. [2019] Braden Hancock, Antoine Bordes, Pierre-Emmanuel MazarĂŠ, and Jason Weston. Learning from dialogue after deployment: Feed yourself, chatbot! In Annual Meeting of the Association for Computational Linguistics, 2019. Hanjie et al. [2022] Austin W. Hanjie, A. Deshpande, and Karthik Narasimhan. Semsup: Semantic supervision for simple and scalable zero-shot generalization. 2022. URL https://api.semanticscholar.org/CorpusID:255595954. Hernandez et al. [2022] Danny Hernandez, Tom B. Brown, Tom Conerly, Nova DasSarma, Dawn Drain, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, T. J. Henighan, Tristan Hume, Scott Johnston, Benjamin Mann, Christopher Olah, Catherine Olsson, Dario Amodei, Nicholas Joseph, Jared Kaplan, and Sam McCandlish. Scaling laws and interpretability of learning from repeated data. ArXiv, abs/2205.10487, 2022. Hinton et al. [2015] Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in a neural network. ArXiv, abs/1503.02531, 2015. Hoffmann et al. [2022] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and L. Sifre. Training compute-optimal large language models. ArXiv, abs/2203.15556, 2022. Ibarz et al. [2018] Borja Ibarz, Jan Leike, Tobias Pohlen, Geoffrey Irving, Shane Legg, and Dario Amodei. Reward learning from human preferences and demonstrations in atari. ArXiv, abs/1811.06521, 2018. URL https://api.semanticscholar.org/CorpusID:53424488. Jain et al. [2015] Ashesh Jain, Shikhar Sharma, Thorsten Joachims, and Ashutosh Saxena. Learning preferences for manipulation tasks from online coactive feedback. The International Journal of Robotics Research, 34:1296 â 1313, 2015. URL https://api.semanticscholar.org/CorpusID:10851113. Jaques et al. [2019] Natasha Jaques, Asma Ghandeharioun, Judy Hanwen Shen, Craig Ferguson, Ăgata Lapedriza, Noah J. Jones, Shixiang Shane Gu, and Rosalind W. Picard. Way off-policy batch deep reinforcement learning of implicit human preferences in dialog. ArXiv, abs/1907.00456, 2019. Kalai and Vempala [2023] Adam Tauman Kalai and Santosh S Vempala. Calibrated language models must hallucinate. arXiv preprint arXiv:2311.14648, 2023. Kaplan et al. [2020] Jared Kaplan, Sam McCandlish, T. J. Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeff Wu, and Dario Amodei. Scaling laws for neural language models. ArXiv, abs/2001.08361, 2020. Kaufmann et al. [2023] Timo Kaufmann, Paul Weng, Viktor Bengs, and Eyke HĂźllermeier. A survey of reinforcement learning from human feedback. arXiv preprint arXiv:2312.14925, 2023. Keneshloo et al. [2018] Yaser Keneshloo, Tian Shi, Naren Ramakrishnan, and Chandan K. Reddy. Deep reinforcement learning for sequence-to-sequence models. IEEE Transactions on Neural Networks and Learning Systems, 31:2469â2489, 2018. Khandelwal et al. [2019] Urvashi Khandelwal, Kevin Clark, Dan Jurafsky, and Lukasz Kaiser. Sample efficient text summarization using a single pre-trained transformer. ArXiv, abs/1905.08836, 2019. Kiegeland and Kreutzer [2021] Samuel Kiegeland and Julia Kreutzer. Revisiting the weaknesses of reinforcement learning for neural machine translation. ArXiv, abs/2106.08942, 2021. Kim et al. [2023] Sungdong Kim, Sanghwan Bae, Jamin Shin, Soyoung Kang, Donghyun Kwak, Kang Min Yoo, and Minjoon Seo. Aligning large language models through synthetic feedback. ArXiv, abs/2305.13735, 2023. Kirk et al. [2023] Robert Kirk, Ishita Mediratta, Christoforos Nalmpantis, Jelena Luketina, Eric Hambro, Edward Grefenstette, and Roberta Raileanu. Understanding the effects of rlhf on llm generalisation and diversity, 2023. Knox and Stone [2008] W. B. Knox and P. Stone. Tamer: Training an agent manually via evaluative reinforcement. 2008 7th IEEE International Conference on Development and Learning, pages 292â297, 2008. Knox et al. [2022] W Bradley Knox, Stephane Hatgis-Kessell, Serena Booth, Scott Niekum, Peter Stone, and Alessandro Allievi. Models of human preference for learning reward functions. arXiv preprint arXiv:2206.02231, 2022. Korbak et al. [2022] Tomasz Korbak, Ethan Perez, and Christopher L. Buckley. Rl with kl penalties is better viewed as bayesian inference. ArXiv, abs/2205.11275, 2022. Korbak et al. [2023] Tomasz Korbak, Kejian Shi, Angelica Chen, Rasika Bhalerao, Christopher L. Buckley, Jason Phang, Sam Bowman, and Ethan Perez. Pretraining language models with human preferences. ArXiv, abs/2302.08582, 2023. Kreutzer et al. [2018a] Julia Kreutzer, Shahram Khadivi, Evgeny Matusov, and Stefan Riezler. Can neural machine translation be improved with user feedback? ArXiv, abs/1804.05958, 2018a. Kreutzer et al. [2018b] Julia Kreutzer, Joshua Uyheng, and Stefan Riezler. Reliability and learnability of human bandit feedback for sequence-to-sequence reinforcement learning. ArXiv, abs/1805.10627, 2018b. Kundu et al. [2023] Sandipan Kundu, Yuntao Bai, Saurav Kadavath, Amanda Askell, Andrew Callahan, Anna Chen, Anna Goldie, Avital Balwit, Azalia Mirhoseini, Brayden McLean, Catherine Olsson, Cassie Evraets, Eli Tran-Johnson, Esin Durmus, Ethan Perez, Jackson Kernion, Jamie Kerr, Kamal Ndousse, Karina Nguyen, Nelson Elhage, Newton Cheng, Nicholas Schiefer, Nova DasSarma, Oliver Rausch, Robin Larson, Shannon Yang, Shauna Kravec, Timothy Telleen-Lawton, Thomas I. Liao, Tom Henighan, Tristan Hume, Zac Hatfield-Dodds, SĂśren Mindermann, Nicholas Joseph, Sam McCandlish, and Jared Kaplan. Specific versus general principles for constitutional ai, 2023. Lagunas et al. [2021] François Lagunas, Ella Charlaix, Victor Sanh, and Alexander M Rush. Block pruning for faster transformers. arXiv preprint arXiv:2109.04838, 2021. Lawrence and Riezler [2018] Carolin (Haas) Lawrence and Stefan Riezler. Improving a neural semantic parser by counterfactual learning from human bandit feedback. In Annual Meeting of the Association for Computational Linguistics, 2018. Lee et al. [2021] Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. Deduplicating training data makes language models better. In Annual Meeting of the Association for Computational Linguistics, 2021. Lewis et al. [2017] Mike Lewis, Denis Yarats, Yann Dauphin, Devi Parikh, and Dhruv Batra. Deal or no deal? end-to-end learning of negotiation dialogues. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2443â2453, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi:10.18653/v1/D17-1259. URL https://aclanthology.org/D17-1259. Li et al. [2016] Jiwei Li, Alexander H. Miller, Sumit Chopra, MarcâAurelio Ranzato, and Jason Weston. Dialogue learning with human-in-the-loop. ArXiv, abs/1611.09823, 2016. Li et al. [2017] Zichao Li, Xin Jiang, Lifeng Shang, and Hang Li. Paraphrase generation with deep reinforcement learning. ArXiv, abs/1711.00279, 2017. Li et al. [2022] Zichao Li, Prakhar Sharma, Xing Han Lu, Jackie Chi Kit Cheung, and Siva Reddy. Using interactive feedback to improve the accuracy and explainability of question answering systems post-deployment. ArXiv, abs/2204.03025, 2022. URL https://api.semanticscholar.org/CorpusID:248006299. Lightman et al. [2023] Hunter Lightman, Vineet Kosaraju, Yura Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Letâs verify step by step. ArXiv, abs/2305.20050, 2023. Likert [1932] Rensis Likert. A technique for the measurement of attitudes. Archives of psychology, 1932. Lin and Och [2004] Chin-Yew Lin and Franz Josef Och. Automatic evaluation of machine translation quality using longest common subsequence and skip-bigram statistics. In Annual Meeting of the Association for Computational Linguistics, 2004. Liu et al. [2018] Bing Liu, GĂśkhan TĂźr, Dilek Z. Hakkani-TĂźr, Pararth Shah, and Larry Heck. Dialogue learning with human teaching and feedback in end-to-end trainable task-oriented dialogue systems. In North American Chapter of the Association for Computational Linguistics, 2018. Liu et al. [2016] Chia-Wei Liu, Ryan Lowe, Iulian Serban, Michael Noseworthy, Laurent Charlin, and Joelle Pineau. How not to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation. ArXiv, abs/1603.08023, 2016. Liu et al. [2023a] Hao Liu, Carmelo Sferrazza, and P. Abbeel. Chain of hindsight aligns language models with feedback. ArXiv, abs/2302.02676, 2023a. Liu et al. [2023b] Ruibo Liu, Chenyan Jia, Ge Zhang, Ziyu Zhuang, Tony X. Liu, and Soroush Vosoughi. Second thoughts are best: Learning to re-align with human values from text edits. ArXiv, abs/2301.00355, 2023b. Liu et al. [2023c] Tianqi Liu, Yao Zhao, Rishabh Joshi, Misha Khalman, Mohammad Saleh, Peter J. Liu, and Jialu Liu. Statistical rejection sampling improves preference optimization, 2023c. Liu et al. [2009] Tie-Yan Liu et al. Learning to rank for information retrieval. Foundations and TrendsÂŽ in Information Retrieval, 3(3):225â331, 2009. Liu et al. [2019] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach, 2019. Luce [1979] R. Duncan Luce. Individual choice behavior: A theoretical analysis. 1979. Madaan et al. [2023] Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Sean Welleck, Bodhisattwa Prasad Majumder, Shashank Gupta, Amir Yazdanbakhsh, and Peter Clark. Self-refine: Iterative refinement with self-feedback. ArXiv, abs/2303.17651, 2023. Markov [1954] Andrei Andreevich Markov. The theory of algorithms. Trudy Matematicheskogo Instituta Imeni VA Steklova, 42:3â375, 1954. Matuszek et al. [2012] Cynthia Matuszek, Nicholas FitzGerald, Luke Zettlemoyer, Liefeng Bo, and Dieter Fox. A joint model of language and perception for grounded attribute learning. In International Conference on Machine Learning, 2012. URL https://api.semanticscholar.org/CorpusID:2408319. Maynez et al. [2020] Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan T. McDonald. On faithfulness and factuality in abstractive summarization. ArXiv, abs/2005.00661, 2020. McFadden [1981] Daniel McFadden. Econometric models of probabilistic choice. 1981. McKenna et al. [2023] Nick McKenna, Tianyi Li, Liang Cheng, Mohammad Javad Hosseini, Mark Johnson, and Mark Steedman. Sources of hallucination by large language models on inference tasks. ArXiv, abs/2305.14552, 2023. URL https://api.semanticscholar.org/CorpusID:258865517. Mei et al. [2022] Jincheng Mei, Wesley Chung, Valentin Thomas, Bo Dai, Csaba Szepesvari, and Dale Schuurmans. The role of baselines in policy gradient optimization. Advances in Neural Information Processing Systems, 35:17818â17830, 2022. Menick et al. [2022] Jacob Menick, Maja Trebacz, Vladimir Mikulik, John Aslanides, Francis Song, Martin Chadwick, Mia Glaese, Susannah Young, Lucy Campbell-Gillingham, Geoffrey Irving, and Nathan McAleese. Teaching language models to support answers with verified quotes. ArXiv, abs/2203.11147, 2022. Mnih et al. [2016] Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference on machine learning, pages 1928â1937. PMLR, 2016. Munos et al. [2023] RĂŠmi Munos, Michal Valko, Daniele Calandriello, Mohammad Gheshlaghi Azar, Mark Rowland, Zhaohan Daniel Guo, Yunhao Tang, Matthieu Geist, Thomas Mesnard, Andrea Michi, Marco Selvi, Sertan Girgin, Nikola Momchev, Olivier Bachem, Daniel Jaymin Mankowitz, Doina Precup, and Bilal Piot. Nash learning from human feedback. ArXiv, abs/2312.00886, 2023. URL https://api.semanticscholar.org/CorpusID:265609682. Murahari et al. [2022] Vishvak Murahari, Carlos E Jimenez, Runzhe Yang, and Karthik R Narasimhan. DataMUX: Data multiplexing for neural networks. In Thirty-Sixth Conference on Neural Information Processing Systems, 2022. URL https://openreview.net/forum?id=UdgtTVTdswg. Murahari et al. [2023] Vishvak Murahari, Ameet Deshpande, Carlos E Jimenez, Izhak Shafran, Mingqiu Wang, Yuan Cao, and Karthik Narasimhan. Mux-plms: Pre-training language models with data multiplexing. arXiv preprint arXiv:2302.12441, 2023. Nakano et al. [2021] Reiichiro Nakano, Jacob Hilton, S. Arun Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. Webgpt: Browser-assisted question-answering with human feedback. ArXiv, abs/2112.09332, 2021. Nakkiran et al. [2019] P Nakkiran, G Kaplun, Y Bansal, et al. Deep double descent: Where bigger models and more data hurt. arxiv: 191202292 [cs, stat]. 2019. Ng et al. [1999] A. Ng, Daishi Harada, and Stuart J. Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In International Conference on Machine Learning, 1999. URL https://api.semanticscholar.org/CorpusID:5730166. Ngo et al. [2022] Richard Ngo, Lawrence Chan, and SĂśren Mindermann. The alignment problem from a deep learning perspective. arXiv preprint arXiv:2209.00626, 2022. Nguyen et al. [2022] Duy-Hung Nguyen, Nguyen-Viet-Dung Nghiem, Bao-Sinh Nguyen, Dung Tien Le, Shahab Sabahi, Minh Le Nguyen, and Hung Le. Make the most of prior data: A solution for interactive text summarization with preference feedback. In NAACL-HLT, 2022. Nguyen et al. [2017] Khanh Nguyen, Hal DaumĂŠ, and Jordan L. Boyd-Graber. Reinforcement learning for bandit neural machine translation with simulated human feedback. ArXiv, abs/1707.07402, 2017. Nguyen et al. [2021] Khanh Nguyen, Dipendra Misra, Robert Schapire, Miro DudĂk, and Patrick Shafto. Interactive learning from activity description. In ICML, 2021. OâConnor [1989] Marcus OâConnor. Models of human behaviour and confidence in judgement: A review. International Journal of Forecasting, 5:159â169, 1989. OpenAI [2022] OpenAI. Chatgpt. https://openai.com/blog/chatgpt, 2022. URL https://openai.com/blog/chatgpt. OpenAI [2023] OpenAI. Gpt-4 technical report. ArXiv, abs/2303.08774, 2023. Oudeyer et al. [2007] Pierre-Yves Oudeyer, F. Kaplan, and Verena V. Hafner. Intrinsic motivation systems for autonomous mental development. IEEE Trans. Evol. Comput., 11:265â286, 2007. URL https://api.semanticscholar.org/CorpusID:260429077. Ouyang et al. [2022] Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke E. Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Francis Christiano, Jan Leike, and Ryan J. Lowe. Training language models to follow instructions with human feedback. ArXiv, abs/2203.02155, 2022. Pang et al. [2022] Richard Yuanzhe Pang, Vishakh Padmakumar, Thibault Sellam, Ankur P. Parikh, and He He. Reward gaming in conditional text generation. ArXiv, abs/2211.08714, 2022. Parmar et al. [2022] Mihir Parmar, Swaroop Mishra, Mor Geva, and Chitta Baral. Donât blame the annotator: Bias already starts in the annotation instructions. In Conference of the European Chapter of the Association for Computational Linguistics, 2022. Pathak et al. [2017] Deepak Pathak, Pulkit Agrawal, Alexei A. Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 488â489, 2017. URL https://api.semanticscholar.org/CorpusID:20045336. Peng et al. [2022] Andi Peng, Besmira Nushi, Emre Kiciman, Kori Inkpen, and Ece Kamar. Investigations of performance and bias in human-ai teamwork in hiring, 2022. Perez et al. [2022] Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nathan McAleese, and Geoffrey Irving. Red teaming language models with language models. In Conference on Empirical Methods in Natural Language Processing, 2022. Plackett [1975] Robin L. Plackett. The analysis of permutations. Journal of The Royal Statistical Society Series C-applied Statistics, 24:193â202, 1975. Pomerleau [1988] Dean A Pomerleau. Alvinn: An autonomous land vehicle in a neural network. Advances in neural information processing systems, 1, 1988. Radford and Narasimhan [2018] Alec Radford and Karthik Narasimhan. Improving language understanding by generative pre-training. 2018. Rae et al. [2021] Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks, Maribeth Rauh, Po-Sen Huang, Amelia Glaese, Johannes Welbl, Sumanth Dathathri, Saffron Huang, Jonathan Uesato, John F. J. Mellor, Irina Higgins, Antonia Creswell, Nathan McAleese, Amy Wu, Erich Elsen, Siddhant M. Jayakumar, Elena Buchatskaya, David Budden, Esme Sutherland, Karen Simonyan, Michela Paganini, L. Sifre, Lena Martens, Xiang Lorraine Li, Adhiguna Kuncoro, Aida Nematzadeh, Elena Gribovskaya, Domenic Donato, Angeliki Lazaridou, Arthur Mensch, Jean-Baptiste Lespiau, Maria Tsimpoukelli, N. K. Grigorev, Doug Fritz, Thibault Sottiaux, Mantas Pajarskas, Tobias Pohlen, Zhitao Gong, Daniel Toyama, Cyprien de Masson dâAutume, Yujia Li, Tayfun Terzi, Vladimir Mikulik, Igor Babuschkin, Aidan Clark, Diego de Las Casas, Aurelia Guy, Chris Jones, James Bradbury, Matthew G. Johnson, Blake A. Hechtman, Laura Weidinger, Iason Gabriel, William S. Isaac, Edward Lockhart, Simon Osindero, Laura Rimell, Chris Dyer, Oriol Vinyals, Kareem W. Ayoub, Jeff Stanway, L. L. Bennett, Demis Hassabis, Koray Kavukcuoglu, and Geoffrey Irving. Scaling language models: Methods, analysis & insights from training gopher. ArXiv, abs/2112.11446, 2021. Rafailov et al. [2023] Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. arXiv preprint arXiv:2305.18290, 2023. Raffel et al. [2019] Colin Raffel, Noam M. Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. ArXiv, abs/1910.10683, 2019. Raichuk et al. [2021] Anton Raichuk, Piotr Stanczyk, Manu Orsini, Sertan Girgin, RaphaĂŤl Marinier, Lâeonard Hussenot, Matthieu Geist, Olivier Pietquin, Marcin Michalski, and Sylvain Gelly. What matters for on-policy deep actor-critic methods? a large-scale study. In International Conference on Learning Representations, 2021. URL https://api.semanticscholar.org/CorpusID:233340556. Ramachandran and Amir [2007] Deepak Ramachandran and Eyal Amir. Bayesian inverse reinforcement learning. In IJCAI, volume 7, pages 2586â2591, 2007. Ramamurthy et al. [2022] Rajkumar Ramamurthy, Prithviraj Ammanabrolu, KiantĂŠ Brantley, Jack Hessel, Rafet Sifa, Christian Bauckhage, Hannaneh Hajishirzi, and Yejin Choi. Is reinforcement learning (not) for natural language processing?: Benchmarks, baselines, and building blocks for natural language policy optimization. ArXiv, abs/2210.01241, 2022. RamĂŠ et al. [2023] Alexandre RamĂŠ, Guillaume Couairon, Mustafa Shukor, Corentin Dancette, Jean-Baptiste Gaya, Laure Soulier, and Matthieu Cord. Rewarded soups: towards pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards. 2023. Ranzato et al. [2015] MarcâAurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. Sequence level training with recurrent neural networks. CoRR, abs/1511.06732, 2015. Rengarajan et al. [2022] Desik Rengarajan, Gargi Nikhil Vaidya, Akshay Sarvesh, Dileep M. Kalathil, and Srinivas Shakkottai. Reinforcement learning with sparse rewards using guidance from offline demonstration. ArXiv, abs/2202.04628, 2022. URL https://api.semanticscholar.org/CorpusID:246679865. Roijers [2016] Diederik M. Roijers. Multi-objective decision-theoretic planning. 2016. URL https://api.semanticscholar.org/CorpusID:124195290. Roijers et al. [2013] Diederik M. Roijers, Peter Vamplew, Shimon Whiteson, and Richard Dazeley. A survey of multi-objective sequential decision-making. ArXiv, abs/1402.0590, 2013. URL https://api.semanticscholar.org/CorpusID:14478191. Santacroce et al. [2023] Michael Santacroce, Yadong Lu, Han Yu, Yuanzhi Li, and Yelong Shen. Efficient rlhf: Reducing the memory usage of ppo, 2023. Santurkar et al. [2023] Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. Whose opinions do language models reflect?, 2023. Schaeffer et al. [2023] Rylan Schaeffer, Mikail Khona, Zachary Robertson, Akhilan Boopathy, Kateryna Pistunova, Jason W Rocks, Ila Rani Fiete, and Oluwasanmi Koyejo. Double descent demystified: Identifying, interpreting & ablating the sources of a deep learning puzzle. arXiv preprint arXiv:2303.14151, 2023. Scheurer et al. [2022] Jâerâemy Scheurer, Jon Ander Campos, Jun Shern Chan, Angelica Chen, Kyunghyun Cho, and Ethan Perez. Training language models with language feedback. 2022. Scheurer et al. [2023] Jâerâemy Scheurer, Jon Ander Campos, Tomasz Korbak, Jun Shern Chan, Angelica Chen, Kyunghyun Cho, and Ethan Perez. Training language models with language feedback at scale. ArXiv, abs/2303.16755, 2023. Schluter [2017] Natalie Schluter. The limits of automatic summarisation according to rouge. In Conference of the European Chapter of the Association for Computational Linguistics, 2017. Schulman et al. [2017] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. ArXiv, abs/1707.06347, 2017. Sellam et al. [2020] Thibault Sellam, Dipanjan Das, and Ankur P. Parikh. Bleurt: Learning robust metrics for text generation. In Annual Meeting of the Association for Computational Linguistics, 2020. Shen et al. [2015] Shiqi Shen, Yong Cheng, Zhongjun He, W. He, Hua Wu, Maosong Sun, and Yang Liu. Minimum risk training for neural machine translation. ArXiv, abs/1512.02433, 2015. Shi et al. [2018] Zhan Shi, Xinchi Chen, Xipeng Qiu, and Xuanjing Huang. Toward diverse text generation with inverse reinforcement learning. In International Joint Conference on Artificial Intelligence, 2018. Shinn et al. [2023] Noah Shinn, Federico Cassano, Beck Labash, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. 2023. URL https://api.semanticscholar.org/CorpusID:258833055. Shumailov et al. [2023] Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Yarin Gal, Nicolas Papernot, and Ross Anderson. The curse of recursion: Training on generated data makes models forget, 2023. Silver et al. [2021] David Silver, Satinder Singh, Doina Precup, and Richard S Sutton. Reward is enough. Artificial Intelligence, 299:103535, 2021. Singhal et al. [2022] K. Singhal, Shekoofeh Azizi, Tao Tu, Said Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Kumar Tanwani, Heather J. Cole-Lewis, Stephen J. Pfohl, P A Payne, Martin G. Seneviratne, Paul Gamble, Chris Kelly, Nathaneal Scharli, Aakanksha Chowdhery, P. A. Mansfield, Blaise Aguera y Arcas, Dale R. Webster, Greg S. Corrado, Yossi Matias, Katherine Hui-Ling Chou, Juraj Gottweis, Nenad Tomavsev, Yun Liu, Alvin Rajkomar, Joelle K. Barral, Christopher Semturs, Alan Karthikesalingam, and Vivek Natarajan. Large language models encode clinical knowledge. Nature, 620:172 â 180, 2022. URL https://api.semanticscholar.org/CorpusID:255124952. Singhal et al. [2023] Prasann Singhal, Tanya Goyal, Jiacheng Xu, and Greg Durrett. A long way to go: Investigating length correlations in rlhf, 2023. Sokolov et al. [2016] Artem Sokolov, Stefan Riezler, and Tanguy Urvoy. Bandit structured prediction for learning from partial feedback in statistical machine translation. ArXiv, abs/1601.04468, 2016. Song et al. [2023] Feifan Song, Bowen Yu, Minghao Li, Haiyang Yu, Fei Huang, Yongbin Li, and Houfeng Wang. Preference ranking optimization for human alignment, 2023. Stiennon et al. [2020] Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan J. Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize from human feedback. ArXiv, abs/2009.01325, 2020. Su et al. [2023] Yushan Su, Vishvak Murahari, Karthik Narasimhan, and Kai Li. Prumux: Augmenting data multiplexing with model compression. arXiv preprint arXiv:2305.14706, 2023. Sun et al. [2023] Zhiqing Sun, Yikang Shen, Qinhong Zhou, Hongxin Zhang, Zhenfang Chen, David D. Cox, Yiming Yang, and Chuang Gan. Principle-driven self-alignment of language models from scratch with minimal human supervision. ArXiv, abs/2305.03047, 2023. Sutton and Barto [2005] Richard S. Sutton and Andrew G. Barto. Reinforcement learning: An introduction. IEEE Transactions on Neural Networks, 16:285â286, 2005. URL https://api.semanticscholar.org/CorpusID:9166388. Sutton and Barto [2018] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018. Sutton et al. [1999] Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. Advances in neural information processing systems, 12, 1999. Sutton [2004] R.S. Sutton. The reward hypothesis. 2004. URL http://incompleteideas.net/rlai.cs.ualberta.ca/RLAI/rewardhypothesis.html. Touvron et al. [2023a] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, TimothĂŠe Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurâelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. ArXiv, abs/2302.13971, 2023a. Touvron et al. [2023b] Hugo Touvron, Louis Martin, Kevin R. Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Daniel M. Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony S. Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel M. Kloumann, A. V. Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, R. Subramanian, Xia Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zhengxu Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. 2023b. Uesato et al. [2022] Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, L. Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. Solving math word problems with process- and outcome-based feedback. ArXiv, abs/2211.14275, 2022. Vaswani et al. [2017] Ashish Vaswani, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, 2017. Wei et al. [2023] Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. Jailbroken: How does llm safety training fail? ArXiv, abs/2307.02483, 2023. URL https://api.semanticscholar.org/CorpusID:259342528. Wei et al. [2021] Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. Finetuned language models are zero-shot learners. ArXiv, abs/2109.01652, 2021. Weng [2018] Lilian Weng. Policy gradient algorithms. lilianweng.github.io, 2018. URL https://lilianweng.github.io/posts/2018-04-08-policy-gradient/. Williams [1992] Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8:229â256, 1992. Wu et al. [2021] Jeff Wu, Long Ouyang, Daniel M. Ziegler, Nissan Stiennon, Ryan Lowe, Jan Leike, and Paul Francis Christiano. Recursively summarizing books with human feedback. ArXiv, abs/2109.10862, 2021. Wu et al. [2023a] Shijie Wu, Ozan Irsoy, Steven Lu, Vadim Dabravolski, Mark Dredze, Sebastian Gehrmann, Prabhanjan Kambadur, David Rosenberg, and Gideon Mann. Bloomberggpt: A large language model for finance. ArXiv, abs/2303.17564, 2023a. URL https://api.semanticscholar.org/CorpusID:257833842. Wu et al. [2023b] Tianhao Wu, Banghua Zhu, Ruoyu Zhang, Zhaojin Wen, Kannan Ramchandran, and Jiantao Jiao. Pairwise proximal policy optimization: Harnessing relative feedback for llm alignment. ArXiv, abs/2310.00212, 2023b. URL https://api.semanticscholar.org/CorpusID:263334045. Wu et al. [2023c] Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A. Smith, Mari Ostendorf, and Hanna Hajishirzi. Fine-grained human feedback gives better rewards for language model training. 2023c. Wu et al. [2023d] Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A Smith, Mari Ostendorf, and Hannaneh Hajishirzi. Fine-grained human feedback gives better rewards for language model training. arXiv preprint arXiv:2306.01693, 2023d. [176] Mengzhou Xia, Tianyu Gao, Zhiyuan Zeng, and Danqi Chen. Sheared llama: Accelerating language model pre-training via structured pruning. Xia et al. [2022] Mengzhou Xia, Zexuan Zhong, and Danqi Chen. Structured pruning learns compact and accurate models. In Association for Computational Linguistics (ACL), 2022. Xie et al. [2023] Sang Michael Xie, Hieu Pham, Xuanyi Dong, Nan Du, Hanxiao Liu, Yifeng Lu, Percy Liang, Quoc V. Le, Tengyu Ma, and Adams Wei Yu. Doremi: Optimizing data mixtures speeds up language model pretraining. 2023. Xu et al. [2022] Jing Xu, Megan Ung, Mojtaba Komeili, Kushal Arora, Y-Lan Boureau, and Jason Weston. Learning new skills after deployment: Improving open-domain internet-driven dialogue with human feedback. ArXiv, abs/2208.03270, 2022. Yan et al. [2023] Lixiang Yan, Lele Sha, Linxuan Zhao, Yuheng Li, Roberto Martinez-Maldonado, Guanliang Chen, Xinyu Li, Yueqiao Jin, and Dragan GaĹĄeviÄ. Practical and ethical challenges of large language models in education: A systematic scoping review. British Journal of Educational Technology, August 2023. ISSN 1467-8535. doi:10.1111/bjet.13370. URL http://dx.doi.org/10.1111/bjet.13370. Yang et al. [2022a] Kevin Yang, Nanyun Peng, Yuandong Tian, and Dan Klein. Re3: Generating longer stories with recursive reprompting and revision. In Conference on Empirical Methods in Natural Language Processing, 2022a. Yang et al. [2022b] Ziqing Yang, Yiming Cui, and Zhigang Chen. TextPruner: A model pruning toolkit for pre-trained language models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 35â43, Dublin, Ireland, May 2022b. Association for Computational Linguistics. doi:10.18653/v1/2022.acl-demo.4. URL https://aclanthology.org/2022.acl-demo.4. Yannakakis and Hallam [2011] Georgios N. Yannakakis and John Hallam. Ranking vs. preference: A comparative study of self-reporting. In Affective Computing and Intelligent Interaction, 2011. URL https://api.semanticscholar.org/CorpusID:48790. Yi et al. [2019] Sanghyun Yi, Rahul Goel, Chandra Khatri, Tagyoung Chung, Behnam Hedayatnia, Anu Venkatesh, Raefer Gabriel, and Dilek Z. Hakkani-TĂźr. Towards coherent and engaging spoken dialog response generation using automatic conversation evaluators. In International Conference on Natural Language Generation, 2019. Yin et al. [2021] Yichun Yin, Cheng Chen, Lifeng Shang, Xin Jiang, Xiao Chen, and Qun Liu. AutoTinyBERT: Automatic hyper-parameter optimization for efficient pre-trained language models. pages 5146â5157, 2021. Yuan et al. [2023] Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Feiran Huang. Rrhf: Rank responses to align language models with human feedback without tears. ArXiv, abs/2304.05302, 2023. Zhang et al. [2022] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. Opt: Open pre-trained transformer language models. ArXiv, abs/2205.01068, 2022. Zhang et al. [2023] Tianjun Zhang, Fangchen Liu, Justin Wong, P. Abbeel, and Joseph Gonzalez. The wisdom of hindsight makes language models better instruction followers. ArXiv, abs/2302.05206, 2023. Zhao et al. [2022] Yao Zhao, Misha Khalman, Rishabh Joshi, Shashi Narayan, Mohammad Saleh, and Peter J. Liu. Calibrating sequence likelihood improves conditional language generation. ArXiv, abs/2210.00045, 2022. Zhao et al. [2023] Yao Zhao, Rishabh Joshi, Tianqi Liu, Misha Khalman, Mohammad Saleh, and Peter J. Liu. Slic-hf: Sequence likelihood calibration with human feedback. ArXiv, abs/2305.10425, 2023. Zheng et al. [2023] Rui Zheng, Shihan Dou, Songyang Gao, Wei Shen, Bing Wang, Yan Liu, Senjie Jin, Qin Liu, Limao Xiong, Luyao Chen, Zhiheng Xi, Yuhao Zhou, Nuo Xu, Wen-De Lai, Minghao Zhu, Rongxiang Weng, Wen-Chun Cheng, Cheng Chang, Zhangyue Yin, Yuan Long Hua, Haoran Huang, Tianxiang Sun, Hang Yan, Tao Gui, Qi Zhang, Xipeng Qiu, and Xuanjing Huang. Secrets of rlhf in large language models part i: Ppo. 2023. Zhou et al. [2023] Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. Lima: Less is more for alignment, 2023. Zhou and Xu [2020] Wangchunshu Zhou and Ke Xu. Learning to compare for better training and evaluation of open domain natural language generation models. In AAAI Conference on Artificial Intelligence, 2020. Zhu et al. [2023a] Banghua Zhu, Jiantao Jiao, and M.I. Jordan. Principled reinforcement learning with human feedback from pairwise or k-wise comparisons. ArXiv, abs/2301.11270, 2023a. Zhu et al. [2023b] Banghua Zhu, Hiteshi Sharma, Felipe Vieira Frujeri, Shi Dong, Chenguang Zhu, M.I. Jordan, and Jiantao Jiao. Fine-tuning language models with advantage-induced policy alignment. ArXiv, abs/2306.02231, 2023b. Ziebart et al. [2008] Brian D. Ziebart, Andrew L. Maas, J. Andrew Bagnell, and Anind K. Dey. Maximum entropy inverse reinforcement learning. In AAAI Conference on Artificial Intelligence, 2008. Ziegler et al. [2019] Daniel M. Ziegler, Nisan Stiennon, Jeff Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. ArXiv, abs/1909.08593, 2019.