Paper deep dive
The History and Risks of Reinforcement Learning and Human Feedback
Nathan Lambert, Thomas Krendl Gilbert, Tom Zick
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 96%
Last extracted: 3/12/2026, 7:33:49 PM
Summary
The paper examines the history, risks, and sociotechnical foundations of Reinforcement Learning from Human Feedback (RLHF). It highlights the ontological gap between traditional control-theory-based reward functions and the complex, often ill-defined nature of human preferences in large language models. The authors argue that current RLHF practices rely on unexamined assumptions and presumptions, calling for greater transparency, multi-stakeholder engagement, and more rigorous evaluation methods for reward models.
Entities (5)
Relation Signals (3)
RLHF â utilizes â Reward Model
confidence 100% ¡ A core piece of the RLHF process is the training and utilization of a model of human preferences that acts as a reward function for optimization.
RLHF â borrowsfrom â Control Theory
confidence 95% ¡ The optimization stack of RLHF is borrowed from control theory
Reward Model â optimizes â LLM
confidence 90% ¡ the language model is optimized with an RL optimizer
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Reinforcement learning from human feedback (RLHF) has emerged as a powerful technique to make large language models (LLMs) easier to use and more effective. A core piece of the RLHF process is the training and utilization of a model of human preferences that acts as a reward function for optimization. This approach, which operates at the intersection of many stakeholders and academic disciplines, remains poorly understood. RLHF reward models are often cited as being central to achieving performance, yet very few descriptors of capabilities, evaluations, training methods, or open-source models exist. Given this lack of information, further study and transparency is needed for learned RLHF reward models. In this paper, we illustrate the complex history of optimizing preferences, and articulate lines of inquiry to understand the sociotechnical context of reward models. In particular, we highlight the ontological differences between costs, rewards, and preferences at stake in RLHF's foundations, related methodological tensions, and possible research directions to improve general understanding of how reward models function.
Tags
Links
- Source: https://arxiv.org/abs/2310.13595
- Canonical: https://arxiv.org/abs/2310.13595
Trouble viewing inline? Open PDF directly â
Full Text
92,950 characters extracted from source content.
Expand or collapse full text
The History and Risks of Reinforcement Learning and Human Feedback Nathan Lambert Allen Institute for AI Berkeley, CA, USA nathanl@allenai.org &Thomas Krendl Gilbert The New York Academy of Sciences New York, NY, USA tgilbert@nyas.org &Tom Zick Berkman Klein Center, Harvard Law Cambridge, MA, USA tzick@jd24.law.harvard.edu (November 28, 2023) Abstract Reinforcement learning from human feedback (RLHF) has emerged as a powerful technique to make large language models (LLMs) easier to use and more effective. A core piece of the RLHF process is the training and utilization of a model of human preferences that acts as a reward function for optimization. This approach, which operates at the intersection of many stakeholders and academic disciplines, remains poorly understood. RLHF reward models are often cited as being central to achieving performance, yet very few descriptors of capabilities, evaluations, training methods, or open-source models exist. Given this lack of information, further study and transparency is needed for learned RLHF reward models. In this paper, we illustrate the complex history of optimizing preferences, and articulate lines of inquiry to understand the sociotechnical context of reward models. In particular, we highlight the ontological differences between costs, rewards, and preferences at stake in RLHFâs foundations, related methodological tensions, and possible research directions to improve general understanding of how reward models function. 1 Introduction Learning from human feedback has become incredibly popular due to the success of large language models (LLMs) such as OpenAIâs ChatGPT (Schulman ., 2022) and Anthropicâs Claude (Bai, Jones ., 2022), which are heavily dependent on human labeled data. These models make use of reinforcement learning from human feedback (RLHF), a technique designed to integrate human preferences where writing an explicit reward function is otherwise challenging (Christiano ., 2017). In the context of language models, RLHF proceeds as follows: first, a reward model is independently trained on aggregate pairwise preferences from many crowdworkers to rate any piece of text; second, the language model is optimized with an RL optimizer (Ouyang ., 2022; Bai, Jones ., 2022; Touvron ., 2023). The final language model is often subject to heavy scrutiny both internally and, more recently, externally through events like DEFCON Red Teaming Village (Bajak, 2023) or coordinated adversarial attacks Zou . ( 2023). The same cannot be said for the intermediate reward model. Historically, reward models have not been released as open-source or evaluated rigorously, obscuring from scrutiny the process through which values are actively encoded into the system. This paper illustrates why reward models are central to understanding the long-term impacts of RLHF, drawing from the rich history, discourse, and tension around how to best quantify human values. RLHF is the intellectual culmination of several distinct domains. The optimization stack of RLHF is borrowed from control theory, a domain in which there are ground truths and reward functions can have an clear notion of success. The primary risk of learning human preferences for LLMs comes through the domain shift from control to language. In language, notions of values are more computationally complex or fundamentally vague Dobbe . ( 2021) in relation to their control counterparts, but the optimization stack nevertheless remains similar. Reinforcement learning is broadly the field of study of sequential decision making, which is built on a substantial literature comprising cognitive biology, optimal control, behavioral economics, and other fields (Sutton Barto, 2018). RLHF combines the social challenges of human data with the techniques of RL â a field with a long history of numerical complexity. Despite the maturity of the domains it draws on, grounding and investigating risks of RLHF requires the development of new tools and research methods. In particular, vestigial assumptions inherited from earlier technologies can re-surface as blind spots in the modern RLHF paradigm. Tracing the history of RL and RLHF as technologies allows us to identify these assumptions and where they matter in particular systems. This paper attempts to provide an exposition of some of this historical context, and to highlight specific sociotechnical areas of opportunity within the reward model specification, beyond the challenges proposed in recent literature (Casper ., 2023). We study the histories of quantification of human preferences and reinforcement learning algorithms, from Port-Royal Logic and Bentham to Markov Decision Processes and Bellman, to highlight potential shortcomings of learning models of human preferences. An initial concern that has been raised with the current deployments of LLMs is the limitations of working with aggregate human data, raising questions as to whose values the model is encoding and prioritizing. Moving beyond this, we study how structural optimization and deployment decisions can impact downstream users. Given the nuance around modeling human preferences, we refer to these artifacts as reward models of preference, or reward models for short, to better match their usage as an optimization target for reinforcement learning algorithms rather than an accurate representation of human values. In order to broaden the scope of study around these reward models, we make the following contributions: ⢠Trace the complex intellectual history of RLHF to illustrate the potential ill-posed assumptions popularized within RLHF. In Sec. 3, we explain the evolution of RL with the history of rational agents and human preferences. In doing so, we distinguish sets of assumptions (explicit premises) and presumptions (implicit premises) made throughout the evolution of RLHF that lead to potential shortcomings of reward models. ⢠Propose a series of questions for contemporary RLHF reward models to increase transparency and opportunities for broader multi-stakeholder engagement in modern LLM development. In Sec. 5, we break these questions down by sections of the machine learning process: data, model, and optimization, and in Sec. 7, we also discuss emerging issues that are not easily classified. ⢠Discuss solutions in Sec. 6 to measure and communicate the values and potential harms of contemporary RLHF reward models. We propose tools that can be used to add rigour to future empirical evaluation work. 2 Related Works 2.1 Reinforcement learning from human feedback RLHF is a set of techniques designed to optimize machine learning models based on human feedback in order to circumvent the need to design a complex reward function. Early work in RLHF focused on soliciting complex behaviors from AI agents in control problems using various environments, feedback methods across trajectories or rankings, and optimizers (Christiano ., 2017; Wirth ., 2017). Recently, developments in RLHF have been centered around its use with LLMs. This branch of study originated with work exploring how technical value alignment may scale with learned reward models (Leike ., 2018). Quoting Leike . ( 2018): We claim that the approach described is agnostic to the ethical paradigm, the userâs preferences, and the legal or social framework, provided we can supply enough feedback (though the preference payload might influence the amount of feedback required). The organization that builds applications where RLHF is used bears the burden of specifying the ethics they used and answering questions about whose preferences are included and how theyâre weighed (Prasad, 2018; Baum, 2020). The development of these methods has accelerated markedly, with many variations on the methodology for integrating feedback into language models (Fernandes ., 2023). Initial work on RLHF for LLMs utilized user choices from a batch of 4 completions (Ziegler ., 2019) to train a reward model across general LLM benchmarks. When comparing recent RLHF work to Ziegler . ( 2019), group preferences were changed to pairwise preferences, and rather than general benchmarks the reward model was focused on the task of summarization (Stiennon ., 2020; J. Wu ., 2021). Next emerged general question-answering models (Ouyang ., 2022) and web crawling agents (Nakano ., 2021), primarily from scaling the initial model and human datasets. Now, RLHF is used to train general chat models across a variety of tasks (Bai, Jones ., 2022; Schulman ., 2022; Touvron ., 2023) and for specific objectives such as harm reduction (Glaese ., 2022) or information accuracy (Menick ., 2022), but methods for collecting the feedback data (from both humans and LLMs) are still burdened by disagreement and other technical challenges (Bansal ., 2023). 2.2 Downstream impacts of optimizing preferences Research venues have encouraged scientists to grapple with these questions around their work and data enrichment through humans, which are particularly relevant for techniques similar to RLHF, but there has been mixed uptake (Hawkins Mittelstadt, 2023). RLHF faces many challenges with its integration of human preferences in an aggregate manner, and potential solutions involving personalization of preferences raise further questions of which values or norms are acceptable to encode in a model (Kirk ., 2023). Specifically, the reward models trained for RLHF are known to be over-optimized during the RL stage, where the language generations continue to shift without the reward model indicating a higher score, without clear measurement of how downstream training signals for LLMs relate to preferences being correlated in the data (Gao ., 2022). Training models based on human preferences also impacts how users interact with the downstream machine learning systems that refer to RLHF reward models as part of their optimization objective. This was illustrated with the launch and widespread adoption of ChatGPT, raising questions regarding the effects of regular communication with RLHF trained LLMs, such as the downstream impact on usersâ moral judgements (KrĂźgel ., 2023) or exposure to value judgements and biases (Johnson ., 2022). Additionally, there are open questions about the stability and robustness of RLHF-trained LLMs, with reports of RLHF modelsâ tone shifting substantially within the course of a single conversation in potentially troubling ways (Nardo, 2023). The issue of downstream model impacts is not new - for instance there is a vast prior literature on how models interface with society. For example, user facing recommendation models have long prompted inquiry around whether agents should respond to our stated or implied preferences (Milli ., 2017). In RLHF, these concerns meet the âreward hackingâ problem endemic to RL. Specifically, as popular models are being tuned based on user experiences, complex feedback dynamics can emerge via the combination of reward mis-specification with the power of RL optimizers (Gilbert, Dean, Zick Lambert, 2022), such as desired capabilities coming and going through repeated training. 3 The Origins of Reward Models: Costs vs. Rewards vs. Preferences In this section, we break down the complex history inspiring the modern use of RLHF. This requires investigation into the intellectual foundations of quantifying human values, reinforcement learning and optimality, as well as behavioral economics as it relates to measuring preferences. The notion of using reinforcement learning to optimize a reward model of preferences combines the history of various once-distanced fields into an intimate optimization built on variegated assumptions about human nature. A high level timeline illustrating the history of this foundational content is shown in Fig. 1. The detailed presumptions and assumptions we reference, are showcased in Fig. 2. Our goal is to unspool the types of uncertainty that designers have grafted to system architectures at various stages of their intellectual history. Modern problem specifications have repeatedly stepped away from domains where optimal solutions are possible and deployed under-specified models as approximate solutions. Throughout, we distinguish between a series of assumptions accepted within theoretically-grounded academic literatures, and relevant presumptions which are common methods of practice for particular subject areas. As we shall see, the unresolved tensions between these assumptions and presumptions are responsible for the current state and outstanding questions of RLHF research. This section does not set out to be a survey but rather interrelates core references to illustrate the modus operandi of RLHF and preference modeling. To begin, all of the following operates on the assumptions that human preferences exist in any form, which emerged in early philosophical discussions, such as Aristotleâs Topics, Book Three. Assumption 1. Human preferences and goals exist. Figure 1: The timeline of the integration of various subfields into the modern version of RLHF. The direct links are continuous developments of specific technologies, and the arrows indicate motivations and conceptual links. 3.1 Specifying objectives: from logic of utility to reward functions The optimization of RLHF explicitly relies only on reward models. In order to use rewards as an optimization target, RLHF presupposes the convergence of ideas from preferences, rewards, and costs. Models of preference, reward functions, and cost landscapes all are tools used by different fields to describe a notion of relative goodness of specific actions and/or states in the domain. The history of these three framings dates back to the origins of probability theory and decision theory. In 1662, The Port Royal Logic introduced the notion of decision making quality (Arnauld, 1662): To judge what one must do to obtain a good or avoid an evil, it is necessary to consider not only the good and evil in itself, but also the probability that it happens or does not happen. This theory has developed along with modern scientific thinking, starting with Benthamâs utilitarian Hedonic Calculus, arguing that everything in life could be weighed (Bentham, 1823). The first quantitative application of these ideas emerged in 1931 with Ramseyâs Truth and Probability (Ramsey, 2016). Assumption 2. Any and all preferences and goals can be quantified and measured. Since these works, quantifying, measuring, and influencing human preferences has been a lively topic in the social and behavioral sciences. These debates have rarely been settled on a theoretical level; rather, different subfields and branches of social science have reached internal consensus on methods and approaches to preference measurement even as they have specialized relative to each other, often developing their own distinct semantics in the process. A minority of economists posit that preferences, if they do exist, are prohibitively difficult to measure because people have preferences over their own preferences, as well as each othersâ preferences (Hirschman, 1984). In this view, which is not reflected in the RLHF process, individual preferences are always embedded within larger social relations, such that the accuracy of any preference model is contingent on the definition and context of the task. Some behavioral economists have even argued that preferences donât existâthey may be less an ontological statement of what people actually value than a methodological tool for indirectly capturing psychological predispositions, perceived behavioral norms and ethical duties, commitments to social order, or legal constraints (Hadfield Weingast, 2014). We address the links of this work to the Von Neumann-Morgenstern (VNM) utility theorem and countering impossibility theorems around quantifying preference in Sec. 3.3. On the other hand, the reinforcement learning optimization methods used today are conceptualized around optimizing estimates of reward-to-go in a trial (Sutton Barto, 2018), which combines the notion of reward with multi-step optimization. The term reward emerged from the study of operant conditioning, animal behavior, and the Law of Effect (Thorndike, 1927; Skinner, 2019), where a reward is a scale of âhow good an action isâ (higher means better). Reward-to-go follows the notion of utility, which is a measure of rationality (Briggs, 2014), modified to measure or predict the reward coming in a future time window. In the context of the mathematical tools used for reinforcement learning, utility-to-go was invented in control theory, specifically in the context of analog circuits in 1960 (Widrow Hoff, 1960). These methods are designed around systems with clear definitions of optimality, or numerical representations of goals of an agent. Reinforcement learning systems are well known for their development with a discount factor, a compounding multiplicative factor, Îłâ[0,1]01Îłâ[0,1]Îł â [ 0 , 1 ], for re-weighting future rewards. Both the original optimal control systems stand and early algorithms for reward stand in heavy contrast to reward models that aggregate multimodal preferences. Specifically, RL systems expect rewards to behave in a specific manner, quoting Singh . ( 2009): Rewards in an RL system correspond to primary rewards, i.e., rewards that in animals have been hard-wired by the evolutionary process due to their relevance to reproductive success. ⌠Further, RL systems that form value functions, ⌠effectively create conditioned or secondary reward processes whereby predictors of primary rewards act as rewards themselves⌠The result is that the local landscape of a value function gives direction to the systemâs preferred behavior: decisions are made to cause transitions to higher-valued states. A close parallel can be drawn between the gradient of a value function and incentive motivation (McClure ., 2003). To summarize, rewards are used in RL systems as a signal to tune behavior towards clearly defined goals. The core thesis is that an learning algorithmâs performance is closely coupled with notions of expected fitness, which permeates the popular view that RL methods are agents that act in environments. This view is linked to the development of reinforcement learning technology, exemplified by claims of the general usefulness of the reward formulation (Silver ., 2021), but is in conflict when many individual desires are reduced to a single function. Assumption 3. Increasing the score of raw reward measurements corresponds to better behaviors (or value functions learned under invariant reward transformation (Ng ., 1999)). 3.2 Implementing optimal utility Modern reinforcement learning methods depend strongly on the Bellman equation (Bellman, 1957; Howard, 1960) to recursively compute estimates of reward-to-go, derived within closed environments that can be modeled as a Markov Decision Process (MDP) (Sutton Barto, 2018). These origins of RL are inspired by dynamic programming methods are were developed solely as optimal control techniques (i.e. RL did not yet exist). The MDP formulation provides theoretical guarantees of performance by structuring the environment as one with a non-changing distribution of state-actions. Assumption 4. Optimal solutions to reward maximization problems exist. The term reinforcement, coming from the psychology literature, became intertwined with modern methods afterwards in the 1960s as reinforcement learning (Mendel McLaren, 1970; Waltz Fu, 1965). Early work reinforcement learning utilized supervised learning of reward signals to solve tasks. Work from Harry Klopf reintroduced the notion of trial-and-error learning (Klopf, 1972), which is crucial to success the field saw in the 1980s and on. Modern RL algorithms build within this formulation of RL as a tool to find optimal behaviors with trial-and-error, but under looser conditions. The notion of temporal-difference (TD) learning was developed to aid agents in both the credit assignment and data collection problems, by directly updating the policy as new data was collected (Sutton, 1988), a concept first applied successfully to Backgammon (Tesauro ., 1995) (rather than updating from a large dataset of cumulative experience, which could be outdated via erroneous past value predictions). The method Q-learning, the basis for many modern forms of RL, learns a model via the Bellman equation that dictates how useful every state-action pair is with a TD update (Watkins Dayan, 1992)111The term âQâ is used in Q-learning to refer to a technical concept the Q-function, which maps from any state-action to a scalar estimate of future reward. A value-function maps from states to this same estimate.. Crucially, these notions of provable usefulness through utility have only been demonstrated for domains cast as MDPs or addressed in tasks with a single closed-form reward function, such as prominent success in games with deep learning (DQN) (Mnih ., 2013). Deep learning allowed the methods to ingest more data and work in high dimensionality environments. As the methods became more general and successful, most prominent developments before ChatGPT had remained motivated within the context of adaptive control, where reward and cost functions have a finite notion of success (Golnaraghi Kuo, 2017), e.g. a minimum energy consumption across an episode in a physical system. Prominent examples include further success in games (Silver ., 2017), controlling complex dynamic systems such as nuclear fusion reactors (Degrave ., 2022), and controlling rapid robotic systems (Kaufmann ., 2023). Most reward or cost functions can return an explicit optimal behavior, whereas models of human preferences cannot. Presumption 1. Optimal solutions can be achieved with finite data in complex environments. Given the successes of deep RL, it is worth noting that the mechanistic understanding of how the methods succeed is not well documented. The field is prone to mistakes of statistical analysis as the methods for evaluation grow more complex (Agarwal ., 2021). In addition, there is little mention of the subfield of inverse reinforcement learning (IRL) in the literature of RLHF. IRL is the problem of learning a reward function based on an agentâs behavior (Ng ., 2000) and highly related to learning a reward model. This primarily reflects the engineering path by which a stable approach to performing RLHF emerged, and motivates further investment and comparison to IRL methods to scale them to the complexity of open-ended conversations. Figure 2: The history covered in Sec. 3 that creates the assumptions and presumptions central to the current deployments of RLHF. The assumptions indicate core theoretical foundations which RLHF builds upon, transposes, prioritizes, or defers to another development stage. The presumptions represent ideas and practices required to build the current renditions of the technology. 3.3 Steering preferences The context in which reinforcement learning was designed means that rewards and costs are assumed to be stable and determinative. Both rewards and costs are expected to be functions, such that if the agent is in a specific state-action pair, then it will be returned a certain value. As we move into preferences, this is no longer the case, as human preferences constantly drift temporally throughout their experiences. The overloading of the term âvalueâ within these two contexts complicates the literature of RLHF that is built on the numerical value updates in Bellman equations with the very different notion of what is a human value, which often refers to moral or ethical principles, but is not well defined in technical literature. An example of where this tension can be seen is how reward models are attempting to map from the text on the screen to a scalar signal, but in reality, dynamics not captured in the problem specification influence the true decision (Salha, 2011; Gilbert, Dean, Zick Lambert, 2022), such as preference shift when labeling many examples sequentially and assuming they are independent. Therein, modeling preferences is at best compressing a multi-reward environment to a single function representation. In theory, the Von Neumann-Morgenstern (VNM) utility theorem gives the designer license to construct such functions, because it ties together the foundations of decision theory under uncertainty, preference theory, and abstract utility functions (Von Neumann Morgenstern, 1947); together, these ideas allow preferences to be modeled in terms of expected value to some individual agent. The MDP formulation used in most RL research has been shown in theory to be modifiable to accommodate the VNM theorem (Pitis, 2019), but this is rarely used in practice. Specifically, the Markovian formulation is limited in its expressivity (Pitis, 2023) and the transition to partially-observed processes, which is needed for language, further challenges the precision of problem specification (Abel ., 2021). However, the VNM utility theorem also invokes a number of assumptions about the nature of preferences and the environment where preferences are being measured that are challenged in teh context of RLHF. Human-computer interaction (HCI) researchers, for example, have emphasized that any numerical model of preference may not capture all the relevant preferences of a scenario. For example, how choices are displayed visually influences peopleâs preferences (Salha, 2011). This means that representing preferences may be secondary to how that representation is integrated within a tool available for people to use. Work from development economics echoes this notion, showing that theories of revealed preferences may just recapitulate Humeâs guillotine (you canât extract an âoughtâ from an âisâ), and in particular the difference between choice (what do I want?) and preference (is X better than Y?) (Sen, 1973). On a mathematical level, well-known impossibility theorems in social choice theory show that not all fairness criteria can be simultaneously met via a given preference optimization technique (Arrow, 1950; Maskin Sen, 2014). Theoretical challenges to these theorems exist, for example by assuming that interpersonal comparison of utility is viable (Harsanyi, 1977). That assumption has inspired a rich line of work in AI safety and value alignment inspired by the principal-agent problem in behavioral economics (Hadfield-Menell ., 2016), and may even include multiple principals (Fickinger ., 2020). However, the resulting utility functions may come into tension with desiderata for corrigibility, i.e. an AI systemâs capacity to cooperate with what its creators regard as corrective interventions (Soares ., 2015). Philosophers have also highlighted that preferences change over time, raising fundamental questions about personal experiences, the nature of human decision-making, and distinct contexts (Pettigrew, 2019). These conflicts around the preference aggregation across people, places, or diverse situations is central to modern RLHF dataset engineering. In practice, the VNM utility theorem ignores the possibility that preferences are also uncertain because of the inherently dynamic and indeterminate nature of valueâhuman decisions are shaped by biology, psychology, culture, and agency in ways that influence their preferences, for reasons that do not apply to a perfectly rational agent. As a result, there are a variety of paths through which theoretical assumptions diverge in practice: ⢠measured preferences may not be transitive or comparable with each other as the environment where they are measured is made more complex; ⢠proxy measurements may be derived from implicit data (page view time, closing tab, repeating question to language model), without interrogating how the measurements may interact with the domain theyâre collected in via future training and deployment of the model; ⢠the number and presentation of input sources may vary the results, e.g. allowing respondents to choose between more than two options, or taking in inputs from the same user at multiple times or in multiple contexts; ⢠relatively low accuracy across respondents in RLHF training data, which may mask differences in context between users that the preference model can aggregate or optimize without resolving. Presumption 2. The temporal- and context-shifting of user preferences does not mitigate the effectiveness of reward functions or notions of optimal utility as an optimization target. Figure 3: Illustrating the training process of an RLHF language reward model. A language model generates text from a distribution of prompts to optimize human preferences for, then humans assign a preference order to them (normally pairwise preferences). These preferences are used to train a score function via a contrastive loss shown in Eq. 1. 4 Background We continue to use assumptions of the literature, grounded in theoretical backing of a subject area, and presumptions, which are commonly accepted methods of practice, to identify blind spots and open questions in reward modeling. 4.1 Reward models of human preferences Data collection Reward models are trained with human preference data collected over a distribution of graphical user interfaces. The most common task is to give the model a prompt, i.e a request or instruction, and rate the completion, or answer. Implementation methods vary; some collect rankings from groups of responses (Ziegler ., 2019), others collect scores and rankings of a group of candidate responses (Ouyang ., 2022) (scores of 1-5 for 6+ outputs then ranking all), yet others present a choice between a pair of responses (Bai, Jones ., 2022) (choose best response between two options), and more remain (Z. Wu ., 2023). Pairwise preferences is often described as the base method (Christiano ., 2017), derived from the Bradley-Terry model (Bradley Terry, 1952). The workers employed are generally given detailed instructions on which styles, occurrences, or values to prioritize in their labels. This data is collected from groups of individuals and sometimes calibrated across per-labelor distributions, but that decision making process is not always documented or shared. In each batch of rankings for a prompt-completion pair, the pairs are often binarized into a chosen element and rejected element (rather than using the corresponding delta as a training objective) to create easy-to-optimize training data. This binarization of chosen and rejected is aggregated independently to measurements of the difficulty of the prompt or mean quality of the responses. Presumption 3. Pairwise preferences can suitably perform as a basis of human values. Model training The reward models trained for RLHF are most often trained as classifiers between a chosen and rejected completion to a prompt before optimizing with RL where they return a scalar value for each piece of text. The loss function of reward models is formulated as a difference between the score for each binarized preference. Given two options for a completion y from a prompt x, and the scores they obtain a scalar output r from a value head of a language model, the loss for the reward model is as follows (Askell ., 2021; Ouyang ., 2022): L=logâ˘(1+erchosenârrejected)log1superscriptsubscriptchosensubscriptrejectedL=log (1+e^r_chosen-r_rejected )L = log ( 1 + eitalic_rchosen - rrejected ) (1) The loss function is designed to increase the distance between the two samples, where variations exist including losses of 4 samples rather than a pair (Ziegler ., 2019), updating the model with batches of pairwise labels on a given prompt (Ouyang ., 2022), or optimizing based on the margin between rchosensubscriptchosenr_chosenrchosen and rrejectedsubscriptrejectedr_rejectedrrejected (Touvron ., 2023). There are multiple popular ways for adapting base language models into reward models. In earlier work, a value head was appended to the model to predict these rewards. A value head can be as simple as adding a linear layer with one output node after the attention blocks of a Transformer model. Today, some work includes an entire copy of the base language model with modified output layers for predicting rewards. Other reward models are trained by removing layers from a base model and/or freezing original model parameters during training. The reward model training component of RLHF is shown in Fig. 3. Presumption 4. Multiple user preferences are successfully represented in training one model by aggregating and comparing individual utilities. Note: Presumption 4 is the convergence of both training practices and motivation of preference models, as we addressed in Sec. 3.3 where economists disagree as to whether preferences can be aggregated. As we discuss, some economists construe this aggregation of preferences as an assumption core to the field, but the practice in RLHF is looser in its rigor due to its evolution from the MDP to the partially observed setting. Presumption 5. The only preferences embedded in the model are from the specifically collected training data. 4.2 Reinforcement Learning on Language Language generation optimized via reinforcement learning, which RLHF is a version of, can be formalized as a partially observable Markov decision process (POMDP) (Spaan, 2012). We define a POMDP âłMM at a per-token level with âł=(,,,,,Îź0,â,Îł)âłsubscript0âM=(S,A,O,T,Z, _% 0,R,Îł)M = ( S , A , O , T , Z , Îź0 , R , Îł ). Here, the state of the system is stâsubscripts_t _t â S, which the agent receives as an observation htâsubscriptâh_t _t â O. The observation is a history of tokens ht=t0,t1,âŚ,tâ1subscriptâsubscript0subscript1âŚsubscript1h_t=\t_0,t_1,âŚ,t_t-1\hitalic_t = t0 , t1 , ⌠, titalic_t - 1 and the action space is the possible set of next-tokens in the vocabulary of the policy model at=tâsubscriptsubscripta_t=t_t _t = titalic_t â A, including the end-of-sequence token aendsubscriptenda_endaend. As in a traditional MPD, TT is the transition function (â |st,at)T(¡|s_t,a_t)T ( â | sitalic_t , aitalic_t ). The goal of the RLHF process is to learn a policy that maps Ď:âŚâ˘():maps-toĎ:O (A)Ď : O ⌠P ( A ). This is done with the reward model, which acts as a reward function Râ˘(st,at)âŚâmaps-tosubscriptsubscriptâR(s_t,a_t) ( sitalic_t , aitalic_t ) ⌠R, used after each sequence is generated. The full sequence, until end-of-sequence token aendsubscriptenda_endaend, is called the action and used to get a scalar reward rtsubscriptr_tritalic_t from the reward model. The rewards for the batch are used for the RL update, where a popular algorithm is Proximal Policy Optimization (Schulman ., 2017). In RLHF, the discount factor of reward is set to 1 and no further actions are taken until the end of a sequence, casting the problem as contextual bandits. Presumption 6. User preferences are extracted uniformly via the RLHF process. 5 Questions In this section, we propose a series of investigations to disentangle the diverse assumptions around using large models of human preferences built on LLMs within the RLHF framework. We focus on the core sections of a machine learning solution: data, model, and the optimization choices that dictate which priorities are encoded in contemporary reward models. The questions are connected to one or more assumptions or presumptions presented in Sec. 3 and Sec. 4. 5.1 Model Questions The reward models of human preference used in RLHF today are built on base, generative LLMs, so issues from the full design and training process of these models applies to reward models. This includes the wide literature on potential sources of harm from LLMs, including a large swath of social biases, such as anti-Muslim bias (Liang ., 2021; Abid ., 2021). Those pretraining LLMs should reckon with the design choices of the base model including environmental costs, training datasets, stakeholder involvement, and more (Bender ., 2021). Base model capability distributions should be tested against, but often are not for engineering and competitive reasons. Given this, two sets of evaluations should be included in the RLHF process: ⢠Base model biases (for Presumption 5): How do different base LLMs used in the reward model training process influence all aspects of reward models? How does this impact training numerical stability, efficiency, training or test loss, and more? Is it possible that many design choices in RLHF are based on a static model and could be solved with a change of base model, as opposed to increased data and optimization? This links to all the questions we ask in the next subsection â ultimately, how does the base model perform as an independent variable? ⢠Sequential model evaluation in RLHF (for Presumption 5, 6): When a reward model is trained, it is making subtle changes to the parameters of a LLM. This is done by appending parameters in parallel, i.e. re-routing the model, so the text that the reward model produces could be observed. How do the biases of a pretrained base model, a downstream RLHF model, and the proxy of an intermediate reward model showcase how RLHF modifies representations of LLMs? Given the common implementation technique of appending a value head to a base LLM and fine-tuning to construct a reward model, the reward models can still generate text that can indicate if training is operating as intended. The base model could also impact which data from the preferences dataset is learned during the RLHF process. 5.2 Data Questions The data used to enable RLHF is often curated by multiple stakeholders in a combination of paid employment and consumer usage. This data, representing a preference between two pieces of text in an individual instance, is capturing a broad and diverse function via extremely limited interactions. Given that the data is sparse in count relative to the complexity it begins to represent, more questions should be openly shared about its curation and impacts. Currently, datasets for the most popular LLMs are being generated by professional work-forces. This opens up many questions around who is creating the data and how the context of their workplace informs it, including: ⢠Data collection contexts (for Presumption 2): Can data involving preferences collected in a professional setting mirror the intent of researchers designing an experiment or provide suitable transfer to downstream users? How does this compare to volunteer workers? How does context inform preferences, how does this data impact a downstream model, how can the impact of a user interface be measured in data? How does repetitive labeling of preference data shift oneâs preferences? Do professional crowd-workers, instructed to follow a set of preferences, follow the instructions or their innate values? ⢠Type of feedback (for Presumption 3): Does the default operating method of RLHF, pairwise preferences capture preferences in its intended form? Can comparisons in RLHF across the same data be made with the default comparisons versus advanced multi-axes feedback mechanisms (Z. Wu ., 2023)? What types of comparisons would reflect how humans communicate preferences in text? ⢠Population demographics (for Presumption 4): Who is completing the data? Is a diverse population maintained? How does a lack of diversity emerge as measurable impacts on the model? What is a minimum number of people required to suitably represent a given population? How are instances of preference annotator disagreement treated â as a source of noise, or a signal? 5.3 Optimization questions The most elusive and potentially insightful questions with respect to reward models and preference data emerge where preferences are extracted by reinforcement learning optimizers and distilled into the LLM. In this vein, further questions should be asked as to whether the outcomes of the optimization match the intended design of the process. Given the complexity of LLMs and the social values of using human data to empower technology, clear attribution is needed for the role of the preference data in the optimization. Questions that can be used to investigate these issues include: ⢠RL optimization of reward model (for Assumption 2, 3, 4, Presumption 1): at a technical level, is the reward signal from the reward model for preference maximized? Should it be maximized? Does this reflect changes in line with the preference data (e.g. is this achieved at the same point where evaluation accuracy peaked). In the RL literature, sharing learning curves is common practice, yet in RLHF it is often omitted, with just the final test set accuracy included. Correlating these common metrics is important to understanding the role of the reward model to the process. ⢠Qualitative alignment (for Assumption 2, Presumption 2, 4, 6): Given the RLHF data process starts with written instructions to data-workers, checks should be done to identify if the model changes align with the intended goals of stated preferences. A suitable method for continued deployment to iteratively refine training and deployment goals would be documentation for RL systems, Reward Reports (Gilbert, Dean, Lambert ., 2022). Beyond this, feedback from annotators, data analysis, and more can quickly provide feedback onto the impacts of RLHF relative to human inputs and intentions. ⢠Weighing preferences (for Presumption 6): Does and should the model uniformly extract information from the dataset? How are different scenarios in the dataset equated (e.g. instructions around healthcare being treated the same as food choices)? Should different reward functions derived in preference literature be used to maintain an inductive bias over preferences, such as Plackett-Luce (Cheng ., 2010). Recent literature shows that aggregate agreement between humans and the reward models varies from approximately 60 to 75% (Stiennon ., 2020; Ouyang ., 2022; Bai, Jones ., 2022), should certain examples be prioritized given an non-uniform distribution of tasks and annotators? Or, can these distributions be broken down by commonly accepted categories such as reasoning, generation, etc.? Such reduction to the mean across groups can squash multi-modal beliefs, especially with underrepresented groups (Prabhakaran ., 2021; Feffer ., 2023). 6 Solutions Evaluation of reward model capabilities The most common evaluation technique for reward models is to hold out data from the training set and measure the final modelâs agreement with the data, but this does not control for the complex nature of the dataset or encourage cross-intuition comparisons. The evaluation score is unlikely to illuminate any conflicts in the data that arise from aggregating over inputs from multiple individuals or contexts. This necessitates building more controlled examples to interpret whether the objective of differentiating better from worse text is achieved. Taking inspiration from recent advancements in evaluating instruction- and chat-tuned LLMs (Zheng ., 2023), where the existing academic benchmarks from the natural language processing literature were incomplete for evaluating modern models like GPT4 (Chang ., 2023), the community should evaluate reward models in a manner analogous to their use. One specific example exists in recent work, a benchmark to evaluate reward model consistency (Shen ., 2023), as its ability to maintain consistent scores over changes to text that do not alter meaning, but many more tools are needed to cover the many use cases of RLHF models. Similar to MT Bench (Zheng ., 2023), a benchmark for LLM capabilities over multi-turn conversations, we can construct representative examples across a variety of categories for the model (e.g. reasoning, generation ability, factuality, etc.) to evaluate the reward signal. Evaluation of reward model safety The primary means for evaluating the safety, toxicity, and potential harms of LLMs is a process known as red-teaming, which entails focused attempts to prompt the language model into a broad array of harmful behaviors (Ganguli ., 2022). Red-teaming for generative models is primarily separated into two axes, attack vectors and harms. The notion of red-teaming for reward models is simpler and less interactive, as the output of the model is a score rather than text to read and comprehend. To do so, the score of reward models across a series of text documents or snippets that have the potential for harm should be compared to âneutralâ texts. Second, reward models should be evaluated for adversarial input strings, e.g. sequences of text that produce unexpectedly high or low scores, which could indicate downstream exploitation during the RL step. As a starting point, practitioners could consider the efforts to identify adversarial prompts that escalate toxicity (but need not necessarily do so) in the evaluation of LLM generations, for instance as in done in the RealToxicityPrompts dataset (Gehman ., 2020). Today, a third-party researcher can play with a closed model and find an adversarial attack even if no red-teaming protocols are shared about the system. The same is not true for a private reward model, which may be re-used in the future to train the next model. Sociotechnical specification of preference The current manner by which reward models are used in RLHF follows the logic of the following statement: the reward model is deemed good when it is good for the downstream task. Given the complex intellectual origins of quantifying and optimizing preferences, specific recommendations should be made around what information a model of human preferences should and should not encode. Doing so may reduce the performance of models in the short term, but increases the ability for multi-stakeholder engagement and reduction of harms with the development of LLMs. 7 Discussions Downstream use of models The final manner by which the assumptions of RLHF are tested is in the increasing alignment of LLM training with the intentions of downstream users, as is the case for ChatGPT (Schulman, 2023). LLMs are technologies used in consumer applications, giving different context and broader use-cases than the limited training data. The developers of RLHF should consider how end user encounters with their model may be different than the preferences collected for training, such as the shift from primarily English training data to multilingual users. Given the direct manner by which human preferences are encoded in models trained with RLHF, users of these models should be presented with the intended preferences of the models for transparency. A potential improved solution is to let users choose a model matching their preferences, but this comes with many technical challenges. Emerging examples of LLM âhubâ interfaces that offer users a menu of LLMs to choose from (like the open-source GPT4All222https://gpt4all.io/index.html, or Metaâs new AI companion character suite333https://about.fb.com/news/2023/09/introducing-ai-powered-assistants-characters-and-creative-tools) offer hope that such approaches might in-fact be tractable and/or accessible to users. Synthetic preference data Recent work on Reinforcement Learning from AI Feedback (RLAIF) uses LLMs in addition to humans to provide critiques of other LLM generations for the RLHF process (Bai, Kadavath ., 2022; Lee ., 2023; Wang ., 2023). Further, these methods are referenced as methods for removing the human bottleneck from alignment (Sutskever, 2023) without existing literature on how methods such as RLHF and reward models succeed in capturing values. Despite the practical attractiveness due to the cost of acquiring human data, synthetic preference data presents different challenges. Preliminary findings have indicated that synthetic supervision data for LLMs can lead to generation instability and lack of robustness (Alemohammad ., ; Gudibande ., ; Shumailov ., ), to say nothing of the potential for further shifting preferences away from the original human data. Direct preference optimization Recent work has shown that direct optimization can be used to extract information from human preference data (Rafailov ., 2023), rather than using an intermediate reward model. The removal of the reward model would make the encoding of values more opaque in the RLHF process, yet at the same time potentially reduce problem misspecification and exploitation by reducing the number of training steps. The questions suggested in this work would still apply, but the methods for auditing them will be different without a distinct reward model. 8 Conclusion Given the importance of RLHF to the deployment and integration of state-of-the-art LLMs, a complete understanding of the motivation, intellectual foundations, and implementation presumptions of reward models is crucial to the safe and equitable rollout of this technology. With the evolution from utility quantification to reinforcement learning to maximizing average preference scores, we have shown that the assumptions at each step are not in-step with how the technology is used in contemporary settings. This paper details the intellectual basis of learning reward models, and argues that more information about existing reward models should be shared with the public via open or partially-open releases. With more development and sharing of reward models, proper evaluation techniques of the models can be designed, which will lead to further technological developments around desired properties such as user tunability, uncertainty quantification, etc. By doing this work, we can better understand the distinct challenges of learning models from incomplete data of human preferences and then optimizing those models to elicit specific behaviors. Acknowledgements Thanks to Aaron Snoswell for providing edits and comments on this work. Credit to Sasha Luccioni and Deep Ganguli for discussions and encouragement towards completing this paper. Credit to Silviu Pitis for providing substantial feedback on the first version of the paper, improving the robustness and depth of the argument. Thanks to Kristian Lum and Dylan Hadfield-Menell for substantial feedback on the early versions of this paper. References Abel . ( 2021) 2021expressivityAPACrefauthorsAbel, D., Dabney, W., Harutyunyan, A., Ho, M ., Littman, M., Precup, D. Singh, S. 2021. the expressivity of markov reward On the expressivity of markov reward. in Neural Information Processing Systems347799â7812. Abid . ( 2021) 2021persistentAPACrefauthorsAbid, A., Farooqi, M. Zou, J. 2021. anti-muslim bias in large language models Persistent anti-muslim bias in large language models. of the 2021 AAAI/ACM Conference on AI, Ethics, and Society Proceedings of the 2021 aaai/acm conference on ai, ethics, and society ( 298â306). Agarwal . ( 2021) 2021deepAPACrefauthorsAgarwal, R., Schwarzer, M., Castro, P ., Courville, A . Bellemare, M. 2021. reinforcement learning at the edge of the statistical precipice Deep reinforcement learning at the edge of the statistical precipice. in neural information processing systems3429304â29320. Alemohammad . ( ) 2023APACrefauthorsAlemohammad, S., Casco-Rodriguez, J., Luzi, L., Humayun, A ., Babaei, H., LeJeune, D. , R . . -Consuming Generative Models Go MAD. Self-Consuming Generative Models Go MAD. APACrefURL [2023-08-01]http://arxiv.org/abs/2307.01850 APACrefDOI 10.48550/arXiv.2307.01850 Arnauld ( 1662) 1861portAPACrefauthorsArnauld, A. 1662. Port-Royal Logic The port-royal logic. Arrow ( 1950) 1950difficultyAPACrefauthorsArrow, K . 1950. difficulty in the concept of social welfare A difficulty in the concept of social welfare. of political economy584328â346. Askell . ( 2021) 2021generalAPACrefauthorsAskell, A., Bai, Y., Chen, A., Drain, D., Ganguli, D., Henighan, T. 2021. general language assistant as a laboratory for alignment A general language assistant as a laboratory for alignment. preprint arXiv:2112.00861. Bai, Jones . ( 2022) 2022trainingAPACrefauthorsBai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., DasSarma, N. 2022. a helpful and harmless assistant with reinforcement learning from human feedback Training a helpful and harmless assistant with reinforcement learning from human feedback. preprint arXiv:2204.05862. Bai, Kadavath . ( 2022) 2022constitutionalAPACrefauthorsBai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A. 2022. AI: Harmlessness from AI Feedback Constitutional ai: Harmlessness from ai feedback. preprint arXiv:2212.08073. Bajak ( 2023) APACrefauthorsBajak, F. 2023. red-teaming A.I. are âbreaking stuff left and right,â but donât expect quick fixes from DefCon: âThere are no good guardrailsâ. Hackers red-teaming a.i. are âbreaking stuff left and right,â but donât expect quick fixes from defcon: âthere are no good guardrailsâ. ://fortune.com/2023/08/13/hackers-red-teaming-ai-defcon-breaking-stuff-but-no-quick-fixes/. : 2023-10-03 Bansal . ( 2023) 2023peeringAPACrefauthorsBansal, H., Dang, J. Grover, A. 2023. Through Preferences: Unraveling Feedback Acquisition for Aligning Large Language Models. Peering through preferences: Unraveling feedback acquisition for aligning large language models. Baum ( 2020) 2020socialAPACrefauthorsBaum, S . 2020. choice ethics in artificial intelligence Social choice ethics in artificial intelligence. & SOCIETY351165â176. Bellman ( 1957) 1957markovianAPACrefauthorsBellman, R. 1957. Markovian decision process A markovian decision process. of mathematics and mechanics679â684. Bender . ( 2021) 2021dangersAPACrefauthorsBender, E ., Gebru, T., McMillan-Major, A. Shmitchell, S. 2021. the dangers of stochastic parrots: Can language models be too big? On the dangers of stochastic parrots: Can language models be too big? of the 2021 ACM conference on fairness, accountability, and transparency Proceedings of the 2021 acm conference on fairness, accountability, and transparency ( 610â623). Bentham ( 1823) 1823hedonicAPACrefauthorsBentham, J. 1823. Introduction to the Principles of Morals and Legislation An introduction to the principles of morals and legislation. Bradley Terry ( 1952) APACrefauthorsBradley, R . Terry, M . 1952. Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons Rank analysis of incomplete block designs: I. the method of paired comparisons. 393/4324â345. APACrefURL [2023-02-13]http://w.jstor.org/stable/2334029 Briggs ( 2014) 2014normativeAPACrefauthorsBriggs, R . 2014. theories of rational choice: Expected utility Normative theories of rational choice: Expected utility. Casper . ( 2023) 2023openAPACrefauthorsCasper, S., Davies, X., Shi, C., Gilbert, T ., Scheurer, J., Rando, J. 2023. problems and fundamental limitations of reinforcement learning from human feedback Open problems and fundamental limitations of reinforcement learning from human feedback. preprint arXiv:2307.15217. Chang . ( 2023) 2023surveyAPACrefauthorsChang, Y., Wang, X., Wang, J., Wu, Y., Zhu, K., Chen, H. 2023. survey on evaluation of large language models A survey on evaluation of large language models. preprint arXiv:2307.03109. Cheng . ( 2010) 2010labelAPACrefauthorsCheng, W., HĂźllermeier, E. Dembczynski, K . 2010. ranking methods based on the Plackett-Luce model Label ranking methods based on the plackett-luce model. of the 27th International Conference on Machine Learning (ICML-10) Proceedings of the 27th international conference on machine learning (icml-10) ( 215â222). Christiano . ( 2017) 2017deepAPACrefauthorsChristiano, P ., Leike, J., Brown, T., Martic, M., Legg, S. Amodei, D. 2017. reinforcement learning from human preferences Deep reinforcement learning from human preferences. in neural information processing systems30. Degrave . ( 2022) 2022magneticAPACrefauthorsDegrave, J., Felici, F., Buchli, J., Neunert, M., Tracey, B., Carpanese, F. 2022. control of tokamak plasmas through deep reinforcement learning Magnetic control of tokamak plasmas through deep reinforcement learning. 6027897414â419. Dobbe . ( 2021) 2021hardAPACrefauthorsDobbe, R., Gilbert, T . Mintz, Y. 2021. choices in artificial intelligence Hard choices in artificial intelligence. Intelligence300103555. Feffer . ( 2023) 2023moralAPACrefauthorsFeffer, M., Heidari, H. Lipton, Z . 2023. Machine or Tyranny of the Majority? Moral machine or tyranny of the majority? preprint arXiv:2305.17319. Fernandes . ( 2023) 2023bridgingAPACrefauthorsFernandes, P., Madaan, A., Liu, E., Farinhas, A., Martins, P ., Bertsch, A. 2023. the gap: A survey on integrating (human) feedback for natural language generation Bridging the gap: A survey on integrating (human) feedback for natural language generation. preprint arXiv:2305.00955. Fickinger . ( 2020) 2020multiAPACrefauthorsFickinger, A., Zhuang, S., Hadfield-Menell, D. Russell, S. 2020. -principal assistance games Multi-principal assistance games. preprint arXiv:2007.09540. Ganguli . ( 2022) 2022redAPACrefauthorsGanguli, D., Lovitt, L., Kernion, J., Askell, A., Bai, Y., Kadavath, S. 2022. teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned. preprint arXiv:2209.07858. Gao . ( 2022) 2022scalingAPACrefauthorsGao, L., Schulman, J. Hilton, J. 2022. Laws for Reward Model Overoptimization Scaling laws for reward model overoptimization. preprint arXiv:2210.10760. Gehman . ( 2020) 2020realtoxicitypromptsAPACrefauthorsGehman, S., Gururangan, S., Sap, M., Choi, Y. Smith, N . 2020. : Evaluating neural toxic degeneration in language models Realtoxicityprompts: Evaluating neural toxic degeneration in language models. preprint arXiv:2009.11462. Gilbert, Dean, Lambert . ( 2022) 2022rewardAPACrefauthorsGilbert, T ., Dean, S., Lambert, N., Zick, T. Snoswell, A. 2022. reports for reinforcement learning Reward reports for reinforcement learning. preprint arXiv:2204.10817. Gilbert, Dean, Zick Lambert ( 2022) 2022choicesAPACrefauthorsGilbert, T ., Dean, S., Zick, T. Lambert, N. 2022. , Risks, and Reward Reports: Charting Public Policy for Reinforcement Learning Systems Choices, risks, and reward reports: Charting public policy for reinforcement learning systems. preprint arXiv:2202.05716. Glaese . ( 2022) 2022improvingAPACrefauthorsGlaese, A., McAleese, N., TrÄbacz, M., Aslanides, J., Firoiu, V., Ewalds, T. 2022. alignment of dialogue agents via targeted human judgements Improving alignment of dialogue agents via targeted human judgements. preprint arXiv:2209.14375. Golnaraghi Kuo ( 2017) 2017automaticAPACrefauthorsGolnaraghi, F. Kuo, B . 2017. control systems Automatic control systems. -Hill Education. Gudibande . ( ) 2023APACrefauthorsGudibande, A., Wallace, E., Snell, C., Geng, X., Liu, H., Abbeel, P. , D. . False Promise of Imitating Proprietary LLMs. The False Promise of Imitating Proprietary LLMs. APACrefURL [2023-08-01]http://arxiv.org/abs/2305.15717 APACrefDOI 10.48550/arXiv.2305.15717 Hadfield Weingast ( 2014) 2014microfoundationsAPACrefauthorsHadfield, G . Weingast, B . 2014. of the Rule of Law Microfoundations of the rule of law. Review of Political Science1721â42. Hadfield-Menell . ( 2016) 2016cooperativeAPACrefauthorsHadfield-Menell, D., Russell, S ., Abbeel, P. Dragan, A. 2016. inverse reinforcement learning Cooperative inverse reinforcement learning. in neural information processing systems29. Harsanyi ( 1977) 1977ruleAPACrefauthorsHarsanyi, J . 1977. utilitarianism and decision theory Rule utilitarianism and decision theory. 11125â53. Hawkins Mittelstadt ( 2023) 2023ethicalAPACrefauthorsHawkins, W. Mittelstadt, B. 2023. ethical ambiguity of AI data enrichment: Measuring gaps in research ethics norms and practices The ethical ambiguity of ai data enrichment: Measuring gaps in research ethics norms and practices. of the 2023 ACM Conference on Fairness, Accountability, and Transparency Proceedings of the 2023 acm conference on fairness, accountability, and transparency ( 261â270). Hirschman ( 1984) 1984againstAPACrefauthorsHirschman, A . 1984. parsimony: Three easy ways of complicating some categories of economic discourse Against parsimony: Three easy ways of complicating some categories of economic discourse. of the American Academy of arts and Sciences37811â28. Howard ( 1960) 1960dynamicAPACrefauthorsHoward, R . 1960. programming and markov processes. Dynamic programming and markov processes. Johnson . ( 2022) 2022ghostAPACrefauthorsJohnson, R ., Pistilli, G., MenĂŠdez-GonzĂĄlez, N., Duran, L ., Panai, E., Kalpokiene, J. Bertulfo, D . 2022. Ghost in the Machine has an American accent: value conflict in GPT-3 The ghost in the machine has an american accent: value conflict in gpt-3. preprint arXiv:2203.07785. Kaufmann . ( 2023) 2023fpvAPACrefauthorsKaufmann, E., Bauersfeld, L., Loquercio, A., MĂźller, M., Koltun, V. Scaramuzza, D. 2023. -level drone racing using deep reinforcement learning Champion-level drone racing using deep reinforcement learning. 6207976982â987. APACrefURL https://doi.org/10.1038/s41586-023-06419-4 APACrefDOI 10.1038/s41586-023-06419-4 Kirk . ( 2023) 2023personalisationAPACrefauthorsKirk, H ., Vidgen, B., RĂśttger, P. Hale, S . 2023. within bounds: A risk taxonomy and policy framework for the alignment of large language models with personalised feedback Personalisation within bounds: A risk taxonomy and policy framework for the alignment of large language models with personalised feedback. preprint arXiv:2303.05453. Klopf ( 1972) 1972brainAPACrefauthorsKlopf, A . 1972. function and adaptive systems: a heterostatic theory Brain function and adaptive systems: a heterostatic theory ( 133). Force Cambridge Research Laboratories, Air Force Systems Command, United âŚ. KrĂźgel . ( 2023) 2023chatgptAPACrefauthorsKrĂźgel, S., Ostermaier, A. Uhl, M. 2023. âs inconsistent moral advice influences usersâ judgment Chatgptâs inconsistent moral advice influences usersâ judgment. Reports1314569. Lee . ( 2023) 2023rlaifAPACrefauthorsLee, H., Phatale, S., Mansoor, H., Lu, K., Mesnard, T., Bishop, C. , A. 2023. : Scaling Reinforcement Learning from Human Feedback with AI Feedback. Rlaif: Scaling reinforcement learning from human feedback with ai feedback. Leike . ( 2018) 2018scalableAPACrefauthorsLeike, J., Krueger, D., Everitt, T., Martic, M., Maini, V. Legg, S. 2018. agent alignment via reward modeling: a research direction Scalable agent alignment via reward modeling: a research direction. preprint arXiv:1811.07871. Liang . ( 2021) 2021towardsAPACrefauthorsLiang, P ., Wu, C., Morency, L . Salakhutdinov, R. 2021. understanding and mitigating social biases in language models Towards understanding and mitigating social biases in language models. Conference on Machine Learning International conference on machine learning ( 6565â6576). Maskin Sen ( 2014) 2014arrowAPACrefauthorsMaskin, E. Sen, A. 2014. Arrow impossibility theorem The arrow impossibility theorem. University Press. McClure . ( 2003) 2003computationalAPACrefauthorsMcClure, S ., Daw, N . Montague, P . 2003. computational substrate for incentive salience A computational substrate for incentive salience. in neurosciences268423â428. Mendel McLaren ( 1970) 1970287APACrefauthorsMendel, J. McLaren, R. 1970. 8 Reinforcement-Learning Control and Pattern Recognition Systems 8 reinforcement-learning control and pattern recognition systems. J. Mendel K. Fu ( ), , Learning and Pattern Recognition Systems Adaptive, learning and pattern recognition systems ( 66, 287-318). . APACrefURL https://w.sciencedirect.com/science/article/pii/S007653920860497X APACrefDOI ://doi.org/10.1016/S0076-5392(08)60497-X Menick . ( 2022) 2022teachingAPACrefauthorsMenick, J., Trebacz, M., Mikulik, V., Aslanides, J., Song, F., Chadwick, M. 2022. language models to support answers with verified quotes Teaching language models to support answers with verified quotes. preprint arXiv:2203.11147. Milli . ( 2017) 2017shouldAPACrefauthorsMilli, S., Hadfield-Menell, D., Dragan, A. Russell, S. 2017. robots be obedient? Should robots be obedient? preprint arXiv:1705.09990. Mnih . ( 2013) 2013playingAPACrefauthorsMnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D. Riedmiller, M. 2013. atari with deep reinforcement learning Playing atari with deep reinforcement learning. preprint arXiv:1312.5602. Nakano . ( 2021) 2021webgptAPACrefauthorsNakano, R., Hilton, J., Balaji, S., Wu, J., Ouyang, L., Kim, C. 2021. : Browser-assisted question-answering with human feedback Webgpt: Browser-assisted question-answering with human feedback. preprint arXiv:2112.09332. Nardo ( 2023) 2023waluigiAPACrefauthorsNardo, C. 2023. waluigi effect (mega-post) The waluigi effect (mega-post). Wrong. APACrefURL https://w.lesswrong.com/posts/D7PumeYTDPfBTp3i7/the-waluigi-effect-mega-post : 2023-09-11 Ng . ( 1999) 1999policyAPACrefauthorsNg, A ., Harada, D. Russell, S. 1999. invariance under reward transformations: Theory and application to reward shaping Policy invariance under reward transformations: Theory and application to reward shaping. Icml ( 99, 278â287). Ng . ( 2000) 2000algorithmsAPACrefauthorsNg, A ., Russell, S. . 2000. for inverse reinforcement learning. Algorithms for inverse reinforcement learning. Icml ( 1, 2). Ouyang . ( 2022) 2022trainingAPACrefauthorsOuyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C ., Mishkin, P. 2022. language models to follow instructions with human feedback Training language models to follow instructions with human feedback. preprint arXiv:2203.02155. Pettigrew ( 2019) 2019choosingAPACrefauthorsPettigrew, R. 2019. for changing selves Choosing for changing selves. University Press. Pitis ( 2019) 2019rethinkingAPACrefauthorsPitis, S. 2019. the discount factor in reinforcement learning: A decision theoretic approach Rethinking the discount factor in reinforcement learning: A decision theoretic approach. of the AAAI Conference on Artificial Intelligence Proceedings of the aaai conference on artificial intelligence ( 33, 7949â7956). Pitis ( 2023) 2023consistentAPACrefauthorsPitis, S. 2023. Aggregation of Objectives with Diverse Time Preferences Requires Non-Markovian Rewards Consistent aggregation of objectives with diverse time preferences requires non-markovian rewards. preprint arXiv:2310.00435. Prabhakaran . ( 2021) 2021releasingAPACrefauthorsPrabhakaran, V., Davani, A . Diaz, M. 2021. releasing annotator-level labels and information in datasets On releasing annotator-level labels and information in datasets. preprint arXiv:2110.05699. Prasad ( 2018) 2018socialAPACrefauthorsPrasad, M. 2018. choice and the value alignment problem Social choice and the value alignment problem. intelligence safety and security291â314. Rafailov . ( 2023) 2023directAPACrefauthorsRafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C . Finn, C. 2023. preference optimization: Your language model is secretly a reward model Direct preference optimization: Your language model is secretly a reward model. preprint arXiv:2305.18290. Ramsey ( 2016) 2016truthAPACrefauthorsRamsey, F . 2016. and probability Truth and probability. in Formal Epistemology: Sourcebook21â45. Salha ( 2011) 2011aestheticsAPACrefauthorsSalha, N. 2011. & Art in the Early Development of Human-Computer Interfaces Aesthetics & art in the early development of human-computer interfaces . de doctorado en ingenierĂa, Universität Bremen]. https://bit. ly/3ZICKZJ. Schulman ( 2023) 2023proxyAPACrefauthorsSchulman, J. 2023. objectives in reinforcement learning from human feedback. Proxy objectives in reinforcement learning from human feedback. APACrefURL https://icml.c/virtual/2023/invited-talk/21549 Conference on Machine Learning (ICML) Schulman . ( 2017) 2017proximalAPACrefauthorsSchulman, J., Wolski, F., Dhariwal, P., Radford, A. Klimov, O. 2017. policy optimization algorithms Proximal policy optimization algorithms. preprint arXiv:1707.06347. Schulman . ( 2022) APACrefauthorsSchulman, J., Zoph, B., Kim, C. more. 2022. : Optimizing Language Models for Dialogue. Chatgpt: Optimizing language models for dialogue. ://openai.com/blog/chatgpt/. : 2023-02-12 Sen ( 1973) 1973behaviourAPACrefauthorsSen, A. 1973. and the Concept of Preference Behaviour and the concept of preference. 40159241â259. Shen . ( 2023) 2023trickleAPACrefauthorsShen, L., Chen, S., Song, L., Jin, L., Peng, B., Mi, H. , D. 2023. Trickle-down Impact of Reward (In-) consistency on RLHF The trickle-down impact of reward (in-) consistency on rlhf. preprint arXiv:2309.16155. Shumailov . ( ) 2023aAPACrefauthorsShumailov, I., Shumaylov, Z., Zhao, Y., Gal, Y., Papernot, N. Anderson, R. . Curse of Recursion: Training on Generated Data Makes Models Forget. The Curse of Recursion: Training on Generated Data Makes Models Forget. APACrefURL [2023-08-01]http://arxiv.org/abs/2305.17493 APACrefDOI 10.48550/arXiv.2305.17493 Silver . ( 2017) 2017masteringAPACrefauthorsSilver, D., Schrittwieser, J., Simonyan, K., Antonoglou, I., Huang, A., Guez, A. 2017. the game of go without human knowledge Mastering the game of go without human knowledge. 5507676354â359. Silver . ( 2021) 2021rewardAPACrefauthorsSilver, D., Singh, S., Precup, D. Sutton, R . 2021. is enough Reward is enough. Intelligence299103535. Singh . ( 2009) 2009rewardsAPACrefauthorsSingh, S., Lewis, R . Barto, A . 2009. do rewards come from Where do rewards come from. of the annual conference of the cognitive science society Proceedings of the annual conference of the cognitive science society ( 2601â2606). Skinner ( 2019) 2019behaviorAPACrefauthorsSkinner, B . 2019. behavior of organisms: An experimental analysis The behavior of organisms: An experimental analysis. Skinner Foundation. Soares . ( 2015) 2015corrigibilityAPACrefauthorsSoares, N., Fallenstein, B., Armstrong, S. Yudkowsky, E. 2015. Corrigibility. at the twenty-ninth AAAI conference on artificial intelligence. Workshops at the twenty-ninth aaai conference on artificial intelligence. Spaan ( 2012) 2012partiallyAPACrefauthorsSpaan, M . 2012. observable Markov decision processes Partially observable markov decision processes. learning: State-of-the-art Reinforcement learning: State-of-the-art ( 387â414). . Stiennon . ( 2020) 2020learningAPACrefauthorsStiennon, N., Ouyang, L., Wu, J., Ziegler, D., Lowe, R., Voss, C. , P . 2020. to summarize with human feedback Learning to summarize with human feedback. H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan H. Lin ( ), in Neural Information Processing Systems Advances in neural information processing systems ( 33, 3008â3021). Associates, Inc. APACrefURL https://proceedings.neurips.c/paper/2020/file/1f89885d556929e98d3ef9b86448f951-Paper.pdf Sutskever ( 2023) APACrefauthorsSutskever, J . 2023. Superalignment. Introducing superalignment. ://openai.com/blog/introducing-superalignment. : 2023-09-27 Sutton ( 1988) 1988learningAPACrefauthorsSutton, R . 1988. to predict by the methods of temporal differences Learning to predict by the methods of temporal differences. learning39â44. Sutton Barto ( 2018) 2018reinforcementAPACrefauthorsSutton, R . Barto, A . 2018. learning: An introduction Reinforcement learning: An introduction. press. Tesauro . ( 1995) 1995temporalAPACrefauthorsTesauro, G. . 1995. difference learning and TD-Gammon Temporal difference learning and td-gammon. of the ACM38358â68. Thorndike ( 1927) 1927lawAPACrefauthorsThorndike, E . 1927. law of effect The law of effect. American journal of psychology391/4212â222. Touvron . ( 2023) 2023llamaAPACrefauthorsTouvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y. 2023. 2: Open foundation and fine-tuned chat models Llama 2: Open foundation and fine-tuned chat models. preprint arXiv:2307.09288. Von Neumann Morgenstern ( 1947) 1947theoryAPACrefauthorsVon Neumann, J. Morgenstern, O. 1947. of games and economic behavior, 2nd rev Theory of games and economic behavior, 2nd rev. Waltz Fu ( 1965) 1965APACrefauthorsWaltz, M. Fu, K. 1965. heuristic approach to reinforcement learning control systems A heuristic approach to reinforcement learning control systems. Transactions on Automatic Control104390-398. APACrefDOI 10.1109/TAC.1965.1098193 Wang . ( 2023) 2023shepherdAPACrefauthorsWang, T., Yu, P., Tan, X ., OâBrien, S., Pasunuru, R., Dwivedi-Yu, J. , A. 2023. : A Critic for Language Model Generation Shepherd: A critic for language model generation. preprint arXiv:2308.04592. Watkins Dayan ( 1992) 1992qAPACrefauthorsWatkins, C . Dayan, P. 1992. -learning Q-learning. learning8279â292. Widrow Hoff ( 1960) 1960adaptiveAPACrefauthorsWidrow, B. Hoff, M . 1960. switching circuits Adaptive switching circuits . Univ Ca Stanford Electronics Labs. Wirth . ( 2017) 2017surveyAPACrefauthorsWirth, C., Akrour, R., Neumann, G., FĂźrnkranz, J. . 2017. survey of preference-based reinforcement learning methods A survey of preference-based reinforcement learning methods. of Machine Learning Research181361â46. J. Wu . ( 2021) 2021recursivelyAPACrefauthorsWu, J., Ouyang, L., Ziegler, D ., Stiennon, N., Lowe, R., Leike, J. Christiano, P. 2021. summarizing books with human feedback Recursively summarizing books with human feedback. preprint arXiv:2109.10862. Z. Wu . ( 2023) 2023fineAPACrefauthorsWu, Z., Hu, Y., Shi, W., Dziri, N., Suhr, A., Ammanabrolu, P. , H. 2023. -Grained Human Feedback Gives Better Rewards for Language Model Training Fine-grained human feedback gives better rewards for language model training. preprint arXiv:2306.01693. Zheng . ( 2023) 2023judgingAPACrefauthorsZheng, L., Chiang, W ., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y. 2023. LLM-as-a-judge with MT-Bench and Chatbot Arena Judging llm-as-a-judge with mt-bench and chatbot arena. preprint arXiv:2306.05685. Ziegler . ( 2019) 2019fineAPACrefauthorsZiegler, D ., Stiennon, N., Wu, J., Brown, T ., Radford, A., Amodei, D. , G. 2019. -tuning language models from human preferences Fine-tuning language models from human preferences. preprint arXiv:1909.08593. Zou . ( 2023) 2023universalAPACrefauthorsZou, A., Wang, Z., Kolter, J . Fredrikson, M. 2023. and transferable adversarial attacks on aligned language models Universal and transferable adversarial attacks on aligned language models. preprint arXiv:2307.15043.