Paper deep dive

Fine-Grained Human Feedback Gives Better Rewards for Language Model Training

Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A. Smith, Mari Ostendorf, Hannaneh Hajishirzi

Year: 2023Venue: NeurIPS 2023Area: Alignment TrainingType: EmpiricalEmbeddings: 99

Models: GPT-2 Large, Longformer-base, T5-Large

Intelligence

Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 95%

Last extracted: 3/12/2026, 8:13:31 PM

Summary

The paper introduces 'Fine-Grained RLHF', a framework for training language models that utilizes dense, category-specific human feedback rather than holistic preference signals. By providing rewards at the sentence or sub-sentence level for specific error types (irrelevance, factual incorrectness, and information incompleteness), the approach improves sample efficiency and allows for customizable model behavior in detoxification and long-form question answering tasks.

Entities (5)

Fine-Grained RLHF · framework · 100%PPO · algorithm · 100%RLHF · methodology · 100%QA-FEEDBACK · dataset · 95%RealToxicityPrompts · dataset · 95%

Relation Signals (3)

Fine-Grained RLHF → evaluatedon → QA-FEEDBACK

confidence 95% · With experiments on long-form QA, we aim to examine training models... for which we construct a long-form QA dataset, QA-FEEDBACK

Fine-Grained RLHF → utilizes → PPO

confidence 95% · We then integrate these reward models into Proximal Policy Optimization (PPO)

Fine-Grained RLHF → improves → RLHF

confidence 90% · how can we improve rewards for LM training via RLHF by using more fine-grained human feedback?

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Language models (LMs) often exhibit undesirable text generation behaviors, including generating false, toxic, or irrelevant outputs. Reinforcement learning from human feedback (RLHF) - where human preference judgments on LM outputs are transformed into a learning signal - has recently shown promise in addressing these issues. However, such holistic feedback conveys limited information on long text outputs; it does not indicate which aspects of the outputs influenced user preference; e.g., which parts contain what type(s) of errors. In this paper, we use fine-grained human feedback (e.g., which sentence is false, which sub-sentence is irrelevant) as an explicit training signal. We introduce Fine-Grained RLHF, a framework that enables training and learning from reward functions that are fine-grained in two respects: (1) density, providing a reward after every segment (e.g., a sentence) is generated; and (2) incorporating multiple reward models associated with different feedback types (e.g., factual incorrectness, irrelevance, and information incompleteness). We conduct experiments on detoxification and long-form question answering to illustrate how learning with such reward functions leads to improved performance, supported by both automatic and human evaluation. Additionally, we show that LM behaviors can be customized using different combinations of fine-grained reward models. We release all data, collected human feedback, and codes at this https URL.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

99,079 characters extracted from source content.

Expand or collapse full text

Fine-Grained Human Feedback Gives Better Rewards for Language Model Training Zeqiu Wu 1∗ Yushi Hu 1∗ Weijia Shi 1 Nouha Dziri 2 Alane Suhr 3 Prithviraj Ammanabrolu 45 Noah A. Smith 12 Mari Ostendorf 1 Hannaneh Hajishirzi 12 1 University of Washington 2 Allen Institute for Artificial Intelligence 3 University of California, Berkeley 4 University of California, San Diego 5 MosaicML Abstract Language models (LMs) often exhibit undesirable text generation behaviors, in- cluding generating false, toxic, or irrelevant outputs. Reinforcement learning from human feedback (RLHF)—where human preference judgments on LM outputs are transformed into a learning signal—has recently shown promise in addressing these issues. However, such holistic feedback conveys limited information on long text outputs; it does not indicate which aspects of the outputs influenced user preference; e.g., which parts contain what type(s) of errors. In this paper, we use fine-grained human feedback (e.g., which sentence is false, which sub-sentence is irrelevant) as an explicit training signal. We introduceFINE-GRAINEDRLHF, a framework that enables training and learning from reward functions that are fine-grained in two respects: (1) density, providing a reward after every segment (e.g., a sentence) is generated; and (2) incorporating multiple reward models associated with different feedback types (e.g., factual incorrectness, irrelevance, and information incom- pleteness). We conduct experiments on detoxification and long-form question answering to illustrate how learning with such reward functions leads to improved performance, supported by both automatic and human evaluation. Additionally, we show that LM behaviors can be customized using different combinations of fine-grained reward models. We release all data, collected human feedback, and codes athttps://FineGrainedRLHF.github.io. 1 Introduction State-of-the-art AI is built on pre-trained language models that are then trained through interaction with humans [29,28,9], with a combination of supervised learning and reinforcement learning. Incorporating human feedback into the process of language model (LM) training has been shown as effective to reduce false, toxic and other undesired model generation outputs [29,3,2,33,10]. Many of these studies adopt reinforcement learning from human feedback (RLHF) [46], a framework that converts human feedback into an effective LM training signal to reach these goals. Specifically, humans are presented with two or more outputs and asked to select one or rank them, and this signal is then used to train a reward model, which computes a single scalar reward for each LM-generated sequence. The LM is then trained with RL to optimize the reward it receives (from the reward model). Such a reward provides a relatively sparse training signal, especially for tasks that require the generation of long-form text—making RLHF in such domains unreliable [33]. Furthermore, previous research [8,18,11,45,25,43] into automated evaluation of generated text shows that it can be challenging for human annotators to reliably compare the overall quality of two or more model outputs when the outputs contain a mixture of diverse undesired behaviors. They demonstrate how categorizing and localizing model errors (i.e., fine-grained evaluation) provides explicit insights about which part of the model output has what type of problem. We thus ask the question: how can we improve rewards for LM training via RLHF by using morefine-grained human feedback? ∗ Equal contribution. Correspondence to <Zeqiu Wu: zeqiuwu1@uw.edu>, <Yushi Hu: yushihu@uw.edu> 37th Conference on Neural Information Processing Systems (NeurIPS 2023). arXiv:2306.01693v2 [cs.CL] 30 Oct 2023 Relevant: + 0.3 Factual: - 0.5 Relevant: + 0.3 Factual: + 0.5 Info. complete: + 0.3 LM outputs: The atmosphere of Earth is a layer of gases retained by Earth’s gravity... The atmosphere is commonly known as air. The top gases by volume that dry air ... Step 1: Collect human feedback and train the reward models Step 2: Fine-tune the policy LM against the reward models using RL LM output: The atmosphere of Earth is a layer of gases retained by Earth’s gravity. The most common gas, by dry air volume, is nitrogen. The second most is oxygen. The third most is carbon dioxide. Irrelevant / Redundant Unverifiable / Untruthful Missing The third most is Argon. Relevance RM What are the 3 most common gasses in earth’s atmosphere? Prompt: What are the 3 most common gasses in earth’s atmosphere? Prompt: Sampled Prompt: Does water boil quicker at high altitudes? It takes longer for water to boil at high altitudes. The reason is that water boils at a lower temperature at higher altitudes. PPO Update policy with rewards Preference Reward: - 0.35 Update policy with rewards PPO (a) Preference-based RLHF(b) Ours: Fine-Grained RLHF Human Feedback Factuality RM Information Completeness RM Preference RM Fine-Grained Human Feedback It takes longer for water to boil at high altitudes. The reason is that water boils at a lower temperature at higher altitudes. Sampled Prompt: Does water boil quicker at high altitudes? The air that surrounds the planet Earth contains various gases. Nitrogen... The atmosphere of Earth is the layer of gases, generally known as air... > = > Figure 1: Comparison of(a) RL with human preferenceand(b) ourFINE-GRAINEDRLHFon long-form QA. Different from (a), which collects human preferences on the overall quality of LM outputs, we ask annotators to mark which part of an output contains what type(s) of errors. We train a fine-grained reward model for each type of error and optimize LM against these reward models. In this example, we provide arelevancerewardand afactualityrewardafter each sentence is generated. There is also a holisticinformationcompletenessrewardafter the whole text is generated. In this paper, we propose that humans give fine-grained feedback to LM output, associatingcategories of undesired behavior (e.g., false or irrelevant generations) and a text span at adensity(e.g., sentence or sub-sentence-level). To enable LMs to learn from such fine-grained feedback, we introduce the FINE-GRAINEDRLHFframework. As shown in Figure 1, we first use collected human feedback to train fine-grained reward models such that each of them focuses on onecategoryand provides rewards at thedensityassociated with that category. We then integrate these reward models into Proximal Policy Optimization (PPO) [37], a commonly used RL algorithm for training LMs with preference-based human feedback (§2). We conduct experiments on two language generation tasks—detoxification [12] (§3) and long-form question answering (QA) [39] (§4). For detoxification, toxicity is the only error category and we explore learning with a dense reward. We adopt PERSPECTIVE[1], a widely used language toxicity detection model trained on millions of human annotations, as our reward model. We use it to calculate a fine-grained reward after the generation of every sentence. Our experimental results show the efficacy and data efficiency of training models with dense reward compared to a holistic sequence-level reward, supported by automatic evaluation results. With experiments on long-form QA, we aim to examine training models with fine-grained rewards at the two granularity dimensions (density and error category), for which we construct a long-form QA dataset,QA-FEEDBACK, along with our collected human feedback. We carefully develop a pipeline to collect fine-grained human feedback on three error categories at different density levels: i) irrelevance, repetition, or incoherence (sub-sentence), i) incorrect or unverifiable facts (sentence), and i) incomplete information (whole sequence; see Figure 1). Our experimental results show improved results in each error category by learning with such fine-grained feedback, supported by both automatic and human evaluation results. In a scenario with multiple reward models representing different error types, we also showFINE-GRAINEDRLHFallows us to combine reward models with different weights and thus control the model training process towards a customized combination of desired behaviors. 2 2FINE-GRAINEDRLHF We introduceFINE-GRAINEDRLHF, a framework that enables us to train fine-grained reward func- tions for generation outputs across different feedback types. We first define the RL environment and learning algorithm. Then we define the fine-grained reward models and describe how to incorporate the fine-grained reward model(s) into an RL algorithm, in contrast to previous RLHF studies that only consider a single reward. Environment: language generation as a MDP.We focus on language generation tasks. For each task, we are given a set of task input promptsD=x n N n=1 . We follow [33] to define language generation as a Markov Decision Process (MDP)⟨S,A,R,P,γ,T max ⟩with a finite vocabularyV. Each MDP episode starts with a sampled promptx= (x 1 ,x 2 ,...,x l )withx i ∈V, and ends when the current time step exceedsT max or an end of sequence token is generated.Sis the state space and s 0 = (x 1 ,x 2 ,...,x l )∈ Sis the initial state. An action in the environmenta t ∈ Ais a generated token (by the policy LM modelP θ ) at timetfromV(a 0 is the begin sequence token). The transition functionP:S×A→∆Sappendsa t at the end of the states t = (x 1 ,x 2 ,...,x l ,a 0 ,a 1 ,...,a t−1 ). This process continues until the end time stepT≤T max is reached, which gives a generated sequence y= (a 1 ,...,a T ). A reward functionR:S ×A→R, which comes from the reward model(s) in FINE-GRAINEDRLHF, provides dense rewards before and whenTis reached.P θ can be initialized with a pre-trained language model, and sometimes also with supervised fine-tuning on task-specific demonstrations. The reward function is defined later. Learning algorithm: proximal policy optimization (PPO).PPO [37] is an actor-critic RL algorithm that is widely used in previous RLHF work to optimize the policy model against a reward model of human feedback. It uses a value modelV ψ (s t )to estimate the value of states t , and optimizes the policy model with a PPO clipped surrogate training objective. The advantageA t at timestep tis estimated by a generalized advantage estimation function [36]:A t = P T t ′ =t (γλ) t ′ −t (r t ′ + γV ψ (s t ′ +1 )−V ψ (s t ′ )), withγas a hyperparameter andλas the discounting factor for rewards.r t is the reward assigned toa t , which in our case is acquired using one or multiple learned reward models. The value modelV ψ (s t )is optimized with an expected squared-error loss with the value target as V targ (s t ) = P T−1 t ′ =t γ t ′ −t r t ′ +γ T−t V ψ old (s T ) , whereV ψ old is the lagging value model. Finally, PPO is trained to optimize both policy (P θ ) and value (V ψ ) models with their respective objectives. No reward model is being optimized during PPO training. See Appendix B for more details. Fine-grained reward models.Previous RLHF work adopts a holistic reward modelR φ that maps input promptxand generated outputyto a single scalar reward representing its overall quality (Figure 1(a)). This single scalar reward is only assigned to the final token in the generated sequence, a T . Formally,r t =R φ (x,y)ift=Tand 0 otherwise. In contrast, we consider a reward function that is derived from one or multiplefine-grainedreward models that (1) provide rewards densely (i.e., for subsequences of the generated output), and (2) compute rewards on distinct categories of undesired behaviors (e.g., false or repetitive generation), where each category is associated with an individual reward model. For a fine-grained reward modelR φ k that gives feedback on error categoryC k , we first segmenty intoL k segments(y k 1 ,y k 2 ,...,y k L k ) corresponding to the density (e.g., sentence-level) ofR φ k , where each segmenty k j ends at timestepT k j .R φ k outputs a rewardR φ k (x,y,j)for each segmenty k j given xandyas the input, which is assigned to the final token iny k j . Additionally, to ensure the fluency of generated outputs, we follow [41] to add an approximate KL divergence penalty to each tokena t with a weightβ, that is not backpropagated through during training. Formally, assuming that we haveKfine-grained reward models that represent different error categories, we will have a combined reward function for each tokena t as: r t = K X k=1 L k X j=1 1(t=T k j )w k R φ k (x,y,j) −βlog P θ (a t |s t ) P θ init (a t |s t ) (1) wherew k ∈Ris a weight assigned to reward modelR φ k . Then we follow the same PPO training algorithm to optimize the policy model. We discuss how we define and train fine-grained reward models for the detoxification and long-form QA task in our experiments in § 3 and § 4 respectively. 3 3 Task 1: Detoxification The task of detoxification aims to reduce the toxicity in the model generationywhen given a promptx. Toxicity is the only undesired behavior in this task, and we aim to explore learning with a dense reward in comparison to asingleholistic reward. We conduct our experiments on REALTOXICITYPROMPTS, a dataset of 100K sentence-level prompts derived from the web that are known to easily elicit problematic generations in GPT-2 [31]. Using a dense sentence-level fine- grained reward, we demonstrate thatour fine-grained reward exhibits greater sample efficiency compared to a holistic reward, achieving lower toxicity with fewer training steps while maintaining better fluency (§3.1). Holistic reward for (non-)Toxicity.We use thePERSPECTIVEAPI [1] as our reward model, which is widely used for language toxicity detection and is trained with millions of examples gathered from several online platforms and annotated by human annotators for toxicity. That means we use an off-policy reward model that is not trained on outputs fromP θ init . The API outputs a score between 0 (non-toxic) and 1 (toxic). Given the entire model outputy, the holistic reward for RL is 1−PERSPECTIVE(y). Sentence-level (fine-grained) reward for (non-)Toxicity.To calculate thefine-grained reward, we query the API after the model generates each sentence instead of generating the full sequence. For each generated sentencey j , we assignPERSPECTIVE([y 1 ,...,y j−1 ]) -PERSPECTIVE([y 1 ,...,y j ]) as the sentence reward (i.e., how much toxicity is changed from generatingy j ). Since there is only one error category, we omit the category superscript, usingy j to denote thej th segment (e.g., sentence) iny. 3.1 Experiments Implementation details.We follow previous work [17,21] and use GPT-2 large model as the initial policy modelP θ init . During both the exploration stage in RL training and inference, we use nucleus sampling decoding withp= 0.9 and temperature = 1.0. The generation length limit is set to 48. The value model used during RL training is initialized with GPT-2-base due to GPU memory constraint. We report RL training parameters in Appendix B. All scores are averaged over 3 independent runs. Compared systems and evaluation.We report the performance ofFINE-GRAINEDRLHF, RLHF with holistic reward (Hol. RLHF), and the state-of-the-art controlled generation approachesGeDi [17] andDEXPERTS[21]. We follow previous work [17,21] to report the toxicity score calculated on each full generation sequence from thePERPLEXITYAPI, as well as other commonly used metrics forREALTOXICITYPROMPTS, including n-gram diversity and GPT-2 XL perplexity (PPL) as a proxy for fluency. The lower the perplexity, the more fluent the generated text. The toxicity score is reported as themaximumscore among 4 sampled model outputs, averaged over all test input prompts. Other metrics are reported as theaveragescore of the same 4 samples. ToxicityFluencyDiversity avg max (↓)PPL (↓)dist-2 (↑) dist-3 (↑) GPT-20.1929.580.9470.931 Controlled Generation GeDi 0.15424.780.9380.938 DEXPERTS0.13622.830.9320.922 Hol. RLHF0.13011.750.9430.926 F.G. RLHF0.0819.770.9490.932 Table 1:Results on theREALTOXICI- TYPROMPTStest set. Figure 2: Curves of toxicity and perplexity on the dev set vs. training steps. Main results.Table 1 shows the experimental results on theREALTOXICITYPROMPTStest set.FINE- GRAINEDRLHFwith sentence-level fine-grained reward attains the lowest toxicity and perplexity among all methods, while maintaining a similar level of diversity. Sample efficiency analysis.Figure 2 shows the max toxicity and average perplexity on the devel- opment set during training.FINE-GRAINEDRLHFhas the toxicity drop much faster while keeping a low-level perplexity. This shows that learning from denser fine-grained reward is more sample efficient than holistic reward. One explanation is that fine-grained reward locates where the toxic 4 content is, which is a stronger training signal compared with a scalar reward for the whole text. The cost is that we have to query the reward model more times per example. 4 Task 2: Long-Form Question Answering (QA) Long-form QA requires an LM to generate a textual response to a question with a comprehensive answer and explanation. To examine learning with fine-grained rewards at the two granularity dimensions (error category and density), we collectQA-FEEDBACK(§4.1), a long-form QA dataset annotated with human feedback on LM-generated responses. We define three error categories at different density levels and train a reward model for each (§4.2). We describe the experimental setup in §4.3. Both human and automatic evaluation show thatFINE-GRAINEDRLHFoutperforms preference-based RLHF and supervised fine-tuning models on all error categories (§4.4). We then show that adjusting the weights of fine-grained reward models during RL training leads to distinct behaviors in LM generation, allowing us to customize the LM for users with different needs (§4.5). Finally, we conduct an in-depth analysis of the fine-grained reward models, revealing that they compete against each other, and provide an analysis of their impact on the resulting policy model. 4.1QA-FEEDBACK: Long Form QA with Human Feedback QA-FEEDBACKis based on ASQA [39], a dataset that focuses on answering ambiguous factoid questions [26] in an open-domain setting. We use their provided oracle knowledge contexts to reformulate the task into a reading comprehension setting: given the inputxthat contains a question qand a set of knowledge passagesP=p 1 ,...,p |P| , generate a long-form responsey. On average, there are roughly 65 words in each gold response. Since ASQA does not release the test set, we create our own train/development/test data split from the original train and development sets. We name our newly constructed data, along with collected human feedback (discussed next),QA-FEEDBACK. Overall, we have 3,853 training, 500 development, and 948 test examples (details in Appendix C). Initial policy and fine-grained human feedback.Before collecting human feedback, we follow [33] to initialize the policy model with supervised fine-tuning on a small set of examples. Specifically, we use 1K training examples to supervise fine-tuning of T5-large (the original baseline for ASQA) [32] to getP θ init . We name this initial policy modelSFT. We then sample outputs from SFT for the remaining training and development examples and collectfine-grainedhuman feedback in three error categories—C 1 :irrelevance,repetition,orincoherence;C 2 :incorrectorunverifiablefactsbased on knowledge passages; andC 3 :incompleteinformation. The collected feedback instances are then used as the training and development examples for training reward models. For each task promptx, we only collect fine-grained feedback foronemodel output. Our data collection has IRB approval and is deemed exempt. We instruct workers to identify any error in each model outputy= (a 1 ,...,a T ), marking the span of text associated with each identified error type. Formally, we define the set of user-annotated feedback for a task promptxand model outputyasF=f i where eachf i =⟨c i ,b i ,e i ⟩represents the user-identified span(a b i ,...,a e i )of the error categoryC c i , wherec i ∈1,2,3. Importantly, we impose three restrictions in the annotation: (1) error spans of categoryC 1 orC 2 should not overlap with each other; (2) only spans that do not have errorC 1 need to be assessed as containing error C 2 or not; (3)C 3 can only apply to whole output sequences. Additionally, we ask workers to mark passage sentences that contain missing information if aC 3 error is annotated. We also ask workers to rewriteyinto a corrected versiony ′ that addresses all annotated feedbackF. Details about the feedback collection interface, instructions, and quality control are in Appendix C. To analyze human-human agreement, a subset of 300 examples receive annotations from two distinct workers. We observe that while exact agreement in error span boundaries is low, workers achieve reasonably high agreement on whether a sub-sentence containsC 1 and whether a sentence contains C 2 . 2 Therefore, we decide to have the density for error typeC 1 ,C 2 , andC 3 as sub-sentence, sentence and full sequence. We provide more data analysis including human agreement in Appendix C. Preference-based human feedback.For comparison purposes, we follow [29] to separately collect pairwisehuman preferencesfrom the same group of workers. We sample 4 model outputs for each promptx, which gives 6 pairs of model outputs. We ask the workers to indicate pairwise preferences 2 We use spaCy [15] to segment generated model outputs into sentences. We then split sentences into sub-sentences using a comma or semicolon. 5 (ties are allowed) based on all errors they can find in each model output. They are not asked to explicitly annotate these errors. Annotation details.On average, both annotation tasks of fine-grained and preference feedback for one question take a worker about 6 minutes to finish. In contrast, [39] report that they spend about 15 minutes to label a human-written response for each question, which is much more time-consuming than our feedback annotation. On average, we pay $1.65 per example for both tasks, leading to $16.50 hourly pay for our workers. We include details of the pay structure in Appendix C. We observe that human annotators can reach a higher agreement in each aspect of fine-grained feedback compared to pairwise comparisons because the feedback definitions are more concrete. 4.2 Fine-Grained Reward Models We train three separate reward modelsR φ 1 ,R φ 2 , andR φ 3 forC 1 ,C 2 , andC 3 error categories respectively with a density of sub-sentence, sentence, and full sequence, respectively. Since reward models provide scalar reward scores and do not perform generation, we use the encoder-only Longformer-base [4] as our backbone model to handle long input sequences (more details of each reward model are in Appendix D). C 1 :Irrelevance,repetition,orincoherence.R φ 1 targets to predict whether each sub-sentence iny contains aC 1 type error. We denotey= (y 1 1 ,...,y 1 L 1 ) , wherey 1 j is thejth segment atR φ 1 ’s density (i.e., sub-sentence), withL 1 segments in total. We add a 2-class token-level classification layer (a single feed-forward layer) on the top of the Longformer encoder. The model input has the format of “question:qanswer: [sep]y 1 1 [sep]y 1 2 . . . ”, and we take the classification output at each [sep]token to indicate whether the followingy 1 j contains aC 1 error. We do not add passages in the model input because, intuitively, the detection ofC 1 errors does not depend on them. To train R φ 1 , we apply a token-level classification loss to each[sep]token beforey 1 j , where its gold labelg j is “has error” if there is af i ∈Fthat has(a b i ,...,a e i )overlapped withy 1 j andc i = 1, and “no error” otherwise. WhenR φ 1 provides a reward during RL training as in Eq. 1, we read a reward R φ 1 (x,y,j) for everyy 1 j givenxandy. We defineR φ 1 (x,y,j) = +1 ifR φ 1 predicts “no error” fory 1 j and−1otherwise. C 2 :Incorrectorunverifiablefacts.R φ 2 is developed for detecting aC 2 error at the sentence level in a similar way. The model input has the format of “question:qcontext:p 1 p 2 . . .answer: [sep]y 2 1 [sep]y 2 2 . . . ”, wherep’s denotes the grounding passages andy 2 j represents thejth sentence. We trainR φ 2 similarly toR φ 1 , with one exception: as we instruct the workers not to annotate aC 2 error for a span that is already labeled as containing aC 1 error, we do not calculate loss on sentences that are labeled as containingC 1 but notC 2 duringR φ 2 training. C 3 :Incompleteinformation.R φ 3 is trained to measure the information completeness ofy, at the full sequence level. Motivated by [19],R φ 3 predicts a single scalar reward and is trained with a pairwise comparison loss [29]: L r (φ) =−E (x, ̄y p , ̄y l )∼D p h log σ(R φ 3 (x, ̄y p )−R φ 3 (x, ̄y l )) i (2) whereR φ 3 (x,y)is the scalar output of the reward model for inputxand outputy; ̄y p and ̄y l are sampled from the same inputx, and ̄y p has less missed information compared with ̄y l ;D p contains the pairwise comparisons bootstraped from human feedback onC 3 errors (see details in Appendix D). Preference-based reward model.The preference-based reward model is trained in a similar way toR φ 3 , with ̄y p representing the human preferred response against ̄y l in the loss function Eq. 2. It outputs a scalar score for the givenxandythat represents the overall response quality. 4.3 Experimental Setup Compared systems.We compare our proposed method,FINE-GRAINEDRLHFwith the initial T5 policy model trained with 1K examples (SFT) and RLHF with holistic preference-based rewards (Preference RLHF). The reward models used in RLHF experiments are trained on 2.8K examples with annotated feedback (but no gold human response). For analysis, we also use the human gold responses of all training examples to finetune a fully supervised T5 model (SFT-Full). Notice that SFT-Full requires much higher annotation cost because it takes longer (15 minutes per example [39]) for annotators to draft long-form responses. 6 Implementation details.Our policy model is based on T5-large [32] and is supervised finetuned on 1K training examples, as explained in §4. During RL exploration, we use top-k (k= 20) sampling decoding with temperature = 0.7, which is set based on previous RLHF work [33]. The value model used during RL training is initialized with T5-base due to GPU memory constraint. The reward model weights we used in FINE-GRAINEDRLHFarew 1 = 0.3,w 2 = 0.5,w 3 = 0.3, unless otherwise specified. Although we use three reward models during RL training, we only observe very small relative additional cost (roughly 1% training time) compared to preference RLHF. During inference, we use greedy decoding to generate responses. We report more details including RL training parameters in Appendix B. All scores reported are averaged over 3 independent runs. Evaluation.We conduct both human and automatic evaluation. Human evaluation is run on 200 randomly sampled test set examples ofQA-FEEDBACKto compareFine-Grained RLHFwith all baselines. Each model output is sampled from inference results of 3 training runs. We use the same protocol of feedback collection to have the same set of workers annotate spans in each model output that contain(1)irrelevance,repetition,orincoherenceerror(rel.)and(2)incorrectorunverifiable facts(fact.). They are also asked to compare theinformationcompleteness(comp.)for each output pair. To report evaluation scores forrel.andfact.error spans, we first map them to their corresponding error type density (sub-sentence and sentence). Then we report the error rate for each error type, measured as the percentage of sub-sentences that contains this type of error. Since spans withrel.error are not checked forfact.error (discussed in §4.1), we exclude sub-sentences with only rel.error when report the error rate offact.error. For automatic evaluation, we report RougeLSum [20] as used for the original ASQA data, as well as the score from each fine-grained reward model (R φ 1 ,R φ 2 , andR φ 3 ). Specifically, we report the percentage of all sub-sentences (or sentences) in the test set predicted as “no error” byR φ 1 (orR φ 2 ). ForR φ 3 , we report the averaged output score for all test examples. 4.4 Main Results Figure 3 shows the human evaluation results forrel.andfact.error types. Table 2 shows the human pairwise comparison results for information completeness (comp.). Error Rate Figure 3: Human evaluation onrel.(left) andfact.(right) error, measured by % of sub-sentences that contain the error type (↓). Ours vs.WinTieLose SFT23.0%65.5%11.5% SFT-Full22.0%61.0%17.0% Pref. RLHF19.5%71.0%9.5% Table 2: Human pairwise compari- son oninformationcompleteness (comp.), where win/lose refers to FINE-GRAINEDRLHF. FINE-GRAINEDRLHFoutperforms SFT and Preference RLHF on all error types.Figure 3 and Table 2 show that ourFINE-GRAINEDRLHFleads to generation that is much more factually correct and contains more complete information, compared to all other systems. It generates fewer irrelevance, repetition, andincoherence errors, compared with SFT and Preference RLHF. In the meantime, Preference RLHF, despite greatly reducing factual errors compared to the initial policy model SFT, generates even more irrelevance, repetition, and incoherence errors than SFT.FINE- GRAINEDRLHFoutperforms Preference RLHF potentially due to more specific and localized training signals. In addition, we ask annotators to compare the overall generation quality ofFINE-GRAINED RLHFand preference RLHF. Although Preference RLHF is trained directly with such preference feedback,FINE-GRAINEDRLHFwas rated better than Preference RLHF in 30.5% of all examples and worse in 24.5% of examples. The annotators indicate a tie in the remaining 45% of cases. Surprisingly, FINE-GRAINEDRLHFoutperforms SFT-Full with more factual and complete generation, despite a much lower annotation cost. RLHF is particularly effective in reducing factual errors.Figure 3 shows that bothFINE- GRAINEDRLHFand Preference RLHF are effective in reducing factual errors in model generation. Meanwhile, we see little or no improvement in reducing irrelevance, repetition, or incoherence errors. We provide more in-depth analysis for this observation in §4.5. Table 3 shows automatic scores on theQA-FEEDBACKtest set, which show similar trends as human evaluation in terms of system comparisons, while all four systems achieve similar Rouge scores. 7 rel.fact.comp. R φ 1 (↑)R φ 2 (↑)R φ 3 (↑)Rouge(↑) SFT-Full0.5080.7560.04449.63 SFT0.5130.749-0.05348.96 + Pref. RLHF0.4820.7810.10149.84 +F.G. RLHF0.5130.8160.13949.93 Table 3: Automatic evaluation on theQA- FEEDBACKtest set. rel.fact.comp.avg. ConfigR φ 1 (↑)R φ 2 (↑)R φ 3 (↑)Rouge(↑)len Short0.6370.760-0.23148.9974.92 Medium0.5130.8160.13949.9398.66 Long0.4250.8600.24148.72109.63 Table 4: Automatic evaluation results (test set) ofFINE-GRAINEDRLHFtrained with different reward model weight configurations. 4.5 LM Customization with FINE-GRAINEDRLHF Since we use multiple reward models inFINE-GRAINEDRLHF, adjusting their weights (see Eq. 1) during RL may lead to different LM behaviors. For example, adding more weight to a reward model associated with one specific desired behavior type (e.g., information completeness) may lead the generation more towards that behavior type compared to others (e.g., information relevance). This flexibility can potentially fit users with diverse needs. Therefore, in this section, we explore FINE-GRAINEDRLHF’s ability to customize the LM behavior. LM customization.As in Table 4, we explore three configurations of reward model weights (w 1 ,w 2 , andw 3 forR φ 1 ,R φ 2 , andR φ 3 ) and name them ‘short’, ‘medium’, and ‘long’ according to the LM’s average generation length. For simplicity, we fixw 2 = 0.5andw 3 = 0.3, and use 0.4, 0.3, and 0.2 forw 1 , which leads to ‘short’, ‘medium’, and ‘long’ generation outputs respectively. We manually inspect 30 random examples and observe that (1) ‘short’ generates more relevant content, but is less factual and complete; (2) ‘long’, in contrast, gives the most factual and complete generation. This reflects that the LM is referencing a large amount of content from passages; (3) The ‘medium’ configuration balances the three rewards and has the highest Rouge score. 24/30 examples follow the above rule. Qualitative analysis and examples of LM customization are in Appendix A. Trade-off between error types.We observe that a higherw 1 leads to a biggerrel.reward, smaller fact.andcomp.rewards, and shorter generated outputs. One interpretation is thatR φ 1 penalizes text spans that are irrelevant to the questions. As such, it encourages answering the question directly and penalizes referencing passages and generating auxiliary information. This reduces the model generation length and information completeness, and induces more factual errors. 4.6 Analysis Figure 4: Dynamics of each type of re- ward during training (reward vs. training steps). All rewards are z-normalized. rel.fact.comp.avg. R φ 1 (↑)R φ 2 (↑)R φ 3 (↑)Rouge(↑)len SFT0.5140.7350.06543.1396.69 F.G. RLHF0.5160.8250.26644.29101.76 w/o.R φ 1 0.2490.7710.74238.52179.31 w/o.R φ 2 0.7160.640-0.17743.1878.08 w/o.R φ 3 0.5650.7990.12343.6193.92 Table 5: Ablation of reward models on the development set.R φ 1 ,R φ 2 , andR φ 3 correspond to the reward model for relevance, factuality, and information completeness. Reward models are competing against each other.In the prior section, we find that there is a trade-off between error types. To further look into this phenomenon, we explore the dynamics of each reward model during training. Figure 4 shows each reward model’s rewards on the development set during training. All rewards are z-normalized for visualization. We see that thefact.reward is consistently increasing. Therel.reward increases rapidly in the first 250 steps and then starts decreasing, while thecomp.reward exhibits an opposite trend, decreasing at first and then starting to increase. As discussed earlier, one interpretation is that relevance (precision) and information completeness (recall) can be adversarial objectives, so the rewards are competing. The three rewards reach an equilibrium point in later steps. Ablation: Does the LM learn from all reward models? What if we remove one reward model? Table 5 explores the policy LM behavior when one of the three reward models is removed during training. Qualitative examples are in Appendix A. First, we observe that the corresponding reward decreases dramatically when the model is removed. When therel.reward model (R φ 1 ) is removed, 8 the outputs become extremely long and thecomp.reward is extremely high. We observe the outputs and find the model is copying a lot of content from the passages. When thefact.reward model (R φ 2 ) is removed, therel.reward becomes the highest. We observe that the LM tends to answer the question directly and not reference the passages, which causes a lot of hallucinations. When the comp.reward model (R φ 3 ) is removed, the outputs are concise and factual but not providing all relevant information to the question. Thus, it has lower information completeness and Rouge score compared with the LM trained with all reward models. Reward model performance.We report and analyze the performance of each reward model in predicting its corresponding error category. Therel.reward modelR φ 1 has a binary classification accuracy of 69.6, and an F1 score (for the “has error” class) of 68.5 on model-generated sub- sentences from the development set. We sample 20 sub-sentences whereR φ 1 predicts the opposite of the human label, and observe that all of them either 1) contain relevant auxiliary information and are marked as “no error” by humans, or 2) are marked as irrelevant by humans but provide closely related background information to the question. In other words,R φ 1 is mostly struggling with predicting the relevance of auxiliary information, and it rarely fails to predict a direct answer as “no error”. Thefact.reward modelR φ 2 has an accuracy of 77.8 and an F1 score of 67.5. We sample 20 sentences whereR φ 2 makes a prediction mistake and we observe that the mistakes often happen when the generated sentence is highly abstractive instead of directly copying information from the passage. We also observe that more than 80% of human labeled factual errors occur when the model generates a direct answer (not auxiliary information) that contains hallucinated information or a random entity from a passage. We notice thatR φ 2 correctly captures more than 80% of such errors. Thecomp.reward modelR φ 3 has an accuracy of 70.9 in pairwise comparison. In contrast, the preference-based reward model only reaches an accuracy of 57.2. This helps confirm our intuition that assessing long-form generation outputs holistically can be more ambiguous and subjective than evaluating the outputs with a focus on a specific undesired behavior type. Comparison with ChatGPT responses.We experiment with answering the questions with ChatGPT. To familiarize ChatGPT with the style of our LFQA task, we prompt it with the task instruction and a single random QA example (due to length limitation). ChatGPT achieves a RougeLSum score of 40.92 on the test set, which is much lower than our models. We do not use our trained reward models to evaluate ChatGPT outputs because reward models trained on T5-large may not generalize well to ChatGPT. We instead manually inspect the ChatGPT responses, and observe that they are mostly concise and factual, yet lack the auxiliary information necessary to clarify ambiguous questions. Qualitative examples are in Appendix A. This shows the difficulty for ChatGPT in learning user-desired behaviors through simple prompting. 5 Related Work Reinforcement learning from human feedback (RLHF).RLHF [46,42,29] aims to optimize the policy language model to generate content that is desired by human. This framework has been explored to improve the model performance on a variety of natural language processing tasks such as text summarization [40], instruction following [29], question answering [24,27] and reducing harmfulness [3,2,22,10]. Most of these studies collect human preferences over pairs of model outputs on one or a set of desired attributes, in order to train a reward model to assign a holistic score for a generation output during RL training. [13] trains separate reward models that assign scores for different desired attributes, but still uses a single reward that combines scores from all reward models. In contrast, we explore RLHF with fine-grained reward models trained on human feedback where each reward model provides dense reward after every small text segment for a specific type of desired behavior. [30] explores using intermediate rewards to improves LM performance on reasoning tasks. Learning from human feedback in NLP.There also exists work that explores non-RL methods to learn from human feedback. [44] trains a reward model that predicts a single score for each model output and selects samples with the highest reward scores for supervised fine-tuning. [38,14,42] train a conversational model to predict both the response and a binary user satisfaction score in order to improve the response generation. Besides such numerical human feedback, natural language (NL) human feedback has also been explored. [23,6] collect and store NL human feedback in a feedback memory for the model to retrieve and then perform the end task conditioning on the retrieved feedback. [5,35,34] use a refinement model to refine model outputs conditioning on NL human feedback and 9 then use a reward model to select the best refined outputs for supervised fine-tuning. Methods for using a reward model to guide LM generation towards desired behaviors at inference time [21,7] can complement our work that aims to improve the LM during training. [16] also explores incorporating human feedback into LM pre-training. 6 Discussion Annotation Costs.It is important to note that the fine-grained human feedback used for training our fine-grained reward models doesnotincur a greater cost than holistic human preference. As outlined in § 4.2, our observations reveal that annotators require a substantial amount of time to compare two lengthy text outputs. For the long-form QA task, both fine-grained feedback and preference-based feedback takes approximately 6 minutes per sample for an annotator. 6.1 Broader Impacts We propose theFINE-GRAINEDRLHFframework that can incorporate multiple reward models to provide dense rewards for RL training, which leads to LM outputs that are optimized towards such rewards. Our framework can be applied to any text generation task, thereby enhancing LM performance by offering more nuanced guidance than holistic feedback. The key advantages of the FINE-GRAINEDRLHFframework are two-fold: Flexibility.Our framework significantly expands the versatility of reward models for RLHF. For example, future work involving fact-checking, sentiment classification, toxicity detection, among others, can all be incorporated within this framework. LMs can be trained against all these reward models via FINE-GRAINEDRLHF. Controllablility.Having multiple reward models that stand for different feedback types allows the end user to exert greater control over RL training (e.g., through different combinations of reward model weights; see details in § 4.5). This leads to customized model behaviors, a benefit particularly valuable for applications like educational tools where model personalization is crucial. 6.2 Limitations and Future Work One major limitation of our framework comes from the additional compute cost of gettingfine- grainedrewards, compared to RLHF with a holistic reward. For instance, in the detoxification task, we need to make multiplePERSPECTIVEAPI calls for each model output depending on how many sentences are generated, while RLHF with a holistic reward only requires one. In the long-form QA task, we need to calculate a dense reward from multiple reward models, which takes more compute time and GPU memory than a single reward model. Another limitation is that different tasks may have different definitions of fine-grained feedback in terms of the feedback types and the density level of each type. Therefore, defining feedback that is well-suited for a task and training reward models accordingly requires non-trivial manual effort. Finally, in this work, we carefully control the quality of annotated feedback, which is then used to train reward models for RL. In practice, when a deployed model is released to the public, end users don’t always give clean feedback. Therefore, how to obtain effective learning signals from noisy human feedback in the wild still needs further investigation. Some other interesting questions to explore in the future include: 1) Can we obtain fine-grained feedback from LMs like GPT-4 instead of humans to improve model performance and reduce annotation costs? 2) How can other non-RL approaches of using human feedback such as controlled generation during inference time complementFINE-GRAINEDRLHF? 3) How would fine-grained reward and value model sizes affect policy model performance during RL training? 7 Conclusion In this work, we introduceFINE-GRAINEDRLHF, a framework that enables LMs to learn from multiple fine-grained reward models trained from human feedback, where each reward model detects a specific error category and provides dense rewards. We conduct experimental analysis on two text generation tasks to illustrate the performance gain ofFINE-GRAINEDRLHFthan RLHF over holistic rewards, supported by both automatic and human evaluation. Furthermore, we show that an LM can be customized for specific needs using different combinations of fine-grained reward models. 10 Acknowledgments We thank Jiacheng Liu for sharing the standard PPO training code, and Yizhong Wang for providing insights during early discussions of the project. We also thank UW TIAL members for participating in our pilot feedback annotation. We extend our thanks to UW NLP members who provided insights or feedback to our project. Lastly, we especially thank all our AMT workers for helping us annotate the high quality feedback data. This research was developed with funding from the Defense Advanced Research Projects Agency (DARPA) under Contract No. FA8650-23-C-7316. This work was also funded in part by the DARPA MCS program through NIWC Pacific (N66001-19-2-4031), NSF IIS-2044660, and ONR N00014-18-1-2826. The views, opinions and/or findings expressed are those of the author and should not be interpreted as representing the official views or policies of the Department of Defense or the U.S. Government. References [1] Perspective API,https://github.com/conversationai/perspectiveapi. [2]Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback.arXiv preprint arXiv:2204.05862, 2022. [3]Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback.arXiv preprint arXiv:2212.08073, 2022. [4]Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150, 2020. [5]Angelica Chen, Jérémy Scheurer, Tomasz Korbak, Jon Ander Campos, Jun Shern Chan, Samuel R Bowman, Kyunghyun Cho, and Ethan Perez. Improving code generation by training with natural language feedback.arXiv preprint arXiv:2303.16749, 2023. [6]Bhavana Dalvi Mishra, Oyvind Tafjord, and Peter Clark. Towards teachable reasoning systems: Using a dynamic memory of user feedback for continual system improvement. InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9465–9480, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. [7]Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosinski, and Rosanne Liu. Plug and play language models: A simple approach to controlled text generation. InInternational Conference on Learning Representations, 2020. [8] Yao Dou, Maxwell Forbes, Rik Koncel-Kedziorski, Noah A. Smith, and Yejin Choi. Is GPT-3 text indistinguishable from human text? scarecrow: A framework for scrutinizing machine text. InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7250–7274, Dublin, Ireland, May 2022. Association for Computational Linguistics. [9]Yao Fu, Litu Ou, Mingyu Chen, Yuhao Wan, Hao Peng, and Tushar Khot. Chain-of-thought hub: A continuous effort to measure large language models’ reasoning performance.arXiv preprint arXiv:2305.17306, 2023. [10]Deep Ganguli, Amanda Askell, Nicholas Schiefer, Thomas Liao, Kamil ̇ e Lukoši ̄ ut ̇ e, Anna Chen, Anna Goldie, Azalia Mirhoseini, Catherine Olsson, Danny Hernandez, et al. The capacity for moral self-correction in large language models.arXiv preprint arXiv:2302.07459, 2023. [11]Tianyu Gao, Howard Yen, Jiatong Yu, and Danqi Chen. Enabling large language models to generate text with citations.arXiv preprint arXiv:2305.14627, 2023. [12]Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. Real- ToxicityPrompts: Evaluating neural toxic degeneration in language models. InFindings of the Association for Computational Linguistics: EMNLP 2020, pages 3356–3369, Online, November 2020. Association for Computational Linguistics. 11 [13]Amelia Glaese, Nat McAleese, Maja Tr ̨ebacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Maribeth Rauh, Laura Weidinger, Martin Chadwick, Phoebe Thacker, et al. Improving alignment of dialogue agents via targeted human judgements.arXiv preprint arXiv:2209.14375, 2022. [14] Braden Hancock, Antoine Bordes, Pierre-Emmanuel Mazare, and Jason Weston. Learning from dialogue after deployment: Feed yourself, chatbot! InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3667–3684, Florence, Italy, July 2019. Association for Computational Linguistics. [15] Matthew Honnibal, Ines Montani, Sofie Van Landeghem, and Adriane Boyd. spaCy: Industrial- strength Natural Language Processing in Python. 2020. [16]Tomasz Korbak, Kejian Shi, Angelica Chen, Rasika Bhalerao, Christopher L Buckley, Ja- son Phang, Samuel R Bowman, and Ethan Perez. Pretraining language models with human preferences.arXiv preprint arXiv:2302.08582, 2023. [17] Ben Krause, Akhilesh Deepak Gotmare, Bryan McCann, Nitish Shirish Keskar, Shafiq Joty, Richard Socher, and Nazneen Fatema Rajani. GeDi: Generative discriminator guided sequence generation. InFindings of the Association for Computational Linguistics: EMNLP 2021, pages 4929–4952, Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. [18]Kalpesh Krishna, Erin Bransom, Bailey Kuehl, Mohit Iyyer, Pradeep Dasigi, Arman Cohan, and Kyle Lo. LongEval: Guidelines for human evaluation of faithfulness in long-form summa- rization. InProceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 1650–1669, Dubrovnik, Croatia, May 2023. Association for Computational Linguistics. [19]Margaret Li, Jason Weston, and Stephen Roller. Acute-eval: Improved dialogue evaluation with optimized questions and multi-turn comparisons.arXiv preprint arXiv:1909.03087, 2019. [20]Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries. InText Summariza- tion Branches Out, pages 74–81, Barcelona, Spain, July 2004. Association for Computational Linguistics. [21]Alisa Liu, Maarten Sap, Ximing Lu, Swabha Swayamdipta, Chandra Bhagavatula, Noah A. Smith, and Yejin Choi. DExperts: Decoding-time controlled text generation with experts and anti-experts. InProceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Vol- ume 1: Long Papers), pages 6691–6706, Online, August 2021. Association for Computational Linguistics. [22]Ximing Lu, Sean Welleck, Liwei Jiang, Jack Hessel, Lianhui Qin, Peter West, Prithviraj Ammanabrolu, and Yejin Choi. Quark: Controllable text generation with reinforced unlearning. InThirty-sixth Conference on Neural Information Processing Systems (NeurIPS), 2022. [23]Aman Madaan, Niket Tandon, Peter Clark, and Yiming Yang. Memory-assisted prompt editing to improve GPT-3 after deployment. InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 2833–2861, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. [24]Jacob Menick, Maja Trebacz, Vladimir Mikulik, John Aslanides, Francis Song, Martin Chad- wick, Mia Glaese, Susannah Young, Lucy Campbell-Gillingham, Geoffrey Irving, et al. Teaching language models to support answers with verified quotes.arXiv preprint arXiv:2203.11147, 2022. [25]Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Wei Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. Factscore: Fine-grained atomic evaluation of factual precision in long form text generation.arXiv preprint arXiv:2305.14251, 2023. [26]Sewon Min, Julian Michael, Hannaneh Hajishirzi, and Luke Zettlemoyer. AmbigQA: Answering ambiguous open-domain questions. InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5783–5797, Online, November 2020. Association for Computational Linguistics. 12 [27]Reiichiro Nakano, Jacob Hilton, S. Arun Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christo- pher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. Webgpt: Browser-assisted question-answering with human feedback.ArXiv, abs/2112.09332, 2021. [28] OpenAI. Gpt-4 technical report, 2023. [29]Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback, 2022. [30] Debjit Paul, Mete Ismayilzada, Maxime Peyrard, Beatriz Borges, Antoine Bosselut, Robert West, and Boi Faltings. Refiner: Reasoning feedback on intermediate representations, 2023. [31]Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019. [32]Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer.The Journal of Machine Learning Research, 21(1):5485–5551, 2020. [33]Rajkumar Ramamurthy*, Prithviraj Ammanabrolu*, Kianté Brantley, Jack Hessel, Rafet Sifa, Christian Bauckhage, Hannaneh Hajishirzi, and Yejin Choi. Is reinforcement learning (not) for natural language processing: Benchmarks, baselines, and building blocks for natural language policy optimization. InInternational Conference on Learning Representations (ICLR), 2023. [34] Jérémy Scheurer, Jon Ander Campos, Jun Shern Chan, Angelica Chen, Kyunghyun Cho, and Ethan Perez. Training language models with natural language feedback.arXiv preprint arXiv:2204.14146, 2022. [35]Jérémy Scheurer, Jon Ander Campos, Tomasz Korbak, Jun Shern Chan, Angelica Chen, Kyunghyun Cho, and Ethan Perez. Training language models with language feedback at scale, 2023. [36]John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. InProceedings of the International Conference on Learning Representations (ICLR), 2016. [37]John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347, 2017. [38]Weiyan Shi, Emily Dinan, Kurt Shuster, Jason Weston, and Jing Xu. When life gives you lemons, make cherryade: Converting feedback from bad responses into good labels.arXiv preprint arXiv:2210.15893, 2022. [39]Ivan Stelmakh, Yi Luan, Bhuwan Dhingra, and Ming-Wei Chang. ASQA: Factoid questions meet long-form answers. InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 8273–8288, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. [40] Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan J. Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize from human feedback. ArXiv, abs/2009.01325, 2020. [41]Jeff Wu, Long Ouyang, Daniel M Ziegler, Nisan Stiennon, Ryan Lowe, Jan Leike, and Paul Christiano. Recursively summarizing books with human feedback.arXiv preprint arXiv:2109.10862, 2021. [42]Jing Xu, Megan Ung, Mojtaba Komeili, Kushal Arora, Y-Lan Boureau, and Jason Weston. Learning new skills after deployment: Improving open-domain internet-driven dialogue with human feedback.arXiv preprint arXiv:2208.03270, 2022. 13 [43]Wenda Xu, Danqing Wang, Liangming Pan, Zhenqiao Song, Markus Freitag, William Yang Wang, and Lei Li. Instructscore: Towards explainable text generation evaluation with automatic feedback.arXiv preprint arXiv:2305.14282, 2023. [44]Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang. Rrhf: Rank responses to align language models with human feedback without tears, 2023. [45]Xiang Yue, Boshi Wang, Kai Zhang, Ziru Chen, Yu Su, and Huan Sun. Automatic evaluation of attribution by large language models.arXiv preprint arXiv:2305.06311, 2023. [46]Daniel M. Ziegler, Nisan Stiennon, Jeff Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences.ArXiv, abs/1909.08593, 2019. 14 Appendices A Qualitative Examples for Long-Form QA16 A.1 Examples on LM Customization . . . . . . . . . . . . . . . . . . . . . . . . . . .16 A.2 Examples on LM Errors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .17 A.3 Examples on Reward Model Ablation . . . . . . . . . . . . . . . . . . . . . . . .19 A.4 Comparison with ChatGPT responses . . . . . . . . . . . . . . . . . . . . . . . .19 B Algorithm and Training Details of FINE-GRAINEDRLHF21 B.1 PPO with Fine-Grained Rewards . . . . . . . . . . . . . . . . . . . . . . . . . . .21 B.2 Implementation Details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .21 C Long-Form QA Data and Human Feedback Annotation22 C.1 Data Construction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .22 C.2 Human Feedback Annotation . . . . . . . . . . . . . . . . . . . . . . . . . . . . .22 C.3 Analysis of Collected Fine-Grained Feedback . . . . . . . . . . . . . . . . . . . .23 D Long-Form QA Reward Model Training Details23 15 A Qualitative Examples for Long-Form QA A.1 Examples on LM Customization As discussed in § 4.5, we can modify the weight of each fine-grained reward model during RL training to get LM with different behaviors. Here, we explore three configurations of reward model weights and name them ‘short’, ‘medium’, and ‘long’ based on the LM’s average generation length. The ‘short’ configuration generates concise and short responses, while the ‘long’ configuration generates detailed and long responses. Table 6 demonstrates the different behaviors of our customized LMs. Given the same question, each LM generates different amount of auxiliary information in the response. Question:When did the French join revolution on colonists’ side? Passages:Article Title: France in the American Revolutionary War French involvement in the American Revolutionary War began in 1775, when France, a rival of the British Empire, secretly shipped supplies to the Continental Army. A Treaty of Alliance in 1778 soon followed, which led to shipments of money and material to the United States. Subsequently, the Spanish Empire and the Dutch Republic also began to send assistance, leaving the British Empire with no allies. France’s help is considered a vital and decisive contribution to the United States’ victory against the British. As a cost of participation in the war, France accumulated over 1 billion livres in debt. After its defeat in the Seven Years’ War in 1763, France lost its vast holdings in North America. Meanwhile, the American colonists and the British government began to fight over whether Parliament in London or the colonial assemblies had primary responsibility for taxation. As part of that conflict, the colonists organized the Boston Tea Party in response to a tax on tea. The British government responded by passing the Intolerable Acts, which included the closing of Boston Harbor and the revocation of Massachusetts’s colonial charter. This conflict exacerbated tensions further. The ideological conflict escalated into open warfare in 1775, at which point the American patriots revolted against British rule. France, who had been rebuilding their Navy and other forces, saw this as an opportunity to seriously weaken her perennial enemy. Article Title: France in the American Revolutionary War France bitterly resented its loss in the Seven Years’ War and sought revenge. It also wanted to strategically weaken Britain. Following the Declaration of Independence, the American Revolution was well received by both the general population and the aristocracy in France. The Revolution was perceived as the incarnation of the Enlightenment Spirit against the "English tyranny." Benjamin Franklin traveled to France in December 1776 in order to rally the nation’s support, and he was welcomed with great enthusiasm. At first, French support was covert: French agents sent the Patriots military aid (predominantly gunpowder) through a company called "Rodrigue Hortalez et Compagnie", beginning in the spring of 1776. Estimates place the percentage of French supplied arms to the Americans in the Saratoga campaign at up to 90%. By 1777, over five million "livres" of aid had been sent to the American rebels. Motivated by the prospect of glory in battle or animated by the sincere ideals of liberty and republicanism, volunteers like Pierre Charles L’Enfant joined the American army. The most famous was Lafayette, a charming young aristocrat who defied the king’s order and enlisted in 1777 at age 20. He became an aide to George Washington and a combat general. More importantly, he solidified a favorable American view of France. Article Title: Treaty of Alliance (1778) On March 17, 1778, four days after a French ambassador informed the British government that France had officially recognized the United States as an independent nation with the signing of the Treaty of Amity and Commerce and the Treaty of Alliance, Great Britain declared war on France, thereby engaging that nation in the American Revolutionary War. French entry into the war would lead to further escalation of the war when Spain entered the fight against Britain as France’s ally, after the signing of the Treaty of Aranjuez on April 12, 1779, and again in December 1780 when Britain declared war on the Dutch Republic after seizing a Dutch merchant ship they claimed was carrying contraband to France during the Affair of Fielding and Bylandt. After the signing of the treaty French supplies of arms, ammunition, and uniforms proved vital for the Continental Army, while their actions in the Anglo French War of 1778-83 in the West Indies and elsewhere forced Britain to redeploy troops and naval units away from the North American colonies to secure their holdings. LM outputs with different reward model configurations: Short: French involvement in the American Revolutionary War began in 1775, when France, a rival of the British Empire, secretly shipped supplies to the Continental Army. A Treaty of Alliance in 1778 soon followed, which led to shipments of money and matériel to the United States. Subsequently, the Spanish Empire and the Dutch Republic also began to send assistance, leaving the British Empire with no allies. Medium:French involvement in the American Revolutionary War began in 1775, when France, a rival of the British Empire, secretly shipped supplies to the Continental Army. A Treaty of Alliance in 1778 soon followed, which led to shipments of money and matériel to the United States. Subsequently, the Spanish Empire and the Dutch Republic also began to send assistance, leaving the British Empire with no allies. On March 17, 1778, four days after a French ambassador informed the British government that France had officially recognized the United States as an independent nation, Great Britain declared war on France, thereby engaging that nation in the American Revolutionary War. Long:French involvement in the American Revolutionary War began in 1775, when France, a rival of the British Empire, secretly shipped supplies to the Continental Army. A Treaty of Alliance in 1778 soon followed, which led to shipments of money and matériel to the United States. Subsequently, the Spanish Empire and the Dutch Republic also began to send assistance, leaving the British Empire with no allies. On March 17, 1778, four days after a French ambassador informed the British government that France had officially recognized the United States as an independent nation with the signing of the Treaty of Amity and Commerce and the Treaty of Alliance, Great Britain declared war on France, thereby engaging that nation in the American Revolutionary War. Table 6: Qualitative examples on LM customization. 16 A.2 Examples on LM Errors Table 7 and Table 8 show examples of LM outputs from all the compared systems (SFT, Pref. RLHF, andFINE-GRAINEDRLHF). We mark the fine-grained errors on the model outputs. Overall, our FINE-GRAINEDRLHFoutperforms SFT and Pref. RLHF in all three error types. Question:Who was the prime minister of Canada in 1920? Passages:Article Title: Robert Borden Not only did Borden’s persistence allow him to represent Canada in Paris as a nation, it also ensured that each of the dominions could sign the Treaty of Versailles in its own right, and receive a separate membership in the League of Nations. During the conference Borden tried to act as an intermediary between the United States and other members of the British Empire delegation, particularly Australia and New Zealand over the issue of Mandates. Borden also discussed with Lloyd George, the possibility of Canada taking over the administration of Belize and the West Indies, but no agreement was reached. At Borden’s insistence, the treaty was ratified by the Canadian Parliament. Borden was the last Prime Minister to be knighted after the House of Commons indicated its desire for the discontinuation of the granting of any future titles to Canadians in 1919 with the adoption of the Nickle Resolution. In 1919 Borden approved the use of troops to put down the Winnipeg general strike, which was feared to be the result of Bolshevik agitation from the Soviet Union. Sir Robert Borden retired from office in 1920. He was the Chancellor of Queen’s University from 1924 to 1930 and also was Chancellor of McGill University from 1918 to 1920 while still Prime Minister. Borden also served as Vice-President of The Champlain Society between 1923 and 1925. He was the Society’s first Honorary President between 1925 and 1938. Article Title: Robert Borden Sir Robert Laird Borden, (June 26, 1854 – June 10, 1937) was a Canadian lawyer and politician who served as the eighth Prime Minister of Canada, in office from 1911 to 1920. He is best known for his leadership of Canada during World War I. Borden was born in Grand-Pré, Nova Scotia. He worked as a schoolteacher for a period and then served his articles of clerkship at a Halifax law firm. He was called to the bar in 1878, and soon became one of Nova Scotia’s most prominent barristers. Borden was elected to the House of Commons of Canada in 1896, representing the Conservative Party. He replaced Charles Tupper as party leader in 1901, and became prime minister after the party’s victory at the 1911 federal election. As prime minister, Borden led Canada through World War I and its immediate aftermath. His government passed the "War Measures Act", created the Canadian Expeditionary Force, and eventually introduced compulsory military service, which sparked the 1917 conscription crisis. On the home front, it dealt with the consequences of the Halifax Explosion, introduced women’s suffrage for federal elections, and used the North-West Mounted Police to break up the 1919 Winnipeg general strike. Article Title: Arthur Meighen Arthur Meighen Arthur Meighen (; 16 June 1874 – 5 August 1960) was a Canadian lawyer and politician who served as the ninth Prime Minister of Canada, in office from July 1920 to December 1921 and again from June to September 1926. He led the Conservative Party from 1920 to 1926 and from 1941 to 1942. Meighen was born in rural Perth County, Ontario. He studied mathematics at the University of Toronto, and then went on to Osgoode Hall Law School. After qualifying to practice law, he moved to Portage la Prairie, Manitoba. Meighen entered the House of Commons of Canada in 1908, aged 34, and in 1913 was appointed to the Cabinet of Prime Minister Sir Robert Borden. Under Borden, Meighen served as Solicitor General (1913–1917), Secretary of State for Canada (1917), Minister of Mines (1917; 1919–1920), Minister of the Interior (1917–1920), and Superintendent-General of Indian Affairs (1917–1920). In 1920, Meighen succeeded Borden as Conservative leader and Prime Minister – the first born after Confederation, and the only one from a Manitoba riding. He suffered a heavy defeat in the 1921 election to Mackenzie King and the Liberal Party, but re-entered Parliament through a 1922 by-election and remained as Opposition Leader. LM outputs SFT:Sir Robert Borden was the eighth Prime Minister of Canada from 1911 to 1920.ArthurMeighenwastheninthPrime MinisterofCanadafrom1920to1926. Pref. RLHF: Sir Robert Laird Borden was a Canadian lawyer and politician who served as the eighth Prime Minister of Canada, in office from 1911 to 1920. Arthur Meighen was a Canadian lawyer and politician who served as the ninth Prime Minister of Canada, in office from July 1920 to December 1921 and again from June to September 1926.HeledtheConservative Partyfrom1920to1926andfrom1941to1942. FINE- GRAINED RLHF Sir Robert Laird Borden was a Canadian lawyer and politician who served as the eighth Prime Minister of Canada, in office from 1911 to 1920. Arthur Meighen was a Canadian lawyer and politician who served as the ninth Prime Minister of Canada, in office from July 1920 to December 1921 and again from June to September 1926. Table 7: Examples of errors in LM outputs of SFT, Pref. RLHF, and ourFINE-GRAINEDRLHF. We mark theC 1 (irrelevance,repetition,orincoherence)andC 2 (incorrectorunverifiablefacts)types of errors in the model outputs. 17 Question:Where does the flat iron cut of meat come from? Passages:Article Title: Flat iron steak Flat iron steak (US), butlers’ steak (UK), or oyster blade steak (Australia and New Zealand) is a cut of steak cut with the grain from the shoulder of the animal. This produces a flavorful cut that is a bit tough because it contains a gristly fascia membrane unless removed. Some restaurants offer it on their menu, often at lower price than the more popular rib-eye and strip steaks of the same grade. This is used, in some places, as a means of selling a less expensive cut from the same animal, for example Kobe beef. This cut of steak is from the shoulder of a beef animal. It is located adjacent to the heart of the shoulder clod, under the seven or paddle bone (shoulder blade or scapula). The steak encompasses the infraspinatus muscles of beef, and one may see this displayed in some butcher shops and meat markets as a "top blade" roast. Anatomically, the muscle forms the dorsal part of the rotator cuff of the steer. This cut is anatomically distinct from the shoulder tender, which lies directly below it and is the teres major. Flat iron steaks usually have a significant amount of marbling. LM outputs SFT:Flat iron steak is a cut of steak cut with the grain from the shoulder of the animal.Thisproducesaflavorfulcutthatisa bittoughbecauseitcontainsagristlyfasciamembraneunlessremoved.Somerestaurantsofferitontheirmenu,oftenat lowerpricethanthemorepopularrib-eyeandstripsteaksofthesamegrade.Thisisused,insomeplaces,asameansof sellingalessexpensivecutfromthesameanimal,forexampleKobebeef. Pref. RLHF: Flat iron steak is a cut of steak cut with the grain from the shoulder of the animal.Thisproducesaflavorfulcutthatisa bittoughbecauseitcontainsagristlyfasciamembraneunlessremoved.Somerestaurantsofferitontheirmenu,oftenat lowerpricethanthemorepopularrib-eyeandstripsteaksofthesamegrade.Thisisused,insomeplaces,asameansof sellingalessexpensivecutfromthesameanimal,forexampleKobebeef. FINE- GRAINED RLHF Flat iron steak is a cut of steak cut with the grain from the shoulder of the animal. This is located adjacent to the heart of the shoulder clod, under the seven or paddle bone (shoulder blade or scapula). This steak encompasses the infraspinatus muscles of beef,andonemayseethisdisplayedinsomebutchershopsandmeatmarketsasa"topbladeroast." Table 8: Qualitative examples on another question. Here we highlight the information in blue that is covered by the LM trained withFINE-GRAINEDRLHFbut missed (C 3 :incompleteinformation error) by SFT and Pref.RLHF. 18 A.3 Examples on Reward Model Ablation As discussed in § 4.6, reward models are competing against each other, and we experiment with removing one of the three reward models during RL training. Table 9 shows an example of how LMs behave in such scenarios. See § 4.6 for our observations. Question:How much does it cost to raise the Costa Concordia? Passages:Article Title: Costa Pacifica Costa Pacifica is a "Concordia"-class cruise ship for Costa Crociere. She was handed over to Costa Crociere on 29 May 2009. Her sister ships, "Costa Concordia" and "Costa Serena", were launched in 2006 and in 2007, with "Costa Favolosa" and "Costa Fascinosa" launched in 2011 and 2012 respectively. Onboard facilities include the Samsara Spa and PlayStation World, an area boasting PS3 consoles, and the first recording studio at sea. "Costa Pacifica" is the third ship of the "Concordia"-class, preceded by sister ships "Costa Concordia" and "Costa Serena" and was followed by "Costa Favolosa" and "Costa Fascinosa" on 2011 and 2012, all part of the expansion program of Costa which entailed an investment of 2.4 billion Euro, and is currently the largest fleet expansion program in the world. "Costa Pacifica" was ordered on 14 December 2005, by Carnival Corporation, the parent company of Costa Crociere. The order was four ships, which includes; "Carnival Dream", for Carnival Cruise Lines; "Nieuw Amsterdam" for Holland America Line; and "Ruby Princess" for Princess Cruises, at a total cost of US $3 billion. Article Title: Costa Concordia disaster On 13 January 2012, the Italian cruise ship "Costa Concordia" ran aground and overturned after striking an underwater rock off Isola del Giglio, Tuscany, resulting in 32 deaths. The eight year old Costa Cruises vessel was on the first leg of a cruise around the Mediterranean Sea when she deviated from her planned route at the Isola del Giglio, sailed closer to the island, and struck a rock formation on the sea floor. A six-hour rescue effort resulted in most of the passengers being brought ashore. "Costa Concordia" was officially declared a "constructive total loss" by the insurance company, and her salvage was "one of the biggest maritime salvage operations". On 16 September 2013, the parbuckle salvage of the ship began, and by the early hours of 17 September 2013, the ship was set upright on its underwater cradle. In July 2014, the ship was refloated by large sponsons (metal tanks) welded to its sides and was towed 320 kilometres (200 miles) to its home port of Genoa for scrapping which was finished in July 2017. The total cost of the disaster, including victims’ compensation, refloating, towing and scrapping costs, is estimated at approximately $2 billion, more than three times the $612 million construction cost of the ship. LM outputs: removing one fine-grained reward model during training All reward models: Costa Concordia is a cruise ship for Costa Crociere. On 13 January 2012, the Italian cruise ship Costa Concordia ran aground and overturned after striking an underwater rock off Isola del Giglio, Tuscany, resulting in 32 deaths. The total cost of the disaster, including victims’ compensation, refloating, towing and scrapping costs, is estimated at approximately $2 billion, more than three times the $612 million construction cost of the ship. Without relevance RMR φ 1 : On 13 January 2012, the Italian cruise ship Costa Concordia ran aground and overturned after striking an underwater rock off Isola del Giglio, Tuscany, resulting in 32 deaths.On16September2013,theparbucklesalvageoftheship began,andbytheearlyhoursof17September2013,theshipwassetuprightonitsunderwatercradle.InJuly2014,the shipwasrefloatedbylargesponsons(metaltanksweldedtoitssidesandwastowed320kilometres(200miles)toits homeportofGenoaforscrappingwhichwasfinishedinJuly2017.The total cost of the disaster, including victims’ compensation, refloating, refloating, towing and scrapping costs, is estimated at approximately $2 billion, more than three times the $612 million construction cost of the ship.Thetotalcostofthedisaster,includingvictims’compensation, refloating,towingandscrappingcosts,isestimatedatapproximately$2billion. Comment: Too much irrelevant information & repetition error. Without factuality RMR φ 2 : ThecosttoraisetheCostaConcordiawasestimatedatapproximately$2billion, more than three times the $612 million construction cost of the ship. Comment: Factuality error. Direct answer without any supporting evidence from the passages. Without info. com- pleteness RMR φ 3 : Costa Concordia is a "Concordia"-class cruise ship for Costa Crociere. She was handed over to Costa Crociere on 29 May 2009. The total cost of the disaster, including victims’ compensation, refloating, towing and scrapping costs, is estimated at approximately $2 billion, more than three times the $612 million construction cost of the ship. Comment: Miss the auxiliaryinformationaboutthedisaster. Table 9: Qualitative examples on the reward model ablation experiment. We show the LM outputs when we use all the reward models and when one of the reward models is removed during training. We also mark the errors in each output. A.4 Comparison with ChatGPT responses We compare the responses generated by ChatGPT (one-shot) 3 and our system in Table 10. As discussed in § 4.6, We find that ChatGPT responses are relevant and factual, yet lack the auxiliary information to answer the ambiguous questions. This shows that it is challenging for ChatGPT to learn user-desired behaviors through prompting and in-context learning. 3 Since the input for each example is very long, we cannot fit more than one in-context example into the model. 19 Question:When did the Rams go to St Louis? Passages:Article Title: History of the Los Angeles Rams The Los Angeles Rams are a professional American football team that play in the National Football League (NFL). The Rams franchise was founded in 1936 as the Cleveland Rams in the short-lived second American Football League before joining the NFL the next year. In 1946, the franchise moved to Los Angeles. The Rams franchise remained in the metro area until 1994, when they moved to St. Louis, and were known as the St. Louis Rams from 1995 to 2015. The Rams franchise returned to Los Angeles in 2016. This article chronicles the franchise’s history during their time in Los Angeles, from playing at the Los Angeles Memorial Coliseum between 1946 and 1979, to playing at Anaheim Stadium (now known as Angel Stadium of Anaheim) in Anaheim from 1980 to 1994, and its return to Southern California beginning with the season. On January 12, 1946, Dan Reeves was denied a request by the other National Football League (NFL) owners to move his team, the Cleveland Rams to Los Angeles and the then-103,000-seat Los Angeles Memorial Coliseum. Reeves threatened to end his relationship with the NFL and get out of the professional football business altogether unless the Rams transfer to Los Angeles was permitted. A settlement was reached and, as a result, Reeves was allowed to move his team to Los Angeles. Article Title: History of the St. Louis Rams The professional American football franchise now known as the Los Angeles Rams played in St. Louis, Missouri, as the St. Louis Rams from the 1995 through the 2015 seasons before relocating back to Los Angeles where the team had played from the 1946 season to the 1994 season. The Rams franchise relocated from Los Angeles to St. Louis in 1995, which had been without a National Football League (NFL) team since the Cardinals moved to Phoenix, Arizona in 1988. The Rams’ first home game in St. Louis was at Busch Memorial Stadium against the New Orleans Saints on September 10, 1995, before the Trans World Dome (later the Edward Jones Dome, and now known as The Dome at America’s Center) was completed for their November 12 game against the Carolina Panthers. Their last game played at the Edward Jones Dome in St. Louis was against the Tampa Bay Buccaneers on December 17, 2015, which they won, 31–23. The Rams’ last game as a St. Louis-based club was on January 3, 2016, against the San Francisco 49ers at Levi’s Stadium, where they lost in overtime 19–16. Comparison between our output and ChatGPT output ChatGPT:The Rams moved to St. Louis in 1995 and were known as the St. Louis Rams from 1995 to 2015. FINE- GRAINED RLHF The Los Angeles Rams are a professional American football team that play in the National Football League (NFL). The Rams franchise was founded in 1936 as the Cleveland Rams in the short-lived second American Football League before joining the NFL the next year. In 1946, the franchise moved to Los Angeles. The Rams franchise remained in the metro area until 1994, when they moved to St. Louis, and were known as the St. Louis Rams from 1995 to 2015. The Rams franchise returned to Los Angeles in 2016. Question:Who sang the song I’m Coming Out? Passages:Article Title: I’m Coming Out There’s also been a cover by Marcia Hines. It was featured as a cover for the video game "Karaoke Revolution" as well. In more recent times, the song was sampled in Keyshia Cole’s remake of "I’m Coming Out" (featuring Iggy Azalea) for the original soundtrack of the 2014 movie The Other Woman, starring Cameron Diaz, Kate Upton and Leslie Mann. The song is sampled in "Break Your Heart Right Back" from Ariana Grande’s sophomore album "My Everything" (2014). This song was also covered for the soundtrack to the 2016 movie "Trolls". In 1997, German comedian Hella von Sinnen sampled "I’m Coming Out" for her single "Mein Coming Out", using the term "coming out" in the context of coming out of the closet, or revealing one’s homosexuality. In 2003, American singer Amerie covered "I’m Coming Out" for the soundtrack to the film "Maid in Manhattan". Her version was released as a single in select European countries and Australia. The final single version to be released was the Loren Dawson remix. The original single version (which is similar to the original) can be found on both CD singles and the original motion picture soundtrack to "Maid in Manhattan". Ross’ version appears on the film’s soundtrack album as well. Article Title: I’m Coming Out "I’m Coming Out" is a song recorded by American singer Diana Ross. It was written and produced by Chic members Bernard Edwards and Nile Rodgers, and released in August 22, 1980 as the second single from Ross’ self-titled tenth album "Diana" (1980). In 1979, Ross commissioned Chic founders Nile Rodgers and Bernard Edwards to create material for a new album after taking her daughters to see the band in concert, following the success of Ross’s final Ashford & Simpson-composed and produced LP," The Boss. "Rodgers got the idea for "I’m Coming Out" after noticing three different drag queens dressed as Diana Ross at a New York club called the G Barnum Room. The lyrics hold an additional meaning to Ross, as she was leaving Motown Records and "coming out" from under Berry Gordy’s thumb. According to the BBC4 documentary "How to Make It in the Music Business", Ross loved the record until she put it in front of Frankie Crocker, who pointed out that ’I’m Coming Out’ is what homosexuals use to announce their homosexuality; she ran back to the studio in tears and demanded to know why Rodgers wanted to ruin her career. The song was another hit from the album, peaking at number five on the U.S. Comparison between our output and ChatGPT output ChatGPT: The song “I’m Coming Out" was recorded by American singer Diana Ross. It was written and produced by Chic members Bernard Edwards and Nile Rodgers, and released in August 22, 1980 as the second single from Ross’ self-titled tenth album “Diana" (1980). FINE- GRAINED RLHF “I’m Coming Out" is a song recorded by American singer Diana Ross. It was written and produced by Chic members Bernard Edwards and Nile Rodgers, and released in August 22, 1980 as the second single from Ross’ self-titled tenth album Diana. In 2003, American singer Amerie covered "I’m Coming Out" for the soundtrack to the film Maid in Manhattan. Her version was released as a single in select European countries and Australia. The final single version to be released was the Loren Dawson remix. Table 10: Comparison between ChatGPT outputs and FINE-GRAINEDRLHFoutputs. 20 B Algorithm and Training Details of FINE-GRAINEDRLHF B.1 PPO with Fine-Grained Rewards The algorithm below shows in detail how PPO updates the policy LMP θ and the value modelV ψ withKfine-grained reward modelsR φ k . Algorithm 1Fine-Grained Reinforcement Learning from Human Feedback (FINE-GRAINEDRLHF) Inputinitial policy modelP θ init ; initial value modelV ψ init ;Kreward modelsR φ k trained from human feedback; task promptsD; hyperparametersγ,λ,ε,β▷§ 2 1:policy modelP θ ←P θ init , value modelV ψ ←V ψ init 2:forstep = 1, . . . , Mdo 3:Sample a batchD b fromD 4:Sample output sequencey n ∼P θ (·|x n )for each promptx n ∈D b 5:Compute rewardsr n t |y n | t=1 for each sampled outputy n by runningR φ k ▷Eq. 1 6:Compute advantagesA t |y n | t=1 and value targetsV targ (s t ) |y n | t=1 for eachy n withV ψ 7:forPPO iteration = 1, . . . ,μdo 8:Update the policy model by maximizing the PPO clipped surrogate objective: θ←arg max θ 1 |D b | |D b | X n=1 1 |y n | |y n | X t=1 min P θ (a t |s t ) P θ old (a t |s t ) A t ,clip(v t ,1−ε,1 +ε)A t 9:Update the value model by minimizing a square-error objective: ψ←arg min ψ 1 |D b | |D b | X n=1 1 |y n | |y n | X t=1 V ψ (s t )−V targ (s t ) 2 OutputP θ B.2 Implementation Details Model architectures.For the detoxification experiments, the policy model is initialized with GPT2- large [31], and the value model is initialized with GPT2-base. For the long-form QA experiments, the policy model is initialized with a supervised fine-tuned T5-large [32], and the value model is initialized with T5-base. This design follows InstructGPT [29], which uses a larger (175B) policy model, and smaller value and reward (6B) models. Training details on detoxification.For both the holistic reward baseline and the sentence-level (fine-grained) reward, we do a hyper-parameter search with the same set of hyper-parameters. For training, we run 200K episodes. The batch size (number of episodes per card during training) is 64. We use Adam optimizer with a linear learning rate scheduler and 10 warmup steps. We perform a hyper-parameter grid-search for peak learning rate∈ 5e−6,1e−5,2e−5, KL coefficient β∈ 0.1,0.2,0.3, discounting factorλ∈ 0.95,0.97,0.99, and the frequency of exploration (number of sampled outputs)∈2,4,8. We find that the higher the KL coefficient, the lower the perplexity, and the higher toxicity. This is consistent with findings from previous RLHF studies ([29], [33]). For a fair comparison, we eventually choose a set of parameters that achieve a similar level of perplexity for both reward models. The optimal set of hyper-parameters for holistic reward is β= 0.3,λ= 0.99. For sentence-level rewardβ= 0.1,λ= 0.95. The learning rate is1e−5, and the exploration frequency is4for both experiments. We choose the checkpoint with the lowest validation set toxicity for evaluation. Regarding computation time, we use2×80G NVIDIA A100 GPU for training, and the run time is about 22 hours. Training details on long-form QA.We conduct a similar hyper-parameter grid search as our detoxification experiments. For long-Form QA, the input length limit is 1024, and the output length limit is 200. Notice that this is much longer than detoxification, so we use a smaller batch size and fewer training episodes. We experiment with multiple combinations of reward model weights. Fixingw 2 = 0.5(factuality reward weight), we perform a grid search onw 1 ,w 3 ∈[0.0,0.5]. 21 We eventually choosew 1 = 0.3,w 2 = 0.5,w 3 = 0.3, which reaches a balance between three reward models and allows all three rewards to increase during training. For training, the batch size (number of episodes per card during training) is 32. We use Adam optimizer with a linear learning rate scheduler and 100 warmup steps. We perform a hyper-parameter grid-search for peak learning rate∈ 5e−6,1e−5,2e−5, KL coefficientβ∈ 0.1,0.2,0.3, discounting factor λ∈ 0.95,0.97,0.99, and the frequency of exploration∈ 2,4,8. The optimal set of hyper- parameters for Pref. RLHF isβ= 0.2,λ= 0.99. ForFINE-GRAINEDRLHF,β= 0.3,λ= 0.95. The learning rate is1e−5, and the exploration frequency is4for both experiments. we run 80K episodes, which is approximately 5 epochs. We choose the checkpoint with the highest validation reward for evaluation. Regarding computation time, we use2×80G NVIDIA A100 GPU for training, and the run time is about 15 hours. A note on the error bars.All results we report in the paper are from 3 independent runs. The scores reported are all averaged across all runs. The error bars are represented as the shades behind each training curve in our figures. It shows the standard error across three runs. C Long-Form QA Data and Human Feedback Annotation C.1 Data Construction ASQA [39] is a long-form QA dataset that focuses on answering ambiguous factoid questions in anopen-domainsetting that requires passage retrieval from a given Wikipedia passage corpus. We reformulate it into areading comprehensionsetting: given the inputxthat contains a questionqand a set of knowledge passagesP=p 1 ,...,p |P| , generate a long-form responsey. To constructP for each inputx, we use the oracle knowledge contexts provided by ASQA for eachx, that are text snippets from the passage corpus. We use BM25 4 to map each knowledge context (text snippet) to the closest passage from the passage corpus and use the resulting passages asP. Our train and dev examples come from the original ASQA train set and our test examples are the original ASQA dev examples. C.2 Human Feedback Annotation Fine-grained feedback.As discussed in § 4.1, we first use 1K randomly sampled training examples to train a T5-large based supervised modelSFTas the initial policy modelP θ init . Then we collect feedback on sampled outputs from SFT for the remaining 2,853 training examples and the 500 development examples, using the Amazon Machanical Turk platform. 5 Figure 5 shows the fine-grained human feedback annotation interface with an example fromQA- FEEDBACK. In addition to the task input—questionqand oracle passagesP, we also provide a human-written response from ASQA to the worker as reference. However, it is important to note that, in practice, the annotation of our fine-grained feedback should not require the human-written response. The only purpose for us to provide the gold response is to have our workers follow the same question interpretation and expected response of the workers who annotate for ASQA, such that our experimental comparison with supervised models (SFTandSFT-Full; details in § 4.3) is fair. However, we still instruct our workers to strictly use the given passages for checking factual errors. For each span error, we ask the worker to select one out of 5 categories shown in Figure 6 (left). 6 However, we collapse these 5 categories into two categories (C 1 andC 2 mentioned in § 4.1) based on whether the error detection depends on the passages or not. When workers mark passage sentences as containing missing information, we instruct them to categorize each sentence as missing “answer”, “major auxiliary information” or “minor auxiliary information,” as shown in Figure 6 (right). Our instruction to the worker is provided in Figure 8. Quality control.Before feedback collection, we design a qualification task to select qualified workers for this feedback annotation task. The qualification task consists of 5 questions with their corresponding passages and model outputs for the workers to annotate. We manually review about 4 https://github.com/castorini/pyserini 5 https://w.mturk.com/ 6 We see very few “incoherence” errors (1%), so the majority of labeled errors are from the other four categories during annotation. 22 70 submissions of the qualification task and select 15 workers whose annotation is marked by us as of high quality. Throughout the actual feedback annotation process, we constantly monitor the annotated data and send feedback to workers. Preference-based feedback.For comparison purposes, we follow [29] to collect pairwise human preferences from the same group of workers we select from the qualification task. We sample four model outputs for each promptx, which gives 6 pairs of model outputs. Similarly, we provide the worker with the human-written response and ask the workers to indicate pairwise preferences (ties are allowed) based on all errors they can find each model output. Figure 7 shows the preference-based human feedback annotation interface with an example from QA-FEEDBACK. Pay structure.We pay a base rate of $1.5 per example for annotating fine-grained or preference feedback. If the example consists of≥3passages to read, we assign an additional $0.3 bonus to the example. On average, we pay roughly $1.65 per example for both tasks, which gives an $16.5 hourly pay for our workers. C.3 Analysis of Collected Fine-Grained Feedback Overall, among allerror spanswe collect, 76% of them areC 1 errors and the remaining 24% are C 2 errors. However, it is important to note that we instruct workers to labelC 2 errors only at places that don’t have aC 1 error. 75% examples are labeled as being incomplete; i.e., containing missing information that can be found in the given passages (C 3 ). Among all marked passage sentences that contain missing information, 31%, 42% and 27% are missing answer, major auxiliary information and minor auxiliary information respectively. To analyze human-human agreement, a subset of 300 examples receive annotations from two distinct workers. We observe that while the exact agreement in error span boundaries is low, workers achieve reasonably high agreement on whether a sub-sentence containsC 1 (reach an agreement for 83% of all sub-sentences) and whether a sentence containsC 2 (92%). 7 The agreement on whether a model output contains complete information or not (C 3 ) is 85%. Therefore, we decide to have the density for error typeC 1 ,C 2 , andC 3 as sub-sentence, sentence and full sequence. D Long-Form QA Reward Model Training Details We train reward models with the 2,835 training examples with feedback collected and select the best model for each error category based on the their performance on the development set. The batch size and training epochs are 24 and 50 forR φ 1 andR φ 2 . Each training is run on a single 80G NVIDIA A100 GPU, taking 1 and 2 hours for trainingR φ 1 andR φ 2 respectively. 8 The batch size and training epochs are 12 (per GPU) and 30 forR φ 3 and the preference-based reward model. Each training is run on2×80G NVIDIA A100 GPU and takes 2 hours. We use Adam optimizer with a linear learning rate scheduler for all reward model training. For each reward model, we search the learning rate over 5e −6 ,1e −5 ,5e −5 , weight decay over0.001,0.01, and warm-up step ratio over0.1,0.2based on the dev set performance. Specifically, we use the model forR φ 1 andR φ 2 that achieve the best binary classification accuracy. ForR φ 3 and the preference-based reward model, we select the model that achieves the best pairwise comparison accuracy. We also provide more training details for each reward model below. R φ 1 forC 1 :Irrelevance,repetition,orincoherence.To train the reward modelR φ 1 that detects error of irrelevance, repetition, or incoherence, we apply a token-level classification loss to each [sep]token beforey 1 j , where its gold labelg j is “has error” if there is af i ∈Fthat hasa b i ,...,e i overlapped withy 1 j andc i = 1, and “no error” otherwise. We observe that most of the spans marked as error typeC 1 that are shorter than 5 words usually carry very little information or are annotated as a result of workers being very careful or strict. Therefore, we filter out such short spans before constructing training examples forR φ 1 . Overall, we get 7379 and 8059 sub-sentences with the “has error” and “no error” label respectively. 7 We use spaCy [15] to segment generated model outputs into sentences. We then split sentences into sub-sentences using a comma or semicolon. 8 Note that trainingR φ 1 takes shorter time as its input does not contain passages. 23 R φ 2 forC 2 :Incorrectorunverifiablefacts.We trainR φ 2 in a similar way as how we trainR φ 1 . Instead of predicting the error for each sub-sentence,R φ 2 is trained to predict at the sentence level (i.e.,y 2 j is thej th sentence iny). Since workers do not annotateC 2 error for spans that are already labeled as havingC 1 error, in order to avoid false negatives in trainingR φ 2 , we do not provide gold label nor calculate loss for sentences that only containC 1 error from training. In other words, all sentences that contain aC 2 error has the gold label “has error” and sentences that contain no error has the gold label “no error”. Overall, we get 1600 and 3411 sentences with the “has error” and “no error” label respectively. R φ 3 forC 3 :Incompleteinformation.Instead of casting this as a classification task,R φ 3 predicts a single scalar reward givenxandyand is trained with a pairwise comparison loss [29]. This is motivated by early work [19] that shows the better reliability of pairwise comparison than error classification when assessing a full generation sequence. To construct training data forR φ 3 , we bootstrap pairwise comparisons from the corrected model outputy ′ as follows. We first map each sub-sentence iny ′ to a passage sentence inPthat has a sub-string with the highest token-level F1 score with the sub-sentence, 9 and denote all mapped sentences asS. We then sample four responses from SFT, for each we do the same sentence mapping to get a set of passages sentencesS ′ . We calculatescore(y) =|S ′ ∩S|/|S|as the information completeness score for each model responsey. We follow [29] to pair up sampled responses forqand denote each sampled response pair as ( ̄y p , ̄y l ), wherescore( ̄y p )> score( ̄y l ). We drop the pairs wherescore( ̄y p ) =score( ̄y l ). Then we follow [29] to trainR φ 3 with the loss function in Eq. 2. We have a total number of 6821 pair examples in training. Preference-based reward model.The preference-based reward model is trained in a similar way as R φ 3 , with ̄y p representing the human preferred response against ̄y l in the loss function Eq. 2. We drop the pairs where a tie is indicated. We have a total number of 14981 pair examples in training. 9 We manually review 50 mapped passage sentences and find over 90% of them are correctly mapped, which indicates frequent extractive behaviors fromP θ init . 24 Figure 5: Fine-grained feedback annotation interface. Figure 6: Error types in the fine-grained feedback annotation interface. Figure 7: Preference feedback annotation interface. The task input is omitted (same as in Figure 5). 25 Figure 8: Fine-grained feedback annotation instructions. 26

Instant research discovery