Paper deep dive

Is Reinforcement Learning (Not) for Natural Language Processing?: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization

Rajkumar Ramamurthy, Prithviraj Ammanabrolu, Kianté Brantley, Jack Hessel, Rafet Sifa, Christian Bauckhage, Hannaneh Hajishirzi, Yejin Choi

Year: 2022Venue: arXiv preprintArea: Alignment TrainingType: BenchmarkEmbeddings: 189

Models: GPT-2, T5-base

Intelligence

Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 96%

Last extracted: 3/12/2026, 8:03:54 PM

Summary

The paper introduces RL4LMs, an open-source library for training language models with reinforcement learning, and the GRUE benchmark, a collection of seven NLP tasks evaluated via reward functions. It also proposes NLPO (Natural Language Policy Optimization), an algorithm designed to improve training stability and performance in large action spaces by dynamically masking tokens, demonstrating superior alignment with human preferences compared to standard PPO.

Entities (5)

GRUE · benchmark · 100%NLPO · algorithm · 100%RL4LMs · software-library · 100%HuggingFace · organizationplatform · 95%PPO · algorithm · 95%

Relation Signals (3)

GRUE → evaluates → NLPO

confidence 95% · We use RL4LMs to test a large range of algorithms on the GRUE benchmark... We compare 3 algorithms... PPO, and NLPO.

NLPO → outperforms → PPO

confidence 95% · NLPO exhibits greater stability and performance than previous policy gradient methods (e.g., PPO)

RL4LMs → supports → NLPO

confidence 95% · RL4LMs provides reliable implementations of popular on-policy RL algorithms... and our own NLPO.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:We tackle the problem of aligning pre-trained large language models (LMs) with human preferences. If we view text generation as a sequential decision-making problem, reinforcement learning (RL) appears to be a natural conceptual framework. However, using RL for LM-based generation faces empirical challenges, including training instability due to the combinatorial action space, as well as a lack of open-source libraries and benchmarks customized for LM alignment. Thus, a question rises in the research community: is RL a practical paradigm for NLP? To help answer this, we first introduce an open-source modular library, RL4LMs (Reinforcement Learning for Language Models), for optimizing language generators with RL. The library consists of on-policy RL algorithms that can be used to train any encoder or encoder-decoder LM in the HuggingFace library (Wolf et al. 2020) with an arbitrary reward function. Next, we present the GRUE (General Reinforced-language Understanding Evaluation) benchmark, a set of 6 language generation tasks which are supervised not by target strings, but by reward functions which capture automated measures of human preference. GRUE is the first leaderboard-style evaluation of RL algorithms for NLP tasks. Finally, we introduce an easy-to-use, performant RL algorithm, NLPO (Natural Language Policy Optimization) that learns to effectively reduce the combinatorial action space in language generation. We show 1) that RL techniques are generally better than supervised methods at aligning LMs to human preferences; and 2) that NLPO exhibits greater stability and performance than previous policy gradient methods (e.g., PPO (Schulman et al. 2017)), based on both automatic and human evaluations.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

188,892 characters extracted from source content.

Expand or collapse full text

Published as a conference paper at ICLR 2023 ISREINFORCEMENTLEARNING(NOT)FORNATURAL LANGUAGEPROCESSING: BENCHMARKS, BASELINES, ANDBUILDINGBLOCKS FORNATURALLANGUAGE POLICYOPTIMIZATION Rajkumar Ramamurthy *♠ Prithviraj Ammanabrolu *♥ Kianté Brantley ♣ Jack Hessel ♥ Rafet Sifa ♠ Christian Bauckhage ♠ Hannaneh Hajishirzi ♦♥ Yejin Choi ♦♥ ♠ Fraunhofer IAIS ♥ Allen Institute for Artificial Intelligence ♣ Cornell University ♦ Paul G. Allen School of Computer Science, University of Washington rajkumar.ramamurthy@iais.fraunhofer.de raja,jackh@allenai.org;kdb82@cornell.edu ABSTRACT We tackle the problem of aligning pre-trained large language models (LMs) with human preferences. If we view text generation as a sequential decision-making problem, reinforcement learning (RL) appears to be a natural conceptual framework. However, using RL for LM-based generation faces empirical challenges, including training instability due to the combinatorial action space, as well as a lack of open- source libraries and benchmarks customized for LM alignment. Thus, a question rises in the research community:is RL a practical paradigm for NLP? To help answer this, we first introduce an open-source modular library,RL4LMs 1 , 2 for optimizing language generators with RL. The library consists of on-policy RL algorithms that can be used to train any encoder or encoder-decoder LM in the HuggingFace library (Wolf et al., 2020) with an arbitrary reward function. Next, we present theGRUE (General Reinforced-language Understanding Evaluation) benchmark, a set of 6 language generation tasks which are supervised not by target strings, but by reward functions which capture automated measures of human preference. GRUE is the first leaderboard-style evaluation of RL algorithms for NLP tasks. Finally, we introduce an easy-to-use, performant RL algorithm,NLPO (Natural Language Policy Optimization)that learns to effectively reduce the combinatorial action space in language generation. We show 1) that RL techniques are generally better than supervised methods at aligning LMs to human preferences; and 2) that NLPO exhibits greater stability and performance than previous policy gradient methods (e.g., PPO (Schulman et al., 2017)), based on both automatic and human evaluations. 1INTRODUCTION The ultimate aim of language technology is to interact with humans. However, most language models are trained without direct signals of human preference, with supervised target strings serving as (a sometimes crude) proxy. One option to incorporate user feedback is via human-in-the-loop, i.e., a user would be expected to provide feedback for each sample online as the model trains, but this degree of dense supervision is often prohibitive and inefficient. Automated metrics offer a promising compromise: models of human preference like pairwise learned preference models (Ouyang et al., 2022), BERTScore (Zhang et al., 2019), BLEURT (Sellam et al., 2020) have significantly improved correlation with human judgment compared to earlier metrics (BLEU, METEOR, etc.), and are cheap to evaluate. But — these functions are usually not per-token differentiable: like humans, metrics * Denotes Equal Contribution 1 Code:https://github.com/allenai/RL4LMs 2 Project Website:https://rl4lms.apps.allenai.org/ 1 arXiv:2210.01241v3 [cs.CL] 1 Mar 2023 Published as a conference paper at ICLR 2023 0.130.580.92 0.510.250.87 I loved the book, but really hated the movie... At first anyway, I warmed slowly as I watched on... An aempt that fell short fell short fell short ... Awful awful awful, I'm going to tell you why ... Exploration Feedback Learning Preference Rewards Naturalness KL Penalty Environment Textual Input Scalar Output Figure 1:Natural Language Policy Optimization (NLPO)in the case of sentiment-guided contin- uation. Here, the LM (i.e., the policy) needs to produce a positive sentiment continuation given a review prompt (we cover other models of human preference in Sec. 3.2). Two objectives are balanced: 1) an automated proxy of human preference that serves as a reward (here: a sentiment classifier); and 2) “naturalness" as measured by a KL divergence from an LM not trained with explicit human feedback. The plots show validation learning curves comparing our NLPO to the popular policy gradient method PPO. (Top plot:) RL methods can easily achieve high reward if the KL penalty is removed, (Bottom:) but at the cost of higher perplexity. NLPO+KL, our proposed approach, succeeds in balancing reward and naturalness more effectively than prior work. can only offer quality estimates for full generations. Reinforcement Learning (RL) offers a natural path forward for optimizing non-differentiable, scalar objectives for LM-based generation when it is cast as a sequential decision-making problem. However, Goodhart’s Law 3 looms: particularly in the case of imperfect metrics that use neural networks, it is easy to find nonsense samples that achieve high-quality estimates. Recent works have shown promising results in aligning LMs to human preferences via RL by constraining preference-based rewards to incorporate notions of fluency (Wu et al., 2021a; Ouyang et al., 2022) but progress in this line of work is heavily hindered by a lack of open-source benchmarks and algorithmic implementations—resulting in perception that RL is a challenging paradigm for NLP (Choshen et al., 2020; Kreutzer et al., 2021). To facilitate research in building RL algorithms to better align LMs, we release a library, a benchmark, and an algorithm. First, we release theRL4LMs library, which enables generative HuggingFace models (e.g., GPT-2 or T5) to be trained using a variety of existing RL methods like PPO/A2C/etc. Next, we apply models trained using RL4LMs to the newGRUE (General Reinforced-language Understanding Evaluation)benchmark: GRUE is a collection of 7 contemporary NLP tasks (see Table1 for details); in contrast to other benchmarks, instead of supervised training, we pair each task with reward function(s). GRUE challenges models to optimize these reward functions while remaining fluent language generators. We train language models via RL—both with and without task specific supervised pre-training—to optimize rewards. Finally, beyond existing RL methods, we introduce a novel on-policy RL algorithm calledNLPO (Natural Language Policy Optimization), that dynamically learns task-specific constraints over the distribution of language at a token level. Experiments on GRUE and human evaluations show that NLPO better balances learning preference rewards while maintaining language fluency compared to alternatives, including PPO (Figure 1). We find that using RL to learn from scalar reward feedback can be more: (1) data efficient than using additional expert demonstrations via supervised learning (though a combination of both is best)—a learned reward function enables greater performance when used as a signal for an RL method than a supervised method trained with 5 times more data, and (2) parameter efficient—enabling a 220 million parameter model trained with a combination of supervision and NLPO to outperform a 3 billion supervised model. We hope that the benchmarks, baselines, and building blocks we release serve to drive forward research in aligning LMs to human preferences. 3 Strathern (1997) paraphrases:When a measure becomes a target, it ceases to be a good measure. 2 Published as a conference paper at ICLR 2023 2RELATEDWORK Imitation learning for NLP.Algorithms such as Schedule Sampling (S) (Bengio et al., 2015), Parallel S (Duckworth et al., 2019), S for Transformers (Mihaylova & Martins, 2019), Diffential S (Goyal et al., 2017), LOLS (Lampouras & Vlachos, 2016; Chang et al., 2015), TextGAIL (Wu et al., 2021b), and SEARNN (Leblond et al., 2017), have been inspired by DAGGER (Ross et al., 2011) and SEARN (Daumé et al., 2009). However, these algorithms are known to suffer from exposure bias in generation (Chiang & Chen, 2021; Arora et al., 2022) and the cliff MDP problem (Huszár, 2015; Agarwal et al., 2019; Swamy et al., 2021). RL for Large Action Spaces.MIXER (Ranzato et al., 2016) combined ideas from schedule sampling and REINFORCE (Williams, 1992). Bahdanau et al. (2016) proposed an actor-critic algorithm to address the variance/large action space problems when using REINFORCE for language generation; follow-up works such as KG-A2C (Ammanabrolu & Hausknecht, 2020), TrufLL (Martin et al., 2022), AE-DQN (Zahavy et al., 2018), and GALAD (Ammanabrolu et al., 2022) addressed similar issues by attempting to eliminate and reduce the action space during exploration. RL for NLP.RL, often in the form of bandit learning, has been used to improve models in machine translation (Wu et al., 2016; Nguyen et al., 2017; Kiegeland & Kreutzer, 2021), summarization (Stiennon et al., 2020; Paulus et al., 2017), dialogue (Li et al., 2016; Zhou et al., 2017; Jaques et al., 2020), image captioning (Rennie et al., 2017), question generation (Pang & He, 2021), text-games (Narasimhan et al., 2015; Hausknecht et al., 2020), and more (Ranzato et al., 2016; Snell et al., 2022). Lu et al. (2022) adapt reward-conditioned transformers (Chen et al., 2021) for several language generation tasks. RL has been the focus of efforts to align LMs with human preferences (Stiennon et al., 2020; Wu et al., 2021a; Nakano et al., 2021; Ziegler et al., 2019), e.g., Ouyang et al. (2022) fine-tuned large language model with PPO Schulman et al. (2017) to align with models of human preference, but their non-public dataset doesn’t enable comparison. Though RL has been successful in some of the use cases described above, it has simultaneously been critiqued for being significantly less stable than supervised LM training (Choshen et al., 2020). As a result, there is relatively little consensus if RL is a worthwhile consideration for training LMs compared to, say, collecting additional supervised data. 3RL4LMS: A LIBRARY FORTRAININGLMS WITHRL We introduce RL4LMs, an open-source library with building blocks for fine-tuning and evaluating RL algorithms on LM-based generation. The library is built on HuggingFace (Wolf et al., 2020) and stable-baselines-3 (Raffin et al., 2021), combining important components from their interfaces. RL4LMs can be used to train any decoder only or encoder-decoder transformer models from Hug- gingFace with any on-policy RL algorithm from stable-baselines-3. Furthermore, we provide reliable implementations of popular on-policy RL algorithms that are tailored for LM fine-tuning such as PPO (Schulman et al., 2017), TRPO (Schulman et al., 2015a), A2C (Mnih et al., 2016), and our own NLPO (§4). The library is modular, which enables users to plug-in customized environments, reward functions, metrics, and algorithms. In the initial release, we provide support for 6 different NLP tasks, 16 evaluation metrics and rewards, and 4 RL algorithms. 3.1ENVIRONMENTS: GENERATION AS ATOKEN-LEVELMDP Each environment is an NLP task: we are given a supervised datasetD=(x i ,y i ) N i=1 ofN examples, wherex∈ Xis an language input andy∈ Yis the target string. Generation can be viewed as a Markov Decision Process (MDP)〈S,A,R,P,γ,T〉using a finite vocabularyV. Each episode in the MDP begins by sampling a datapoint(x,y)from our dataset and ends when the current time steptexceeds the horizonTor an end of sentence (EOS) token is generated. The input x= (x 0 ,·,x m )is a task-specific prompt that is used as our initial states 0 = (x 0 ,·,x m ), where s 0 ∈ SandSis the state space withx m ∈ V. An action in the environmenta t ∈ Aconsists of a token from our vocabularyV. The transition functionP:S×A→∆(S)deterministically appends an actiona t to the end of the states t−1 = (x 0 ,·,x m ,a 0 ,·,a t−1 ). This continues until the end of the horizont≤Tand we obtain a states T = (x 0 ,·,x m ,a 0 ,·,a T ). At the end of an episode a rewardR:S×A×Y →R 1 that depends on the (s T ,y) (e.g., an automated metric like PARENT Dhingra et al. (2019)) is emitted. RL4LMs provides an OpenAI gym (Brockman et al., 2016) style 3 Published as a conference paper at ICLR 2023 API for an RL environment that simulates this LM-Based MDP formulation. This abstraction allows for new tasks to be added quickly with compatibility across all implemented algorithms. 3.2REWARDFUNCTIONS ANDEVALUATIONMETRICS Because RL4LMs provides a generic interface for per-token or per-sequence generation rewards, it is possible to quickly apply a wide array of RL algorithms to a similarly diverse range of textual metrics-as-rewards. Specifically, we provide interfaces to 1)n-gram overlap metricsmetrics such as ROUGE (Lin, 2004), BLEU (Papineni et al., 2002), SacreBLEU (Post, 2018), METEOR (Banerjee & Lavie, 2005); (2)model-based semantic metricssuch as BertScore (Zhang et al., 2019) and BLEURT (Sellam et al., 2020) which generally provide higher correlation with human judgment; 3)task-specific metricssuch as CIDER (Vedantam et al., 2015), SPICE (Anderson et al., 2016) (for captioning/commonsense generation), PARENT (Dhingra et al., 2019) (for data-to-text) and SummaCZS (Laban et al., 2022) (for factuality of summarization); 4)diversity/fluency/naturalness metrics such as perplexity, Mean Segmented Type Token Ratio (MSSTR) (Johnson, 1944), Shannon entropy over unigrams and bigrams (Shannon, 1948), the ratio of distinct n-grams over the total number of n-grams (Distinct-1, Distinct-2) and count of n-grams that appear only once in the entire generated text (Li et al., 2015); 5)task-specific, model-based human preference metricssuch as classifiers trained on human preference data collected in the methodology of Ouyang et al. (2022). 3.3ON-POLICYACTOR-CRITICALGORITHMS RL4LMs supports fine-tuning and training LMs from scratch via on-policy actor-critic algorithms on language environments. Formally, this class of algorithms allows us to train a parameterized control policy defined asπ θ :S →∆(A), a function that attempts to select an action in a given state so as to maximize long term discounted rewards over a trajectoryE π [ ∑ T t=0 γ t R(s t ,a t )] . Our benchmark experiments focus on fine-tuning a pre-trained LM denoted asπ 0 from which we initial our agent’s policyπ θ =π 0 . Similarly, the value networkV φ used to estimate the value function is also initialized fromπ 0 except for the final layer which is randomly initialized to output a single scalar value. As with other deep RL actor-critic algorithms, we define our value and Q-value functions as V π t =E a t ∼π [ ∑ T τ=t γR(s τ ,a τ ,y)],Q π t (s t ,a t ) =R(s t ,a t ,y) +γE s t+1 ∼P [V π t+1 (s t+1 )] leading to a definition of our advantage function asA π t (s,a) =Q π t (s,a)−V π t . To increase training stability, advantage is appoximated using Generalized Advantage Estimation (Schulman et al., 2015b). Given an input-output pair(x,y)and generation predictions from our agent; because the environment rewards are sequence-level and sparse, following Wu et al. (2021a) we regularize the reward function using a token-level KL penalty for all on-policy algorithms, to prevent the model from deviating too far from the initialized LMπ 0 . Formally, the regularized reward function is: ˆ R(s t ,a t ,y) =R(s t ,a t ,y)−βKL(π θ (a t |s t )||π 0 (a t |s t ))(1) where ˆ Ris the regularized KL reward,yis gold-truth predictions,KL(π θ (a t |s t )||π 0 (a t |s t )) = (logπ 0 (a t |s t )−logπ θ (a t |s t ))and the KL coefficientβis dynamically adapted (Ziegler et al., 2019). Further details on actor-critic methods can be found in Appendix A. 4NLPO: NATURALLANGUAGEPOLICYOPTIMIZATION Language generation action spaces are orders of magnitude larger than what most discrete action space RL algorithms are designed for (Ranzato et al., 2016; Ammanabrolu, 2021), e.g., GPT-2/3 and T5 have a vocabulary size of 50K and 32K respectively. We hypothesize that the size of the action space is a core cause of instability when training LMs with existing RL methods. To address this issue, we introduce NLPO (Natural Language Policy Optimization), which is inspired by work on action elimination/invalid-action masking (Zahavy et al., 2018; Huang & Ontañón, 2020; Ammanabrolu & Hausknecht, 2020). NLPO, a parameterized-masked extension of PPO, learns to mask out less relevant tokens in-context as it trains. NLPO accomplishes this via top-psampling, which restricts tokens to the smallest possible set whose cumulative probability is greater than the probability parameterp(Holtzman et al., 2018). Specifically, NLPO maintains amasking policyπ ψ : the masking policy is a copy of the current policy (π θ ), but is updated only everyμsteps. A parameterized-invalid-mask is created fromπ ψ by first 4 Published as a conference paper at ICLR 2023 selecting the top-ptokens from the vocabulary, 4 and then applying an invalid-mask to the remaining tokens—i.e. setting their probabilities to zero when sampling actions fromπ θ during training; this periodic updating policyπ ψ is inspired by off-policy Q-learning algorithms (Andrychowicz et al., 2017), providing the policyπ θ with an additional constraint that balances between the benefits of containing more task relevant information than the KL penalty derived fromπ 0 and the risk of reward hacking. We provide pseudocode in Algorithm 1 (green portions highlight the differences with PPO). Algorithm 1NLPO - Natural Languge Policy Optimization Input:DatasetD=(x i ,y i ) N i=1 of sizeN Input:initial policy parametersπ θ 0 Input:initial LMπ 0 Input:initial value function parametersV φ 0 Input:initialize parameterized masked policyπ ψ 0 (·|·,π θ 0 )with parameterized top-ppolicyπ θ 0 Input:policy update frequencyμ repeat Sample mini-batchD m =(x m ,y m ) M m=1 fromD Collect trajectoriesT m =τ i by running policyπ ψ n in for batchD m in env..Eq.6 Compute Preference and KL penalty rewards ˆ R t .Eq. 1 Compute the advantage estimate ˆ A t .Sec. 3.3 Update the policy by maximizing the PPO-Clip objective: π θ m+1 =argmax θ 1 |D m |T ∑ τ∈D m T ∑ τ=0 min ( r t (θ)A π θ m ,clip(r t (θ),1−,1 +)A π θ m ) ) wherer t (θ) = π θ (a t |s t ) π θ m (a t |s t ) . Update the value function: V φ m+1 =argmin φ 1 |D m |T ∑ τ∈D m T ∑ t=0 ( V φ (s t )− ˆ R t ) 2 Update the parameterized masked policy everyμiterations: π ψ n+1 (·|·,π θ m+1 ) untilconvergence andreturnπ θ 5GRUE (GENERALREINFORCED-LANGUAGEUNDERSTANDINGEVAL) GRUE is a collection of 7 generative NLP tasks. To combat reward hacking for any single metric, each task is evaluated at test time according to a task-specific mix of metrics, detailed in Table 1. The metrics span two categories.Task preference metricscapture how well the models produce generations that satisfy the desiderata of the specific generation task, e.g., for Commongen, if the generations contain all the required words, or for IMDB, how positive the generated completions are.Naturalness metricscapture fluency, readability, etc. and provide perspective on factors beyond semantics. At training time, there are no special restrictions: models are free to use the supervised data, compute metrics on intermediate generations, etc. Train/val/test splits follow the original works. All results are averaged over multiple seeds, with exact counts being found in Appendix B. Experimental Setup.We use RL4LMs to test a large range of algorithms on the GRUE benchmark. Specifically: We compare 3 algorithms for direct fine-tuning — Supervised, PPO, 5 and NLPO. In 4 π ψ could be trained with alternate sampling techniques like top-kor beam search (or even hard-coded via rules by domain experts), though we find top-psampling to be most effective in practice. 5 We consider PPO representative of the present state-of-the-art — in particular, we do not consider the popular REINFORCE (Willianms, 1988; Williams, 1992), as recent works have shown PPO to be strictly superior to REINFORCE in multiple domains (Schulman et al., 2017) 5 Published as a conference paper at ICLR 2023 DatasetTaskInputOutputTask Preference Metrics(s) Naturalness Metrics(s) IMDB (Maas et al., 2011) Text Continua- tion Partial Movie Review A positive completion of the movie review. Learned Sentiment Classifier Perplexity (GPT-2) CommonGEN (Lin et al., 2020) Generative Commonsense Concept Set A sentence coherently using all input concepts. CIDER; ROUGE-2,L; BLEU-3,4; METEOR; Coverage SPICE CNN Daily Mail (Hermann et al., 2015)SummarizationNews ArticleSummarized article. SummaCZS; ROUGE-1, 2, L, LSum; METEOR; BLEU BertScore ToTTo (Parikh et al., 2020) Data to Text Highlighted Wiki Table Factually accurate text describing the information. SacreBLEU; PARENT BLEURT WMT-16 (en-de) (Bojar et al., 2016) Machine Trans- lation Text (English)Translated text (German). TER; cHRF; ROUGE-1, 2, L, LSum, METEOR; SacreBLEU, BLEU BertScore NarrativeQA (Ko ˇ cisk ` y et al., 2018) Question An- swering Question Context (a Story) Abstractive answer to the question. ROUGE-1, 2, L, LSum, LMax; METEOR; BLEU; SacreBLEU BertScore DailyDialog (Li et al., 2017) Chitchat Dia- logue Dialogue HistoryA conversational responseMETEOR; Learned Intent ClassifierBertScore Table 1:GRUE Benchmark using RL4LMsshowing the various tasks, input and output types, and the metrics used. We note that we test RL algorithms on these tasks for a wider range of possible rewards than just the task specific ones shown here. Unless specified, datasets are in English. addition, we consider a hybrid approach of supervised learning and our RL methods by applying PPO and NLPO on checkpoints that have been fine-tuned in a supervised fashion—we call these Supervised+PPO, Supervised+NLPO. As an additional baseline, we additionally run zero-shot evaluations where we design prompts which aim to elicit task-specific generations, but with no training data or parameter updates. For each task, to isolate the effect of training method, we select a single pre-trained LM backbone. For IMDB text continuation we use GPT-2 (117m parameters), and for the rest of the tasks we use T5-base (220m parameters). For our RL models (PPO, NLPO, Supervised+PPO, Supervised+NLPO), for a thorough investigation of how reward-hacking might interplay with GRUE, we run a separate set of experiments optimizing multiple task rewards for each task independently, e.g., for Commongen which has 6 task rewards (CIDER, ROUGE-2, ROUGE-L, BLEU-3, BLEU-4, METEOR) we run 6 different experiments optimizing each metric independently and report all possible metrics seen in Table 1 regardless of which individual metric was being optimized for. Human Participant Study.We gather human judgments for five of the tasks in GRUE. In doing so, our goals are 1) to validate that the automated metrics we selected for GRUE correlate with human judgments with respect to relative ranking between models; and 2) to provide additional empirical comparisons regarding NLPO vs. PPO, ablations to study the effects of the KL naturalness penalty, etc. We specifically consider IMDB, Commongen, ToTTo, DailyDialog, and CNN Daily Mail. For each individual sample in a task, we ask 3 unique human raters to provide Likert judgments of 1) quality, i.e., for the specific task, how correct/appropriate is the generation, given the context, and 2) fluency, i.e., how well-written is the generation. We used Amazon Mechanical Turk, and paid crowdworkers a minimum of $15/hr. More details, including qualification information, interface screenshots, instructions, etc. are given in the corresponding Appendicies. 5.1RESULTS ONGRUE: WHICHALGORITHMSHOULD BEUSED TOLEARNPREFERENCES? Figures 2(a), 2(b) present the results on GRUE, split into task metrics and naturalness metrics, and Tables 2, 3 highlight key results via ablation studies. Full results are available in Appendix B. For text continuation and summarization, with non-trivial zero-shot performance, RL tends to perform better than supervised training, but for tasks like Commongen and ToTTo, which have very low zero-shot performance, supervised training performs best—with both approaches outperforming zero-shot. However,using RL+Supervised learning in conjunction works best;NLPO+supervised and PPO+supervised usually always outperforms NLPO/PPO (or supervised in isolation) across both task metrics and naturalness metrics. Supervised warm-starting is particularly effective for Commongen and ToTTo, which our results suggest are more prone to reward hacking. The one exception to this trend is DailyDialog where the RL models outperform warm-started Supervised+RL models likely due to the low performance of the Supervised models. We note that Supervised+NLPO using a 6 Published as a conference paper at ICLR 2023 Commongen IMDB NarrativeQA WMT16 ToTTo CNN DailyDialog 0.320.350.390.43 0.55 0.57 0.59 0.61 0.13 0.19 0.25 0.31 0.49 0.49 0.5 0.5 0.3 0.36 0.42 0.48 0.37 0.38 0.39 0.4 0.31 0.33 0.34 0.35 (a) Automated Task Metrics Commongen IMDB NarrativeQA WMT16 ToTTo CNN DailyDialog 0.190.220.250.28 Supervised PPO Supervised+PPO NLPO Supervised+NLPO 0.95 0.96 0.97 0.98 0.86 0.89 0.92 0.95 0.88 0.88 0.89 0.89 0.04 0.08 0.13 0.17 0.64 0.67 0.7 0.72 0.85 0.86 0.86 0.87 (b) Automated Naturalness Metrics Commongen CNN IMDB ToTTo DailyDialog 2.532.963.383.81 3.79 3.85 3.9 3.96 3.54 3.59 3.65 3.7 2.99 3.39 3.79 4.19 3.36 3.43 3.49 3.56 (c) Human Study Task Metrics Commongen CNN IMDB ToTTo DailyDialog 2.472.953.443.92 4.0 4.08 4.16 4.24 3.14 3.28 3.42 3.56 2.71 3.18 3.66 4.13 3.99 4.04 4.1 4.15 (d) Human Study Naturalness Metrics Figure 2: Summarized results via automated metrics across all 7 GRUE tasks for each of the 5 algorithms we consider, and human participant studies for the 5 tasks suitable for human studies. Test results are averaged over all the respective metrics seen in Table 1. QuestionsTasks IMDBCommonGenCNN/DMToTTOWMT16NarQADialog Needs Warm Start7373737 Easily reward hackable?3377777 RL>Sup (auto)? 3777773 RL>Sup (human)?3777--3 Sup+RL>Sup (auto)?3333337 Sup+RL>Sup (human)?3733--7 Sup+NLPO>Sup+PPO (auto)?3333333 Sup+NLPO>Sup+PPO (human)?3333--3 Table 2:Key questions answered using GRUE + RL4LMs:This table summarizes the results found in the ablations and Fig. 2 and provides an overview of the ques- tions we ask in Section 5: which tasks require warm starts or are easily reward hackable; when to use RL over Super- vised, when to use both; and when to use NLPO over PPO. All conclusions drawn are the result of statistical analysis as discussed in the experimental setup. AblationSentimentPerplexity Zero Shot0.48932.171 Supervised0.53935.472 PPO0.60233.816 NLPO0.61133.832 Warm Starting (Sec. 5.1) PPO+Supervised0.62635.049 NLPO+Supervised0.62034.816 Data Budget (Reward trained on 10% of data, Sec. 5.3) PPO0.59835.929 NLPO0.59933.536 Removing NLPO Top-pConstraints (Sec. 5.2) (p= 1is equivalent to PPO,p= 0.9is NLPO) NLPOp= 0.10.57932.451 NLPOp= 0.50.58832.447 Removing KL Constraints (Sec. 5.2) PPO-no-KL0.83841.897 NLPO-no-KL0.85841.429 Discount Ablations (γ= 1) (Sec. 5.4) PPO0.65141.035 NLPO0.62443.720 Table 3: IMDB Ablation Results. T5-base (220m parameter) LM currently outperforms all the models on the ToTTo leaderboard, many of which have≥3b parameter supervised models—suggesting that RL is parameter efficient as well. In these cases, it is critical that the initial policy already contain (some) signal for the task due to it being used as a KL constraint and masking constraint in NLPO. If the mask contains no initial priors about task specific language, it will be eliminating the wrong actions—a better initial policy leads to better RL performance downstream. 7 Published as a conference paper at ICLR 2023 Human agreement with automated metrics.As human judgments can be noisy, we run additional statistical analysis such as measuring inter-annotator agreement, via Krippendorf’s alpha score, and using a one-way ANOVA followed by a post-hoc Tukey HSD test to measure if differences in means of average scores between pairs of models are significant. We find that trends in our human evaluations generally match those seen in the automated metrics for both task and naturalness metrics (see Figures 2(c), 2(d) which summarize Appendix Tables 10,15,21,26, 35—Supervised+NLPO> Supervised≥Supervised+PPO>NLPO≥PPO>Zero-shot—with the exception of Supervised outperforming Supervised+PPO on 2 out of 5 tasks when automated metrics would indicate that Supervised+PPO outperforms Supervised on all of the tasks. We draw two conclusions from this: (1) if the generated text is above a certain threshold of naturalness, the automated metricsusually correlate with human judgements; (2) usually but not always as seen in the relative performance of Supervised and Supervised+PPO, potentially indicating reward hacking behaviors undetected by automated metrics but caught by human preference feedback. 5.2PREFERENCEREWARDLEARNING, SELECTION,ANDHACKING While the GRUE benchmark’s metric for each task is an average over several measures, the RL models we trained optimized only a single metric independently. Thus, we can empirically investigate which metric for which GRUE produces the best results. We observe that many possible single metric rewards provide task performance gains over supervised methods (results shown in Fig. 3(a), 2(c) are averaged across these reward functions) with the condition that the text is also coherent and natural. Which constraints best prevent reward hacking?The reward function in Equation 1 balances a task-specific reward with a KL constraint — models are penalized from straying too far from a base LM in their pursuit of high reward (Table 3 and Appendix Table 5) clearly show that if KL constraints are removed entirely, models reward hack). But which model works best as a base regularizing LM? When the initial policy (i.e., the raw, pretrained model) has low performance on the task, the KL penalty pushes the policy towards nonsense, e.g. on Commongen and ToTTo the trained policy learns to simply repeat portions of the input (as seen in Tables B.4.5, B.6.4). This behavior is mitigated if the base regularizing LM is the supervised model—the reward encourages the policy to balance the task-specific reward and a more reasonable regularization term. Deriving KL penalties from warm-started initial policies is critical for performance on such tasks. PPO vs. NLPO. Figure 2 shows that NLPO generally outperforms PPO and supervised, especially when applied after supervised training. We hypothesize that the primary reason for NLPO’s improved performance and stability is because the masking policy provides an additional constraint for the current policy. This constraint is not based on the initial untuned policy like the KL penalty but of the policy fromμiterations ago and likely contains more task-relevant information learned during RL training. Table 3 (and Appendix Table 8) shows how performance increases up to a point and then decreases aspin top-psampling is increased for the masking policy, relaxing the constraint by eliminating less tokens at each step, implying that there is a balance to be found in how much the model should be constrained during RL training. Human Preference Reward Learning.To this point, our experiments have largely focused on optimizing evaluation metrics that correlate with human judgments, e.g., METEOR. Here: we additionally test how well preferences can be learned from direct human feedback. For this, we focus on Commongen — a GRUE dataset well-suited for displaying differences due to human preferences. First, we randomly select prompts from the Commongen train dataset and sample a single completion from both the Supervised and Supervised+NLPO models. We then present the prompt and the two completion candidates to 3 unique crowdworkers and ask them to select which one they prefer with respect to commonsense/fluency for 417 unique pairs (Krippendorfα=.28). We use this data to train a reward model, T5-11B Raffel et al. (2020), on the balanced binary classification task of predicting which of the pair was preferred by a majority of 3 annotators, conditioned on the prompt and completion. The resulting model achieved 69.5 test ROC AUC suggesting it indeed captures average human preferences. Additional details on this process are found in Appendix B.4.4. We train Supervised+RL with a METEOR-only reward as a baseline, and compare it to a reward function that uses the fine-tuned T5-11B model. Finally, we rerun the same pairwise preference collection procedure—this time sampling from Commongen test—with human participants to compare the generations from a preference optimized RL policy to the previously best Supervised+NLPO policy. Comparing the METEOR-only to the preference model, the generations produced by the human 8 Published as a conference paper at ICLR 2023 feedback model are preferred in 682 cases, compared to the METEOR-only model which is preferred in 587 cases (p <0.01the models are equally preferred). This implies that this pipeline of collecting preferences, training a reward, and further tuning the policy improves alignment to human preferences. 5.3DATABUDGET: IMPROVE YOURREWARD ORGATHERMOREDEMONSTRATION? Given a fixed data collection budget, is it more efficient to gather feedback to improve a learned reward function or to gather more expert demonstrations? We use the IMDB text continuation task as a case study. In the IMDB task, a model is given a partial movie review as a prompt, and is asked to continue it as positively as possible (even if the prompt was negative). The original dataset consists of movie reviews and sentiment labels of positive, negative, or neutral. A DistilBERT (Sanh et al., 2019) classifier is trained on these labels and used to provide sentiment scores on how positive a given piece of text is, which serves as the task reward. The trade-off is between gathering more: 1) sentiment labels (improving the reward); or 2) positive sentiment reviews (improving supervised training). We train a classifier on varying amounts of training data and evaluate on the held out test dataset— finding as expected that more training data improves test accuracy and so results in a higher quality reward. We then use each of these rewards of varying quality during RL training, and evaluate using the same metric as GRUE (i.e., a classifier trained with the entire training set). As seen in Table 3, we find that improving the reward quality improves LM performance as well. Further, we trained a supervised model with at least as many samples used to train each of these reward classifiers. We find thata learned reward function enables greater performance when used as a signal for an RL method than a supervised method trained with 5 times more data.This implies that improving reward models can be more data efficient than collection expert demonstrations for a task—and that’s not accounting for the fact that assigning sentiment labels is likely a simpler task than writing full demonstrations. Further details on this ablation are found in Appendix Table 7. 5.4PRACTICALCONSIDERATIONS: WHICHIMPLEMENTATIONDETAILSMATTERMOST? Generation as a token-level MDP, not a bandit environment.Most recent works that tune LMs using RL do so by calculating a reward for all the tokens in the sentence (Wu et al., 2021a; Ouyang et al., 2022; Lu et al., 2022). This setting is equivalent to a bandit feedback environment where the action space is the space of all possible generations for the task (Sutton & Barto, 2018). This type of environment can be simulated within our RL formulation by setting the discount factorγ= 1. Table 3 (and Appendix Table 6) shows that this causes instability in training with respect to naturalness in both PPO and NLPO for IMDB. Our standard setting isγ= 0.95when calculating discounted rewards-to-go in the token-level MDP formulation, which reduces the magnitude of the reward that is applied to tokens selected at the beginning. The sentiment scores are approximately the same between both settings but the naturalness of language in the bandit setting is significantly less—indicating that discounting rewards withγ <1via a token-level MDP formulation is at least sometimes more effective for language generation. Dropout and Sampling. We found two other implementation details to be critical for stability of RL training. The first is dropout, which in its standard form was found to cause instability in policy gradient methods in continuous control settings by Hausknecht & Wagener (2022). We find a similar effect when using dropout when RL training LMs as well, with training loss often diverging for dropout>0in training. The second important detail, particularly affecting the machine translation task, is sampling methods. We find that using the same sampling methods during exploration and inference is critical to translating training performance to test performance–else the model exhibits high train rewards but low test metrics. 6CONCLUSIONS We’re hopeful that the GRUE benchmark and the RL4LMs library can push progress in aligning language models to human preferences via RL methods by providing the community with a standard means of comparing methods. Furthermore, we’re optimistic that, as the stability and consistency of training improves, our methods provide a path towards iterative improvement of language technolo- gies, with deployment, user feedback collection, and re-optimization enabling better user experiences when interacting with generative models. 9 Published as a conference paper at ICLR 2023 7ACKNOWLEDGEMENTS We’d like to acknowledge the support of DARPA MCS program through NIWC Pacific (N66001-19- 2-4031), Google Cloud Compute, and the ReViz team at the Allen Institute for AI. KB is supported by NSF under grant No. 2127309 to the Computing Research Association for the CIFellows Project. REFERENCES Alekh Agarwal, Nan Jiang, Sham M Kakade, and Wen Sun. Reinforcement learning: Theory and algorithms.CS Dept., UW Seattle, Seattle, WA, USA, Tech. Rep, p. 10–4, 2019. Prithviraj Ammanabrolu.Language Learning in Interactive Environments. PhD thesis, Georgia Institute of Technology, 2021. Prithviraj Ammanabrolu and Matthew Hausknecht. Graph constrained reinforcement learning for natural language action spaces. InInternational Conference on Learning Representations, 2020. URLhttps://openreview.net/forum?id=B1x6w0EtwH. Prithviraj Ammanabrolu, Liwei Jiang, Maarten Sap, Hannaneh Hajishirzi, and Yejin Choi. Aligning to social norms and values in interactive narratives. InNAACL, 2022. Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. Spice: Semantic propositional image caption evaluation. InEuropean conference on computer vision, p. 382–398. Springer, 2016. Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAI Pieter Abbeel, and Wojciech Zaremba. Hindsight experience replay. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.),Advances in Neural Information Processing Systems, volume 30. Curran As- sociates, Inc., 2017. URLhttps://proceedings.neurips.c/paper/2017/file/ 453fadbd8a1a3af50a9df4df899537b5-Paper.pdf. Kushal Arora, Layla El Asri, Hareesh Bahuleyan, and Jackie Cheung. Why exposure bias matters: An imitation learning perspective of error accumulation in language generation. InFindings of the Association for Computational Linguistics: ACL 2022, p. 700–710, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.findings-acl.58. URL https://aclanthology.org/2022.findings-acl.58. Dzmitry Bahdanau, Philemon Brakel, Kelvin Xu, Anirudh Goyal, Ryan Lowe, Joelle Pineau, Aaron Courville, and Yoshua Bengio. An actor-critic algorithm for sequence prediction.arXiv preprint arXiv:1607.07086, 2016. Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. InProceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, p. 65–72, 2005. Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. Scheduled sampling for sequence prediction with recurrent neural networks.Advances in neural information processing systems, 28, 2015. Ond ˇ rej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Varvara Logacheva, Christof Monz, Matteo Negri, Aurélie Névéol, Mariana Neves, Martin Popel, Matt Post, Raphael Rubino, Carolina Scarton, Lucia Specia, Marco Turchi, Karin Verspoor, and Marcos Zampieri. Findings of the 2016 conference on machine translation. InProceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers, p. 131–198, Berlin, Germany, August 2016. Association for Computational Linguistics. doi: 10.18653/v1/W16-2301. URLhttps://aclanthology.org/W16-2301. Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym.arXiv preprint arXiv:1606.01540, 2016. 10 Published as a conference paper at ICLR 2023 Kai-Wei Chang, Akshay Krishnamurthy, Alekh Agarwal, Hal Daumé I, and John Langford. Learning to search better than your teacher. InInternational Conference on Machine Learning, p. 2058– 2066. PMLR, 2015. Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. InNeurIPS, 2021. Ting-Rui Chiang and Yun-Nung Chen. Relating neural text degeneration to exposure bias. In Proceedings of the Fourth BlackboxNLP Workshop on Analyzing and Interpreting Neural Net- works for NLP, p. 228–239, Punta Cana, Dominican Republic, November 2021. Associa- tion for Computational Linguistics. doi: 10.18653/v1/2021.blackboxnlp-1.16. URLhttps: //aclanthology.org/2021.blackboxnlp-1.16. Leshem Choshen, Lior Fox, Zohar Aizenbud, and Omri Abend. On the weaknesses of reinforcement learning for neural machine translation. InInternational Conference on Learning Representations, 2020. Hal Daumé, John Langford, and Daniel Marcu. Search-based structured prediction.Machine learning, 75(3):297–325, 2009. Bhuwan Dhingra, Manaal Faruqui, Ankur Parikh, Ming-Wei Chang, Dipanjan Das, and William W Cohen. Handling divergent reference texts when evaluating table-to-text generation.arXiv preprint arXiv:1906.01081, 2019. Daniel Duckworth, Arvind Neelakantan, Ben Goodrich, Lukasz Kaiser, and Samy Bengio. Parallel scheduled sampling.arXiv preprint arXiv:1906.04331, 2019. Kartik Goyal, Chris Dyer, and Taylor Berg-Kirkpatrick. Differentiable scheduled sampling for credit assignment.arXiv preprint arXiv:1704.06970, 2017. Matthew Hausknecht and Nolan Wagener. Consistent dropout for policy gradient reinforcement learning.arXiv preprint arXiv:2202.11818, 2022. Matthew Hausknecht, Prithviraj Ammanabrolu, Marc-Alexandre Côté, and Xingdi Yuan. Interactive fiction games: A colossal adventure. InThirty-Fourth AAAI Conference on Artificial Intelligence (AAAI), 2020. URLhttps://arxiv.org/abs/1909.05398. Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend.Advances in neural information processing systems, 28, 2015. Ari Holtzman, Jan Buys, Maxwell Forbes, Antoine Bosselut, David Golub, and Yejin Choi. Learning to write with cooperative discriminators.arXiv preprint arXiv:1805.06087, 2018. Shengyi Huang and Santiago Ontañón. A closer look at invalid action masking in policy gradient algorithms.arXiv preprint arXiv:2006.14171, 2020. Ferenc Huszár. How (not) to train your generative model: Scheduled sampling, likelihood, adversary? arXiv preprint arXiv:1511.05101, 2015. Natasha Jaques, Judy Hanwen Shen, Asma Ghandeharioun, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind Picard. Human-centric dialog training via offline reinforcement learning. InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), p. 3985–4003, Online, 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.emnlp-main.327. Wendell Johnson. Studies in language behavior: A program of research.Psychological Monographs, 56(2):1–15, 1944. Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Han- naneh Hajishirzi. UNIFIEDQA: Crossing format boundaries with a single QA system. InFindings of the Association for Computational Linguistics: EMNLP 2020, p. 1896–1907, Online, Novem- ber 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.findings-emnlp.171. 11 Published as a conference paper at ICLR 2023 Samuel Kiegeland and Julia Kreutzer. Revisiting the weaknesses of reinforcement learning for neural machine translation. InProceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, p. 1673–1681, Online, June 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.naacl-main. 133. Tomáš Ko ˇ cisk ` y, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. The narrativeqa reading comprehension challenge.Transactions of the Association for Computational Linguistics, 6:317–328, 2018. Julia Kreutzer, Stefan Riezler, and Carolin Lawrence. Offline reinforcement learning from human feedback in real-world sequence-to-sequence tasks. InProceedings of the 5th Workshop on Structured Prediction for NLP (SPNLP 2021), p. 37–43, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.spnlp-1.4. URLhttps://aclanthology. org/2021.spnlp-1.4. Philippe Laban, Tobias Schnabel, Paul N. Bennett, and Marti A. Hearst. SummaC: Re-Visiting NLI- based Models for Inconsistency Detection in Summarization.Transactions of the Association for Computational Linguistics, 10:163–177, 02 2022. ISSN 2307-387X. doi: 10.1162/tacl_a_00453. URLhttps://doi.org/10.1162/tacl_a_00453. Gerasimos Lampouras and Andreas Vlachos. Imitation learning for language generation from unaligned data. InProceedings of COLING 2016, the 26th International Conference on Computa- tional Linguistics: Technical Papers, p. 1101–1112. The COLING 2016 Organizing Committee, 2016. Rémi Leblond, Jean-Baptiste Alayrac, Anton Osokin, and Simon Lacoste-Julien. Searnn: Training rnns with global-local losses.arXiv preprint arXiv:1706.04499, 2017. Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. A diversity-promoting objective function for neural conversation models.arXiv preprint arXiv:1510.03055, 2015. Jiwei Li, Will Monroe, Alan Ritter, Dan Jurafsky, Michel Galley, and Jianfeng Gao. Deep reinforcement learning for dialogue generation. InProceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, p. 1192–1202, Austin, Texas, Novem- ber 2016. Association for Computational Linguistics. doi: 10.18653/v1/D16-1127. URL https://aclanthology.org/D16-1127. Yanran Li, Hui Su, Xiaoyu Shen, Wenjie Li, Ziqiang Cao, and Shuzi Niu. DailyDialog: A manually labelled multi-turn dialogue dataset. InProceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), p. 986–995, Taipei, Taiwan, November 2017. Asian Federation of Natural Language Processing. URLhttps://aclanthology. org/I17-1099. Bill Yuchen Lin, Wangchunshu Zhou, Ming Shen, Pei Zhou, Chandra Bhagavatula, Yejin Choi, and Xiang Ren. CommonGen: A constrained text generation challenge for generative commonsense reasoning. InFindings of the Association for Computational Linguistics: EMNLP 2020, p. 1823–1840, Online, 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020. findings-emnlp.165. Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. InText summarization branches out, p. 74–81, 2004. Siqi Liu, Zhenhai Zhu, Ning Ye, Sergio Guadarrama, and Kevin Murphy. Improved image captioning via policy gradient optimization of spider. InProceedings of the IEEE international conference on computer vision, p. 873–881, 2017. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach.arXiv preprint arXiv:1907.11692, 2019. Ilya Loshchilov and Frank Hutter.Decoupled weight decay regularization.arXiv preprint arXiv:1711.05101, 2017. 12 Published as a conference paper at ICLR 2023 Ximing Lu, Sean Welleck, Liwei Jiang, Jack Hessel, Lianhui Qin, Peter West, Prithviraj Am- manabrolu, and Yejin Choi. Quark: Controllable text generation with reinforced unlearning.arXiv e-prints, p. arXiv–2205, 2022. Andrew Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng, and Christopher Potts. Learning word vectors for sentiment analysis. InProceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies, p. 142–150, 2011. Alice Martin, Guillaume Quispe, Charles Ollion, Sylvain Le Corff, Florian Strub, and Olivier Pietquin. Learning natural language generation with truncated reinforcement learning. InProceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, p. 12–37, 2022. Tsvetomila Mihaylova and André FT Martins. Scheduled sampling for transformers.arXiv preprint arXiv:1906.07651, 2019. Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. InInternational conference on machine learning, p. 1928–1937. PMLR, 2016. Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser-assisted question-answering with human feedback.arXiv preprint arXiv:2112.09332, 2021. Karthik Narasimhan, Tejas D. Kulkarni, and Regina Barzilay. Language understanding for text- based games using deep reinforcement learning. InEMNLP, p. 1–11, 2015. URLhttp: //aclweb.org/anthology/D/D15/D15-1001.pdf. Khanh Nguyen, Hal Daumé I, and Jordan Boyd-Graber. Reinforcement learning for bandit neural machine translation with simulated human feedback. InProceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, p. 1464–1474, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi: 10.18653/v1/D17-1153. URL https://aclanthology.org/D17-1153. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback.arXiv preprint arXiv:2203.02155, 2022. Richard Yuanzhe Pang and He He. Text generation by learning from demonstrations. InInternational Conference on Learning Representations, 2021. URLhttps://openreview.net/forum? id=RovX-uQ1Hua. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. InProceedings of the 40th annual meeting of the Association for Computational Linguistics, p. 311–318, 2002. Ankur Parikh, Xuezhi Wang, Sebastian Gehrmann, Manaal Faruqui, Bhuwan Dhingra, Diyi Yang, and Dipanjan Das. ToTTo: A controlled table-to-text generation dataset. InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), p. 1173–1186. Association for Computational Linguistics, 2020. doi: 10.18653/v1/2020.emnlp-main.89. Romain Paulus, Caiming Xiong, and Richard Socher. A deep reinforced model for abstractive summarization.arXiv preprint arXiv:1705.04304, 2017. Matt Post. A call for clarity in reporting bleu scores.arXiv preprint arXiv:1804.08771, 2018. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer.Journal of Machine Learning Research, 21(140):1–67, 2020. URL http://jmlr.org/papers/v21/20-074.html. Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. Stable-baselines3: Reliable reinforcement learning implementations.Journal of Machine Learning Research, 2021. 13 Published as a conference paper at ICLR 2023 Marc’Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. Sequence level training with recurrent neural networks. InICLR, 2016. Steven J Rennie, Etienne Marcheret, Youssef Mroueh, Jerret Ross, and Vaibhava Goel. Self-critical sequence training for image captioning. InCVPR, 2017. Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. InProceedings of the fourteenth international conference on artificial intelligence and statistics, p. 627–635. JMLR Workshop and Conference Proceedings, 2011. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter.arXiv preprint arXiv:1910.01108, 2019. John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. InInternational conference on machine learning, p. 1889–1897. PMLR, 2015a. John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation.arXiv preprint arXiv:1506.02438, 2015b. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347, 2017. Thibault Sellam, Dipanjan Das, and Ankur Parikh. BLEURT: Learning robust metrics for text generation. InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics, p. 7881–7892, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.acl-main.704. URLhttps://w.aclweb.org/anthology/2020. acl-main.704. Claude Elwood Shannon. A mathematical theory of communication.The Bell system technical journal, 27(3):379–423, 1948. Charlie Snell, Ilya Kostrikov, Yi Su, Mengjiao Yang, and Sergey Levine. Offline rl for natural language generation with implicit language q learning.arXiv preprint arXiv:2206.11871, 2022. Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback.Advances in Neural Information Processing Systems, 33:3008–3021, 2020. Marilyn Strathern. ‘improving ratings’: audit in the british university system.European review, 5(3): 305–321, 1997. Richard S Sutton and Andrew G Barto.Reinforcement learning: An introduction. MIT press, 2018. Gokul Swamy, Sanjiban Choudhury, J Andrew Bagnell, and Steven Wu. Of moments and matching: A game-theoretic framework for closing the imitation gap. InInternational Conference on Machine Learning, p. 10022–10032. PMLR, 2021. Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus-based image description evaluation. InProceedings of the IEEE conference on computer vision and pattern recognition, p. 4566–4575, 2015. Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning.Machine learning, 8(3):229–256, 1992. RJ Willianms. Toward a theory of reinforcement-learning connectionist systems.Technical Report NU-CCS-88-3, Northeastern University, 1988. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Transformers: State-of-the-art natural language processing. InProceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations, p. 38–45, 2020. 14 Published as a conference paper at ICLR 2023 Jeff Wu, Long Ouyang, Daniel M Ziegler, Nisan Stiennon, Ryan Lowe, Jan Leike, and Paul Christiano. Recursively summarizing books with human feedback.arXiv preprint arXiv:2109.10862, 2021a. Qingyang Wu, Lei Li, and Zhou Yu. Textgail: Generative adversarial imitation learning for text generation. InThe Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, p. online. AAAI Press, 2021b. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. Google’s neural machine translation sys- tem: Bridging the gap between human and machine translation.arXiv preprint arXiv:1609.08144, 2016. Tom Zahavy, Matan Haroush, Nadav Merlis, Daniel J Mankowitz, and Shie Mannor. Learn what not to learn: Action elimination with deep reinforcement learning.Advances in neural information processing systems, 31, 2018. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert.arXiv preprint arXiv:1904.09675, 2019. Li Zhou, Kevin Small, Oleg Rokhlenko, and Charles Elkan. End-to-end offline goal-oriented dialog policy learning via policy gradient.CoRR, abs/1712.02838, 2017. URLhttp://arxiv.org/ abs/1712.02838. Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences.arXiv preprint arXiv:1909.08593, 2019. 15 Published as a conference paper at ICLR 2023 CONTENTS 1 Introduction1 2 Related Work3 3 RL4LMs: A Library for Training LMs with RL3 3.1Environments: Generation as a Token-level MDP . . . . . . . . . . . . . . . . . .3 3.2Reward Functions and Evaluation Metrics . . . . . . . . . . . . . . . . . . . . . .4 3.3On-policy Actor-critic Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . .4 4 NLPO: Natural Language Policy Optimization4 5 GRUE (General Reinforced-language Understanding Eval)5 5.1Results on GRUE: Which Algorithm Should be Used to Learn Preferences? . . . .6 5.2Preference Reward Learning, Selection, and Hacking . . . . . . . . . . . . . . . .8 5.3Data Budget: Improve your Reward or Gather More Demonstration? . . . . . . . .9 5.4Practical Considerations: Which Implementation Details Matter Most? . . . . . . .9 6 Conclusions9 7 Acknowledgements10 A On-policy Algorithm Implementation Details18 A.1 PPO Details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .18 A.2 NLPO Details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .18 B Experimental Details19 B.1 Crowdworking Details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .19 B.2 GRUE Experiment Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .19 B.3 IMDB . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .20 B.3.1Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .20 B.3.2Results and Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . .20 B.3.3Human Participant Study . . . . . . . . . . . . . . . . . . . . . . . . . . .23 B.3.4Qualitative Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .26 B.4 CommonGen . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .28 B.4.1Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .28 B.4.2Results and Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . .29 B.4.3Human Participant Study . . . . . . . . . . . . . . . . . . . . . . . . . . .32 B.4.4Human Preference Learning Experiments . . . . . . . . . . . . . . . . . .32 B.4.5Qualitative Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . .33 B.5 CNN Daily Mail . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .35 16 Published as a conference paper at ICLR 2023 B.5.1Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .35 B.5.2Results and Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . .35 B.5.3Human Participant Study . . . . . . . . . . . . . . . . . . . . . . . . . . .39 B.5.4Qualitative Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . .39 B.6 ToTTo . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .43 B.6.1Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .43 B.6.2Results and Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . .43 B.6.3Human Participant Study . . . . . . . . . . . . . . . . . . . . . . . . . . .47 B.6.4Qualitative Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . .48 B.7 Narrative QA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .49 B.7.1Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .49 B.7.2Results and Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . .51 B.7.3Qualitative Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .51 B.8 Machine Translation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .53 B.8.1Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .53 B.8.2Results and Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . .53 B.8.3Qualitative Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .56 B.9 Daily Dialog . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .57 B.9.1Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .57 B.9.2Results and Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . .59 B.9.3Human Participant Study . . . . . . . . . . . . . . . . . . . . . . . . . . .59 B.9.4Qualitative Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . .60 17 Published as a conference paper at ICLR 2023 AON-POLICYALGORITHMIMPLEMENTATIONDETAILS A.1PPO DETAILS Given discussion and equations in Section 3.3, we further note that we follow (Ziegler et al., 2019) and dynamically adapt the KL coefficientβduring training where, e t =clip ( KL(π(a t |s t )||π 0 (a t |s t ))−KL target KL target ,−0.2,0.2 ) (2) β t+1 =β t (1 +K β e t )(3) whereKL target is user-specified KL divergence between initial modelhand current policyπandK β is rate of update which we generally set to0.2in our experiments. To increase stability during training, we further use Generalized Advantage Estimation (GAE) (Schul- man et al., 2015b) and define the advantage estimator ˆ A(s n ,a n ) based on the Temporal Difference residual as: δ t =r(s t ,a t ) +V φ (s t+1 )−V φ (s t ).(4) ˆ A(s n ,a n ) = ∞ ∑ t=0 λ t δ n+t ,(5) whereλprovides the trade-off between bias and variance. A.2NLPO DETAILS NLPO learns to mask irrelevant language by maintaining amasking policyπ ψ : the masking policy is a copy of the current policy (π θ ), but is updated only everyμsteps. GivenZ(π θ ) = ∑ a∈V π θ 0 (a|s) the normalization value of the sum of probabilities of all actiona∈ Agiven a particular State s∈ S, let the parameterized top-pvocabularyV p π θ ⊂ Vbe the subset of the vocab, consisting of the top-phighest probability vocabulary tokens with respect toπ θ . Formally, letZ p be the normalization value for the parameterized top-pvocabulary, can be defined as the subset of tokens that maximizesZ p (π θ ) = ∑ a∈V k π θ π θ (a|s) . Then optimizing a policy according to the parameterized top-pvocabulary can be defined as: π ψ (·|s,π θ ) = π θ (·|s)/Z p (π θ )ifa∈V p π θ andZ(π θ ) 0otherwise. (6) 18 Published as a conference paper at ICLR 2023 BEXPERIMENTALDETAILS B.1CROWDWORKINGDETAILS Qualification roundWe ran a qualification round using the IMDB task. We opened the qualifica- tion around to users fromAU, CA, NZ, GB, USwith 5K prior approved HITs and a minimum acceptance rate of 97% on their previous HITs. We gathered judgments over 600 generations from 3 annotators per generation. One of the authors of this paper also completed 17 random HITs to serve as a proxy for “ground truth." After gathering these annotations, we selected workers who: 1) didn’t significantly disagree with other annotators on the same instance more than 20% of the time; 2) who completed at least 5 HITs; 3) who didn’t disagree with the author annotator on the 17 HITs by more than 1 point; and 4) (likely) spent a reasonable amount of time reading the instructions/examples provided. In the end, 56 annotators were qualified. Additional per-task details are provided in the per-task sections of the Appendix. Compensation detailsAs per Amazon Mechanical Turk policy, annotators were compensated on a per-HIT basis. In addition, we used a timing script to estimate hourly wages to ensure our target of $15/hr was met. In cases where this minimum hourly rate was not met, we manually assigned bonuses. B.2GRUE EXPERIMENTSETUP We benchmark 5 training algorithms on 6 tasks (see Table 1) using either an encoder model (eg. GPT-2) or encoder-decoder model (eg. T5). We train policies using PPO, NLPO with variations of whether supervised pre-training is applied before RL fine-tuning and compare against supervised policy. The choice of LM is based on the type of task. For IMDB text continuation, we use GPT-2 and T5 for rest of the tasks. We use two separate LM models as actor and critics networks (i.e. no shared layers) in which the critic network has an additional linear layer mapping last token’s hidden representation to a scalar value. We use AdamW optimizer Loshchilov & Hutter (2017) with fixed learning rate and no scheduling. Commongen IMDB NarrativeQA WMT16 ToTTo CNN DailyDialog 0.320.350.390.43 0.55 0.57 0.59 0.61 0.13 0.19 0.25 0.31 0.49 0.49 0.5 0.5 0.3 0.36 0.42 0.48 0.37 0.38 0.39 0.4 0.31 0.33 0.34 0.35 (a) Automated Task Metrics Commongen IMDB NarrativeQA WMT16 ToTTo CNN DailyDialog 0.190.220.250.28 Supervised PPO Supervised+PPO NLPO Supervised+NLPO 0.95 0.96 0.97 0.98 0.86 0.89 0.92 0.95 0.88 0.88 0.89 0.89 0.04 0.08 0.13 0.17 0.64 0.67 0.7 0.72 0.85 0.86 0.86 0.87 (b) Automated Naturalness Metrics Figure 3: Summarized results via automated metrics across all 7 GRUE tasks for each of the 5 algorithms we consider, and human participant studies for the 5 tasks suitable for human studies. We break up the metrics into task-specific, e.g. average positive sentiment for IMDB task, and naturalness metrics, such as perplexity and human perceived coherence for the human rated metrics. This plot differs from Figure 2 as this one averages over over multiple reward functions per each task. 19 Published as a conference paper at ICLR 2023 Model Paramsvalue supervisedbatch size:64 epochs:10 learning rate:0.00001 pposteps per update:1280 total number of steps:64000 batch size:64 epochs per update:5 learning rate:0.000001 discount factor:0.99 gae lambda:0.95 clip ratio:0.2 value function coeff:0.5 nlposteps per update:1280 total number of steps:64000 batch size:64 epochs per update:5 learning rate:0.000001 discount factor:0.99 gae lambda:0.95 clip ratio:0.2 top mask ratio:0.9 target update iterations:5 decodingsampling: true top k:50 min length:48 max new tokens:48 tokenizerpadding side: left truncation side: left max length:64 Table 4:IMDB Hyperparams: Table shows a list of all hyper-parameters and their settings B.3IMDB B.3.1SETUP We consider IMDB dataset for the task of generating text with positive sentiment. The dataset consists of 25k training, 5k validation and 5k test examples of movie review text with sentiment labels of positive and negative. The input to the model is a partial movie review text (upto 64 tokens) that needs to be completed (generating 48 tokens) by the model with a positive sentiment while retaining fluency. For RL methods, we use a sentiment classifier Sanh et al. (2019) that is trained on pairs of text and labels as a reward model which provides sentiment scores indicating how positive a given piece of text is. For supervised Seq2Seq baselines, we consider only the examples with positive labels. We chose GPT-2 as LM for this task as it is more suited for text continuation than encoder-decoder LMs (eg. T5). We use top-k sampling withK= 50as the decoding method and for fair comparison, we keep this setting for all methods. For PPO and NLPO models, we train for64ksteps in total and update policy and value networks every1280steps with a mini-batch size of64and epochs of5per update. We apply adaptive KL controllers with different target KLs of0.02,0.05,0.1,infwith an initial KL co-efficient ofβ= 0.1. Table 4 provides an in-depth summary of all hyperparameters and other implementation details. B.3.2RESULTS ANDDISCUSSION Target KL ablation Fig 4 shows learning curves for PPO and NLPO in terms of episodic training reward, corpus level sentiment scores and perplexity scores on validation set averaged for 5 random seeds. It is seen that higher target KL of0.1is desired to achieve higher rewards but results in drifting 20 Published as a conference paper at ICLR 2023 01000020000300004000050000 steps 0.45 0.50 0.55 0.60 0.65 0.70 0.75 0.80 0.85 rollout_info/ep_rew alg/kl_div/target_kl 0.02 0.05 0.1 None (a) PPO Episodic total reward 01020304050 epoch 0.50 0.55 0.60 0.65 0.70 0.75 0.80 0.85 0.90 semantic/learned_automodel_metric alg/kl_div/target_kl 0.02 0.05 0.1 None (b) PPO Val avg sentiment score 01020304050 epoch 34 36 38 40 42 44 fluency_metrics/perplexity alg/kl_div/target_kl 0.02 0.05 0.1 None (c) PPO Val perplexity 01000020000300004000050000 steps 0.50 0.55 0.60 0.65 0.70 0.75 0.80 0.85 rollout_info/ep_rew alg/kl_div/target_kl 0.02 0.05 0.1 None (d) NLPO Episodic total reward 01020304050 epoch 0.50 0.55 0.60 0.65 0.70 0.75 0.80 0.85 0.90 semantic/learned_automodel_metric alg/kl_div/target_kl 0.02 0.05 0.1 None (e) NLPO Val avg sentiment score 01020304050 epoch 34 36 38 40 42 44 fluency_metrics/perplexity alg/kl_div/target_kl 0.02 0.05 0.1 None (f) NLPO Val perplexity Figure 4:Learning Curves: Averaged learning curves over 5 different runs by varying target KL, shaded regions indicate one standard deviation. (a) shows the rollout episodic total reward during training (b) shows evolution of sentiment scores on the validation split (c) shows evolution of perplexity on the validation split. From (a) and (b), it is seen that higher target KL (0.1) is desired to achieve higher rewards. However, this setting drifts away from the original LM too much and loses fluency. Therefore a lower target KL (0.02 or 0.05) is required to keep the model closer to original LM. Similar trends hold for NLPO but when compared to PPO, it retains lower perplexities and is more stable even with higher KL targets Target-KLSemantic and Fluency MetricsDiversity Metrics Sentiment Score↑Perplexity↓MSTTRDistinct 1 Distinct 2 H 1 H 2 Unique 1 Unique 2 Zero-Shot0.489±0.00632.171±0.1370.682±0.0010.042±0.0010.294±0.0018.656±0.00413.716±0.0035063±14.83247620±238 Supervised0.539±0.00435.472±0.0740.682±0.0010.047±0.0010.312±0.0028.755±0.01213.806±0.0165601±5751151±345 PPO 0.020.546±0.02233.127±0.0920.680±0.0030.044±0.0010.297±0.0048.665±0.02913.685±0.0765332±18448380±733 0.050.594±0.02233.765±0.3670.671±0.0050.043±0.0010.286±0.0098.588±0.06613.519±0.1035171±19046336±1872 0.10.602±0.01233.816±0.2330.664±0.0070.042±0.0010.278±0.0058.529±0.03713.366±0.1195108±20445158±961 inf0.838±0.06141.897±1.8060.577±0.0590.034±0.0030.197±0.0367.737±0.51411.866±0.9934214±26031181±5524 PPO+supervised 0.10.626±0.01435.049±0.3470.668±0.0040.048±0.0020.307±0.0088.704±0.05313.656±0.0665757±32450522±1514 inf0.796±0.00442.916±1.7160.617±0.0170.038±0.0030.233±0.0178.149±0.18312.733±0.3164563±32737040±2507 NLPO 0.020.564±0.04333.477±0.5780.679±0.0020.043±0.0010.294±0.0018.649±0.00713.688±0.045232±9647732±184 0.050.582±0.03733.470±0.4530.675±0.0030.043±0.0010.293±0.0048.63±0.03313.656±0.0855200±10147484±822 0.10.611±0.02333.832±0.2830.670±0.0020.043±0.0020.286±0.0068.602±0.04913.53±0.0765179±19646294±1072 inf0.858±0.02941.429±1.8250.575±0.0480.035±0.0050.201±0.0287.755±0.37911.862±0.8084389±60931714±4500 NLPO+supervised 0.10.620±0.01434.816±0.3400.672±0.0060.048±0.0020.31±0.0128.725±0.0913.709±0.1745589±14050734±1903 inf0.777±0.04241.035±0.6010.636±0.0230.043±0.0050.265±0.0348.373±0.26912.947±0.3595173±58943342±6828 Table 5:Target KL Ablations: Mean and standard deviations over 5 random seeds is reported for sentiment scores along with fluency and diversity metrics on validation set. It is seen from perplexity scores that a lower target KL constraint is desired to keep the model closer to the original model. On the otherhand, a higher target KL yields higher sentiment scores at the cost of fluency. inf KL penalty (target KL of inf), model simply learns to generate positive phrases (eg: "I highly recommend this movie to all!", "worth watching") regardless of the context. NLPO achieves better sentiment and perplexity scores than PPO. away from pre-trained LM and loses fluency. Therefore, a lower target KL (0.02 or 0.05) is required to keep the LM closer to original LM. This is also seen in Table 5 where we presented a comparative analysis of final performance of all models. 21 Published as a conference paper at ICLR 2023 Training data size ablationWe vary the amount of data used to train the reward classifier and the supervised baseline model to understand whether it is more efficient to gather data to improve reward model or to gather expert demonstrations for supervised learning. As observed in Table 7, improving the quality of reward function increases the performance on the overall task better than training with more data for supervised training, indicating that improving reward models is efficient than collect expert demonstrations for supervised training from a data efficiency perspective. Discount factor ablationTo understand the effect of discounted vs undiscounted (bandit) envi- ronments, we report sentiment and perplexity scores for different values of discount factor (0.5, 0.95and1.0) in Table 6 and observe that using a bandit environment (discount factor of1.0) results in performance loss in the case of NLPO and reward hacking in the case of PPO, indicating that discounted setting (with0.95) is desired. NLPO paramsTable. 8 shows ablation on different hyperparameters in NLPO algorithm. GammaSemantic and Fluency MetricsDiversity Metrics Sentiment Score↑Perplexity↓MSTTRDistinct 1 Distinct 2 H 1 H 2 Unique 1 Unique 2 Zero-Shot0.489±0.00632.371±0.1370.682±0.0010.042±0.0010.294±0.0018.656±0.00413.716±0.0035063±14.83247620±238 PPO 0.50.511±0.02335.945±0.920.69±0.0010.044±0.0020.304±0.0078.726±0.04113.793±0.0555304±28549668±1496 0.950.605±0.02333.497±0.4470.666±0.0130.043±0.0020.287±0.0088.575±0.07313.484±0.2445230±36346483±1318 1.00.651±0.0541.035±2.8850.691±0.0170.042±0.0040.295±0.0318.697±0.23713.563±0.3965127±46048319±5650 NLPO 0.50.49±0.0137.279±5.1370.688±0.010.045±0.0020.312±0.0168.746±0.11313.873±0.255395±19250828±2506 0.950.637±0.01332.667±0.6310.677±0.0140.044±0.0020.288±0.0108.588±0.10013.484±0.2365205±18946344±2688 1.00.624±0.03943.72±2.4750.662±0.0190.05±0.0070.3±0.0388.624±0.27713.360±0.5376337±92149441±6520 Table 6:Evaluation of GPT2 with different algorithms on IMDB sentiment text continuation task, discount factor ablations: Mean and standard deviations over 5 random seeds is reported for sentiment scores along with fluency and diversity metrics. This table measures performance differences for the discount factor. We note that most NLP approaches using RL follow the style of Li et al. (2016); Wu et al. (2021a) and use a discount factor of 1. This is equivalent to reducing the generation MDP to a bandit feedback environment and causes performance loss (in the case of NLPO) and reward hacking and training instability (in the case of PPO). Perc Data (size)Semantic and Fluency MetricsDiversity Metrics Sentiment Score↑Perplexity↓MSTTRDistinct 1 Distinct 2 H 1 H 2 Unique 1 Unique 2 Zero-Shot0.489±0.00632.371±0.1370.682±0.0010.042±0.0010.294±0.0018.656±0.00413.716±0.0035063±14.83247620±238 Supervised 0.0 (0k)0.489±0.00632.371±0.1370.682±0.0010.042±0.0010.294±0.0018.656±0.00413.716±0.0035063±1447620±238 0.1 (1k)0.531±0.00534.846±0.1230.685±0.0010.045±0.0010.313±0.0048.775±0.02313.854±0.0325215±6251125±685 0.5 (5k)0.536±0.00635.008±0.2290.684±0.0010.047±0.0000.314±0.0028.764±0.01013.837±0.01785489±4451284±576 1.0 (10k)0.539±0.00435.472±0.0740.682±0.0010.047±0.0010.312±0.0028.755±0.01213.806±0.0165601±5751151±345 PPO 0.0 (0k)0.492±0.0133.57±0.3230.69±0.020.047±0.0010.321±0.0158.816±0.14913.866±0.365629±24052911±1786 0.1 (2k)0.598±0.01735.929±1.3970.698±0.0090.051±0.0030.339±0.0128.968±0.08314.013±0.1586173±36055918±2641 0.5 (10k)0.593±0.02635.95±2.1770.666±0.0730.049±0.0030.314±0.0468.635±0.63413.432±1.1735882±35651403±9297 1.0 (20k)0.605±0.02333.497±0.4470.666±0.0130.043±0.0020.287±0.0088.575±0.07313.484±0.2445230±36346483±1318 NLPO 0.0 (0k)0.487±0.0132.572±0.1650.685±0.0030.043±0.0010.299±0.0038.691±0.02313.787±0.0345126±17748475±491 0.1 (2k)0.599±0.00733.536±0.3780.67±0.010.043±0.0010.289±0.0098.608±0.06113.576±0.1925125±22046755±1449 0.5 (10k)0.617±0.02133.409±0.3540.668±0.0050.041±0.0010.281±0.0068.552±0.04413.533±0.0914926±18345256±1022 1.0 (20k)0.637±0.01332.667±0.6310.677±0.0140.044±0.0020.288±0.0108.588±0.10013.484±0.2365205±18946344±2688 Table 7:Evaluation of GPT2 with different algorithms on IMDB sentiment text continuation task, data budget ablations: Mean and standard deviations over 5 random seeds is reported for sen- timent scores along with fluency and diversity metrics. This table measures performance differences as a function of the fraction of the dataset that has been used. In the case of the RL approaches, this measures how much data is used to train the reward classifier, and for the supervised method it directly measures fraction of positive reviews used for training. We note that using even a small fraction of data to train a reward classifier proves to be effective in terms of downstream task performance while this is not true for supervised approaches. This lends evidence to the hypothesis that adding expending data budget on a reward classifier is more effective than adding more gold label expert demonstrations. 22 Published as a conference paper at ICLR 2023 HyperparamsSemantic and Fluency MetricsDiversity Metrics Sentiment Score↑Perplexity↓MSTTRDistinct 1 Distinct 2 H 1 H 2 Unique 1 Unique 2 Target Update Iterationsμ 10.594±0.01832.671±0.2010.669±0.0080.042±0.0020.284±0.0078.575±0.06413.503±0.1814986±26545916±1168 100.622±0.01432.729±0.5670.659±0.0190.042±0.0020.274±0.0078.489±0.10613.31±0.2725138±38543989±1120 200.637±0.01332.667±0.6310.677±0.0140.044±0.0020.288±0.0108.588±0.10013.484±0.2365205±18946344±2688 500.603±0.01533.397±0.3250.67±0.0060.043±0.0010.287±0.0048.605±0.04113.54±0.1165228±11346418±685 Top-p mask 0.10.579±0.02132.451±0.2430.67±0.0080.042±0.0010.283±0.018.569±0.08413.515±0.1955018±4745760±1579 0.30.588±0.01932.451±0.3030.666±0.0070.043±0.0010.285±0.0048.568±0.03213.482±0.1725201±24746357±539 0.50.588±0.0132.447±0.3930.669±0.0010.044±0.0030.291±0.0088.614±0.05313.535±0.065305±38447251±1226 0.70.619±0.01332.373±0.3290.663±0.0080.043±0.0010.28±0.0068.533±0.04313.366±0.1295186±21645149±1452 0.90.637±0.01332.667±0.6310.677±0.0140.044±0.0020.288±0.0108.588±0.10013.484±0.2365205±18946344±2688 Table 8:Evaluation of GPT2 with different algorithms on IMDB sentiment text continuation task, NLPO hyperparameter ablations : Mean and standard deviations over 5 random seeds is reported for sentiment scores along with fluency and diversity metrics. This table shows results of NLPO’s stability to the unique hyperparameters introduced in the algorithm - all other parameters held constant from the best PPO model. The number of iterations after which the masking model syncs with the policy and the top-p nucleus percentage for the mask model itself. We see that in general, the higher the top-p mask percentage, the better the performance. For target update iterations, performance is low if the mask model is not updated often enough or if it updated too often. AlgorithmUnique N CoherenceSentiment ValueAlphaSkewValueAlphaSkew NLPO with KL273.490.1963.4973.610.23.601 NLPO without KL293.160.213.1584.410.1584.403 PPO without KL273.160.173.1634.360.1964.363 PPO with KL293.460.1243.4623.580.1163.575 Zero Shot283.60.1623.5913.10.133.097 Supervised293.510.1923.5123.430.23.428 Human274.130.1594.1283.010.313.017 Supervised+PPO223.450.2113.1473.640.213.161 Supervised+NLPO223.480.1813.2263.730.223.047 Table 9: Results of the human subject study showing the number of participants N, average Likert scale value for coherence and sentiment, Krippendorf’s alpha showing inter-annotator agreement, and Skew. For each model a total of 100 samples were drawn randomly from the test set and rated by 3 annotators each, resulting in 300 data points per algorithm. B.3.3HUMANPARTICIPANTSTUDY Figure 5 shows the IMDB instructions, example, and interface used both for the qualification round, and then later, for the human evaluation experiments. Tables 9, 10 show averaged results, annotator agreement, and the results of statistical significance tests to determine which models output better generations when rated by humans. 23 Published as a conference paper at ICLR 2023 Figure 5: Instructions, example, and interface for the IMDB sentiment completion task. 24 Published as a conference paper at ICLR 2023 Group 1Group 2 CoherenceSentiment Diff (G2-G1)p-valuesDiff (G2-G1)p-values PPO with KLPPO without KL-0.30.0350.7830.001 PPO with KLNLPO with KL0.030.90.0270.9 PPO with KLNLPO without KL-0.30.0350.8270.001 PPO with KLSupervised0.050.9-0.150.591 PPO with KLHuman0.6670.001-0.5670.001 PPO with KLZero Shot0.1370.776-0.4830.001 PPO without KLNLPO with KL0.330.013-0.7570.001 PPO without KLNLPO without KL0.0010.90.0430.9 PPO without KLSupervised0.350.006-0.9330.001 PPO without KLHuman0.9670.009-1.350.001 PPO without KLZero Shot0.4370.001-1.2670.001 NLPO with KLNLPO without KL-0.330.0130.80.001 NLPO with KLSupervised0.020.9-0.1770.404 NLPO with KLHuman0.6370.001-0.5930.001 NLPO with KLZero Shot0.1070.9-0.510.001 NLPO without KLSupervised0.350.006-0.9770.001 NLPO without KLHuman0.9670.001-1.3930.001 NLPO without KLZero Shot0.4370.001-1.310.001 SupervisedHuman0.6170.001-0.4170.001 SupervisedZero Shot0.0870.9-0.3330.0027 HumanZero Shot-0.530.0010.0830.9 Supervised+PPOSupervised+NLPO0.030.90.090.035 Supervised+PPONLPO with KL0.040.9-0.030.9 Supervised+PPONLPO without KL-0.290.0010.770.001 Supervised+PPOPPO without KL-0.290.0060.720.001 Supervised+PPOPPO with KL0.010.9-0.060.001 Supervised+PPOZero Shot0.150.035-0.540.001 Supervised+PPOSupervised0.060.001-0.210.001 Supervised+PPOHuman0.680.001-0.630.001 Supervised+NLPONLPO with KL0.010.9-0.120.001 Supervised+NLPONLPO without KL-0.320.0010.680.001 Supervised+NLPOPPO without KL-0.320.0350.630.001 Supervised+NLPOPPO with KL-0.020.9-0.150.006 Supervised+NLPOZero Shot-0.120.001-0.630.001 Supervised+NLPOSupervised0.030.9-0.30.001 Supervised+NLPOHuman0.650.001-0.720.006 Table 10: Results of an post-hoc Tukey HSD Test for difference in means between pairs of algorithms (Group 2 - Group 1) and correspondingp-values. Individually statistically significant results are bolded and are used to discuss results in the analysis. Overallp-values showing that there is a significant difference in means between the models via a one-way ANOVA test are significant with p0.05for both coherence and sentiment. 25 Published as a conference paper at ICLR 2023 B.3.4QUALITATIVERESULTS We show sample generations from each of the algorithms for three randomly picked prompts below. Sample1 Prompt: A man and his wife are not getting along because of the wife’s jealousy and drinking problems. When the wife disappears, her sister begins a vigorous search involving the Zero-Shot: husband’s sister, which leads several men to a remote cabin and then a boat. The wife and her husband then start to see the same things. Finally, a strange woman is driving out to her cabin and discovers the husband’s disappearance but References: local police, state government, and a television crime show. What she finds out stuns the entire community. Good drama with lots of fine performances; based on a true story PPO: police and the local public. At the end, the woman is killed and the wife is found naked and with a red head tattooed on her back. This is part of a special feature called"On the Road, Hidden" which was PPO-no-KL: police and other local governments. Written by G. Scott Fitzgerald, who also stars in this production, "A Very Merry Christmas" is a wonderful family movie with a wonderful soundtrack and wonderful characters. It brings home the true beauty of small- NLPO: mother while the brother goes under. After all, his sister, who was apparently kidnapped but who turns out to be a man, is no longer in her place. If a film that was made in the 1950s NLPO-no-KL: friends of the wife, and also the friend of the wife of the woman that is missing, and eventually he begins to find that she has gone to Germany and he has made a very good friend. There is a good scene between John and Supervised: amily tree. The widower, now widowed, returns, the daughter, daughter is not pleased when his wife is mistaken for one of her sons, and the patriarch, then a wealthy man, falls victim to a jealous husband. The Sample2 Prompt: A very credible and unsettling movie portraying the effects of war on the general public, Shame gives us a chilling understanding oh Ingmar Bergman’s beliefs. The absence of color and sound (as in a soundtrack) helps to give the film a more realistic feeling. There is no soft or hard lighting or a variety of camera angles to give the story a charming or dramatic effect. Everything is very simply portrayed as it would be in reality. At times the storyline was confusing, but perhaps in order to give a sense of the characters’ frame of mind how they too do now know what is happening or why. The simplicity of the Zero-Shot: camera work is at once apparent and fascinating in the manner in which it affects the way things are portrayed. In other films these two have been referred to from time to time in a form and manner more suited to the screen. The film shows References: film made it somewhat boring, but I could understand how the director is trying not to glamorize the story and points. He depicted the harsh and unsettling events and effects of war. In the beginning the characters were introduced and shown as content with the lighter settings PPO: characters relationships is so engrossing and the great and unique story presented by each pair of brothers and sisters creates a truly gripping and enjoyable experience. To put it simply, I strongly recommend to everyone who plays the role to buy this DVD!!! PPO-no-KL: movie helps it to flow along in a believable way. Rating 4 out of 5 Great Movie! I highly recommend these movies! I highly recommend these movies! Rating 5 out of 5 Great Movie! I highly recommend these movies! Rating 10 of" NLPO: novel makes it one of the first books I have read by a British actor. It is a fun way for someone who likes their books, like me, to sit through a novel. If you like to read novels, check it out. NLPO-no-KL: setting makes the final product very enjoyable to watch. Highly recommended!! ~ Michael Bowerman, SPSA Film Festival Host, SPSA, WA (December 17, 2006), 8/17/2006 Great photography, good acting, Supervised: "Dakota Clan" and his sister, the two become friends with Ted Masters who soon finds the chance to fly back to their home land and go back to save his brother’s life and his family. The Sample3 Prompt: Shakespearean language is wild and rambling, saturated in multiple meanings, word play and metaphor. To be understood it must be wrangled and tamed by an actor with the strength and knowledge to do so. When an actor fails, the words pour forth in a torrent of incomprehensible words, but when he succeeds, the English language springs to life with an immediacy and vibrancy that takes your breath away. Al Pacino is one such actor, and here displays an incredible level of clarity and control that, were there any justice, Zero-Shot: might still be seen in the world today. This movie, like "Vacation", is an effort at telling the tale of two families from different races. One is African but never grew up the other. One of them, a man References: would sweep every award in the offering. He meets the challenge of presenting Shylock head on, and delivers an extraordinarily subtle and nuanced performance. It would be a crime if we never got the opportunity to see what he does with King Lear.<br /><br />The supporting cast PPO: would have made him an instant-classic. This movie captures the spirit of childhood and has everyone in the world watching it. It’s a love story and it’s really heartbreaking. And at the same time, it’s a classic. It 26 Published as a conference paper at ICLR 2023 PPO-no-KL: will bring victory to the fight against hatred and extremism. Al Pacino is a very beautiful, well executed actor who brings an absolute intensity and sensibility to his portrayal of love and loyalty. A must see! Recommended! Also, an Excellent NLPO: I would watch and enjoy. 7/10, Oscar Wilde is always a pleasure to see. A must see. Highly recommend. Highly recommend! One of my top ten films of all time. A must see!! NLPO-no-KL: the whole film would have fallen to the same fate, just as it did just a couple hundred years ago. Don’t miss it. It’s a real classic. Highly Recommended. * outta five stars for it! Supervised: his performance (so far) would seem mere shadow. He is truly in the middle of a movie, and this film is one of those films where he can be convincing in it (and his trademark acting, as you can see in the 27 Published as a conference paper at ICLR 2023 B.4COMMONGEN B.4.1SETUP CommonGen (Lin et al., 2020) deals with task of generating coherent sentences describing an input set of concepts (eg. "a man is throwing a frisbee"). For training RL methods, we consider 3 traditional lexical rewards namely Rouge-1, Rouge-avg (which is an average of Rouge-1, 2 and L) and meteor. Additionally, we also train with task-specific rewards such as CIDEr (Vedantam et al., 2015), SPICE (Anderson et al., 2016) and SPiDer (Liu et al., 2017) which is a just a linear combination of both with equal weights. We chose T5-base as the base LM since it is well-suited for structure to text tasks. We additionally note that concept set inputs are prefixed with "generate a sentence with:" to encourage exploration. During our initial experiments when fine-tuning directly on LM, we observed that policy learns to repeat the prompted concepts in order to maximize rewards resulting in a well-known problem of reward hacking. To mitigate this, we add a penalty score of−1to final task reward if the n-grams of prompt text overlaps with generated text. In contrast, when initialized with a supervised policy, this problem is not seen and hence penalty score is not applied. We use beam search as the decoding method during evaluation whereas for rollouts, we use top k sampling to favor exploration over exploitation. Table 11 provides an in-depth summary of setting of hyperparameter values along with other implementation details. Model Paramsvalue supervisedbatch size:8 epochs:4 learning rate:0.00001 learning rate scheduler: cosine weight decay:0.01 ppo/ nlposteps per update:1280 total number of steps:256000 batch size:64 epochs per update:5 learning rate:0.000002 entropy coefficient:0.01 initial kl coeff:0.001 target kl:2.0 discount factor:0.99 gae lambda:0.95 clip ratio:0.2 value function coeff:0.5 top mask ratio:0.9 target update iterations:20 supervised+ ppo (or nlpo)steps per update:1280 total number of steps:128000 batch size:64 epochs per update:5 learning rate:0.000002 entropy coefficient:0.01 initial kl coeff:0.01 target kl:1.0 discount factor:0.99 gae lambda:0.95 clip ratio:0.2 value function coeff:0.5 top mask ratio:0.9 target update iterations:20 decodingnum beams:5 min length:5 max new tokens:20 tokenizerpadding side: left max length:20 Table 11:CommonGen Hyperparams: Table shows a list of all hyper-parameters and their settings 28 Published as a conference paper at ICLR 2023 B.4.2RESULTS ANDDISCUSSION Tables 13, 12 presents our benchmarking results with 6 reward functions along with supervised baseline performances on dev and test sets respectively. Our main finding is that warm-started initial policies are crucial for learning to generate coherent sentences with common sense. Without warm- start, policies suffer from reward hacking despite application of repetition penalty and task-specific metrics such as CIDer etc. Further, we find that RL fine-tuned models obtain very high concept coverage which is also seen in Table B.4.5. Supervised models often tend to miss few concepts in its generation compared to RL methods. Tasks_Lexical and Semantic Metrics AlgLMReward functionRouge-2Rouge-LBleu (n=3)Bleu (n=4)MeteorCIDErSPICECoverage CommonGen Zero-ShotT50.0160.2640.0290.0060.2036.2000.11591.070 PPOT5Rouge-10.085±0.0080.354±0.0040.161±0.0110.087±0.0090.235±0.0028.673±0.2340.157±0.00188.544±2.36 T5Rouge-Avg0.093±0.0050.351±0.0010.169±0.0320.097±0.0170.224±0.0128.212±1.3290.159±0.01182.584±2.569 T5Meteor0.091±0.0080.308±0.0070.166±0.0160.088±0.0130.220±0.0067.251±0.4530.161±0.00779.718±2.267 T5SPice0.065±0.0030.302±0.0020.115±0.0630.067±0.0410.193±0.0146.571±1.3120.175±0.01169.340±3.617 T5CiDer0.066±0.0030.304±0.0020.132±0.0570.074±0.0360.211±0.0096.877±1.2180.143±0.01780.114±4.852 T5SPider0.117±0.0050.352±0.0070.224±0.0140.137±0.0110.226±0.019.162±0.5390.186±0.00673.374±6.073 NLPOT5Rouge-10.087±0.0020.339±0.0090.127±0.0480.069±0.0350.213±0.0026.962±0.8830.145±0.02280.89±9.544 T5Rouge-Avg0.095±0.0010.338±0.0020.159±0.020.093±0.0130.216±0.0097.55±0.6880.153±0.00877.944±2.770 T5Meteor0.110±0.0050.332±0.0030.214±0.0070.124±0.0070.235±0.0048.669±0.1640.173±0.00282.007±1.012 T5SPice0.014±0.0060.242±0.0010.037±0.0110.018±0.0070.156±0.0074.685±0.2830.168±0.00856.998±3.548 T5CiDer0.046±0.0010.241±0.0030.078±0.0280.043±0.0160.143±0.0183.964±0.7920.103±0.01249.606±7.971 T5SPider0.060±0.0060.258±0.0010.090±0.0080.056±0.0050.151±0.0224.411±0.8370.123±0.02249.230±10.468 SupervisedT50.215±0.0010.438±0.0010.444±0.0010.329±0.0010.321±0.00116.385±0.0460.299±0.00194.476±0.172 Supervised + PPOT5Rouge-10.232±0.0020.453±0.0020.454±0.0060.338±0.0060.320±0.00216.233±0.1590.288±0.00496.412±0.424 T5Rouge-Avg0.230±0.0010.450±0.0010.448±0.0050.334±0.0050.319±0.00116.069±0.1670.287±0.00396.116±0.679 T5Meteor0.234±0.0020.450±0.0030.462±0.0070.342±0.0070.327±0.00116.797±0.1520.295±0.00197.690±0.371 T5SPice0.227±0.0040.447±0.0030.450±0.0070.336±0.0080.319±0.00216.208±0.2490.288±0.00396.492±0.29 T5CiDer0.224±0.0030.446±0.0030.427±0.0120.309±0.010.316±0.00415.497±0.4280.283±0.00496.344±0.547 T5SPider0.226±0.0030.448±0.0020.436±0.0050.319±0.0040.317±0.00315.678±0.1920.281±0.00396.154±0.426 Supervised + NLPOT5Rouge-10.229±0.0020.450±0.0010.454±0.0050.338±0.0040.320±0.00316.206±0.1750.289±0.00296.342±0.572 T5Rouge-Avg0.232±0.0030.451±0.0020.458±0.010.342±0.0090.321±0.00316.351±0.3350.290±0.00595.998±0.496 T5Meteor0.231±0.0030.449±0.0020.454±0.0070.334±0.0080.326±0.00216.574±0.2690.292±0.00397.374±0.457 T5SPice0.223±0.0020.442±0.0010.435±0.0110.321±0.0100.315±0.00415.747±0.4010.283±0.00596.25±0.313 T5CiDer0.226±0.0020.447±0.0040.433±0.0070.315±0.0080.318±0.00315.741±0.1700.285±0.00196.354±0.971 T5SPider0.226±0.0040.447±0.0030.434±0.0060.316±0.0060.319±0.00215.739±0.3110.284±0.00396.333±0.644 Table 12:CommonGen test evaluationTable shows official scores obtained from CommonGen hold-out evaluation. The most important result is that RL fine-tuning on a supervised model yields better performance across most metrics especially Coverage which indicates the ratio of concepts covered in generated texts 29 Published as a conference paper at ICLR 2023 Tasks _ Lexical and Semantic Metrics Diversity Metrics Alg Reward Function Top k LM Rouge-1 Rouge-2 Rouge-L Rouge-LSum Meteor BLEU BertScore Cider Spice MSTTR Distinct 1 Distinct 2 H 1 H 2 Unique 1 Unique 2 Mean Output Length CommonGen Zero-Shot T5 0.415 0.016 0.270 0.270 0.179 0.0 0.854 0.640 0.231 0.430 0.090 0.335 5.998 7.957 345 1964 8.797 PPO Rouge-1 50 T5 0.537 ± 0.004 0.093 ± 0.012 0.380 ± 0.006 0.380 ± 0.006 0.235 ± 0.005 0.016 ± 0.002 0.896 ± 0.001 0.950 ± 0.015 0.318 ± 0.016 0.526 ± 0.020 0.128 ± 0.005 0.518 ± 0.036 6.679 ± 0.132 10.572 ± 0.234 437.4 ± 42.017 2418.8 ± 167.947 7.214 ± 0.374 Rouge-Avg 50 T5 0.519 ± 0.0185 0.102 ± 0.007 0.377 ± 0.013 0.376 ± 0.014 0.225 ± 0.024 0.020 ± 0.002 0.897 ± 0.005 0.921 ± 0.102 0.328 ± 0.009 0.536 ± 0.069 0.141 ± 0.022 0.510 ± 0.056 6.777 ± 0.539 10.348 ± 0.134 458.6 ± 19.734 2244.4 ± 162.855 6.887 ± 1.006 Meteor 50 T5 0.411 ± 0.009 0.090 ± 0.008 0.304 ± 0.006 0.304 ± 0.006 0.210 ± 0.005 0.029 ± 0.004 0.875 ± 0.007 0.638 ± 0.048 0.259 ± 0.017 0.547 ± 0.012 0.147 ± 0.003 0.529 ± 0.014 7.62 ± 0.127 11.464 ± 0.151 1039.4 ± 63.276 5197.2 ± 280.004 13.660 ± 0.324 SPice 50 T5 0.439 ± 0.035 0.079 ± 0.045 0.323 ± 0.036 0.323 ± 0.036 0.183 ± 0.022 0.012 ± 0.009 0.891 ± 0.005 0.777 ± 0.140 0.400 ± 0.012 0.546 ± 0.054 0.149 ± 0.019 0.545 ± 0.072 6.721 ± 0.441 10.492 ± 0.330 409.2 ± 41.605 1878.4 ± 167.492 5.706 ± 0.678 CiDer 50 T5 0.453 ± 0.038 0.081 ± 0.037 0.326 ± 0.033 0.326 ± 0.033 0.203 ± 0.022 0.017 ± 0.009 0.885 ± 0.008 0.770 ± 0.134 0.291 ± 0.036 0.597 ± 0.081 0.195 ± 0.040 0.639 ± 0.106 7.732 ± 0.682 11.131 ± 0.502 777.0 ± 144.676 3350.8 ± 503.419 7.393 ± 0.572 SPider 50 T5 0.512 ± 0.008 0.141 ± 0.007 0.388 ± 0.002 0.388 ± 0.003 0.242 ± 0.007 0.032 ± 0.003 0.902 ± 0.001 1.045 ± 0.034 0.380 ± 0.006 0.482 ± 0.015 0.133 ± 0.003 0.472 ± 0.021 6.372 ± 0.221 10.303 ± 0.228 502.6 ± 33.422 2281.4 ± 252.471 7.489 ± 0.358 NLPO Rouge-1 50 T5 0.499 ± 0.012 0.089 ± 0.003 0.328 ± 0.007 0.328 ± 0.007 0.198 ± 0.002 0.021 ± 0.001 0.872 ± 0.005 0.815 ± 0.009 0.305 ± 0.008 0.559 ± 0.01 0.148 ± 0.003 0.555 ± 0.012 7.059 ± 0.067 10.657 ± 0.105 457.9 ± 11.108 2349.6 ± 60.345 6.586 ± 0.094 Rouge-Avg 50 T5 0.47 ± 0.01 0.096 ± 0.004 0.312 ± 0.006 0.312 ± 0.006 0.202 ± 0.008 0.025 ± 0.002 0.843 ± 0.013 0.816 ± 0.026 0.299 ± 0.007 0.512 ± 0.019 0.146 ± 0.011 0.513 ± 0.012 6.781 ± 0.15 10.424 ± 0.156 484.18 ± 17.303 2357.54 ± 152.113 7.131 ± 0.487 Meteor 50 T5 0.389 ± 0.013 0.1 ± 0.004 0.293 ± 0.008 0.293 ± 0.008 0.226 ± 0.024 0.035 ± 0.004 0.832 ± 0.018 0.691 ± 0.04 0.266 ± 0.016 0.503 ± 0.003 0.132 ± 0.005 0.471 ± 0.008 7.146 ± 0.192 10.727 ± 0.313 648.05 ± 33.963 3536.0 ± 444.638 11.062 ± 1.301 SPice 50 T5 0.329 ± 0.015 0.036 ± 0.008 0.247 ± 0.013 0.247 ± 0.013 0.137 ± 0.009 0.006 ± 0.002 0.817 ± 0.024 0.515 ± 0.033 0.323 ± 0.021 0.543 ± 0.023 0.174 ± 0.004 0.568 ± 0.026 7.176 ± 0.212 10.551 ± 0.216 479.45 ± 19.77 2065.8 ± 288.843 5.785 ± 0.431 CiDer 50 T5 0.515 ± 0.006 0.143 ± 0.008 0.387 ± 0.006 0.308 ± 0.006 0.19 ± 0.001 0.019 ± 0.001 0.865 ± 0.015 0.726 ± 0.018 0.282 ± 0.009 0.55 ± 0.02 0.179 ± 0.005 0.576 ± 0.014 7.286 ± 0.125 10.812 ± 0.089 661.46 ± 21.776 2726.32 ± 71.253 7.13 ± 0.223 SPider 50 T5 0.393 ± 0.008 0.086 ± 0.012 0.297 ± 0.007 0.297 ± 0.007 0.183 ± 0.007 0.02 ± 0.003 0.842 ± 0.019 0.717 ± 0.026 0.297 ± 0.019 0.525 ± 0.024 0.167 ± 0.009 0.537 ± 0.025 6.986 ± 0.262 10.451 ± 0.171 530.14 ± 16.805 2263.4 ± 166.221 6.687 ± 0.372 Supervised T5 0.503 ± 0.001 0.175 ± 0.001 0.411 ± 0.001 0.411 ± 0.001 0.309 ± 0.001 0.069 ± 0.001 0.929 ± 0.000 1.381 ± 0.011 0.443 ± 0.001 0.509 ± 0.001 0.101 ± 0.001 0.339 ± 0.001 6.531 ± 0.006 10.079 ± 0.016 503.600 ± 6.530 2158.8 ± 24.514 10.934 ± 0.020 Supervised + PPO Rouge-1 50 T5 0.537 ± 0.004 0.198 ± 0.005 0.433 ± 0.002 0.433 ± 0.002 0.314 ± 0.003 0.070 ± 0.002 0.930 ± 0.001 1.426 ± 0.018 0.449 ± 0.001 0.527 ± 0.007 0.112 ± 0.001 0.393 ± 0.004 6.680 ± 0.044 10.289 ± 0.040 498.2 ± 8.931 2317.0 ± 22.609 9.667 ± 0.105 Rouge-Avg 50 T5 0.536 ± 0.001 0.198 ± 0.002 0.433 ± 0.002 0.433 ± 0.002 0.311 ± 0.002 0.070 ± 0.002 0.929 ± 0.001 1.421 ± 0.028 0.446 ± 0.004 0.526 ± 0.004 0.114 ± 0.002 0.395 ± 0.005 6.682 ± 0.0297 10.274 ± 0.042 506.4 ± 6.829 2326.4 ± 41.778 9.614 ± 0.102 Meteor 50 T5 0.540 ± 0.005 0.204 ± 0.005 0.436 ± 0.004 0.436 ± 0.004 0.329 ± 0.003 0.076 ± 0.003 0.930 ± 0.001 1.474 ± 0.022 0.447 ± 0.004 0.514 ± 0.004 0.105 ± 0.002 0.378 ± 0.008 6.631 ± 0.053 10.270 ± 0.064 507.0 ± 17.146 2424.6 ± 72.550 10.551 ± 0.271 SPice 50 T5 0.532 ± 0.006 0.194 ± 0.007 0.430 ± 0.005 0.430 ± 0.005 0.311 ± 0.004 0.068 ± 0.003 0.929 ± 0.001 1.415 ± 0.029 0.458 ± 0.001 0.532 ± 0.008 0.113 ± 0.0038 0.392 ± 0.009 6.736 ± 0.058 10.338 ± 0.057 507.4 ± 14.319 2313.8 ± 27.694 9.742 ± 0.208 CiDer 50 T5 0.530 ± 0.004 0.191 ± 0.003 0.427 ± 0.004 0.427 ± 0.004 0.309 ± 0.008 0.063 ± 0.002 0.928 ± 0.001 1.337 ± 0.040 0.444 ± 0.002 0.518 ± 0.009 0.110 ± 0.003 0.382 ± 0.006 6.614 ± 0.082 10.166 ± 0.053 490.4 ± 9.457 2295.4 ± 51.554 9.838 ± 0.265 SpiDer 50 T5 0.536 ± 0.002 0.197 ± 0.002 0.430 ± 0.002 0.430 ± 0.002 0.313 ± 0.002 0.064 ± 0.002 0.928 ± 0.001 1.374 ± 0.018 0.445 ± 0.003 0.524 ± 0.007 0.112 ± 0.001 0.394 ± 0.004 6.673 ± 0.066 10.247 ± 0.066 504.8 ± 7.440 2361.8 ± 20.856 9.761 ± 0.121 Supervised + NLPO Rouge-1 50 T5 0.545 ± 0.002 0.197 ± 0.002 0.432 ± 0.001 0.432 ± 0.001 0.31 ± 0.002 0.068 ± 0.001 0.929 ± 0.0 1.41 ± 0.012 0.449 ± 0.001 0.529 ± 0.002 0.114 ± 0.002 0.399 ± 0.005 6.705 ± 0.018 10.301 ± 0.03 498.86 ± 8.594 2311.46 ± 33.451 9.463 ± 0.111 Rouge-Avg 50 T5 0.541 ± 0.003 0.2 ± 0.003 0.435 ± 0.002 0.435 ± 0.002 0.313 ± 0.002 0.07 ± 0.002 0.93 ± 0.001 1.424 ± 0.023 0.447 ± 0.003 0.53 ± 0.006 0.113 ± 0.002 0.396 ± 0.008 6.708 ± 0.05 10.318 ± 0.074 493.64 ± 10.068 2319.42 ± 55.738 9.596 ± 0.123 Meteor 50 T5 0.537 ± 0.003 0.201 ± 0.004 0.431 ± 0.002 0.431 ± 0.002 0.326 ± 0.002 0.074 ± 0.003 0.93 ± 0.0 1.464 ± 0.025 0.448 ± 0.002 0.516 ± 0.006 0.106 ± 0.002 0.377 ± 0.008 6.634 ± 0.044 10.26 ± 0.077 506.04 ± 3.502 2401.32 ± 38.569 10.453 ± 0.194 SPice 50 T5 0.535 ± 0.007 0.193 ± 0.008 0.429 ± 0.005 0.429 ± 0.005 0.3 ± 0.003 0.064 ± 0.002 0.927 ± 0.001 1.333 ± 0.017 0.459 ± 0.003 0.553 ± 0.013 0.12 ± 0.004 0.415 ± 0.014 6.908 ± 0.118 10.445 ± 0.057 508.075 ± 4.669 2343.3 ± 53.274 9.249 ± 0.225 CiDer 50 T5 0.533 ± 0.003 0.197 ± 0.004 0.43 ± 0.003 0.43 ± 0.004 0.316 ± 0.004 0.066 ± 0.001 0.929 ± 0.001 1.381 ± 0.014 0.446 ± 0.004 0.516 ± 0.009 0.108 ± 0.003 0.379 ± 0.01 6.583 ± 0.077 10.165 ± 0.084 490.78 ± 9.734 2304.52 ± 62.068 9.923 ± 0.213 SPider 50 T5 0.532 ± 0.006 0.196 ± 0.006 0.431 ± 0.004 0.431 ± 0.004 0.314 ± 0.004 0.066 ± 0.002 0.929 ± 0.0 1.371 ± 0.011 0.448 ± 0.002 0.521 ± 0.005 0.109 ± 0.002 0.385 ± 0.005 6.623 ± 0.034 10.223 ± 0.049 485.325 ± 5.683 2297.575 ± 21.271 9.798 ± 0.179 Table 13: CommonGen dev evaluation : Table shows lexical, semantic and diversity metrics for best performing models found in each algorithm-reward function combinations along with best performing supervised baseline models. Generated text from these models are submitted to official CommonGen test evaluation toobtain test scores presented in Table 12 30 Published as a conference paper at ICLR 2023 AlgorithmUnique N CoherenceCommonsense ValueAlphaSkewValueAlphaSkew PPO+Supervised254.140.0734.1374.030.1374.023 NLPO+Supervised264.250.0364.2534.160.0024.163 Zero Shot242.150.3912.1542.290.3422.291 PPO242.840.162.8493.030.0813.027 Supervised234.390.1594.3874.210.2254.209 NLPO2420.3352.0032.130.2652.124 Table 14: Results of the human subject study showing the number of participants N, average Likert scale value for coherence and sentiment, Krippendorf’s alpha showing inter-annotator agreement, and Skew. For each model a total of 100 samples were drawn randomly from the test set and rated by 3 annotators each, resulting in 300 data points per algorithm. Group 1Group 2 CoherenceCommonsense Diff (G2-G1)p-valuesDiff (G2-G1)p-values NLPOPPO0.8470.0010.8970.001 NLPOSupervised2.3970.0012.0830.001 NLPONLPO+Supervised2.2570.0012.0330.001 NLPOPPO+Supervised2.1430.0011.8970.001 NLPOZero Shot0.1530.5150.1570.624 PPOSupervised1.5500.0011.1870.001 PPONLPO+Supervised1.4100.0011.1370.001 PPOPPO+Supervised1.2970.0011.0000.001 PPOZero Shot-0.6930.001-0.7400.001 SupervisedNLPO+Supervised-0.1400.601-0.0500.900 SupervisedPPO+Supervised-0.2530.050-0.1870.045 SupervisedZero Shot-2.2430.001-1.9270.001 NLPO+SupervisedPPO+Supervised-0.1130.008-0.1370.007 NLPO+SupervisedZero Shot-2.1030.001-1.8770.001 PPO+SupervisedZero Shot-1.9900.001-1.7400.001 Table 15: Results of an post-hoc Tukey HSD Test for difference in means between pairs of algorithms (Group 2 - Group 1) and correspondingp-values. Individually statistically significant results are bolded and are used to discuss results in the analysis. Overallp-values showing that there is a significant difference in means between the models via a one-way ANOVA test are significant with p0.05for both coherence and sentiment. 31 Published as a conference paper at ICLR 2023 Figure 6: Instructions, examples, and interface for the Commongen task. B.4.3HUMANPARTICIPANTSTUDY Figure 6 shows the commongen instructions, examples, and interface used for the human evaluation experiments. Different from the other human evaluations, we didn’t provide any prompt because knowing the set of words to be used isn’t required for rating either of the axes. Tables 14, 15 show averaged results, annotator agreement, and the results of statistical significance tests to determine which models output better generations when rated by humans. B.4.4HUMANPREFERENCELEARNINGEXPERIMENTS First, we randomly select prompts from the Commongen train dataset and sample a single completion from both the Supervised and Supervised+NLPO models. Next, we filter to prompts where both models at least attempted to use all input concepts. This filtration step was conducted because if a model fails to use all concepts, it may generate a more natural/fluent sentence, but,a priori, it shouldn’t be preferred by crowdworkers; instead of training crowdworkers to prefer sentences with all concepts, we perform this filter. Figure 7 shows the task presented to the crowdworkers. We then present the prompt and the two completion candidates to 3 unique crowdworkers and ask them to 32 Published as a conference paper at ICLR 2023 Figure 7: Instructions and interface for the pairwise Commongen HIT. select which one they prefer with respect to commonsense/fluency; We gathered 3 annotations on 417 pairs (Krippendorfα=.28), and split into 60/20/20 train/val/test split. We then trained a reward model, T5-11B Raffel et al. (2020), on the balanced binary classification task of predicting which of the pair was preferred by a majority of 3 annotators, conditioned on the prompt and completion. The resulting model achieved 69.5 test ROC AUC suggesting it indeed captures average human preferences. The model is then used as a reward function. We train Supervised+RL with a METEOR- only reward as a baseline, and compare it to a reward function that uses the fine-tuned T5-11B model. We design the reward function based on the preference model asr=meteor+pref/(1 +|miss|) wheremissis a set of concepts not covered in the generated text, in an attempt to mimic the data collection process that humans are instructed to follow. This reward function accounts for both the task of using all concepts and also human’s preferences for how a sentence should look within the constraints stipulated by the task. Finally, we rerun the same pairwise preference collection procedure—this time sampling from Commongen test—with human participants to compare the generations from a preference optimized RL policy to the previously best Supervised+NLPO policy. Comparing the METEOR-only to the preference model head-to-head, the generations produced by the human feedback model are preferred in 682 cases, compared to the METEOR-only model which is preferred in 587 cases (p <0.01the models are equally preferred). B.4.5QUALITATIVEANALYSIS This section shows sample generations from different algorithms for three randomly picked prompts. Sample1 Prompt: generate a sentence with: apron cut hat kitchen sausage Zero-Shot: generate a sentence with: apron cut hat kitchen sausage. PPO: sausage in the kitchen on an apron. NLPO: sausage cut hat cut hat cut hat cut apron cut hat Supervised: A man is cutting sausage on an apron in a kitchen. Supervised+PPO: A man cutting sausage from a hat on an apron in the kitchen. Supervised+NLPO: A man cuts sausage in a hat on an apron in the kitchen. Sample2 33 Published as a conference paper at ICLR 2023 Prompt: generate a sentence with: backhand competition player reach shuttlecock Zero-Shot: backhand competition player reach shuttlecock. PPO: the ball in the backhand and the competition player in the shuttlecock. NLPO: backhanded competition player reaches shuttlecock. Supervised: tennis player reaches for a shuttlecock during a competition. Supervised+PPO: football player reaches for a shuttlecock in backhand during the competition. Supervised+NLPO: football player reaches for a shuttlecock after hitting a backhand during the competition. Sample3 Prompt: generate a sentence with: bowl combine cucumber onion pasta Zero-Shot: cucumber onion pasta. PPO: the cucumber onion pasta in a small bowl. NLPO: the in a bowl Combine cucumber onion pasta in a large bowl. Toss pasta Supervised: A bowl containing pasta, tomatoes, cucumbers, and onions. Supervised+PPO: A bowl containing pasta topped with cucumbers, onions, and peppers. Supervised+NLPO: A bowl containing a mixture of pasta, cucumber, and onion. 34 Published as a conference paper at ICLR 2023 B.5CNN DAILYMAIL B.5.1SETUP As a representative of the summarization task, we consider CNN/DM dataset consisting of long news articles and their highlights written by news authors. The dataset consists of 287k training, 13k validation and 11k test examples. We trained RL methods using 3 different automated metrics, namely Rouge-1, Rouge-avg and Meteor. We chose T5 as our base LM as it is pre-trained in a unified text-to-text framework and relishes Zero-Shot capabilities. For decoding, we use multinomial sampling with a temperature of0.7for all the models. Model Paramsvalue supervisedbatch size:16 epochs:2 learning rate:0.0001 learning rate scheduler: cosine weight decay:0.1 ppo/ nlposteps per update:5120 total number of steps:512000 batch size:64 epochs per update:5 learning rate:0.000002 entropy coefficient:0.0 initial kl coeff:0.001 target kl:0.2 discount factor:0.99 gae lambda:0.95 clip ratio:0.2 value function coeff:0.5 rollouts top k: sweep of (50,100) top mask ratio:0.9 target update iterations: sweep of (10,20,30) supervised+ppo/ nlposteps per update: 5120 total number of steps:256000 batch size:64 epochs per update:5 learning rate:0.000002 entropy coefficient:0.0 initial kl coeff:0.01 target kl:0.2 discount factor:0.99 gae lambda:0.95 clip ratio:0.2 value function coeff:0.5 rollouts top k: sweep of (50,100) top mask ratio:0.9 target update iterations: sweep of (10,20,30) decodingsampling: True temperature:0.7 min length:50 max new tokens:100 tokenizerpadding side: left truncation side: right max length: 512 Table 16:CNN/DM Hyperparams: Table shows a list of all hyper-parameters and their settings B.5.2RESULTS ANDDISCUSSION Table 17 presents benchmarking results on test set reporting a wide range of metrics: lexical, semantic, factual correctness and diversity metrics. As baselines, we report lead-3 which selects first three sentences as the summary, Zero-Shot and a supervised model. PPO and NLPO models are on par with supervised performance on several metrics including Rouge-2, Rouge-L, and Bleu. On fine-tuning on top of supervised model, performance improves consistently on all metrics indicating that RL fine-tuning is beneficial. Another interesting finding is that, RL fine-tuned models are factually consistent as measured by SummaCZS metric. For ablations on PPO params, NLPO params, we refer to Tables 18,19. 35 Published as a conference paper at ICLR 2023 Tasks _ Lexical and Semantic Metrics Factual Consistency Diversity Metrics Alg Reward Function LM Rouge-1 Rouge-2 Rouge-L Rouge-LSum Meteor BLEU BertScore SummaCZS MSTTR Distinct 1 Distinct 2 H 1 H 2 Unique 1 Unique 2 Mean Output Length CNN/DM Lead-3 0.401 0.175 0.250 0.363 0.333 0.099 0.874 0.993 0.750 0.0482 0.386 10.481 16.631 21465 273153 84 Zero-Shot T5 0.372 0.145 0.247 0.311 0.256 0.077 0.864 0.654 0.725 0.061 0.414 10.285 16.183 19113 193999 55 PPO Rouge-1 T5 0.410 0.182 0.283 0.349 0.276 0.095 0.876 0.622 0.760 0.068 0.464 10.661 16.437 18189 191383 47 Rouge-Avg T5 0.396 0.176 0.273 0.338 0.270 0.095 0.874 0.622 0.773 0.071 0.490 10.830 16.664 19478 209140 48 Meteor T5 0.408 0.178 0.276 0.342 0.301 0.109 0.873 0.527 0.765 0.060 0.447 10.699 16.688 20528 234386 61 NLPO Rouge-1 T5 0.404 0.180 0.278 0.344 0.275 0.096 0.875 0.636 0.771 0.069 0.480 10.789 16.618 18677 201971 48 Rouge-Avg T5 0.404 0.177 0.279 0.344 0.274 0.094 0.874 0.586 0.765 0.066 0.476 10.744 16.620 18179 206368 50 Meteor T5 0.405 0.180 0.277 0.343 0.292 0.108 0.872 0.578 0.772 0.064 0.471 10.802 16.766 20212 231038 56 Supervised T5 0.411 0.177 0.276 0.343 0.309 0.108 0.876 0.654 0.727 0.057 0.401 10.459 16.410 21096 230343 68 Supervised + PPO Rouge-1 T5 0.417 0.189 0.294 0.358 0.278 0.101 0.882 0.722 0.750 0.070 0.459 10.595 16.389 18184 184220 46 Rouge-Avg T5 0.425 0.194 0.297 0.363 0.296 0.114 0.882 0.728 0.747 0.066 0.445 10.589 16.458 18939 200617 52 Meteor T5 0.426 0.194 0.293 0.361 0.316 0.125 0.880 0.726 0.741 0.059 0.420 10.532 16.491 20395 224432 63 Supervised + NLPO Rouge-1 T5 0.421 0.193 0.297 0.361 0.287 0.108 0.882 0.740 0.748 0.067 0.446 10.528 16.313 18204 185561 48 Rouge-Avg T5 0.424 0.193 0.296 0.363 0.295 0.115 0.882 0.743 0.744 0.065 0.443 10.570 16.444 18747 201705 53 Meteor T5 0.429 0.194 0.293 0.361 0.319 0.124 0.880 0.743 0.745 0.059 0.422 10.574 16.516 20358 226801 63 Table 17: CNN/Daily Mail test evaluation : Table presents a wide range of metrics: lexical, semantic, factual correctness and diversity metrics on test set. As baselines, we report lead-3 which selects first three sentences as the summary, Zero-Shot and a supervised model. PPO and NLPO models are on par with supervisedperformance on several metrics including Rouge-2, Rouge-L, and Bleu. On fine-tuning on top of supervised model, performance improves consistently on all metricsindicating that RL fine-tuning is beneficial. Another interesting finding is that, RL fine-tuned models are factually consistent as measured by SummaCZS metric. 36 Published as a conference paper at ICLR 2023 _Lexical and Semantic Metrics AlgReward FunctionTop kRouge-1Rouge-2Rouge-LRouge-LSumMeteorBLEUBertScore PPO Rouge-1500.4040.1810.2800.3460.2730.0950.874 1000.4120.1860.2860.3540.2760.0940.876 Rouge-Avg500.4010.1770.2760.3420.2710.0920.873 1000.3990.1790.2750.3420.2700.0940.874 Meteor500.4130.1820.2790.3480.3010.1100.873 1000.4090.1790.2760.3450.2960.1080.871 Supervised+PPO Rouge-1500.4140.1900.2930.3580.2720.0970.881 1000.4200.1930.2950.3620.2770.1000.881 Rouge-Avg500.4260.1960.2980.3660.2940.1140.881 1000.4270.1960.2980.3660.2940.1130.881 Meteor500.4290.1970.2970.3670.3060.1220.881 1000.4320.1990.2970.3670.3170.1310.879 Table 18:PPO Ablation/Model Selection: Evaluation of PPO models on validation set with different reward functions and top k values for rollouts. For each alg-reward combo, best model (top k ) is chosen. _Lexical and Semantic Metrics AlgReward FunctionTop k (rollout)Top p (Action mask)target updaten i tersRouge-1Rouge-2Rouge-LRouge-LSumMeteorBLEUBertScore NLPO Rouge-1500.9100.4000.1780.2750.3430.2690.0940.872 200.3960.1730.2740.3400.2570.0820.873 300.3960.1740.2730.3390.2650.0910.872 1000.9100.4070.1770.2790.3470.2650.0850.875 200.4060.1820.2810.3470.2730.0940.874 300.4050.1800.2790.3470.2690.0910.875 Rouge-Avg500.9100.4000.1800.2760.3430.2710.0960.873 200.3490.1470.2410.2980.2370.0780.858 300.3930.1730.2720.3360.2670.0920.870 1000.9100.3960.1740.2740.3390.2650.0880.872 200.4060.1790.2800.3470.2720.0920.874 300.4000.1780.2790.3440.2660.0870.874 Meteor500.9100.4040.1770.2740.3430.2860.1020.872 200.4060.1800.2760.3430.2920.1070.871 300.4010.1720.2710.3370.2880.0990.870 1000.9100.4050.1780.2760.3430.2940.1070.870 200.4060.1760.2760.3430.2910.1060.872 300.4090.1840.2800.3480.2910.1080.873 Supervised + NLPO Rouge-1500.9100.4250.1960.2990.3660.2850.1060.882 200.4170.1910.2950.3600.2760.1000.881 300.4180.1920.2960.3610.2780.1010.881 1000.9100.4240.1960.2990.3660.2860.1060.882 200.4230.1960.2990.3650.2890.1100.881 300.4200.1930.2960.3620.2790.1020.881 Rouge-Avg500.9100.4260.1970.2980.3670.2940.1150.881 200.4250.1960.2980.3660.2920.1120.881 300.4240.1940.2970.3650.2870.1070.881 1000.9100.4240.1960.2980.3650.2910.1130.881 200.4280.1980.3000.3680.2960.1150.882 300.4290.1990.3000.3690.2960.1160.882 Meteor500.9100.4300.1970.2940.3640.3200.1300.879 200.4320.1980.2970.3670.3180.1300.880 300.4230.1910.2930.3610.2970.1160.879 1000.9100.4350.2000.2980.3690.3200.1310.881 200.4330.1980.2970.3680.3190.1300.879 300.4340.2000.2970.3690.3240.1320.879 Table 19:NLPO Ablation/Model Selection: Evaluation of NLPO models on validation set with different reward functions, top k values for rollouts and target update iterations. For each alg-reward combo, best model is chosen 37 Published as a conference paper at ICLR 2023 AlgorithmUnique N CoherenceQuality ValueAlphaSkewValueAlphaSkew PPO+Supervised224.210.1984.2243.970.2563.98 NLPO+Supervised194.30.264.3083.980.0894 Zero Shot173.730.13.7573.690.253.722 Supervised194.250.1164.2413.990.23.986 NLPO174.030.134.0423.830.1913.832 PPO213.940.1113.9453.760.1293.767 Human193.890.2773.9023.770.0293.769 Table 20: Results of the human subject study showing the number of participants N, average Likert scale value for coherence and sentiment, Krippendorf’s alpha showing inter-annotator agreement, and Skew. For each model a total of 50 samples were drawn randomly from the test set and rated by 3 annotators each, each resulting in 150 data points per algorithm. CoherenceQuality Group 1Group 2Diff (G2-G1)p-valuesDiff (G2-G1)p-values HumanNLPO0.1470.7550.0600.900 HumanNLPO+Supervised0.4130.0010.2130.047 HumanPPO0.0530.900-0.0070.900 HumanPPO+Supervised0.3270.0240.2000.544 HumanSupervised0.3600.0080.2200.043 HumanZero Shot-0.1600.679-0.0800.900 NLPONLPO+Supervised0.2670.0120.1530.008 NLPOPPO-0.0930.900-0.0670.900 NLPOPPO+Supervised0.1800.5640.1400.860 NLPOSupervised0.2130.3610.1600.754 NLPOZero Shot-0.3070.044-0.1400.860 NLPO+SupervisedPPO-0.3600.008-0.2200.043 NLPO+SupervisedPPO+Supervised-0.0870.009-0.0130.009 NLPO+SupervisedSupervised-0.0530.0090.0070.900 NLPO+SupervisedZero Shot-0.5730.001-0.2930.012 PPOPPO+Supervised0.2730.1060.2070.508 PPOSupervised0.3070.0440.2270.394 PPOZero Shot-0.2130.361-0.0730.900 PPO+SupervisedSupervised0.0330.9000.0200.900 PPO+SupervisedZero Shot-0.4870.001-0.2800.155 SupervisedZero Shot-0.5200.001-0.3000.101 Table 21: Results of an post-hoc Tukey HSD Test for difference in means between pairs of algorithms (Group 2 - Group 1) and correspondingp-values. Individually statistically significant results are bolded and are used to discuss results in the analysis. Overallp-values showing that there is a significant difference in means between the models via a one-way ANOVA test are significant with p0.05for both coherence and sentiment. 38 Published as a conference paper at ICLR 2023 Figure 8: Instructions and interface for the summarization task. B.5.3HUMANPARTICIPANTSTUDY Figure 8 shows the summarization instructions and interface used for the human evaluation exper- iments. Participants weren’t required to read the entire article, but to encourage some reading, a minimum time on the window of 15s was enforced via hiding the sliders. Tables 20, 21 show averaged results, annotator agreement, and the results of statistical significance tests to determine which models output better generations when rated by humans. B.5.4QUALITATIVEANALYSIS We show sample generations from each of the algorithms for three randomly picked prompts below. Sample1 Prompt: Manchester City are confident UEFAâ ̆ A ́ Zs punishment for breaching financial fairplay regulations will be lifted this summer which would allow them to bid for stellar names like Raheem Sterling, Gareth Bale, Kevin de Bruyne and Ross Barkley. City boss Manuel Pellegrini has been hampered over the past year by UEFA restricting them to a net transfer spend of 49million in each window and keeping the clubâ ̆ A ́ Zs overall wage bill to its current level of 205million-a-year. UEFAâ ̆ A ́ Zs settlement with City published in May stated those penalties would remain in place until the end of the 2015/16 season but the clubâ ̆ A ́ Zs latest financial figures showed drastically-reduced losses of 23million which they feel proves they are now compliant with FPP regulations. Manuel Pellegrini is hoping that the financial restrictions imposed by UEFA for a breach of FFP rules will be lifted at the end of this season . Manchester City have been limited to a net spend of 49 million in the last two transfer windows - they spent 25m bringing Wilfried Bony in from Swansea in January . Ahead of Monday nightâ ̆ A ́ Zs trip to Crystal Palace, Pellegrini was certainly talking like a man excited at the prospect of signing ’crack’ players this summer. â ̆ AŸI think that next season we donâ ̆ A ́ Zt have any restrictions so we will be in the same position that all the other English clubs have,â ̆ A ́ Z said Pellegrini. â ̆ AŸItâ ̆ A ́ Zs important. You have so many strong teams here in England and in Champions League, you can not allow them to keep the advantage every year; having less players to put in your squad or spending less money. We spend money, of course we always spend money, but they spent more.â ̆ A ́ Z Manchester United, Barcelona, Liverpool and Arsenal have all paid more in transfer fees in the past 12 months than City who were traditionally Europeâ ̆ A ́ Zs biggest 39 Published as a conference paper at ICLR 2023 spenders after the club was taken over by Abu Dhabi owners in 2008. Uefa also ordered City to play with a reduced squad from 25 players to 21 in the Champions League this season and while that restriction has now ended, any time reduction in the penalties on spending and wages is more controversial. Arsenal have paid more in transfer fees than City in the last 12 months, including 30m on Alexis Sanchez . The document published last May by UEFAâ ̆ A ́ Zs Club Financial Control Body investigative chamber explicitly said Cityâ ̆ A ́ Zs financial penalties would run for two seasons at least and there has been no official deviation from that decision. The published statement said at the time: â ̆ AŸManchester City agrees to significantly limit spending in the transfer market for the seasons 2014/15 and 2015/16. It means City will have to argue their case with Uefa that as they have been financially compliant over the past year, they deserve to be free of restrictions moving forward. They have successfully argued their case with UEFA before. Last summer they persuaded the governing body to allow them to bypass the normal quota of eight homegrown players as their Champions League squad had been reduced. Eliaquim Mangala joined the champions from Porto for 32m last summer . The reigning Premier League champions have only paid money for Fernando, Willie Caballero, Eliaquim Mangala and Wilfried Bony in the last two transfer windows and that was part-paid by the sales of Javi Garcia and Jack Rodwell. Pellegrini admitted they werenâ ̆ A ́ Zt in a position to compete with neighbours United for established world stars like Angel di Maria because of the FFP penalties. It has cost City on the pitch. Even if they win their remaining eight Premier League fixtures their points tally will be lower than last season and they crashed out once again at the last-16 stage of the Champions League. Pellegrini and director of football Txiki Begiristain both accept the squad needs replenishing and they want to be in the market for the top available players. Manchester City have been heavily linked with a move for former Chelsea midfielder Kevin de Bruyne . The club also want to be involved in any race for Liverpool contract rebel Raheem Sterling . They have been strongly linked with former Chelsea winger Kevin de Bruyne who has been one of the best players in the Bundesliga with Wolfsburg. And if Liverpool do allow Sterling to leave, City want to be involved in the chase for one of Englandâ ̆ A ́ Zs most outstanding talents to help fit in with quotas for homegrown players. â ̆ AŸIf you are the manager or a player of a big tem, you are always open to criticism because everyone expects a high level of performance,â ̆ A ́ Z he said. â ̆ AŸThe major issue with our team always seems to be the money but this season I think we were the team that spent less than any other team. That is the weird thing. â ̆ AŸFor this club trying to be a big team in so few years, maybe we are paying the cost for that.â ̆ A ́ Z Since the transfer penalties were introduced, City have spent 91million on players in the last two windows ( Net spend 68million). That compares to Manchester United 145.5million, Liverpool 113 million, Arsenal 92.5million and Chelsea 82.6million. Over the same time period Barcelona spent 118.3million on players and Real Madrid 81.2million though they also broke the world transfer record for Gareth Bale the previous summer. Zero-Shot: manuel Pellegrini hoping UEFAâ ̆ A ́ Zs punishment for breaching financial fairplay regulations will be lifted this summer. city have been restricted to a net transfer spend of 49million in each window. a total of 23million has been lost in the last year. References: Manchester City have been restricted to a net transfer spend of 49m .Club’s also had to keep overall wage bill to its current level of 205m .Punishments imposed by UEFA for breaching Financial Fair Play rules .The spending restrictions were set for this season and the next one .But City are confident they will be lifted early after their compliance . PPO: Manuel Pellegrini is hoping UEFAâ ̆ A ́ Zs punishment for breaching financial fairplay rules will be lifted this summer. Manchester City have been limited to a net spend of 49million in each transfer window. the club also keep the club’s overall wage bill to its current level of 205million-a-year. NLPO: Manchester City are confident UEFAâ ̆ A ́ Zs punishment for breaching financial fairplay regulations will be lifted this summer. that would allow them to bid for stellar names like Raheem Sterling, Kevin de Bruyne and Ross Barkley. Supervised: Manchester City have been limited to a net transfer spend of 49million in each window. City’s overall wage bill is 205m-a-year due to a breach of UEFA rules. City are confident the financial restrictions will be lifted at the end of the current season in order to bid for stars such as Raheem Sterling. Manuel Pellegrini is hoping that the financial restrictions will be lifted at the end of the 2015/16 season Supervised+PPO: Manchester City have been restricted to a net spend of 49million in each window and kept their overall wage bill at 205m. Manuel Pellegrini believes these restrictions will be lifted at the end of the season. Arsenal have spent more in transfer fees than City in the last 12 months, including 30m on Alexis Sanchez. Supervised+NLPO: Manchester City are confident UEFAâ ̆ A ́ Zs punishment for a breach of financial fairplay regulations will be lifted this summer. The club have been restricted to a net transfer spend of 49million over the past two window. They aim to bid for stars like Raheem Sterling, Gareth Bale and Kevin de Bruyne. City lost 23m on transfers in the last 12 months due to financial restrictions imposed by UEFA for breach Sample2 Prompt: (CNN)Larry Johnson remembers the fear and feeling of helplessness from being on the SkyWest Airlines flight that made an emergency landing in Buffalo, New York. "I thought we were done," he said Thursday, one day later. "There was no doubt in my mind that we weren’t going to land." Johnson was flying with his brother, his girlfriend and his 8- month-old son when he says a flight attendant came over the speaker asking for someone who was medically trained to help with a sick passenger. Minutes later, Johnson says, the attendant announced there was a pressurization problem and told passengers to prepare for the emergency landing. "You’re going to feel dizzy and woozy and lightheaded, and you’ 40 Published as a conference paper at ICLR 2023 re not going to be able to breathe," Johnson recalled the flight attendant saying. The Federal Aviation Administration on Wednesday initially reported a pressurization problem with SkyWest Flight 5622, and said it would investigate. It later issued a statement that did not reference any pressurization issues. "It was like being trapped and you couldn’t do anything. You have no control over it. Your body just felt pressured," Johnson said. Johnson said his girlfriend was seeing spots and that "when I was blinking, it was really bright. I could barely see." According to Marissa Snow, spokeswoman for SkyWest, three passengers reported a loss of consciousness while on the flight. Fifteen passengers were examined by medical personnel at the Buffalo airport, but no one was transported to the hospital, she said. The spokeswoman said that maintenance personnel found no indication of a pressurization problem with the aircraft, an Embraer E170, and that the airline continues to investigate the cause. An official with the National Transportation Safety Board told CNN that the agency is in communication with the FAA and SkyWest to gather information on the incident to better understand what took place. Mary Cunningham was the nurse on board who attended to the first sick passenger. "There was a woman, very lethargic, very nauseous, turning gray, just not looking good," Cunningham said. After administering oxygen to the woman, Cunningham, a nurse at Yale-New Haven Hospital in Connecticut, returned to her seat. Soon after, Cunningham said, she was called up again by the flight attendant after another women near the first started to lose consciousness. "It was something that was affecting oxygen in that area," which was near the middle of the plane, she said. Cunningham said she had to return to her seat after feeling lightheaded. There were 84 passengers on board the plane, including nine crew members. Flight 5622 was originally scheduled to fly from Chicago to Hartford, Connecticut. The plane descended 28,000 feet in three minutes. "It would feel like a roller coaster -- when you’re coming over the top and you’re going down," CNN aviation analyst Mary Schiavo said, describing how such a descent would feel. "You know that these pilots knew they were in a very grave and very serious situation." Johnson, flying to visit his parents with their only grandchild, was glad that roller coaster ride ended safely. "It was a shaky ride. It wasn’t anything I would like to experience again," he said. CNN’s Jean Casarez, Josh Gaynor, Dana Ford and Stephanie Gallman contributed to this report. Zero-Shot: "you’re going to feel dizzy and woozy and lightheaded," Johnson says. "it was like being trapped and you couldn’t do anything," Johnson says. "the plane was trying to get back on the ground," passenger says of fear. References: Three passengers report a loss of consciousness on SkyWest flight .But officials say there is no evidence of a pressurization problem . PPO: SkyWest flight 5312 made emergency landing in Buffalo, new york. passengers were told they were not going to be able to breathe, passenger recalls attendant. the faa initially reported a pressurization problem with the flight 5622. NLPO: SkyWest flight 5622 made emergency landing in buffalo, new york.Larry Johnson says flight attendant announced there was a pressurization problem. three passengers reported loss of consciousness while on flight; no one transported to hospital. Supervised: The FAA says it is investigating the cause of the emergency landing. SkyWest Flight 5622 made an emergency landing in Buffalo, New York. The flight was carrying a family of three people. The flight attendant told the passengers to prepare for the landing. Supervised+PPO: "I thought we were done," the father of one passenger says. The flight attendant announced there was a pressurization problem. The flight made an emergency landing in Buffalo, New York. Three passengers reported a loss of consciousness while on the flight. Supervised+NLPO: SkyWest Airlines flight made an emergency landing in Buffalo, New York, on Thursday. "There was no doubt in my mind that we weren’t going to land," passenger says. "You’re going to feel dizzy and woozy and lightheaded...you’re not going to be able to breathe," he says." Sample3 Prompt: For Inverness the latest chapter in their remarkable story featured not one, but two, plot-changing twists. One featured penalty Celtic didn’t get for 2-0 in which Josh Meekings handball should have also led to a sending off. The other the spot kick they did, followed by a red card for Craig Gordon. ’I’ve not seen it yet, but going by the reaction of the Celtic players we got away with a penalty and a sending off and that was probably the turning point in the game,’ acknowledged Caley manager John Hughes after. Inverness’s Josh Meekings appears to get away with a handball on the line in their win over Celtic . Caley boss John Hughes says the break, which could have meant a penalty and red card, was a turning point . ’I’ve not spoken to Josh. I haven’t seen it - but going by the media it was definitely a hand ball. We look at the referee behind the line and all that and I know Ronny will feel aggrieved - because I certainly would. ’But it’s part and parcel of football and you need a wee bit of luck to beat Celtic. ’This was their biggest game of the season because they will go on and win the league and if they had beaten us today there was a good chance they would have gone on and won the Scottish Cup. ’But when Marley Watkins was clipped by Craig Gordon and they were down to 10 men that was advantage Inverness. ’We weren’t going to give Celtic the ball back, they had to come and get it and we had to be patient. ’When big Edward put us into the lead we thought it was going to be our day on the back of things that had happened. ’Celtic equalised with another free kick but it’s typical of Inverness that we don’t do anything easy. ’We do it the hard way and we came up with the winner through David Raven.’ Hughes hauled Raven, his Scouse defender, from his backside as extra-time beckoned. Offended by the sight of one of his players resting he had a message to impart. Caley players celebrate after upsetting Celtic in a Scottish Cup semi-final 3-2 thriller . Celtic, depleted by games and absentees, were virtually on their knees after a relentless programme of midweek games. In last season’s League Cup Final Inverness had been passive and unambitious prior 41 Published as a conference paper at ICLR 2023 to losing on penalties. This was no time to repeat the mistake. ’I tried to emphasise to the players they would never have a better time to go on and beat Celtic, down to 10 men in the semi final of a cup. We needed to go for it,’ Hughes said. ’Before Raven scored at the back post I was looking to change it. I was going to bring on another winger, Aaron Doran, and put him in the full-back position over on the right, but more advanced so he could take their left back on. Thankfully I didn’t do that and David Raven came up with the goal. Virgil Van Dijk (centre) fired Celtic into an early lead with a superb free-kick in the 18th minute . ’I didn’t realise this is the first time the club have been in the final of the Scottish Cup and that’s a remarkable achievement given it was only formed 20 years ago. ’It is a great story isn’t it? It’s an absolutely fantastic story. It is 20 odd years since the amalgamation. We are a small provincial club up there in the Highlands. ’We have lost a real inspirational skipper in Richie Foran right from the start of the season. He has never played. We have had to adjust to that. ’We had to sell Billy McKay, our top goalscorer, at Christmas. We have had to go again and adjust. I am a very humble guy and I am grateful and thankful that injuries have never caught up with us.’ There is remarkable irony in the fact Falkirk will be the opponents for the final. A former Bairns captain, he was manager of the club in 2009 when they lost to Rangers at Hampden. Former Falkirk captain and manager John Hughes will take on his former club in the final . ’I had a lot of great times at Falkirk. So much so that it is possibly my favourite time in my playing career. I am still friendly with an awful lot of the characters who were in that dressing room. Neil Oliver is a good friend of mine from my Falkirk days. He comes along and sits on the bench and gives me a wee hand out. ’That is the spirit that we had at that club. I have met some great guys. Brian Rice, my ex-coach, Davie Weir, Ian McCall, the list is endless. I was just talking the other day about that day at Falkirk. There are times even now when I see. ’I have a real special, special feel for the place. I am not kidded Falkirk is a right good side, trust me on that. No side goes to Tynecastle and beats Hearts without being a good side. No side beats Hibs the way they were playing without being a good side. You might say we are favourites I say it is even stevens and I am just looking forward to when the days comes.’ David Raven (second right) celebrates with his Inverness team-mates following his extra-time winner . An Inverness and Falkirk final would not have been the choice of sponsors William Hill, you suspect. Vast swathes of Hampden’s west stand were empty, a 12.15 kick-off for the benefit of broadcasters proving unhelpful . There will be no ticket rows this time. Not over a final unlikely to raise a crowd much higher than 35,000 in a stadium of 52,000. The contract with Hampden plc is signed, with no question of moving it. If the bitter amalgamation of 21 years ago produced a fine football club it also disenfranchised many of those who followed the old ones. ’I hope this encourages the next generation to follow Inverness,’ added Hughes. ’In the peer groups it is still Celtic and Rangers they follow but I hope the next ones will want to support their local club. ’If we can add another few hundred or thousand to the support then we are playing our part. Zero-Shot: inverness beat Celtic 2-0 in their opening game of the season. Josh Meekings was the subject of a handball after the break. the break could have meant a penalty and a red card for Craig Gordon. References: Josh Meekings handball should have been a penalty and a red card .If the penalty was awarded and converted Celtic would have gone to 2-0 .Caley manager John Hughes admitted his side were fortunate in win .Virgil van Dijk scored Celtic’s opener with a superb free-kick .Celtic keeper Craig Gordon was sent off early in the second half .Greg Tansey, Edward Ofere and finally Daven Raven scored for Caley . PPO: Josh Meekings handball could have meant sending off for Celtic in extra-time win. John Hughes says the break could have meant an penalty and red card. Caley boss also admits he would feel aggrieved had Celtic won the championship. NLPO: Caley beat Celtic 2-0 in their Scottish Cup clash. Josh Meekings handball could have led to a penalty and red card. John Hughes says the break was probably the turning point in the game. defender does not believe defensive striker can get away with being shown the yellow card - despite playing in same half we never scored. Supervised: Inverness beat Celtic 2-0 in their Scottish Cup clash. Josh Meekings appeared to get away with a handball on the line. Another player was sent off for a spot kick which ended in a red card. Caley manager John Hughes says the break could have meant a penalty. Hughes says he ’would feel aggrieved’ after the break. Supervised+PPO: Caley beat Celtic 1-0 in their Scottish Cup encounter at the Etihad. Josh Meekings’ handball on the line should have also seen a sending off. But Craig Gordon got away with a spot kick and sending off. John Hughes said it was a turning point in the game for Inverness. Supervised+NLPO: Josh Meekings appears to get away with a handball after Inverness’s win. Inverness were down to 10 men when Craig Gordon was sent off. John Hughes says the break could have meant a penalty and red card. Hughes says Celtic will go on and win the Scottish Cup with win 42 Published as a conference paper at ICLR 2023 B.6TOTTO B.6.1SETUP ToTTo (Parikh et al., 2020) is a controlled table-to-text generation task in which the goal is to produce one-sentence description of highlighted table cells. For training RL methods, we consider 5 different reward functions: BLEU, SacreBLEU, METEOR, PARENT and a combination of Meteor and PARENT. We chose T5 as our base LM here too, as they are more suitable for structure to text tasks. For decoding, we use beam search during inference and for generating rollouts, we use top k sampling. Other implementation details are captured in Table 22. Model Paramsvalue supervisedbatch size:8 epochs:4 learning rate:0.0001 learning rate scheduler: constant with warm up weight decay:0.1 ppo/nlposteps per update:2560 total number of steps:256000 batch size:64 epochs per update:5 learning rate:0.000002 entropy coefficient:0.0 initial kl coeff:0.001 target kl:2.0 discount factor:0.99 gae lambda:0.95 clip ratio:0.2 rollouts top k :0 value function coeff:0.5 top mask ratio:0.9 target update iterations:20 supervised+ ppo (or nlpo)steps per update:2560 total number of steps:256000 batch size:64 epochs per update:5 learning rate:0.0000005 entropy coefficient:0.0 initial kl coeff:0.01 target kl:0.2 discount factor:0.99 gae lambda:0.95 clip ratio:0.2 rollouts top k :50 value function coeff:0.5 top mask ratio:0.9 target update iterations:20 decodingnum beams:5 min length:10 max new tokens:50 tokenizerpadding side: left truncation side: right max length: 512 Table 22:ToTTO Hyperparams: Table shows a list of all hyper-parameters and their settings B.6.2RESULTS ANDDISCUSSION Tables 24, 23 presents our benchmarking results with 5 reward functions along with supervised baseline performances on dev and test sets respectively. Similar to other tasks, our main finding is that warm-started initial policies are crucial for learning to generate descriptions from highlighted cells. Without warm-start, policies suffer from reward hacking and resulting in sub-optimal solutions despite application of task-specific metrics such as PARENT etc. We find that Supervised+NLPO method outperforms all models on ToTTo leaderboard in terms of PARENT metric. 43 Published as a conference paper at ICLR 2023 Tasks_Lexical and Semantic MetricsFactual Consistency AlgLMReward functionSacreBleuBLEURTPARENT OverallOverlapNon-OverlapOverallOverlapNon-OverlapOverallOverlapNon-Overlap ToTTo Zero-ShotT50.0360.0400.032-1.392-1.387-1.3970.1160.1190.112 PPOT5bleu0.0650.0670.063-1.074-1.045-1.0980.2460.2460.244 T5sacrebleu0.0860.0900.083-0.979-0.955-1.0030.2930.2920.294 T5meteor0.1440.1550.132-0.769-0.713-0.8260.3560.3610.351 T5parent0.1460.1530.128-0.721-0.688-0.7530.3360.3350.339 T5meteor + parent0.1610.1690.152-0.891-0.861-0.9220.3450.3420.348 NLPOT5bleu0.0620.0650.059-1.077-1.057-1.0970.2350.2360.233 T5sacrebleu0.0850.0880.083-0.945-0.917-0.9720.3140.3150.313 T5meteor0.1020.1080.097-1.044-1.009-1.0790.3290.3280.330 T5parent0.1590.1660.152-0.710-0.675-0.7450.3570.3510.363 T5meteor + parent0.1660.1750.158-0.704-0.668-0.7400.3650.3620.368 SupervisedT50.4570.5350.3770.2040.3270.0810.5830.6310.534 Supervised + PPOT5bleu0.4730.5480.3950.2000.3230.0780.5900.6380.542 T5sacrebleu0.4740.5570.3890.2090.3400.0770.5730.6200.525 T5meteor0.4680.5410.3920.2030.3250.0820.5900.6380.542 T5parent0.4690.5470.3880.1750.3000.0500.5950.6410.549 T5meteor + parent0.4730.5470.3920.1920.3140.0690.5950.6420.549 Supervised + NLPOT5bleu0.4750.5480.3990.2080.3300.0850.5930.6390.546 T5sacrebleu0.4750.5570.3920.2080.3350.0810.5770.6250.529 T5meteor0.4680.5410.3920.2010.3220.0790.5940.6410.546 T5parent0.4740.5500.3920.1920.3150.0680.5960.6430.550 T5meteor + parent0.4710.5460.3930.2040.3260.0810.5920.6400.544 Table 23:ToTTo test evaluation: Table shows lexical, semantic and factual correctness metric scores of algorithms with different reward functions on hold-out test set. Without supervised pre-training, both PPO and NLPO results in sub-optimal solutions, with NLPO better than PPO. With supervised pre-training, PPO and NLPO achieve better scores across all metrics showing RL fine-tuning is beneficial. Most importantly, RL fine-tuned models produce more factually consistent text as seen in higher PARENT scores. Another observation, fine-tuning with a task-specific metric PARENT is better than training on task-agnostic lexical rewards 44 Published as a conference paper at ICLR 2023 Tasks _ Lexical and Semantic Metrics Factual Consistency Diversity Metrics Alg LM Reward function Rouge-1 Rouge-2 Rouge-L Rouge-LSum Meteor BertScore SacreBleu PARENT Overall Overlap Non-Overlap Overall Overlap Non-Overlap MSTTR Distinct 1 Distinct 2 H 1 H 2 Unique 1 Unique 2 Mean Output Length ToTTo Zero-Shot T5 0.131 0.055 0.127 0.127 0.057 0.805 0.038 0.042 0.034 0.118 0.119 0.116 0.428 0.084 0.238 6.703 9.933 8387 26490 19.964 Supervised T5 0.410 0.279 0.388 0.388 0.223 0.953 0.458 0.533 0.387 0.586 0.633 0.540 0.715 0.162 0.511 9.995 14.468 15168 54706 17.791 PPO T5 bleu 0.274 0.138 0.249 0.249 0.139 0.844 0.068 0.071 0.066 0.251 0.250 0.251 0.403 0.091 0.308 10.659 14.511 7536 34232 28.545 T5 sacrebleu 0.341 0.166 0.300 0.300 0.165 0.858 0.09 0.094 0.086 0.300 0.299 0.300 0.469 0.121 0.407 11.071 14.880 10138 48195 26.612 T5 meteor 0.322 0.157 0.286 0.286 0.173 0.888 0.147 0.163 0.133 0.358 0.367 0.350 0.625 0.136 0.482 10.189 14.910 12346 54925 21.484 T5 parent 0.268 0.125 0.251 0.251 0.119 0.890 0.150 0.158 0.143 0.337 0.332 0.342 0.764 0.202 0.646 11.068 14.988 13068 50313 13.035 T5 meteor + parent 0.266 0.128 0.251 0.251 0.130 0.886 0.165 0.175 0.155 0.348 0.346 0.350 0.702 0.181 0.594 10.096 14.432 14422 55770 15.354 NLPO T5 bleu 0.267 0.134 0.24 0.24 0.137 0.84 0.068 0.071 0.065 0.238 0.239 0.237 0.448 0.1 0.359 11.259 14.623 9029 47209 28.472 T5 sacrebleu 0.341 0.168 0.297 0.297 0.183 0.863 0.089 0.093 0.085 0.32 0.324 0.317 0.494 0.111 0.373 11.007 15.032 9455 43379 27.977 T5 meteor 0.322 0.157 0.286 0.286 0.173 0.888 0.147 0.163 0.133 0.358 0.367 0.350 0.625 0.136 0.482 10.189 14.910 12346 54925 21.484 T5 parent 0.283 0.132 0.264 0.264 0.133 0.894 0.163 0.174 0.153 0.36 0.357 0.364 0.824 0.223 0.691 11.493 15.127 14344 55542 14.204 T5 meteor + parent 0.299 0.14 0.276 0.276 0.142 0.896 0.171 0.181 0.161 0.369 0.365 0.372 0.779 0.214 0.674 11.072 15.275 14939 58737 15.141 Supervised + PPO T5 bleu 0.408 0.283 0.388 0.388 0.222 0.954 0.477 0.549 0.405 0.596 0.644 0.550 0.722 0.167 0.525 10.080 14.524 15203 54724 17.296 T5 sacrebleu 0.395 0.275 0.378 0.378 0.211 0.955 0.477 0.554 0.401 0.577 0.621 0.535 0.728 0.174 0.539 10.086 14.518 14846 52327 16.063 T5 meteor 0.410 0.282 0.389 0.389 0.223 0.954 0.469 0.540 0.398 0.593 0.642 0.547 0.718 0.165 0.516 10.037 14.467 15182.0 54446 17.542 T5 parent 0.401 0.277 0.382 0.382 0.215 0.953 0.470 0.543 0.394 0.598 0.647 0.550 0.732 0.174 0.545 10.209 14.660 15379.0 55421 16.826 T5 meteor + parent 0.406 0.281 0.386 0.387 0.220 0.954 0.473 0.544 0.399 0.600 0.648 0.553 0.727 0.170 0.532 10.143 14.586 15330 55211 17.185 Supervised + NLPO T5 bleu 0.410 0.283 0.388 0.388 0.222 0.954 0.476 0.548 0.404 0.597 0.644 0.552 0.721 0.167 0.524 10.077 14.532 15213 54948 17.408 T5 sacrebleu 0.397 0.276 0.38 0.38 0.214 0.955 0.477 0.555 0.401 0.581 0.628 0.535 0.729 0.174 0.54 10.124 14.544 14940 52986 16.334 T5 meteor 0.411 0.283 0.389 0.39 0.224 0.954 0.474 0.547 0.403 0.6 0.649 0.554 0.727 0.171 0.536 10.156 14.612 15341 55292 17.637 T5 parent 0.405 0.28 0.386 0.386 0.219 0.954 0.469 0.541 0.398 0.598 0.645 0.552 0.716 0.165 0.519 10.019 14.5 15218 54793 17.095 T5 meteor + parent 0.405 0.28 0.386 0.386 0.219 0.954 0.474 0.547 0.398 0.598 0.646 0.552 0.727 0.171 0.536 10.156 14.612 15341 55292 17.095 Table 24: ToTTo dev evaluation : Table shows lexical, semantic and factual correctness metric scores of algorithms with different reward functions on dev set. Without supervised pre-training, both PPO and NLPO results in sub-optimal solutions, with NLPO better than PPO. With supervised pre-training, PPO and NLPO achieve better scores across all metrics showing RL fine-tuning is beneficial. Most importantly, RL fine-tuned models produce more factually correct text as seen inhigher PARENT scores. Another observation, fine-tuning with a task-specific metric PARENT is better than training just on task-agnostic lexical metrics 45 Published as a conference paper at ICLR 2023 AlgorithmUnique N CoherenceCorrectness ValueAlphaSkewValueAlphaSkew Zero Shot251.630.7181.6421.930.5031.946 PPO+Supervised244.570.2214.5794.480.0984.483 PPO262.750.4272.7533.230.2143.227 NLPO282.250.4012.2472.610.4192.613 Supervised244.590.1734.5924.540.1894.537 NLPO+Supervised264.580.2444.6014.570.1444.581 Table 25: Results of the human subject study showing the number of participants N, average Likert scale value for coherence and sentiment, Krippendorf’s alpha showing inter-annotator agreement, and Skew. For each model a total of 50 samples were drawn randomly from the test set and rated by 3 annotators each, resulting in 150 data points per algorithm. Group 1Group 2 CoherenceCorrectness Diff (G2-G1)p-valuesDiff (G2-G1)p-values PPONLPO-0.5070.001-0.6130.001 PPONLPO+Supervised1.8270.0011.3400.001 PPOSupervised1.8330.0011.3130.001 PPOPPO+Supervised1.8130.0011.2530.001 PPOZero Shot-1.1200.001-1.2930.001 NLPONLPO+Supervised2.3330.0011.9530.001 NLPOSupervised2.3400.0011.9270.001 NLPOPPO+Supervised2.3200.0011.8670.001 NLPOZero Shot-0.6130.001-0.6800.001 NLPO+SupervisedSupervised0.0070.9-0.0270.009 NLPO+SupervisedPPO+Supervised-0.0130.009-0.0870.009 NLPO+SupervisedZero Shot-2.9470.001-2.6330.001 SupervisedPPO+Supervised-0.0200.009-0.0600.009 SupervisedZero Shot-2.9530.001-2.6070.001 PPO+SupervisedZero Shot-2.9330.001-2.5470.001 Table 26: Results of an post-hoc Tukey HSD Test for difference in means between pairs of algorithms (Group 2 - Group 1) and correspondingp-values. Individually statistically significant results are bolded and are used to discuss results in the analysis. Overallp-values showing that there is a significant difference in means between the models via a one-way ANOVA test are significant with p0.05for both coherence and sentiment. 46 Published as a conference paper at ICLR 2023 Figure 9: Instructions, two examples, and interface for the ToTTo table description task. B.6.3HUMANPARTICIPANTSTUDY Figure 9 shows the ToTTo instructions, example, and interface used for the human evaluation experiments. We made small modifications to the original code release’s HTML renderer to make the tables display in our HITs. Tables 25, 26 show averaged results, annotator agreement, and the results of statistical significance tests to determine which models output better generations when rated by humans. 47 Published as a conference paper at ICLR 2023 B.6.4QUALITATIVEANALYSIS We show sample generations from each of the algorithms for three randomly picked prompts below. Sample1 Prompt: <table> <cell> August 1, 1992 <col_header> Date </col_header> <col_header> North America </col_header> <col_header> Australasia </col_header> <col_header> Asia </ col_header> <col_header> North America </col_header> </cell> <cell> Allentown <col_header> City </col_header> <col_header> North America </col_header> <col_header> Australasia </ col_header> <col_header> Asia </col_header> <col_header> North America </col_header> </ cell> <cell> Allentown Fairgrounds Grandstand <col_header> Country </col_header> < col_header> North America </col_header> <col_header> Australasia </col_header> < col_header> Asia </col_header> <col_header> North America </col_header> </cell> </table>< page_title> Under My Spell Tour </page_title> <section_title> Tour dates </section_title> <table> <cell> August 1, 1992 <col_header> Date </col_header> <col_header> North America </col_header> <col_header> Australasia </col_header> <col_header> Asia </col_header> < col_header> North America </col_header> </cell> <cell> Allentown <col_header> City </ col_header> <col_header> North America </col_header> <col_header> Australasia </ col_header> <col_header> Asia </col_header> <col_header> North America </col_header> </ cell> <cell> Allentown Fairgrounds Grandstand <col_header> Country </col_header> < col_header> North America </col_header> <col_header> Australasia </col_header> < col_header> Asia </col_header> <col_header> North America </col_header> </cell> </table> Zero-Shot: table> cell> Under My Spell Tour /section_title> Tour dates /cell> cell> August 1, 1992 col_header> Date /col_header> PPO: Under My Spell Tour, August 1, 1992. August 1, 1992. NLPO: Under My Spell Tour dates Under My Spell Tour dates Allentown Fairgrounds Grandstand August 1, 1992. Under My Spell Tour dates Under My Spell Tour dates Allentown Fairgrounds Grandstand August 1, 1992 table> Supervised: The Under My Spell Tour began on August 1, 1992 at the Allentown Fairgrounds Grandstand in Allentown. Supervised+PPO: On August 1, 1992, Under My Spell Tour was held at the Allentown Fairgrounds Grandstand in Allentown. Supervised+NLPO: On August 1, 1992, Under My Spell Tour was held at the Allentown Fairgrounds Grandstand in Allentown. Sample2 Prompt: <table> <cell> Philip IV the Fair, the Iron King (Philippe) <col_header> Name </ col_header> </cell> <cell> 5 October 1285 <col_header> King from </col_header> </cell> < cell> King of France and of Navarre (Roi de France et de Navarre) <col_header> Title </ col_header> </cell> </table><page_title> List of French monarchs </page_title> < section_title> House of Capet (987â ̆ A ̧S1328) </section_title> <table> <cell> Philip IV the Fair, the Iron King (Philippe) <col_header> Name </col_header> </cell> <cell> 5 October 1285 <col_header> King from </col_header> </cell> <cell> King of France and of Navarre ( Roi de France et de Navarre) <col_header> Title </col_header> </cell> </table> Zero-Shot: > </table><page_title> List of French monarchs < PPO: Philip IV the Fair, the Iron King (Philippe) 5 October 1285. Title: King of France and of Navarre (Roi de France et de Navarre NLPO: King (Philippe) Philip IV the Fair, the Iron King (Philippe) 5 October 1285 Supervised: Philip IV the Fair, the Iron King (Philippe) was King of France and Navarre from 1285. Supervised+PPO: Philip IV the Fair, the Iron King was King of France and Navarre from 1285. Supervised+NLPO: Philip IV the Fair, the Iron King (Philippe) was King of France and Navarre (Roi de France et de Navarre) in 1285. Sample3 Prompt: <table> <cell> Benjamin A. Smith I (D) <col_header> Vacator </col_header> </cell> < cell> Successor elected November 6, 1962. <col_header> Reason for change </col_header> </ cell> <cell> Ted Kennedy (D) <col_header> Successor </col_header> </cell> <cell> November 7, 1962 <col_header> Date of successor’s formal installation </col_header> </cell> </ table><page_title> 87th United States Congress </page_title> <section_title> Senate </ section_title> <table> <cell> Benjamin A. Smith I (D) <col_header> Vacator </col_header> </cell> <cell> Successor elected November 6, 1962. <col_header> Reason for change </ col_header> </cell> <cell> Ted Kennedy (D) <col_header> Successor </col_header> </cell> < cell> November 7, 1962 <col_header> Date of successor’s formal installation </col_header> </cell> </table> Zero-Shot: cell> Ted Kennedy (D) col_header> Vacator /col_header> /cell> cell> November 7, 1962 col_header> Reason for change PPO: Ted Kennedy (D) succeeding November 6, 1962 as Vacator of the 87th United States Congress. NLPO: Ted Kennedy (D) Successor elected November 6, 1962. table> cell Benjamin A. Smith I (D) col_header Reason for change: Date of successor’s formal installation November 7, 1962. Supervised: Benjamin A. Smith I (D) served as senate until November 6, 1962 which was later served by Ted Kennedy (D) from November 7, 1962. Supervised+PPO: Benjamin A. Smith I (D) served until November 6, 1962 and Ted Kennedy (D) succeeded him from November 7, 1962. Supervised+NLPO: Benjamin A. Smith I (D) served until November 6, 1962 and Ted Kennedy (D) succeeded him from November 7, 1962. 48 Published as a conference paper at ICLR 2023 B.7NARRATIVEQA B.7.1SETUP NarrativeQA (Ko ˇ cisk ` y et al., 2018) deals with task of generating answers to questions about a given story. For training RL methods, we consider 2 traditional lexical rewards namely Rouge Combined and Rouge-L-Max. We chose T5-base as the base LM since it has been shown to do well at question answering in prior work (Khashabi et al., 2020). We note that the supervised models we use are trained on the UnifiedQA dataset, which contains other QA datasets, and is shown by Khashabi et al. (2020) to outperform supervised fine-tuning only on NarrativeQA. Hyperparams for our models can be found in Table 27. Model Paramsvalue ppo/nlposteps per update:5120 total number of steps:512000 batch size:64 epochs per update:5 learning rate:0.000002 entropy coefficient:0.0 initial kl coeff:0.001 target kl:1.0 discount factor:0.99 gae lambda:0.95 clip ratio:0.2 rollouts top k :50 value function coeff:0.5 top mask ratio:0.9 target update iterations:20 supervised+ ppo (or nlpo)steps per update:2560 total number of steps:512000 batch size:64 epochs per update:5 learning rate:0.0000005 entropy coefficient:0.0 initial kl coeff:0.001 target kl:0.2 discount factor:0.99 gae lambda:0.95 clip ratio:0.2 rollouts top k :50 value function coeff:0.5 top mask ratio:0.9 target update iterations:20 decodingnum beams:4 max new tokens:50 tokenizerpadding side: left truncation side: right max length: 512 Table 27:NarQA Hyperparams: Table shows a list of all hyper-parameters and their settings 49 Published as a conference paper at ICLR 2023 Tasks Lexical and Semantic Metrics Diversity Metrics Alg Reward Function LM Rouge-1 Rouge-2 Rouge-L Rouge-LSum Rouge-LMax Meteor BLEU BertScore MSTTR Distinct 1 Distinct 2 H 1 H 2 Unique 1 Unique 2 Mean Output Length NarQA Zero Shot T5 0.095 0.022 0.084 0.084 0.117 0.095 0.009 0.835 0.415 0.026 0.097 9.641 13.468 1880 11495 31.688 PPO Rouge Combined T5 0.101 0.025 0.088 0.088 0.122 0.099 0.01 0.837 0.462 0.03 0.125 9.759 13.789 2522 17806 32.352 Rouge-L Max T5 0.099 0.025 0.087 0.087 0.122 0.099 0.01 0.835 0.439 0.029 0.119 9.653 13.618 2292 15816 31.479 NLPO Rouge Combined T5 0.097 0.023 0.085 0.085 0.118 0.098 0.009 0.836 0.418 0.025 0.096 9.652 13.528 1816 10980 32.117 Rouge-L Max T5 0.102 0.026 0.089 0.089 0.124 0.1 0.01 0.837 0.445 0.029 0.118 9.776 13.75 2181 14569 31.555 Supervised T5 0.378 0.190 0.367 0.367 0.581 0.099 0.209 0.931 0.609 0.156 0.534 9.807 13.657 3250 14995 4.923 Supervised + PPO Rouge Combined T5 0.38 0.177 0.371 0.371 0.585 0.09 0.229 0.931 0.64 0.174 0.559 10.132 13.547 3326 13785 4.353 Rouge-L Max T5 0.368 0.18 0.36 0.36 0.585 0.083 0.239 0.931 0.641 0.187 0.576 10.201 13.452 3287 12436 3.913 Supervised + NLPO Rouge Combined T5 0.398 0.21 0.393 0.373 0.589 0.096 0.24 0.971 0.679 0.185 0.595 10.304 13.694 3371 15067 4.728 Rouge-L Max T5 0.381 0.194 0.383 0.383 0.588 0.093 0.243 0.932 0.645 0.187 0.59 10.2 13.397 3287 12171 3.889 Table 28: Evaluation of NarrativeQA : Reference Metrics, supervised is based on UnifiedQA (Khashabi et al., 2020). 50 Published as a conference paper at ICLR 2023 B.7.2RESULTS ANDDISCUSSION Table 28 presents our benchmarking results with 2 reward functions along with supervised baseline performances on the NarrativeQA test set. Similar to other methods, our main finding is that warm- started initial policies are crucial for learning to generate answers that successfully use the input context. B.7.3QUALITATIVERESULTS We show sample generations from each of the algorithms for three randomly picked prompts below. Sample1 Prompt: who is mark hunter? mark hunter (slater), a high school student in a sleepy suburb of phoenix, arizona, starts an fm pirate radio station that broadcasts from the basement of his parents’ house. mark is a loner, an outsider, whose only outlet for his teenage angst and aggression is his unauthorized radio station. his pirate station’s theme song is " everybody knows" by leonard cohen and there are glimpses of cassettes by such alternative musicians as the jesus and mary chain, camper van beethoven, primal scream, soundgarden, ice-t, bad brains, concrete blonde, henry rollins, and the pixies. by day, mark is seen as a loner, hardly talking to anyone around him; by night, he expresses his outsider views about what is wrong with american society. when he speaks his mind about what is going on at his school and in the community, more and more of his fellow students tune in to hear his show.nobody knows the true identity of "hard harry" or "happy harry hard-on ," as mark refers to himself, until nora diniro (mathis), a fellow student, tracks him down and confronts him the day after a student named malcolm commits suicide after harry attempts to reason with him. the radio show becomes increasingly popular and influential after harry confronts the suicide head-on, exhorting his listeners to do something about their problems instead of surrendering to them through suicideÃ ́ cÂ ̆ AÂ ˇ Tat the crescendo of his yelled speech, an overachieving student named paige woodward (who has been a constant listener) jams her various medals and accolades into a microwave and turns it on. she then sits, watching the awards cook until the microwave explodes, injuring her. while this is happening, other students act out in cathartic release.eventually, the radio show causes so much trouble in the community that the fcc is called in to investigate. during the fracas, it is revealed that the school’s principal (annie ross) has been expelling " problem students," namely, students with below-average standardized test scores, in an effort to boost the district’s test scores while still keeping their names on the rolls ( a criminal offense) in order to retain government funding.realizing he has started something huge, mark decides it is up to him to end it. he dismantles his radio station and attaches it to his mother’s old jeep, creating a mobile transmitter so his position can’t be triangulated. pursued by the police and the fcc, nora drives the jeep around while mark broadcasts. the harmonizer he uses to disguise his voice breaks, and with no time left to fix it, mark decides to broadcast his final message as himself. they finally drive up to the crowd of protesting students, and mark tells them that the world belongs to them and that they should make their own future. the police step in and arrest mark and nora. as they are taken away, mark reminds the students to "talk hard." as the film ends, the voices of other students (and even one of the teachers) speak as intros for their own independent stations, which can be heard broadcasting across the country. Zero-Shot: hunter (slater) is a high school student in a sleepy suburb of phoenix, arizona. he starts an fm pirate radio station that broadcasts from the basement of his References: he is a high school student in phoenix 2) loner and outsider student with a radio station PPO: hunter (slater), a high school student in a sleepy suburb of phoenix, arizona, starts an fm pirate radio station that broadcasts from the basement of his parents’ house NLPO: hunter (slater), a high school student in a sleepy suburb of phoenix, arizona, starts an fm pirate radio station that broadcasts from the basement of his parents’ house Supervised: a high school student Supervised+PPO: a high school student Supervised+NLPO: a high school student Sample2 Prompt: what was the principle doing with the problem students? mark hunter (slater), a high school student in a sleepy suburb of phoenix, arizona, starts an fm pirate radio station that broadcasts from the basement of his parents’ house. mark is a loner, an outsider, whose only outlet for his teenage angst and aggression is his unauthorized radio station. his pirate station’s theme song is "everybody knows" by leonard cohen and there are glimpses of cassettes by such alternative musicians as the jesus and mary chain, camper van beethoven, primal scream, soundgarden, ice-t, bad brains, concrete blonde, henry rollins, and the pixies. by day, mark is seen as a loner, hardly talking to anyone around him; by night, he expresses his outsider views about what is wrong with american society. when he speaks his mind about what is going on at his school and in the community, more and more of his fellow students tune in to hear his show.nobody knows the true identity of "hard harry" or "happy harry hard-on," as mark refers to himself, until nora diniro ( mathis), a fellow student, tracks him down and confronts him the day after a student named malcolm commits suicide after harry attempts to reason with him. the radio show becomes increasingly popular and influential after harry confronts the suicide head-on, exhorting his listeners to do something about their problems instead of surrendering to them through suicideÂ ̆ AÂ ˇ Tat the crescendo of his yelled speech, an overachieving student named paige woodward (who has been a constant listener) jams her various medals and 51 Published as a conference paper at ICLR 2023 accolades into a microwave and turns it on. she then sits, watching the awards cook until the microwave explodes, injuring her. while this is happening, other students act out in cathartic release.eventually, the radio show causes so much trouble in the community that the fcc is called in to investigate. during the fracas, it is revealed that the school’s principal (annie ross) has been expelling "problem students," namely, students with below-average standardized test scores, in an effort to boost the district’s test scores while still keeping their names on the rolls (a criminal offense) in order to retain government funding.realizing he has started something huge, mark decides it is up to him to end it. he dismantles his radio station and attaches it to his mother’s old jeep, creating a mobile transmitter so his position can’t be triangulated. pursued by the police and the fcc, nora drives the jeep around while mark broadcasts. the harmonizer he uses to disguise his voice breaks, and with no time left to fix it, mark decides to broadcast his final message as himself. they finally drive up to the crowd of protesting students, and mark tells them that the world belongs to them and that they should make their own future. the police step in and arrest mark and nora. as they are taken away, mark reminds the students to "talk hard." as the film ends, the voices of other students (and even one of the teachers) speak as intros for their own independent stations, which can be heard broadcasting across the country. Zero-Shot: mark hunter (slater) starts an fm pirate radio station that broadcasts from the basement of his parents’ house. by day, mark is seen as a loner, an outsider, whose only outlet References: 1) expelling them. 2) expelling the students PPO: mark hunter (slater) starts an fm pirate radio station that broadcasts from the basement of his parents’ house. mark is a loner, an outsider, whose only outlet for his teenage ang NLPO: mark hunter (slater) starts an fm pirate radio station that broadcasts from the basement of his parents’ house. mark is a loner, an outsider, whose only outlet for his teenage ang Supervised: he was taking them to a psychiatric hospital. Supervised+PPO: addressing them Supervised+NLPO: he was expelling them Sample3 Prompt: what happens to the people maskull meets in his travels to various lands? maskull, a man longing for adventures, accepts an invitation from krag, an acquaintance of his friend nightspore, to travel to tormance after a seance. the three set off in a crystal ship from an abandoned observatory in scotland but maskull awakens to find himself alone on tormance. in every land he passes through he usually meets only one or two persons; these meetings often (though not always) end in the death of those he meets, either at his own hand or by that of another. he learns of his own impending death, meets krag again, and dies shortly after learning that he is in fact nightspore himself. the book concludes with a final revelation from krag (who claims to be known on earth as "pain") to nightspore about the origin of the universe. the author turns out to support a variation of the doctrine of the demiurge, somewhat similar to that propounded by some gnostics.all of the characters and lands are types used to convey the author’s critique of several philosophical systems. on tormance, most such viewpoints or ways of life are accompanied by corresponding new bodily sense organs or modifications of the same, thus each distinct weltanschauung landscape has its corresponding sensorium. Zero-Shot: what happens to the people maskull meets in his travels to various lands? n maskull, a man longing for adventures, accepts an invitation from krag to travel to tormance after References: 1) they usually die by his hand or anothers. 2) they die PPO: what happens to the people maskull meets in his travels to various lands? n maskull, a man longing for adventures, accepts an invitation from krag to travel to tormance after NLPO: maskull meets krag, an acquaintance of nightspore, to travel to tormance after a seance. maskull awakens to find himself alone on tormance. in every land Supervised: they die either from his own hand or from another person’s death. Supervised+PPO: they end up dying either at his own hand or by another’s. Supervised+NLPO: they end up dying either at his own hand or by another’s 52 Published as a conference paper at ICLR 2023 B.8MACHINETRANSLATION B.8.1SETUP WMT-16 We pick two languages, English and German, and frame this task similarly to other machine translation tasks—requiring the models to translate from English to German. We train models on 4 rewards: SacreBLEU, chRF, TER, and BertScore. Model Paramsvalue supervisedbatch size:64 epochs:5 learning rate:0.00001 learning rate scheduler: constant weight decay: 0.1 ppo/nlposteps per update:5120 total number of steps:256000 batch size:64 epochs per update:5 learning rate:0.0.000001 entropy coefficient:0.0 initial kl coeff:0.001 target kl:0.2 discount factor:0.99 gae lambda:0.95 clip ratio:0.2 rollouts top k :10 value function coeff:0.5 top mask ratio:0.5 target update iterations:20 supervised+ ppo (or nlpo)steps per update:2560 total number of steps:256000 batch size:64 epochs per update:5 learning rate:0.0000005 entropy coefficient:0.0 initial kl coeff:0.001 target kl:0.2 discount factor:0.99 gae lambda:0.95 clip ratio:0.2 rollouts top k :10 value function coeff:0.5 top mask ratio:0.5 target update iterations:20 decodingnum beams:4 length penalty:0.6 max new tokens:128 tokenizerpadding side: left truncation side: right max length: 128 Table 29:NMT Hyperparams: Table shows a list of all hyper-parameters and their settings B.8.2RESULTS ANDDISCUSSION Tables 30, 31 presents our benchmarking results with 4 reward functions along with supervised baseline performances on test set. Our main finding is that NLPO + Supervised performs better than PPO and supervised models. 53 Published as a conference paper at ICLR 2023 DatasetsLexical and Semantic Metrics AlgLMReward FunctionRouge-1Rouge-2Rouge-LRouge-LSumMeteorBLEUSacreBLEUchRfTERBertScore WMT16 Zero-ShotT50.6350.4140.5910.5910.4830.2940.3480.6130.5430.882 PPOT5SacreBLEU0.6360.4150.5910.5910.4820.2940.3480.6140.5390.882 T5chRF 0.6350.4140.5910.5910.4810.2910.3460.6120.5400.882 T5TER0.6380.4160.5950.5940.4840.2940.3500.6160.5340.883 T5BertScore 0.6370.4170.5930.5930.4790.2940.3470.6130.5340.882 NLPOT5SacreBLEU0.6350.4150.5920.5920.4840.2970.3520.6150.5420.882 T5chRF 0.6340.4130.590.590.4810.2910.3450.6120.5400.882 T5TER0.6330.4120.590.590.4770.2860.3410.6080.5400.881 T5BertScore 0.6220.3970.580.5810.4580.2690.3230.5910.5460.876 SupervisedT50.6350.4110.5900.5900.4820.2940.3500.6170.5400.882 Supervised + PPOT5SacreBLEU0.6400.4160.5950.5950.4870.2980.3550.6200.5330.883 T5chRF0.6400.4160.5960.5960.4860.2980.3540.6210.5320.883 T5TER 0.6370.4140.5940.5940.4830.2950.3520.6180.5330.882 T5BertScore0.6370.4130.5930.5940.4820.2940.3500.6160.5330.882 Supervised + NLPOT5SacreBLEU0.6420.4190.5960.5960.4970.2970.3550.6210.5330.888 T5chRF0.6360.4120.5920.5920.4920.2930.3490.6170.5340.886 T5TER0.6370.4140.5940.5940.4910.2920.3490.6150.5310.886 T5BertScore0.640.4170.5980.5980.4990.2870.3490.620.5380.887 IWSLT2017 Zero-ShotT5 0.6190.3860.5880.5870.4450.2540.3080.5770.5730.870 PPOT5SacreBLEU0.6210.3830.5870.5870.4480.2430.2960.5750.5830.869 T5chRF0.6220.3850.5900.5900.4480.2480.3010.5780.5750.870 T5TER0.6230.3840.5910.5910.4430.2460.3030.5720.5680.869 T5BertScore0.5330.3260.5070.5070.3210.1430.1740.4060.5730.839 NLPOT5SacreBLEU0.6240.3850.590.590.450.2450.2990.5780.5780.87 T5chRF0.6240.3860.590.590.4510.2480.3020.5810.5760.87 T5TER0.6220.3840.590.590.4430.2460.3030.5730.570.869 T5BertScore0.6110.3770.580.580.4250.2390.2910.5550.5730.866 SupervisedT50.6380.4000.6100.6090.4610.2800.3370.5930.5380.878 Supervised + PPOT5SacreBLEU0.6400.4070.6100.6100.4650.2770.3320.5960.5420.877 T5chRF0.6390.4060.6090.6090.4640.2770.3310.5960.5430.877 T5TER0.6370.4060.6090.6090.4570.2740.3310.5890.5350.876 T5BertScore0.6120.3810.5850.5850.4180.2400.2910.5480.5590.867 Supervised + NLPOT5SacreBLEU0.6410.4180.6140.6140.4740.2890.3430.5970.5350.877 T5chRF0.6430.4180.6210.6210.4640.2910.3450.5960.5390.877 T5TER0.6390.4190.6210.6210.4710.2890.3460.5930.5350.877 T5BertScore0.6330.4010.6060.6060.4480.2670.3230.5800.5370.875 Table 30:WMT-16 and IWSLT test evaluation - lexical and semantic: Table shows lexical, semantic metrics for RL algorithms with different reward functions bench-marked against supervised baseline models 54 Published as a conference paper at ICLR 2023 TasksDiversity Metrics AlgReward FunctionLM MSTTRDistinct 1 Distinct 2 H 1 H 2 Unique 1 Unique 2 Mean Output Length WMT16 Zero-ShotT50.7320.1930.67510.10014.56172903369120.533 PPOT5SacreBLEU0.7380.1980.68710.16614.61375033414020.375 T5chRF 0.7380.1960.68710.17514.61173763411620.337 T5TER0.7360.1960.68310.13214.58874473397720.356 T5BertScore0.7360.1950.68510.12914.57472723347720.035 NLPOT5SacreBLEU0.7350.1930.6810.12514.59273953427620.672 T5chRF0.7380.1960.68610.16414.60673993405620.351 T5TER0.740.20.69410.20414.6375223423420.151 T5BertScore0.7390.20.69810.19414.60872033316919.482 SupervisedT50.7290.1900.66910.04814.53072053343020.622 Supervised + PPOT5SacreBLEU0.7320.1910.67410.08014.55272223372320.605 T5chRF0.7350.1920.67710.09314.56973193392320.586 T5TER0.7320.1920.67610.07914.55372653363520.441 T5BertScore0.7320.1920.67710.08214.55071873338520.305 Supervised + NLPOT5SacreBLEU0.7340.1910.67510.08914.56873083394120.686 T5chRF0.7350.1940.68110.11214.57173723381420.348 T5TER0.7370.1940.68210.10514.56672433348220.159 T5BertScore0.7370.2270.74210.04214.17954382257412.63 IWSLT2017 Zero-ShotT50.6620.0970.47009.27614.52683125294718.739 PPOT5SacreBLEU0.6570.0950.4649.23014.49882855300019.069 T5chRF0.6600.0960.4689.25314.52682435314218.912 T5TER0.6590.0970.4749.24414.53681295191418.268 T5BertScore0.6730.1200.5419.28814.38866423726711.602 NLPOT5SacreBLEU0.6560.0940.4639.20714.48382405282219.043 T5chRF0.6580.0950.4649.23314.50282305316719.073 T5TER0.6610.0980.4769.27114.55282235243818.344 T5BertScore0.6670.1020.4919.3114.57681345074017.162 SupervisedT50.6550.0950.4679.21014.49279705143018.440 Supervised + PPOT5SacreBLEU0.6540.0940.4619.17614.46780615184018.803 T5chRF 0.6560.0940.4649.20214.49780545219818.794 T5TER0.6580.0970.4759.23914.52979695125518.048 T5BertScore0.6650.1020.4959.27014.52474954762916.051 Supervised + NLPOT5SacreBLEU0.6550.0940.4619.18214.46980735179718.777 T5chRF0.6570.0950.4669.21214.50380745205618.642 T5TER0.6580.0960.4739.23114.52180255138918.133 T5BertScore0.6610.0980.4829.2614.54778525067217.496 Table 31:WMT-16 and IWSLT test evaluation - diversity metricsTable shows diversity metrics for RL algorithms with different reward functions bench-marked against supervised baseline models 55 Published as a conference paper at ICLR 2023 B.8.3QUALITATIVERESULTS We show sample generations from each of the algorithms for three randomly picked prompts from IWSLT below. Sample1 Prompt: translate English to German: Several years ago here at TED, Peter Skillman introduced a design challenge called the marshmallow challenge. Zero-Shot: Vor einigen Jahren hat Peter Skillman hier bei TED eine Design-Herausforderung namens Marshmallow Challenge vorgestellt. PPO: Vor einigen Jahren, hier bei TED, stellte Peter Skillman einen Design-Wettbewerb namens " Die Marshmallow-Herausforderung" vor NLPO: Vor einigen Jahren hier bei TED fÃijhrte Peter Skillman eine Design-Herausforderung namens Marshmallow Challenge ein. Supervised: Vor einigen Jahren hier bei TED fÃijfchrte Peter Skillman eine Design- Herausforderung namens Marshmallow Challenge ein. Supervised+PPO: Vor einigen Jahren hier bei TED, fÃijhrte Peter Skillman eine Design- Herausforderung namens Marshmallow-Herausforderung ein. Supervised+NLPO: Vor einigen Jahren hier bei TED fÃijchrte Peter Skillman eine Design- Herausforderung ein, die Marshmallow-Herausforderung. Sample2 Prompt: translate English to German: They spend the majority of their time assembling the sticks into ever-growing structures. Zero-Shot: Sie verbringen den grÃ ̋uÃ§ten Teil ihrer Zeit damit, die StÃ ̋ucke in immer wachsende Strukturen zu montieren. References: Sie investieren einen GroÃ§teil ihrer Zeit in den Zusammenbau stÃd’ndig wachsender Strukturen PPO: Sie verbringen den GroÃ§teil ihrer Zeit damit, die StÃ ̋ucke in immer grÃ ̋uÃ§er werdende Strukturen zu montieren. NLPO: Sie verbringen den GroÃ§teil ihrer Zeit, die StÃ ̋ucke in immer wachsende Strukturen zu montieren. Supervised: Sie verbringen den GroÃ§teil ihrer Zeit damit, die StÃ ̋ucke in immer wachsende Strukturen zu montieren. Supervised+PPO: Sie verbringen den GroÃ§teil ihrer Zeit damit, die StÃ ̋ucke in immer wachsende Strukturen zu montieren. Supervised+NLPO: Sie verbringen den GroÃ§teil ihrer Zeit damit, die StÃ ̋ucke in immer wachsende Strukturen zu montieren. Sample3 Prompt: translate English to German: And of course there are teams that have a lot more "ta-da " structures, and among the best are recent graduates of kindergarten. Zero-Shot: Und natÃijrlich gibt es Teams, die viel mehr "ta-da" Strukturen haben, und zu den besten gehÃ ̋uren jÃijngste Absolventen des Kindergartens. References: Und natÃijrlich gibt es auch Teams die viel mehr "ta-da"-Gebilde haben und unter den Besten sind frische Absolventen des Kindergartens. PPO: Und natÃijrlich gibt es Teams, die viel mehr "ta-da" Strukturen, und unter den besten sind jÃijngste Absolventen des Kindergartens. NLPO: Und natÃijrlich gibt es Teams, die viel mehr "t a-da" Strukturen haben, und unter den besten sind jÃijngste Absolventen des Kindergartens Supervised: Und natÃijrlich gibt es Teams, die viel mehr "ta-da"-Strukturen haben, und unter den besten sind jÃijngste Absolventen des Kindergartens. Supervised+PPO: Und natÃijrlich gibt es Teams, die viel mehr "ta-da"-Strukturen haben, und unter den besten sind jÃijngste Absolventen des Kindergartens. Supervised+NLPO: Und natÃijlich gibt es Teams, die viel mehr "ta-da"-Strukturen haben, und unter den besten sind jÃijngste Absolventen des Kindergartens. 56 Published as a conference paper at ICLR 2023 B.9DAILYDIALOG B.9.1SETUP We consider DailyDialog (Li et al., 2017) as the test bed for the dialogue generation task. The dataset includes conversations written by human on various topics. In addition, each utterance contains labels of intent and emotional information. For simplicity, we focus only on generating the next utterance, given the dialogue context. We chose a context window of size5, which results in35k training,3k and3k utterances. The input to the model is dialogue history in which utterances are concatenated using a <EOU> token. We picked GPT-2 as the LM as they are more suited for text continuation than encoder-decoder LMs. For a fair comparison, we use top-k sampling withk= 20as the decoding method for all methods. For RL methods, we use a linear combination of meteor score and intent match score (whether the generated text’s intent matches with the reference’s intent) as the reward function. The coefficients for meteor and intent are chosen based on both lexical scores and intent accuracy on the validation set. For this purpose, we trained an intent classifier (fine-tuned RoBERTa (Liu et al., 2019)) that classifies given text into intent categories such asinform,question,directive andcommisive, etc. Table 32 provides a summary of hyperparameters and implementation details. Model Paramsvalue ppo/nlposteps per update:1280 total number of steps:128000 batch size:64 epochs per update:5 learning rate:0.000001 entropy coefficient:0.0 initial kl coeff:0.2 target kl:0.5 discount factor:0.99 gae lambda:0.95 clip ratio:0.2 rollouts top k :20 value function coeff:0.5 meteor coeff:0.25 intent coeff:0.75 top mask ratio:0.9 target update iterations:20 supervised+ ppo (or nlpo)steps per update:1280 total number of steps:64000 batch size:64 epochs per update:5 learning rate:0.000001 entropy coefficient:0.0 initial kl coeff:0.2 target kl:0.5 discount factor:0.99 gae lambda:0.95 clip ratio:0.2 rollouts top k :20 value function coeff:0.5 meteor coeff:0.5 0.25 intent coeff:0.5 0.75 top mask ratio:0.9 target update iterations:20 decodingtop k:20 min length:2 max new tokens:50 tokenizerpadding side: left truncation side: right max length: 128 Table 32:DailyDialog Hyperparams: Table shows a list of all hyper-parameters and their settings 57 Published as a conference paper at ICLR 2023 Tasks Lexical and Semantic Metrics Diversity Metrics Alg Reward Function LM Rouge-1 Rouge-2 Rouge-L Rouge-LSum Meteor SacreBLEU BertScore Intent Accuracy MSTTR Distinct 1 Distinct 2 H 1 H 2 Unique 1 Unique 2 Mean Output Length Dialog Zero Shot GPT-2 0.157 0.012 0.131 0.131 0.191 0.066 0.854 0.427 0.608 0.055 0.316 7.787 11.831 1574 12327 18.685 Supervised GPT-2 0.162 0.020 0.138 0.138 0.186 0.064 0.855 0.437 0.635 0.065 0.342 8.051 12.119 1925 13952 18.919 PPO Meteor + Intent GPT-2 0.168 0.012 0.142 0.142 0.221 0.085 0.861 0.474 0.581 0.058 0.310 7.653 11.437 1719 12156 18.538 NLPO Meteor + Intent GPT-2 0.169 0.013 0.142 0.142 0.221 0.087 0.860 0.490 0.568 0.059 0.309 7.630 11.351 1718 11946 18.397 Supervised + PPO Meteor + Intent GPT-2 0.169 0.021 0.144 0.144 0.198 0.071 0.857 0.455 0.626 0.068 0.348 8.056 12.015 1983 14170 18.829 Supervised + NLPO Meteor + Intent GPT-2 0.171 0.020 0.146 0.146 0.205 0.074 0.858 0.454 0.624 0.070 0.349 8.044 11.990 2051 14213 18.763 Table 33: Evaluation of Daily Dialog : Table shows lexical, semantic metrics for RL algorithms bench-marked against supervised baseline models 58 Published as a conference paper at ICLR 2023 AlgorithmUnique N CoherenceQuality ValueAlphaSkewValueAlphaSkew Zeroshot313.840.2254.1813.20.1253.352 NLPO304.180.1144.173.350.1593.318 PPO324.180.1124.0323.320.1633.478 Supervised+PPO313.990.1484.1333.480.1663.58 Supervised+NLPO314.130.1863.9533.580.1783.597 Supervised313.960.2493.8343.590.2363.196 Table 34: Results of the human subject study showing the number of participants N, average Likert scale value for coherence and sentiment, Krippendorf’s alpha showing inter-annotator agreement, and Skew. For each model a total of 100 samples were drawn randomly from the test set and rated by 3 annotators each, each resulting in 300 data points per algorithm. B.9.2RESULTS ANDDISCUSSION Table 33 presents our benchmarking results of RL methods along with supervised baseline perfor- mances on test sets. Our main finding is that RL methods generally achieve better intent accuracy and automatic metric scores, in particular NLPO variants perform better than all other methods. B.9.3HUMANPARTICIPANTSTUDY Figure 10 shows the Daily Dialogue instructions and interface used for the human evaluation ex- periments. Tables 34, 35 show averaged results, annotator agreement, and the results of statistical significance tests to determine which models output better generations when rated by humans. Group 1Group 2 CoherenceQuality Diff (G2-G1)p-valuesDiff (G2-G1)p-values NLPOPPO-0.0030.900-0.0300.900 NLPOSupervised-0.2270.0430.2380.020 NLPOSupervised+NLPO-0.0500.9000.2340.022 NLPOSupervised+PPO-0.1940.0130.1270.803 NLPOZero Shot-0.3450.001-0.1540.655 PPOSupervised-0.2240.0490.2680.010 PPOSupervised+NLPO-0.0470.9000.2640.011 PPOSupervised+PPO-0.1910.1440.1570.636 PPOZero Shot-0.3410.001-0.1240.822 SupervisedSupervised+NLPO0.1770.021-0.0030.900 SupervisedSupervised+PPO0.0330.900-0.1100.896 SupervisedZero Shot-0.1170.645-0.3910.002 Supervised+NLPOSupervised+PPO-0.1440.444-0.1070.009 Supervised+NLPOZero Shot-0.2940.002-0.3880.003 Supervised+PPOZero Shot-0.1510.390-0.2810.008 Table 35: Results of an post-hoc Tukey HSD Test for difference in means between pairs of algorithms (Group 2 - Group 1) and correspondingp-values. Individually statistically significant results are bolded and are used to discuss results in the analysis. Overallp-values showing that there is a significant difference in means between the models via a one-way ANOVA test are significant with p0.05for both coherence and sentiment. 59 Published as a conference paper at ICLR 2023 Figure 10: Instructions and interface for the Daily Dialogue task. B.9.4QUALITATIVEANALYSIS We show sample generations from each of the algorithms for three randomly picked prompts below. Sample1 Prompt: Right . And China is the homeland of tea . <EOU> Yes , Chinese people love drinking tea so much . Some even claim they can’t live without tea . <EOU> Do you know there are several catagories of Chinese tea ? <EOU> Yes , I believe there are green teas , black teas and scented teas . Any Others ? <EOU> Well , have you ever heard of Oulong tea and compressed tea ? <EOU> References: Oh , yeah . Oulong tea is good for one’s health . isn’t it ? Zero-Shot: I know that Oulong tea can be enjoyed with very few meals, and there are very few PPO: I have, but it’s not as strong as Oulong tea. NLPO: I haven’t. I don’t have tea. Supervised: Yes. I hear they do good. And I heard this Taijiquan Tea is very good Supervised+PPO: Actually, I have had Oulong tea, but I haven’t had any for awhile now. Supervised+NLPO: No, I’ve never heard of that. Sample2 60 Published as a conference paper at ICLR 2023 Prompt: I think I might have a dahl . <EOU> what’s in a dahl ? <EOU> it’s got chickpeas and vegetables in a spicy curry sauce with rice . <EOU> that sounds nice . Do you want to share some kebabs as well ? <EOU> ok . How about some lamb kebabs ? <EOU> References: that’s my favourite . Do you want to have some wine or beer ? Zero-Shot: they’re very salty PPO: well you might want to take a peek. NLPO: well I like that. Supervised: ok, here they are. I’m so thirsty. I can’t wait until the next bus to Supervised+PPO: I’ve tried that a few times, but they’re too dry and I don’t want any of Supervised+NLPO: oh, ok. That’s a good choice. I’l take it. Sample3 Prompt: Really ? Have you had an auto-reverse break before ? <EOU> Yes , twice now . I think it’s too high-tech , and so it’s the first part that breaks . <EOU> Hmm . You know , the problem might be dust . Did you clean the unit often ? <EOU> Well . No , I didn’t clean it.But still , I only had the last one two months , and it broke . I don’t trust auto- reverse . <EOU> Alright . Well . We have many units without auto-reverse.Here is a good unit . Very reliable.It has an AM / FM band , built-in microphone , recording capability , and no auto-reverse . <EOU> References: How much is it ? Zero-Shot: Oh. So you don’t have the AM / FM band? PPO: You said AM / FM. Are you sure about that? NLPO: Well. Well. I had no idea when it was, and didn’t care. Supervised: Good.Well, what else I need? Supervised+PPO: But I think it lacks the CD player or its CD player accessories. Supervised+NLPO: Cool! So what’s the trouble with it? 61

Instant research discovery