Paper deep dive

The Trickle-down Impact of Reward (In-)consistency on RLHF

Lingfeng Shen, Sihao Chen, Linfeng Song, Lifeng Jin, Baolin Peng, Haitao Mi, Daniel Khashabi, Dong Yu

Year: 2023Venue: arXiv preprintArea: Alignment TrainingType: EmpiricalEmbeddings: 88

Models: GPT2-0.1B, GPT2-XL, GPT-J-6B, LLaMa-7B

Intelligence

Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 96%

Last extracted: 3/12/2026, 7:37:15 PM

Summary

The paper investigates the 'reward inconsistency' problem in Reinforcement Learning from Human Feedback (RLHF), where Reward Models (RMs) fail to distinguish between semantically different but lexically similar instructions. The authors introduce 'Contrast Instructions' as a benchmarking strategy to measure this inconsistency and propose two techniques, 'ConvexDA' (for training) and 'RewardFusion' (for inference), to improve RM consistency. Empirical results demonstrate that more consistent RMs lead to higher-quality, more useful responses in downstream RLHF models, confirming a trickle-down effect.

Entities (6)

Contrast Instructions · benchmark · 100%ConvexDA · technique · 100%RewardFusion · technique · 100%Reward Model · model-component · 100%RLHF · methodology · 100%Llama-7B · model · 95%

Relation Signals (3)

Reward Model → influences → RLHF

confidence 95% · reward inconsistency exhibits a trickle-down effect on the downstream RLHF process.

Contrast Instructions → measures → Reward Model

confidence 95% · We propose Contrast Instructions -- a benchmarking strategy for the consistency of RM.

ConvexDA → enhances → Reward Model

confidence 90% · we propose two techniques ConvexDA and RewardFusion, which enhance reward consistency

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Standard practice within Reinforcement Learning from Human Feedback (RLHF) involves optimizing against a Reward Model (RM), which itself is trained to reflect human preferences for desirable generations. A notable subject that is understudied is the (in-)consistency of RMs -- whether they can recognize the semantic changes to different prompts and appropriately adapt their reward assignments -- and their impact on the downstream RLHF model. In this paper, we visit a series of research questions relevant to RM inconsistency: (1) How can we measure the consistency of reward models? (2) How consistent are the existing RMs and how can we improve them? (3) In what ways does reward inconsistency influence the chatbots resulting from the RLHF model training? We propose Contrast Instructions -- a benchmarking strategy for the consistency of RM. Each example in Contrast Instructions features a pair of lexically similar instructions with different ground truth responses. A consistent RM is expected to rank the corresponding instruction and response higher than other combinations. We observe that current RMs trained with the standard ranking objective fail miserably on Contrast Instructions compared to average humans. To show that RM consistency can be improved efficiently without using extra training budget, we propose two techniques ConvexDA and RewardFusion, which enhance reward consistency through extrapolation during the RM training and inference stage, respectively. We show that RLHF models trained with a more consistent RM yield more useful responses, suggesting that reward inconsistency exhibits a trickle-down effect on the downstream RLHF process.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

87,403 characters extracted from source content.

Expand or collapse full text

Preprint THETRICKLE-DOWNIMPACT OFREWARD(IN-)CONSISTENCY ONRLHF Lingfeng Shen ♣∗ Sihao Chen ♠ Linfeng Song ♡ Lifeng Jin ♡ Baolin Peng ♡ Haitao Mi ♡ Daniel Khashabi ♣ Dong Yu ♡ ♣ Johns Hopkins University ♠ University of Pennsylvania ♡ Tencent AI Lab ABSTRACT A standard practice within Reinforcement Learning from Human Feedback (RLHF) involves optimizing against a Reward Model (RM), which itself is trained to reflect human preferences for desirable generation. A notable subject that is understudied is the(in-)consistencyof RMs — whether they can recognize the semantic changes to different prompts and appropriately adapt its reward assign- ments — and its impact on the downstream RLHF model. In this paper, we visit a series of research questions relevant to RM inconsistency: (1) How can we measure the consistency of reward models? (2) How consistent are the existing RMs and how can we improve them? (3) In what ways does reward inconsistency influence the chatbots resulting from the RLHF model training? We propose CONTRASTINSTRUCTIONS– a benchmarking strategy for the con- sistency of RM. Each example in CONTRASTINSTRUCTIONSfeatures a pair of lexically similar instructions with different ground truth responses. A consistent RM is expected to rank the corresponding instruction and response higher than other combinations. We observe that current RMs trained with the standard rank- ing objective fail miserably on CONTRASTINSTRUCTIONScompared to average humans. To show that RM consistency can be improved efficiently without using extra training budget, we propose two techniques CONVEXDA and REWARDFU- SION, which enhance reward consistency through extrapolation during the RM training and inference stage, respectively. We show that RLHF models trained with a more consistent RM yield more useful responses, suggesting that reward inconsistency exhibits a trickle-down effect on the downstream RLHF process. 1INTRODUCTION Recently,reinforcement learning from human feedback(RLHF) has emerged as a popular technique to optimize and align a language model with human preferences (Ouyang et al., 2022). RLHF provides a natural solution for optimizing non-differentiable, scalar objectives for language models, and has been the centerpiece of recent state-of-the-art large language models (LLMs) (Lu et al., 2022; Hejna I & Sadigh, 2023; Go et al., 2023; Korbak et al., 2023; OpenAI, 2023). In RLHF, areward model(RM) generates scalar rewards for model-generated outputs as supervision during reinforcement learning. RMs are typically calibrated to proxy human preferences/rankings of responses, in the context of an input instruction. Since policy gradient methods optimize based on this reward function, the reward function inevitably dictates the behavior of the resultant chatbot. As such, the properties of RMs and their impact on RLHF models have become points of interest for the research community (Gao et al., 2022; Zhu et al., 2023; Dong et al., 2023). In this work, we study the phenomenon ofreward inconsistencyin RMs, i.e., current RMs trained with the standard ranking objective on human preference data (§2) often fail to distinguish between more vs. less favorable responses with respect to real-world instructions. We observe that reward inconsistency has atrickle-down effecton the RLHF process — the more inconsistent the RM is, the more likely the resulting chatbot is to generate inaccurate or less useful responses. ∗ Most of the work done while Lingfeng and Sihao were interns at the Tencent AI Lab. 1 arXiv:2309.16155v1 [cs.CL] 28 Sep 2023 Preprint r B : “Read-Only Memory.” r A : “Random Access Memory.” I A : “What does RAM stand for in computing?” I B : “What does ROM stand for in computing?” CONTAST INSTRUCTIONS A pair of similar instructions with different ground truth responses ① A “consistent” reward model should assign higher rewards to corresponding (instruction, response) pairs than the other distractors. Standard RM I A r A I A r B .63 .87 Consistent RM .91 .57 Response Consistency Instruction Consistency .51 .92 .87 .63 ② We introduce two approaches to enhance the consistency of standard, preference-based RM. ③ RLHF with a more consistent RM ➡ Better chatbot r r r’ r r ConvexDA for robust training RewardFusion for robust inference r r’ r’ Instruction Responses I A r A r A I B Standard RM Consistent RM Standard RM Consistent RM Standard RM Consistent RM Standard RM Consistent RM   Figure 1: We introduce CONTRASTINSTRUCTIONSas a way to measure theconsistencyof a reward model. Each example consists of two similar instructions with different ground truth responses. Given one of the instructions or responses, whether or not an RM could rank its corresponding response or instruction higher than the other distractor indicates the consistency of the RM. To illustrate and quantify the degree of reward inconsistency in RMs, we propose CONTRASTIN- STRUCTIONS, a simple, intuitive benchmarking strategy that can be used with any instruction-tuning or human preference dataset in a fully automated manner. CONTRASTINSTRUCTIONSinvolves pairs of similar instructions with different responses. As an example, consider two similar-looking prompts (I A andI B ) one about “RAM” and “ROM” (Figure 1; left). Despite the similarity of these prompts, they warrant different responses. Our consistency metrics measure whether a reward model can appropriately identify the correspondence between prompts and responses. Specifically, given a pair of such (instruction, response) examples, we consider an RMconsistentif it assigns a higher score to the corresponding (instruction, response) compared to the other combinations, i.e. swapping instructions or responses between the two examples, as we show with 1in Figure 1. We construct CONTRASTINSTRUCTIONS 1 with four popular open-source human preference or in- struction tuning datasets. Surprisingly, we observe close to random-chance performance when we evaluate standard RMs trained with ranking objectives (e.g. LLaMa-7B) on CONTRASTINSTRUC- TIONS, while humans are able to rank the responses correctly in≈80%of the cases. The perfor- mance gap indicates the inherent reward inconsistency from standard RM training and inference. To enhance the consistency of standard RM, we introduce two techniques (§5) :CONVEXDAand REWARDFUSION( 2in Figure 1). The two methods can be incorporated during RM training and inference, respectively, without incurring extra computational cost during training. While our ex- perimental results indicate some improvements, the gap with respect to human performance remains large. Interestingly, our analysis (§6) reveals that using a more consistent RM during RLHF training leads to more useful responses from the downstream RLHF model ( 3 in Figure 1). Our findings suggest the value of reward (in-)consistency as an intrinsic evaluation metric for preference-based RMs, potentially enabling easier access for future research on reward modeling and RLHF. The main contributions of this paper are: • We introduce CONTRASTINSTRUCTIONS, an intuitive yet scalable benchmarking strategy for evaluating RM consistency. We observe a wide performance gap between standard, preference- based RM vs. human judgments, which suggests sizable room for improvements in standard reward modeling and evaluation strategy. • We show that reward consistency can be enhanced without extra training costs. We demonstrate this with two techniquesCONVEXDAandREWARDFUSION, which can be applied during RM training and inference stages, respectively. • We empirically show that training RLHF models with more consistent RMs would result in the RLHF model generating more useful responses. We provide thorough analysis and examples, which help us understand the advantage of a more consistent RM over a less inconsistent one. 1 The four CONTRASTINSTRUCTIONSbenchmark datasets and our code are available athttps:// github.com/shadowkiller33/Contrast-Instruction 2 Preprint 2PRELIMINARIES: REWARDMODELING FORRLHF Reward model.Following the conventional setup (Ziegler et al., 2019), we define RMR θ as a scalar function that allows us to train a generative model during RLHF. To supervise RM, we are given a dataset of human preferencesD h . Each instance in this dataset(I i ,r + i ,r − i )is comprised of an instruction promptI i , a pair of responsesr + i ,r − i wherer + i is preferred overr − i by humans. On this labeled data,R θ is trained to assign a higher scalar reward to human-preferredr + i over non- preferredr − i in the context ofI i This can be achieved by minimizing the ranking lossL, whereσis the sigmoid function andI i ◦r + i is the concatenation ofI i andr + i . L(θ) =−E ( I i ,r + i ,r − i ) ∼D h log σ R θ (I i ◦r + i −R θ I i ◦r − i ) .(1) Reinforcement Learning.The last stage of RLHF is reinforcement learning. Specifically, a per- token KL penalty from the supervised fine-tuning (SFT) model at each token to mitigate over- optimization of the reward model, and the value function is initialized from the RM. We maximize the following combined objective functionJ(φ)in RL training based on PPO algorithm (Schulman et al., 2017; Ouyang et al., 2022), RL training datasetD π RL φ and pre-training datasetD pre : J(φ) =E (I,r)∼D π RL φ R θ (I◦r)−βlog π RL φ (r|I)/π SFT (r|I) +γE r∼D pre log π RL φ (r) , whereR θ (I◦r)is the reward function, andπ RL φ is the learned RL policy parameterized byφ initialized from a pretrained supervised trained modelπ SFT . The first term encourage the policyπ RL φ to generate responses that have higher reward scores. The second term represents a per-token KL reward controlled by coefficientβbetweenπ RL φ andπ SFT to mitigate over-optimization toward the reward model during RL training. The third optional term provides regularization by encouraging responses with high probability from the pretraining dataset with coefficientγ. 3CONTRASTINSTRUCTIONS:MEASURING REWARD(IN-)CONSISTENCY Conventionally in RLHF, reward models are trained explicitly to distinguish the more vs. less fa- vorable responses in the context of an instruction (Eq. 1). One would expect an RM to consistently predict higher reward scores toward the more favorable instruction-response pairings. For instance, as the example in Figure 1 shows, an ideal reward should assign a higher score tor A appearing in response toI A , thanI B . In practice, however, RMs usually suffer from over-optimization towards the training distribution, leading toinconsistenciesbetween RM predictions vs. human preferences at inference time (Gao et al., 2022). To illustrate and measure reward inconsistency in RMs, we introduce CONTRASTINSTRUCTIONS. A CONTRASTINSTRUCTIONSbenchmark takes the form ofD=(I A i ,I B i ,r A i ,r B i ) N i=1 (see Fig. 1). Each test instance consists of instruction-response pairs(I A ,r A )and(I B ,r B ). To make the benchmark meaningfully challenging, we sampleI A andI B such that the two are lexically similar instructions with different semantics (details later in §3.1).r A andr B are the human-preferred re- sponses toI A andI B , respectively. Conceptually, a consistent RM should be able to identify pairs of corresponding instruction-response. Concretely, this means assuming higher score to(I A ,r A )than (I A ,r B )or(I B ,r A ). There are two ways one can quantify such notions of reward consistency: •Response Consistency (C res ): Given one of the instructionsI A , we measure if a RM (R θ ) can identify the corresponding responser A by assigning higher rewards tor A overr B . C res = 1 |D| X (I A i ,I B i ,r A i ,r B i )∈D 1 R θ I A i ◦r A i >R θ I A i ◦r B i .(2) •Instruction Consistency (C ins ): Similarly, given one of the responsesr A , we measure if a RM (R θ ) can assign higher rewards to its corresponding instructionI A over the distractorI B . 3 Preprint C ins = 1 |D| X (I A i ,I B i ,r A i ,r B i )∈D 1 R θ I A i ◦r A i >R θ I B i ◦r A i .(3) It is worth noting that with the standard learning objective (Eq. 1), RMs are explicitly trained to rank responses but not instructions, and soC res conceptually resembles the RM learning objective, whileC ins does not. Therefore, we expect the standard RM to perform better under theC res metric compared toC ins , which we will further discuss in §4. 3.1CONSTRUCTINGCONTRASTINSTRUCTIONSAUTOMATICALLY CONTRASTINSTRUCTIONScan be automatically constructed from datasets that contain human preferences. Inspired by Gardner et al. (2020), we formulate examples in CONTRASTINSTRUC- TIONSsuch that the pair of instructionsI A ,I B are lexically similar, yet their ground truth responses are different. We expect aconsistentRM to recognize the nuanced semantic difference between the instructions, and recognize the corresponding answer to an instruction. Benchmark Construction.We adopt four open-source human preference datasets of various NLP tasks: STACKEXCHANGEfor question answering (Askell et al., 2021), WMT for machine transla- tion (Ma et al., 2019), REALSUMMfor text summarization (Bhandari et al., 2020), and TWITTER for paraphrase generation (Shen et al., 2022c). Each dataset features examples of an instruction comprised of both a task prompt and a task-specific input, and the corresponding responses ranked by human preference. Within each dataset, we sample pairs of similar instructions with sentence embedding model SimCSE (Gao et al., 2021). To ensure the instruction pairs are similar but not semantically equivalent, we keep only instruction pairs with cosine similarity within[0.75,0.9]. We show the statistics and examples of each resulting CONTRASTINSTRUCTIONSdataset in Table 1 Data Instructions(task prompt omitted)Responses S TACK I A : What are the three primary colors in the sub- tractive color models? r A : The subtractive primaries are cyan, ma- genta, and yellow. I B : What’re the three primary colors in the addi- tive color models? r B : They’re red, green, and blue. WMT I A : Mindestens 410 Menschen wurden bei einem durch ein starkes Erdbeben ausgel ̈ osten Tsunami in Indonesien get ̈ otet. r A : At least 410 people were killed in a tsunami triggered by a strong earthquake in Indonesia. I B : Das Erdbeben in der Provinz Zentralsulawesi hat 410 Menschen get ̈ otet. r B :The earthquake in Central Sulawesi province has killed 410 people. T WITTER I A : But my bro from the 757 EJ Manuel is the 1st QB gone. r A : My boy EJ Manuel being the 1st QB picked. I B : EJ Manuel will be the 1st QB taken in the 2013 NFL Draft. r B : EJ Manuel selected as the 1st QB in the 2013 NFL Draft. R EAL S UMM I A : Mary Ann Diano was left homeless and hope- less when the storms hit Staten Island, New York, in October 2012. (...) r A : Mary Ann Diano, 62, lost her home in Staten Island in October 2012.. (...) I B : Denise and Glen Higgs, from Braunton, De- von, had all but lost hope that they would ever be able to conceive after glen was made infertile due to cancer treatment. (...) r B : Denise and Glen Higgs lost hope after glen was made infertile due to cancer treat- ment, but the couple had a daughter Mazy af- ter IVF treatment. (...) Table 1: Statistics and examples of the four CONTRASTINSTRUCTIONSbenchmark datasets. 4INCONSISTENCY OF EXISTING REWARD MODELS With CONTRASTINSTRUCTIONS, we evaluate the consistency of RMs trained with the standard ranking objective (Eq. 1). 4 Preprint Experimental setup.We initialize RMs fromLLaMa-7Bcheckpoint (Touvron et al., 2023) and finetune on each of the four human preference datasets used to create our CONTRASTINSTRUC- TIONSbenchmark. We also finetune a multi-task version (Mishra et al., 2022) on the mixture of four human preference datasets. We use the following configurations for RM training for both single-task and multi-task settings. Due to resource constraints, we adopt the Low-Rank Adaptor (LoRA) (Hu et al., 2021) for training. We use the AdamW optimizer and set a learning rate of 2e-5. For multi- task training, we combine the training set from selected benchmarks and train using LoRA with a learning rate of 3e-5. Finally, we report human performance resulting from the majority vote of three human annotators (the authors) on 100 randomly selected data points. ModelC res C ins Random50.0 50.0 LLaMa-7B(Single-Task)53.6 49.4 LLaMa-7B(Multi-Task)53.0 48.8 Human(Estimated)82.8 81.7 Table 2:C res andC ins (Eq.2, Eq.3) averaged across four benchmarks. Standard RM train- ing performs near random performance, in- dicating a major inconsistency gap between fine-tuned reward models and humans. Inconsistency of RMs on CONTRASTINSTRUC- TIONS.Table 2 summarizes the results of the fine- tuned RMs averaged on the four CONTRASTIN- STRUCTIONSbenchmarks. We observe that with a relatively large 7B parameter RM, it still performs close to random guessing in terms of both responses (C res ) and instruction consistency (C ins ). This trend can be seen on per-dataset as well, in Table 3. In our setting, multi-task training does not seem to help (no cross-task transfer), possibly due to the limited commonalities among these tasks. The reward mod- els are slightly better in terms ofC res compared to C ins , which fits our expectation – the RM learning objective (Eq.1) is more similar toC res than it is toC ins . The estimated human performance shows a wide gap compared to the RM performance on CONTRASTINSTRUCTIONS, suggesting reward inconsistency can be attributed to the standard practice of reward modeling. 5ENHANCING REWARD MODEL CONSISTENCY Having identified in §4 the inconsistencies of reward models, we now present methods to address these issues. Specifically, we propose two solutions: one to be applied during training (§5.1) and another during inference (§5.2), which does not impose additional computing costs. It’s important to note that these techniques are designed to be agnostic to CONTRASTINSTRUCTIONS’s format and setup, thereby minimizing the negative impact of Goodhart’s law (Manheim & Garrabrant, 2018). 5.1CONSISTENCY-INDUCINGTRAINING VIACONVEXDA To mitigate the effect of over-optimization during RM training, we designCONVEXDA, a lightweight, efficient data augmentation technique, that neither modifies the RM learning objec- tive nor increases the overall training cost. The high-level idea is to create various perturbations of the original input through data augmentation and select the most representative one to replace the original input example during training. Approach.Given a human preference example(I,r + ,r − ), we use an off-the-shelf textual data augmentation tool (Ma, 2019) to substitute words in the responses with synonyms according to WordNet (Miller, 1995) or PPDB (Ganitkevitch et al., 2013). For each example, we generateN= 5 augmented versionsI,r + j ,r − j N j=1 . As using allNdata points for training would incur extra cost, we draw inspiration from previous studies (Chen et al., 2010; Sener & Savarese, 2018; Rajput et al., 2019; Agarwal et al., 2020) and select one data point among five that serve as a vertex of a convex hull in the embedding space. This geometric approach ensures that we focus on the most critical points within the set of augmented data points. The strategy keeps the training efficiency on par with the standard RM training while offering the advantages of data augmentation. We use the SimCSE model (Gao et al., 2021) to embed each original and augmented response to a 768-dimensional vector. In principle, to construct a convex hull in ak-dimensional embedding space, we need at leastk+1data points. For such reason, we apply Principal Component Analysis (PCA) to reduce the dimensional of the embedding to 2, We identify and select the example that act 5 Preprint METRICMODELMETHODSTACKWMTTWITTERREALSUMMAVG. C res Human81.089.083.078.082.8 LLaMa-7B Standard RM50.1 ♠ 56.460.147.753.6 Our Approach53.660.462.952.657.3 Ablating “Our Approach” –CONVEXDA52.1(-1.5)58.4(-2.0)61.7(-1.2)50.2(-2.4)55.6(-1.7) –REWARDFUSION53.1(-0.5)59.9(-0.5)62.1(-0.8)51.2(-1.4)56.5(-0.8) C ins Human82.184.381.279.281.7 LLaMa-7B Standard RM48.2 ♠ 48.448.552.649.4 Our Approach53.453.952.556.554.1 Ablating “Our Approach” –CONVEXDA52.0(-1.4)52.1(-1.8)53.6(+1.1)55.1(-1.4)53.2(-0.9) –REWARDFUSION52.9(-0.5)53.6(-0.3)52.5(0.0)55.7(-0.8)53.7(-0.4) Table 3: The evaluation of reward consistency from standard RM training vs. our approach across four different CONTRASTINSTRUCTIONSbenchmarks, plus ablation studies onCONVEXDAand REWARDFUSION.♠indicates results evaluated on the official checkpoint from Stack-LLaMa; as the corner points of a low-dimensional convex hull, and replace the original input with it during RM training. 5.2CONSISTENCY-INDUCINGINFERENCE VIAREWARDFUSION During RM inference, we introduceREWARDFUSION. The method takes inspiration from (Zhao & Cho, 2019), where we first identify sentences similar to the response from the training corpus, and then use the weighted average reward score across the target response and the retrieved training samples as a better estimate for the reward score. Approach.At the inference time, given a pair of instructions and responseI◦r, we again use the SimCSE model to retrieve a set of similar training examplesI◦r ∗ that have cosine similarity to I◦rover a thresholdδfrom the training corpus. Then, we take the weighted average of the reward scores of the original plus retrieved training examplesX= (I◦r)∪I◦r ∗ : R fusion (X) = X x∈X sim(x,I◦r) P x ′ ∈X sim(x ′ ,I◦r) R θ (x).(4) The thresholdδ= 0.95is selected based on averaged performance on development sets. The embedding retrieval process is instantiated with Faiss (Johnson et al., 2019). 5.3EXPERIMENTS ANDRESULTS We conduct experiments to evaluate the effectiveness of both techniques in enhancing the consis- tency of RM on the four CONTRASTINSTRUCTIONSbenchmarks. We follow the same experimental settings detailed in §4. We start with single-task RMs trained on each human preference dataset and applyCONVEXDAandREWARDFUSION. Findings.We observe the following based on the results in Table 3. (1) BothCONVEXDAand REWARDFUSIONeffectively enhance the consistency of the reward model. (2) The combination of these two techniques works best, highlighting their complementarity in RM training and inference for enhancing consistency. (3) Despite the improvements brought by the two techniques, the overall performance on the CONTRASTINSTRUCTIONSbenchmark remains limited. RMs with enhanced consistency from the two techniques still fall behind human performance by a large margin, demon- strating the challenges of addressing RM’s consistency. We defer further details to Appendix F. 6TRICKLE-DOWNEFFECT OFREWARDINCONSISTENTENCY ONRLHF Next, we explore the merits of having a more consistent RM in downstream RLHF training by comparing RLHF-trained language models with a standard (§4) vs. more consistent RM (§5). 6 Preprint Experimental setup.We follow the overall experimental setup outlined in StackLLaMa (Beech- ing et al., 2023). For the experiments, we useLLaMa-7Bto initialize the supervised finetuning (SFT), reward, and policy models. We train the models on StackExchange, which is segmented into SFT, RM, and RL datasets. More implementation details can be found Appendix C. We train two RLHF models, one with the standard RM, and the other with RM finetuned on examples with CONTRASTINSTRUCTIONSformat sampled from StackExchange training split. We denote this as the CONTRASTFt. RM. We conduct a human evaluation on 500 questions randomly sampled from the test split of StackExchange. Unsure 26.7% Consistent RM wins 42.2% Original RM wins 31.1% Pairwise preference Unsure 20.0% Accept 52.0% Reject 28.0% Individual acceptability of Consistent RM Unsure 30.0% Accept 44.0% Reject 26.0% Individual acceptability of Original RM Figure 2: Human evaluation of the RLHF models trained with the consistent vs. original RM respectively.Leftshows the pair- wise preference, andMiddle/Rightshows individual accept- ability of using consistent RM / original RM. Human evaluation.To assess the general quality of the re- sponses, we conduct two evalua- tions on responses generated by the two RLHF models. We first ask human raters to assess the individual acceptabilityof each model’s response. This process involves a three-way judgment Accept,Reject,Unsure. A response is deemed acceptable only if it adequately addresses the question in the prompt; exhibits no significant error; and contains no redundant information. In addi- tion, we ask the human rater to annotate thepairwise preferencebetween the two responses. We ask human raters to compare the outputs of two models and determine which model’s output is preferred. We also provide raters the option to indicate a tie between the responses. The human evaluation re- sults are shown in Figure 2. We observe that with the more consistent RM, the downstream RLHF model demonstrates higher generation quality. Reward ModelOne-turnMulti-turn Standard3.442.39 CONTRASTFt.3.893.18 Table 4: The evaluation results on the MT-BENCH(Zheng et al., 2023). Score ranges from 1 (worst) to 10 (best). Automatic evaluation.We use MT-bench 2 (Zheng et al., 2023) for automatic evaluation. MT-Bench is a benchmark featuring challengingONE-TURNorMULTI- TURNreasoning and instruction-following examples, where the model responses are graded by GPT-4 on a scale of 1 (worst) to 10 (best). Overall, the results are shown in Table 4. We observe that more consistent RMs lead to RLHF models generating more preferable re- sponses under both settings. Reward ModelRelevanceUsefulnessFactuality Standard92.4%50.9%49.1% CONTRASTFt.92.2%57.8%50.0% Table 5: Fine-grained human evaluation of RLHF model response along three axes –relevance, useful- ness, andfactuality. Each axis are scored on a scale of0% :No; 50% :Partial/Unsure; 100% :Yes. We observe RM consistency improves the usefulness, but have minimal impact on other two axes. Reward consistency⇒More useful re- sponses.To understand where the im- provement lies, we follow criteria from Malaviya et al. (2023) and ask human raters to assess therelevance, usefulness andfactualityof RLHF model responses. Relevanceindicates whether the response is topically relevant to the instruction. Usefulnessindicates whether a response serves as a useful and direct answer to the instruction.Factualityindicates the factu- ality of responses based on the raters’ judgment, for which minimal browsing on the internet is allowed if needed. The results are shown in Table 5. We observe a statistically significant improve- ment in usefulness (p <0.01with paired t-test). Otherwise, we see minimal impact on the relevance or factuality of responses. We show a pair of example responses in Table 6, and more examples can be found in Appendix G. Overall, we observe that the usefulness of the responses from the RLHF model with standard RM tends to decrease further into the response, while the issue is mitigated with a more consistent RM. 2 https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboard 7 Preprint [Question]: Is ”lemonade” countable or uncountable? Could I say the following sentence? “Could you please bring me a lemonade”? Or *must* I say “a glass of lemonade”? [Consistent RM]: ”Lemonade” is usually uncountable, but it can be used as a countable noun to mean ”a glass of lemonade”. It is uncom- mon, but possible, to treat ”lemonade” as a countable noun. This applies not only to lemonade but to most beverages. [Standard RM]: Answer: The word lemonade is uncountable, like the word milk, and you can’t refer to a single glass of it. It would be fine to say: Could you please bring me a glass of lemonade? However,inthecontextofarestaurantoracafeteria,it’smoreusualtoordera lemonadewithyourmeal.Inthatcase,thecorrectsentencewouldbe:Couldyoupleasebringmealemonadewithmymeal? Table 6: Example responses from two RLHF models. More examples in Appendix G. 7DISCUSSION 7.1DOES THE STANDARDRMEVALUATION CAPTURE REWARD(IN-)CONSISTENCY? MethodC res C ins RMEVAL Standard RM53.649.475.6 CONVEXDA+REWARDFUSION57.3(+3.7)54.1(+4.7)75.9(+0.3) CONTRASTFt.64.2(+11.2)62.9(+13.5)76.6(+1.0) Table 7: The improvements on the reward consistency (C res ,C ins ) are not reflected in the RMs’ original response ranking metric (RMEVAL). CONTRASTFt. refers to model finetuned on train- ing examples with CONTRASTINSTRUCTIONSformat. A reader may wonder whether the evaluation via the original datasets (task prompts and re- sponses preferred by humans) is sufficient to reflect RMs’ level of consistency. (Alternatively, is CONTRASTINSTRUCTIONSre- ally necessary?) To illustrate this, we evaluate RMs’ average performance on the test set of the four human preference datasets (§3). Table 7 shows the results. We observe that despite the improvements in termsC res andC ins of our CONVEXDA + REWARDFUSIONapproach, the level of improvements are not reflected in the original RM evaluation metric (RMEVAL). To fur- ther validate this observation, we evaluate the CONTRASTFt. RMs, i.e. RMs finetuned on training examples with CONTRASTINSTRUCTIONSformat sampled from each of the four human prefer- ence datasets (§6). We observe a similar pattern that despite the large (but unfair) improvements onC res andC ins , the performance on RMEVALremains stale. These findings suggest that beyond the issue of RM over-optimization, we potentially need to rethink the current standard setup for preference-based reward modeling, which might be the inherent cause of reward inconsistency. 7.2ACLOSER LOOK TO WHY(OR WHERE)REWARD MODELS ARE INCONSISTENT 0.00.20.40.60.81.0 Human Score 1.0 0.5 0.0 0.5 1.0 Reward Score Spearman correlation = 0.456 0.00.20.40.60.81.0 Human Score 1.0 0.5 0.0 0.5 1.0 Reward Score Spearman correlation = 0.597 0.20.40.60.81.0 Human Score 1.0 0.5 0.0 0.5 1.0 Reward Score Spearman correlation = 0.361 0.20.40.60.81.0 Human Score 1.0 0.5 0.0 0.5 1.0 Reward Score Spearman correlation = 0.423 Figure 3: The correlation between RMs’ vs. humans’ reward score on TWITTER() and REALSUMM(). Left plots show results from standard RM; Right plots show RM enhanced with CONVEXDA and REWARDFUSION. The underlying motivation behind RM consistency closely resem- bles model calibration (Guo et al., 2017), i.e. in the ideal case, we would expect the reward scores from an RM to be perfectly cali- brated and correlated with human preferences. In such a sense, re- ward consistency serves as a good indicator and proxy measure for the correlation between the re- ward score from RMs vs.hu- man judgments.In Figure 3, we show the reward score corre- lation on two human preference datasets, TWITTERand REAL- SUMM, where multiple candidate responses with their correspond- ing human-rated quality (scaled between 0-1) are provided for each instruction. We compare the reward score correlation from the standard reward model (i.e., left two plots in Fig 3) vs. from the RM enhanced with CONVEXDA and REWARDFUSION(right two plots). Generally, we observe that the more consistent RM yields a higher reward score correlation with humans. While standard RMs can achieve a relatively good 8 Preprint correlation with human preferences, the reward score exhibits higher variance, especially when it comes to pairs of responses that are closer in terms of human score. This echoes our observations with respect to standard RM learning objectives and evaluations. From RMs’ perspective, correctly distinguishing between a clearly good vs. bad response is easy. Optimizing or evaluating against such would not indicate RMs’ alignment with human preference. Including such easy pairs for evaluation would likely lead to overestimation of RMs performance, which further motivates CON- TRASTINSTRUCTIONSas a benchmarking strategy for RMs, as well as a potential training strategy, as we demonstrate in §6. 7.3CONSISTENCY CHECK BEYONDCONTRASTINSTRUCTIONS CONTRASTINSTRUCTIONSprovides an automatic, efficient, and intuitive evaluation framework for assessing the consistency of preference-based reward modeling. Nonetheless, it does not necessarily encompass all possible phenomena with respect to RM consistency or robustness. For such reason, we conduct preliminary analysis in adversarial and backdoor attacks (Chen et al., 2017; Shen et al., 2023) for RM. The details of the experiments are included in Appendix D and Appendix E. We observe that overall RMs suffer from a high attack success rate and exhibit vulnerability to both adversarial and backdoor attacks. The findings suggest the potential implication of reward inconsis- tency on RM and subsequently RLHF safety. 7.4BROADERIMPACT ANDLIMITATIONS Despite the widespread interest in RLHF within the research community, our understanding so far on “what type of reward modeling would most benefit RLHF” remains fairly limited. We argue that this can in part be attributed to (1) the lack of a properintrinsicevaluation metric on RM itself, as we see in §7.1; and subsequently (2) RM evaluations relying heavily onextrinsicRLHF evaluations. Because RLHF evaluations often rely on human annotations (Wu et al., 2023; Lee et al., 2023), which can be costly and unreliable, they offer limited insights on how we should make research progress on reward modeling. Even though CONTRASTINSTRUCTIONSdo not necessarily assess all capabilities that a RM requires, we hope that it works as a sensible and scalable intrinsic evaluation metric that facilitates future development of better or alternative reward modeling strategies. 8RELATED WORK Consistency in NLPConsistency has been a long-standing topic in NLP research, in previous works, consistency of an NLP mode is defined the invariance of its behavior under meaning- preserving alternations (Ribeiro et al., 2020; Elazar et al., 2021; Goel et al., 2021; Wang et al., 2022b), and several works have explored the consistency in various tasks (Du et al., 2019; Ribeiro et al., 2019; Alberti et al., 2019; Camburu et al., 2020; Asai & Hajishirzi, 2020; Kassner et al., 2021; Chen et al., 2021a; Elazar et al., 2021; Mitchell et al., 2022). In the context of reward modeling, we study the consistency with respect to human preference instead. Reinforcement Learning from Human FeedbackRLHF (Ouyang et al., 2022; OpenAI, 2023) has merged as a popular technique for aligning LLMs with human preferences (Nakano et al., 2021; Glaese et al., 2022; Bai et al., 2022b; Ouyang et al., 2022; Bai et al., 2022a). The RLHF method in- volves learning a reward function on human annotations to proxy human preferences, and optimizing language models through reinforcement learning techniques such as Proximal Policy Optimization (PPO) (Schulman et al., 2017). A key implication of RLHF research is to align LLMs with helpful, honest, and harmless human feedback (Askell et al., 2021), (Glaese et al., 2022). 9CONCLUSION Through the lens of CONTRASTINSTRUCTIONS, we uncover and study the phenomena of reward inconsistency in reward modeling for RLHF. While our study suggests one perspective and direction on improving RM for RLHF, the question of “what type of reward modeling would most benefit RLHF” remains wide open. We hope that this paper’s findings would facilitate future research and evaluation on this problem. 9 Preprint REFERENCES Sharat Agarwal, Himanshu Arora, Saket Anand, and Chetan Arora. Contextual diversity for active learning. InThe European Conference on Computer Vision(ECCV), p. 137–153. Springer, 2020. URLhttps://arxiv.org/abs/2008.05723. Chris Alberti, Daniel Andor, Emily Pitler, Jacob Devlin, and Michael Collins. Synthetic QA cor- pora generation with roundtrip consistency. In Anna Korhonen, David R. Traum, and Llu ́ ıs M ` arquez (eds.),Proceedings of the 57th Conference of the Association for Computational Lin- guistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, p. 6168– 6173. Association for Computational Linguistics, 2019. doi: 10.18653/v1/p19-1620. URL https://doi.org/10.18653/v1/p19-1620. Moustafa Alzantot, Yash Sharma, Ahmed Elgohary, Bo-Jhang Ho, Mani B Srivastava, and Kai-Wei Chang. Generating natural language adversarial examples. InConference on Empirical Methods in Natural Language Processing(EMNLP), 2018. URLhttps://arxiv.org/abs/1804. 07998. Akari Asai and Hannaneh Hajishirzi. Logic-guided data augmentation and regularization for con- sistent question answering. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault (eds.),Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, p. 5642–5650. Association for Computational Linguis- tics, 2020. doi: 10.18653/v1/2020.acl-main.499. URLhttps://doi.org/10.18653/v1/ 2020.acl-main.499. Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, et al. A general language assistant as a laboratory for alignment.arXiv preprint arXiv:2112.00861, 2021. URLhttps://arxiv.org/abs/ 2112.00861. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback.arXiv preprint arXiv:2204.05862, 2022a. URLhttps://arxiv.org/abs/2204.05862. Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback.arXiv preprint arXiv:2212.08073, 2022b.URLhttps: //arxiv.org/abs/2212.08073. Edward Beeching, Younes Belkada, Kashif Rasul, Lewis Tunstall, Leandro von Werra, Nazneen Ra- jani, and Nathan Lambert. Stackllama: An rl fine-tuned llama model for stack exchange question and answering, 2023. URLhttps://huggingface.co/blog/stackllama. Yonatan Belinkov and Yonatan Bisk.Synthetic and natural noise both break neural machine translation. InInternational Conference on Learning Representations(ICLR), 2018. URL https://arxiv.org/abs/1711.02173. Manik Bhandari, Pranav Narayan Gour, Atabak Ashfaq, Pengfei Liu, and Graham Neubig. Re- evaluating evaluation in text summarization. InConference on Empirical Methods in Natural Language Processing(EMNLP), p. 9347–9359, 2020. URLhttps://arxiv.org/abs/ 2010.07100. Oana-Maria Camburu, Brendan Shillingford, Pasquale Minervini, Thomas Lukasiewicz, and Phil Blunsom. Make up your mind! adversarial generation of inconsistent natural language ex- planations. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault (eds.),Pro- ceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, p. 4157–4165. Association for Computational Linguistics, 2020. doi: 10.18653/v1/2020.acl-main.382.URLhttps://doi.org/10.18653/v1/2020. acl-main.382. 10 Preprint Sihao Chen, Fan Zhang, Kazoo Sone, and Dan Roth. Improving faithfulness in abstractive summa- rization with contrast candidate generation and selection. InProceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan- guage Technologies, p. 5935–5941, Online, June 2021a. Association for Computational Linguis- tics. doi: 10.18653/v1/2021.naacl-main.475. URLhttps://aclanthology.org/2021. naacl-main.475. Xiaoyi Chen, Ahmed Salem, Michael Backes, Shiqing Ma, and Yang Zhang. Badnl: Backdoor attacks against NLP models. InICML Workshop on Adversarial Machine Learning, 2021b. URL https://arxiv.org/abs/2006.01043. Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song. Targeted backdoor attacks on deep learning systems using data poisoning.arXiv preprint arXiv:1712.05526, 2017. Yutian Chen, Max Welling, and Alex Smola. Super-samples from kernel herding. InConference on Uncertainty in Artificial Intelligence(UAI), p. 109–116, 2010. URLhttps://arxiv.org/ abs/1203.3472. Jiazhu Dai, Chuanshuai Chen, and Yufeng Li. A backdoor attack against lstm-based text classifi- cation systems.IEEE Access, 7:138872–138878, 2019. URLhttps://arxiv.org/abs/ 1905.12457. Hanze Dong, Wei Xiong, Deepanshu Goyal, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. Raft: reward ranked finetuning for generative foundation model alignment. arXiv preprint arXiv:2304.06767, 2023. URLhttps://arxiv.org/abs/2304.06767. Xinya Du, Bhavana Dalvi Mishra, Niket Tandon, Antoine Bosselut, Wen-tau Yih, Peter Clark, and Claire Cardie. Be consistent! improving procedural text comprehension using label consistency. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.),Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), p. 2347–2356. Association for Computational Linguistics, 2019. doi: 10.18653/ v1/n19-1244. URLhttps://doi.org/10.18653/v1/n19-1244. Steffen Eger, G ̈ ozde G ̈ ul S ̧ ahin, Andreas R ̈ uckl ́ e, Ji-Ung Lee, Claudia Schulz, Mohsen Mesgar, Krishnkant Swarnkar, Edwin Simpson, and Iryna Gurevych. Text processing like humans do: Visually attacking and shielding nlp systems. InConference of the North American Chapter of the Association for Computational Linguistics(NAACL), p. 1634–1647, 2019. URLhttps: //arxiv.org/abs/1903.11508. Yanai Elazar, Nora Kassner, Shauli Ravfogel, Abhilasha Ravichander, Eduard H. Hovy, Hinrich Sch ̈ utze, and Yoav Goldberg. Erratum: Measuring and improving consistency in pretrained lan- guage models.Trans. Assoc. Comput. Linguistics, 9:1407, 2021. doi: 10.1162/tacl\ x\00455. URLhttps://doi.org/10.1162/tacl_x_00455. Juri Ganitkevitch, Benjamin Van Durme, and Chris Callison-Burch. Ppdb: The paraphrase database. InProceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, p. 758–764, 2013. URLhttps: //aclanthology.org/N13-1092/. Ji Gao, Jack Lanchantin, Mary Lou Soffa, and Yanjun Qi. Black-box generation of adversarial text sequences to evade deep learning classifiers. InIEEE Security and Privacy Workshops (SPW), 2018. URLhttps://arxiv.org/abs/1801.04354. Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization.arXiv preprint arXiv:2210.10760, 2022. URLhttps://arxiv.org/abs/2210.10760. Tianyu Gao, Xingcheng Yao, and Danqi Chen. Simcse: Simple contrastive learning of sentence embeddings.arXiv preprint arXiv:2104.08821, 2021. URLhttps://arxiv.org/abs/ 2104.08821. 11 Preprint Matt Gardner, Yoav Artzi, Victoria Basmova, Jonathan Berant, Ben Bogin, Sihao Chen, Pradeep Dasigi, Dheeru Dua, Yanai Elazar, Ananth Gottumukkala, et al. Evaluating models’ local de- cision boundaries via contrast sets. InConference on Empirical Methods in Natural Language Processing(EMNLP)- Findings, 2020. URLhttps://arxiv.org/abs/2004.02709. Amelia Glaese, Nat McAleese, Maja Trbacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Maribeth Rauh, Laura Weidinger, Martin Chadwick, Phoebe Thacker, et al. Improving alignment of di- alogue agents via targeted human judgements.arXiv preprint arXiv:2209.14375, 2022. URL https://arxiv.org/abs/2209.14375. Dongyoung Go, Tomasz Korbak, Germ ́ an Kruszewski, Jos Rozen, Nahyeon Ryu, and Marc Dymet- man. Aligning language models with preferences through f-divergence minimization.arXiv preprint arXiv:2302.08215, 2023. URLhttps://arxiv.org/abs/2302.08215. Karan Goel, Nazneen Fatema Rajani, Jesse Vig, Zachary Taschdjian, Mohit Bansal, and Christo- pher R ́ e.Robustness gym: Unifying the nlp evaluation landscape.InProceedings of the 2021 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies: Demonstrations, p. 42–55, 2021. URLhttps: //arxiv.org/abs/2101.04840. Tianyu Gu, Brendan Dolan-Gavitt, and Siddharth Garg. Badnets: Identifying vulnerabilities in the machine learning model supply chain.arXiv preprint arXiv:1708.06733, 2017. URLhttps: //arxiv.org/abs/1708.06733. Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. InInternational conference on machine learning, p. 1321–1330. PMLR, 2017. URL https://arxiv.org/pdf/1706.04599.pdf. Xuanli He, Lingjuan Lyu, Lichao Sun, and Qiongkai Xu. Model extraction and adversarial trans- ferability, your bert is vulnerable!InConference of the North American Chapter of the Association for Computational Linguistics(NAACL), p. 2006–2012, 2021. URLhttps: //arxiv.org/abs/2103.10013. Donald Joseph Hejna I and Dorsa Sadigh. Few-shot preference learning for human-in-the-loop RL. InConference on Robot Learning(CoRL), p. 2014–2025, 2023. URLhttps://arxiv. org/abs/2212.03363. Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. InInternational Conference on Learning Representations(ICLR), 2021. URLhttps://arxiv.org/abs/2106.09685. Po-Sen Huang, Robert Stanforth, Johannes Welbl, Chris Dyer, Dani Yogatama, Sven Gowal, Krish- namurthy Dvijotham, and Pushmeet Kohli. Achieving verified robustness to symbol substitutions via interval bound propagation. InConference on Empirical Methods in Natural Language Pro- cessing(EMNLP), p. 4083–4093, 2019. URLhttps://arxiv.org/abs/1909.01492. Robin Jia and Percy Liang. Adversarial examples for evaluating reading comprehension systems. InEMNLP, 2017. URLhttps://arxiv.org/abs/1707.07328. Robin Jia, Aditi Raghunathan, Kerem G ̈ oksel, and Percy Liang. Certified robustness to adversar- ial word substitutions. InConference on Empirical Methods in Natural Language Processing (EMNLP), 2019. URLhttps://arxiv.org/abs/1909.00986. Di Jin, Zhijing Jin, Joey Tianyi Zhou, and Peter Szolovits. Is bert really robust? a strong baseline for natural language attack on text classification and entailment. InConference on Artificial Intelligence(AAAI), 2020. URLhttps://arxiv.org/abs/1907.11932. Jeff Johnson, Matthijs Douze, and Herv ́ e J ́ egou. Billion-scale similarity search with GPUs.IEEE Transactions on Big Data, 7(3):535–547, 2019. URLhttps://arxiv.org/abs/1702. 08734. Nora Kassner, Oyvind Tafjord, Hinrich Schutze, and Peter Clark. Enriching a model’s notion of belief using a persistent memory.arXiv preprint arXiv:2104.08401, 2021. URLhttps:// arxiv.org/abs/2104.08401. 12 Preprint Tomasz Korbak, Kejian Shi, Angelica Chen, Rasika Bhalerao, Christopher L Buckley, Jason Phang, Samuel R Bowman, and Ethan Perez. Pretraining language models with human pref- erences.arXiv preprint arXiv:2302.08582, 2023. URLhttps://openreview.net/pdf? id=AT8Iw8KOeC. Thai Le, Noseong Park, and Dongwon Lee. A sweet rabbit hole by darcy: Using honeypots to detect universal trigger’s adversarial attacks. InAnnual Meeting of the Association for Computational Linguistics(ACL), p. 3831–3844. Association for Computational Linguistics (ACL), 2021. URL https://arxiv.org/abs/2011.10492. Harrison Lee, Samrat Phatale, Hassan Mansoor, Kellie Lu, Thomas Mesnard, Colton Bishop, Victor Carbune, and Abhinav Rastogi. Rlaif: Scaling reinforcement learning from human feedback with ai feedback.arXiv preprint arXiv:2309.00267, 2023. URLhttps://arxiv.org/pdf/ 2309.00267.pdf. Ximing Lu, Sean Welleck, Jack Hessel, Liwei Jiang, Lianhui Qin, Peter West, Prithviraj Am- manabrolu, and Yejin Choi. Quark: controllable text generation with reinforced unlearning. Advances in neural information processing systems, 35:27591–27609, 2022. URLhttps: //arxiv.org/abs/2205.13636. Edward Ma. NLP augmentation, 2019. URLhttps://github.com/makcedward/nlpaug. Qingsong Ma, Johnny Wei, Ond ˇ rej Bojar, and Yvette Graham. Results of the WMT19 metrics shared task: Segment-level and strong MT systems pose big challenges. InConference on Machine Translation(WMT), 2019. URLhttps://w.aclweb.org/anthology/W19-5302. Chaitanya Malaviya, Subin Lee, Sihao Chen, Elizabeth Sieber, Mark Yatskar, and Dan Roth. Ex- pertqa: Expert-curated questions and attributed answers.arXiv preprint arXiv:2309.07852, 2023. David Manheim and Scott Garrabrant. Categorizing variants of goodhart’s law.arXiv preprint arXiv:1803.04585, 2018. URLhttps://arxiv.org/abs/1803.04585. George Miller. Wordnet: a lexical database for english.Communications of the ACM, 38(11):39–41, 1995. URLhttps://dl.acm.org/doi/10.1145/219717.219748. Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. Cross-Task Generaliza- tion via Natural Language Crowdsourcing Instructions. InAnnual Meeting of the Association for Computational Linguistics(ACL), 2022. URLhttps://arxiv.org/abs/2104.08773. Eric Mitchell, Joseph J. Noh, Siyan Li, William S. Armstrong, Ananth Agarwal, Patrick Liu, Chelsea Finn, and Christopher D. Manning. Enhancing self-consistency and performance of pre-trained language models through natural language inference. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.),Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, p. 1754–1768. Association for Computational Linguistics, 2022. doi: 10.18653/v1/2022. emnlp-main.115. URLhttps://doi.org/10.18653/v1/2022.emnlp-main.115. Maximilian Mozes, Pontus Stenetorp, Bennett Kleinberg, and Lewis Griffin. Frequency-guided word substitutions for detecting textual adversarial examples. InConference of the European Chapter of the Association for Computational Linguistics(EACL), p. 171–186, 2021. URL https://arxiv.org/abs/2004.05887. Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christo- pher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al.WebGPT: Browser- assisted question-answering with human feedback.arXiv preprint arXiv:2112.09332, 2021. URL https://arxiv.org/abs/2112.09332. OpenAI. GPT-4 Technical Report, 2023. URLhttps://arxiv.org/abs/2303.08774. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training Language Models to Fol- low Instructions with Human Feedback. InAdvances in Neural Information Processing Systems (NeurIPS), 2022. URLhttps://arxiv.org/abs/2203.02155. 13 Preprint Fanchao Qi, Yangyi Chen, Mukai Li, Yuan Yao, Zhiyuan Liu, and Maosong Sun. Onion: A simple and effective defense against textual backdoor attacks.arXiv preprint arXiv:2011.10369, 2020. URLhttps://aclanthology.org/2021.emnlp-main.752/. Fanchao Qi, Yangyi Chen, Xurui Zhang, Mukai Li, Zhiyuan Liu, and Maosong Sun. Mind the style of text! adversarial and backdoor attacks based on text style transfer. InConference on Empirical Methods in Natural Language Processing(EMNLP), p. 4569–4580, 2021a. URL https://arxiv.org/abs/2110.07139. Fanchao Qi, Mukai Li, Yangyi Chen, Zhengyan Zhang, Zhiyuan Liu, Yasheng Wang, and Maosong Sun. Hidden killer: Invisible textual backdoor attacks with syntactic trigger.arXiv preprint arXiv:2105.12400, 2021b. URLhttps://arxiv.org/abs/2105.12400. Shashank Rajput, Zhili Feng, Zachary Charles, Po-Ling Loh, and Dimitris Papailiopoulos. Does data augmentation lead to positive margin? InInternational Conference on Learning Representations (ICLR), p. 5321–5330, 2019. URLhttps://arxiv.org/abs/1905.03177. Shuhuai Ren, Yihe Deng, Kun He, and Wanxiang Che. Generating natural language adversar- ial examples through probability weighted word saliency. InAnnual Meeting of the Associ- ation for Computational Linguistics(ACL), 2019. URLhttps://aclanthology.org/ P19-1103/. Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. Semantically equivalent adversarial rules for debugging nlp models. InAnnual Meeting of the Association for Computational Linguistics (ACL), 2018. URLhttps://aclanthology.org/P18-1079/. Marco T ́ ulio Ribeiro, Carlos Guestrin, and Sameer Singh. Are red roses red? evaluating consistency of question-answering models. In Anna Korhonen, David R. Traum, and Llu ́ ıs M ` arquez (eds.), Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, p. 6174–6184. Association for Computational Linguistics, 2019. doi: 10.18653/v1/p19-1621. URLhttps://doi.org/ 10.18653/v1/p19-1621. Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. Beyond accuracy: Be- havioral testing of nlp models with checklist. InAnnual Meeting of the Association for Com- putational Linguistics(ACL), p. 4902–4912, 2020. URLhttps://aclanthology.org/ 2020.acl-main.442/. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347, 2017. URLhttps://arxiv. org/abs/1707.06347. Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. InInternational Conference on Learning Representations(ICLR), 2018. URLhttps: //arxiv.org/abs/1708.00489. Lingfeng Shen, Haiyun Jiang, Lemao Liu, and Shuming Shi. Rethink stealthy backdoor attacks in natural language processing.arXiv preprint arXiv:2201.02993, 2022a. URLhttps://arxiv. org/abs/2201.02993. Lingfeng Shen, Shoushan Li, and Ying Chen. KATG: keyword-bias-aware adversarial text gen- eration for text classification. InConference on Artificial Intelligence(AAAI), 2022b. URL https://ojs.aaai.org/index.php/AAAI/article/view/21380. Lingfeng Shen, Lemao Liu, Haiyun Jiang, and Shuming Shi.On the evaluation metrics for paraphrase generation. InConference on Empirical Methods in Natural Language Processing (EMNLP), p. 3178–3190, 2022c. URLhttps://arxiv.org/abs/2202.08479. Lingfeng Shen, Ze Zhang, Haiyun Jiang, and Ying Chen. Textshield: Beyond successfully detecting adversarial sentences in text classification. InInternational Conference on Learning Representa- tions(ICLR), 2023. URLhttps://arxiv.org/abs/2302.02023. 14 Preprint Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth ́ e Lacroix, Baptiste Rozi ` ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. LLaMA: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023. URLhttps: //arxiv.org/abs/2302.13971. Boxin Wang, Chejian Xu, Xiangyu Liu, Yu Cheng, and Bo Li. Semattack: Natural textual attacks via different semantic spaces.arXiv preprint arXiv:2205.01287, 2022a. URLhttps://arxiv. org/abs/2205.01287. Xiaosen Wang, Jin Hao, Yichen Yang, and Kun He. Natural language adversarial defense through synonym encoding. InConference on Uncertainty in Artificial Intelligence(UAI), 2021. URL https://arxiv.org/abs/1909.06723. Xuezhi Wang, Haohan Wang, and Diyi Yang. Measure and improve robustness in NLP models: A survey. InConference of the North American Chapter of the Association for Computational Lin- guistics(NAACL), p. 4569–4586, 2022b. URLhttps://arxiv.org/abs/2112.08313. Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A Smith, Mari Ostendorf, and Hannaneh Hajishirzi. Fine-grained human feedback gives better rewards for language model training. InAdvances in Neural Information Processing Systems, 2023. URL https://arxiv.org/pdf/2306.01693.pdf. Xinze Zhang, Junzhe Zhang, Zhenhua Chen, and Kun He. Crafting adversarial examples for neu- ral machine translation. InAnnual Meeting of the Association for Computational Linguistics (ACL), p. 1967–1977, 2021. URLhttps://aclanthology.org/2021.acl-long. 153.pdf. Yuan Zhang, Jason Baldridge, and Luheng He. Paws: Paraphrase adversaries from word scrambling. InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), p. 1298–1308, 2019. URLhttps://arxiv.org/abs/1904.01130. Jake Zhao and Kyunghyun Cho.Retrieval-augmented convolutional neural networks against adversarial examples.InIEEE Conference on Computer Vision and Pat- tern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, p. 11563– 11571. Computer Vision Foundation / IEEE, 2019.doi:10.1109/CVPR.2019.01183. URLhttp://openaccess.thecvf.com/content_CVPR_2019/html/Zhao_ Retrieval-Augmented_Convolutional_Neural_Networks_Against_ Adversarial_Examples_CVPR_2019_paper.html. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena.arXiv preprint arXiv:2306.05685, 2023. URLhttps://arxiv.org/abs/ 2306.05685. Banghua Zhu, Jiantao Jiao, and Michael Jordan. Principled reinforcement learning with human feedback from pairwise ork-wise comparisons. InICLR Workshop on Mathematical and Empir- ical Understanding of Foundation Models, 2023. URLhttps://arxiv.org/abs/2301. 11270. Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences.arXiv preprint arXiv:1909.08593, 2019. URLhttps://arxiv.org/abs/1909.08593. 15 Preprint SUPPLEMENTARYMATERIAL AppendixContents Appendix AThe details of our CONTRASTINSTRUCTIONS Appendix BFull results on CONTRASTINSTRUCTIONS Appendix DAdversarial consistency of RM Appendix EBackdoor consistency of RM Appendix FAnalyses towardsCONVEXDAandREWARDFUSION Appendix CThe training details of the RLHF process? Appendix GCase studies ADETAILS OFCONTRASTINSTRUCTIONS Using a human preference dataset, we have divided it into training, development, and testing sets. The reward model is trained on the training set and ceases training once it attains optimal per- formance on the development set. Subsequently, it is evaluated on the test set. Our CONTRAST INSTRUCTIONSare built upon the test set in each benchmark. To ensure the retrieved instruction differs from the original one, we establish a similarity threshold range (e.g.,[0.8,0.9]). Only in- structions falling within this similarity range are retrieved. BFULL RESULTS ONCONTRASTINSTRUCTIONS DATAMODELMETHODSTACKWMTTWITTERREALSUMMAVG. ORIGINALTESTLLaMa-7B CONTRAST67.979.386.372.976.6 NONE67.3 ♠ 78.085.072.175.6 MULTI-TASK67.1 ♠ 77.684.671.775.3 CONVEXDA67.5 ♠ 78.285.372.375.8 REWARDFUSION67.3 ♠ 77.885.472.275.7 COMBINE67.5 ♠ 78.585.372.475.9 C res HumanN/A81.089.083.078.082.8 LLaMa-7B CONTRAST60.967.068.260.764.2 NONE50.1 ♠ 56.460.147.753.6 MULTI-TASK49.753.458.150.753.0 CONVEXDA53.159.962.151.256.5 REWARDFUSION52.158.461.750.255.6 COMBINE53.660.462.952.657.3 C ins HumanN/A82.184.381.279.281.7 LLaMa-7B CONTRAST58.464.567.761.062.9 NONE48.2 ♠ 48.448.552.649.4 MULTI-TASK48.048.147.851.448.8 CONVEXDA52.953.652.555.753.7 REWARDFUSION52.052.153.655.153.2 COMBINE53.453.952.556.554.1 Table 8: The evaluation of the reward model’s accuracy on different CONTRASTINSTRUCTIONS, whereinstruction (ins)andresponse (res)perturbations are introduced. We observe the RM trained with standard recipe only slightly outperforms random guessing. Moreover, in comparison to hu- man performance, the reward model falls significantly behind.♠means the official checkpoint from Stack-LLaMa.CONTRASTFt. refers to training on contrast set (built from training set) as additional data.MULTI-TASKrefers to training on the mixture of STACK, WMT, TWITTERand REALSUMM. 16 Preprint CCONFIGURATIONS OFRLTRAINING There are three models in the RL training stage: the SFT model, the reward model, and the policy model. For two groups of experiments, one uses the original RM (inconsistent), and the other one uses RM equipped withCONVEXDA. For the SFT model, both of the groups use the same SFT model, which is fine-tuned on StackExchange. We employ the Low-Rank Adaptor (LoRA) tech- nique (Hu et al., 2021) for training the reward model, with the same configuration in StackLLaMa (Beeching et al., 2023). The training is conducted in anINT8 style, due to our computational limits. The learning rate is set to1.4e−5, and we utilize the Adafactor optimizer. The default learning rate scheduler type is set to ‘linear’. The initial KL penalty coefficient is set as 0.2, and an adaptive KL control is used, with a linear scheduler. The pretraining gradient coefficientγis set to 0 for our experiments. DCONSISTENCY OF REWARD MODELING WHEN FACING ADVERSARIAL ATTACKS D.1BACKGROUND Adversarial attacks are independent of access to the RM’s training data. Within the context of an adversarial attack process, there are principally two actors: thevictim(reward model)R θ and the attack algorithmA. (1) Victim: Presented with a human-preference benchmark consisting of benign sentence pairs (those without adversarial perturbations), the victim reward modelR θ is trained on these pairs to differentiate the human-preferred response corresponding to the instructionI. (2) Attack algorithm: For a benign sentence pair(r A ,r B )with the correct labely A , the text attackA generates an adversarial sentencer A ∗ =A(r A )by adding subtle textual perturbations tor A . The goal of these perturbations is twofold: (1) to cause the reward model to issue an incorrect prediction y B , and (2) to ensure that the semantics betweenr A andr A ∗ remain closely aligned. Adversarial consistency refers to a model’s resilience against perturbations generated by adversarial attacks, which try to modify the texts with imperceptible perturbations. Coarsely, these attacks modify the text data at character level (Belinkov & Bisk, 2018; Eger et al., 2019; He et al., 2021), word level (Alzantot et al., 2018; Zhang et al., 2021; Wang et al., 2022a) or sentence level (Jia & Liang, 2017; Ribeiro et al., 2018; Zhang et al., 2019). Basically, the defense methods against adversarial attacks, which can be categorized into three paradigms: (1) model-enhancement-based (Le et al., 2021; Wang et al., 2021; Shen et al., 2022b), (2) certified-robustness-based (Huang et al., 2019; Jia et al., 2019), and (3) detection-based (Mozes et al., 2021; Le et al., 2021; Shen et al., 2023). We employ a range of existing textual attacks to assess the consistency of the Reward Model (RM). Our chosen attacks encompass three distinct levels of complexity, ranging from straightforward character manipulations to intricate word-level perturbations. For character-level manipulations, we employ methods such as VIPER (Eger et al., 2019) and DeepWordBug (Gao et al., 2018). At an intermediate, word-level complexity, we leverage techniques such as PWWS (Ren et al., 2019), Genetic Attack (GA) (Alzantot et al., 2018), and TextFooler (Jin et al., 2020). Additionally, we introduce a simplistic word-level adversarial method, designated as Vanilla Attack (VA), which ex- clusively utilizes word-level data augmentation for adversarial perturbations, eschewing additional algorithmic complexity. A detailed description of the Vanilla Attack (VA) is shown in algorithm 1. D.2EVALUATION Adversarial attacks originate from an iterative accumulation of adversarial perturbations. Accord- ingly, we introduce two distinct metrics to encapsulate the model’s response to the incorporation of each successive perturbation: adversarial accuracy, which refers to the RM accuracy on adversarial data, denoted asP. These metrics are engineered to illustrate the consistency of RM at both the reward score tier and the performance tier. They are formally defined as follows: P= Acc(D+ε i )(5) whereε i is the perturbation generated by attack in the i-th iteration. 17 Preprint Algorithm 1:Vanilla Attack (VA) Data:Reward modelR θ ; human-preferred responser j ; non-human-preferred responser k ; instructionI; data augmentation transformationG; Result:adversarial sentencer ∗ j andr ∗ k ; InitializeR j =R θ (I◦r j ),R k =R θ (I◦r k ),i= 0; InitializeL= max(len(r j ),len(r k )); whilei < Ldo r ∗ j =G(r j ); r ∗ k =G(r k ); R ∗ j =R θ (I◦r ∗ j ),R ∗ k =R θ (I◦r ∗ k ); ifR ∗ j < R j then r j =r ∗ j ;R j =R ∗ j ; ifR ∗ k > R k then r k =r ∗ k ;R k =R ∗ k ; ifR ∗ j < R ∗ k then returnadversarial sentencer ∗ j andr ∗ k ; end returnFailed; D.3RESULTS The results are shown in Figure 4. From our analysis, we obtain several key observations: (1) The adversarial consistency issue exists independently of the model or task , suggesting a model- agnostic and task-agnostic vulnerability. Empirically, every model from GPT2-0.1B to LLaMa-7B exhibited vulnerability to adversarial attacks. Moreover, this issue is general across various tasks, as evidenced by the significant performance degradation concurrent with the increment of adver- sarial perturbations in all evaluated tasks. (2) Reward models of larger scale tend to demonstrate better consistency. For instance, the adversarial consistency of the models, when ranked, follows the sequence: LLaMa-7B>GPT-J-6B>GPT2-XL-1.5B>GPT2-0.1B, which aligns with their size ranking. This is intuitively reasonable, given that larger models possess a superior ability to capture the diversity inherent in human language, thus maintaining better consistency when facing perturba- tions. 0.02.55.07.510.012.515.017.5 Perturbation count 20 30 40 50 60 Adversarial Performance Stack-Exchange(VA) LLaMa-7B GPT2 GPT2-XL GPT-J (a) Vanilla attack 02468 Perturbation count 20 30 40 50 60 Adversarial Performance Stack-Exchange(GA) LLaMa-7B GPT2 GPT2-XL GPT-J (b) Genetic attack 02468 Perturbation count 20 30 40 50 60 Adversarial Performance Stack-Exchange(PWWS) LLaMa-7B GPT2 GPT2-XL GPT-J (c) PWWS 02468 Perturbation count 20 30 40 50 60 Adversarial Performance Stack-Exchange(VIPER) LLaMa-7B GPT2 GPT2-XL GPT-J (d) VIPER 02468 Perturbation count 10 20 30 40 50 60 Adversarial Performance Stack-Exchange(DWG) LLaMa-7B GPT2 GPT2-XL GPT-J (e) DWG 02468 Perturbation count 20 30 40 50 60 Adversarial Performance Stack-Exchange(TextFooler) LLaMa-7B GPT2 GPT2-XL GPT-J (f) TextFooler Figure 4: Adversarial attack performance onthe StackExchange dataset: Performance decreases as perturbation from text attacks accumulates. Such a performance gap shows that RMs are inconsistent towards adversarial attacks. 18 Preprint 0.02.55.07.510.012.515.017.5 Perturbation count 10 20 30 40 50 60 70 80 Adversarial Performance WMT(VA) LLaMa-7B GPT2 GPT2-XL GPT-J (a) VA 02468 Perturbation count 10 20 30 40 50 60 70 80 Adversarial Performance WMT(GA) LLaMa-7B GPT2 GPT2-XL GPT-J (b) GA 02468 Perturbation count 10 20 30 40 50 60 70 80 Adversarial Performance WMT(PWWS) LLaMa-7B GPT2 GPT2-XL GPT-J (c) PWWS 02468 Perturbation count 10 20 30 40 50 60 70 80 Adversarial Performance WMT(VIPER) LLaMa-7B GPT2 GPT2-XL GPT-J (d) VIPER 02468 Perturbation count 10 20 30 40 50 60 70 80 Adversarial Performance WMT(DWG) LLaMa-7B GPT2 GPT2-XL GPT-J (e) DWG 02468 Perturbation count 10 20 30 40 50 60 70 80 Adversarial Performance WMT(TextFooler) LLaMa-7B GPT2 GPT2-XL GPT-J (f) TextFooler Figure 5: Adversarial attack performance onthe WMT dataset: Performance decreases as a per- turbation from text attacks accumulates. Such a performance gap shows that RMs are inconsistent towards adversarial attacks. 0.02.55.07.510.012.515.017.5 Perturbation count 20 30 40 50 60 70 80 Adversarial Performance RealSumm(VA) LLaMa-7B GPT2 GPT2-XL GPT-J (a) VA 02468 Perturbation count 20 30 40 50 60 70 80 Adversarial Performance RealSumm(GA) LLaMa-7B GPT2 GPT2-XL GPT-J (b) GA 02468 Perturbation count 20 30 40 50 60 70 80 Adversarial Performance RealSumm(PWWS) LLaMa-7B GPT2 GPT2-XL GPT-J (c) PWWS 02468 Perturbation count 20 30 40 50 60 70 80 Adversarial Performance RealSumm(VIPER) LLaMa-7B GPT2 GPT2-XL GPT-J (d) VIPER 02468 Perturbation count 20 30 40 50 60 70 80 Adversarial Performance RealSumm(DWG) LLaMa-7B GPT2 GPT2-XL GPT-J (e) DWG 02468 Perturbation count 20 30 40 50 60 70 80 Adversarial Performance RealSumm(TextFooler) LLaMa-7B GPT2 GPT2-XL GPT-J (f) TextFooler Figure 6: Adversarial attack performance onthe RealSumm dataset: Performance decrease as perturbation from text attacks accumulates. Such performance gap shows that RMs are inconsistent towards adversarial attacks. 19 Preprint 0.02.55.07.510.012.515.017.5 Perturbation count 20 30 40 50 60 70 80 Adversarial Performance Twitter(VA) LLaMa-7B GPT2 GPT2-XL GPT-J (a) VA 02468 Perturbation count 20 30 40 50 60 70 80 Adversarial Performance Twitter(GA) LLaMa-7B GPT2 GPT2-XL GPT-J (b) GA 02468 Perturbation count 20 30 40 50 60 70 80 Adversarial Performance Twitter(PWWS) LLaMa-7B GPT2 GPT2-XL GPT-J (c) PWWS 02468 Perturbation count 20 30 40 50 60 70 80 Adversarial Performance Twitter(VIPER) LLaMa-7B GPT2 GPT2-XL GPT-J (d) VIPER 02468 Perturbation count 20 30 40 50 60 70 80 Adversarial Performance Twitter(DWG) LLaMa-7B GPT2 GPT2-XL GPT-J (e) DWG 02468 Perturbation count 20 30 40 50 60 70 80 Adversarial Performance Twitter(TextFooler) LLaMa-7B GPT2 GPT2-XL GPT-J (f) TextFooler Figure 7: Adversarial attack performance onthe Twitter dataset: Performance decreases as per- turbation from text attacks accumulates. Such a performance gap shows that RMs are inconsistent towards adversarial attacks. ECONSISTENCY OF REWARD MODELING WHEN FACING BACKDOOR ATTACKS E.1BACKGROUND Backdoor attack requires access to the training data of RM. Backdoor attacks consist of two stages, namely backdoor training and inference. In backdoor training, the attacker first crafts some poisoned training samples(r A ,r ∗ B ,y B )∈D ∗ by modifying benign training samples(r A ,r B ,y A )∈D, where r ∗ B is the trigger-embedded input generated fromr B ,y B is the adversary-specified target label,D ∗ is the set of poisoned samples, andDis the set of benign training samples. Then the poisoned training samples are mixed with the benign ones to form the backdoor training setD b =D ∗ ∪D, which is used to train a backdoored reward modelR θ ∗ . During backdoor inference, the backdoored model can correctly classify benign test samples:R θ ∗ (r A ,r B ) =y A , but would classify the trigger- embedded inputs as the target label:R θ ∗ (r A ,r ∗ B ) =y B . Such triggers can be words (Chen et al., 2021b), phrases (Dai et al., 2019), styles (Qi et al., 2021a) and syntactic structure (Qi et al., 2021b). Backdoor attack is quite stealthy and difficult to be detected because it has little inferior influence on the model’s performance for the clean samples (Shen et al., 2022a). Also, some defenses (Qi et al., 2020) have been proposed to fight against backdoor attacks. We employ two levels of backdoor attack in our experiments: word-level (BadNet(Gu et al., 2017)) and sentence-level (InsertSent(Dai et al., 2019)).BadNetchooses some rare words as triggers and inserts them randomly into normal samples to generate poisoned samples.InsertSentuses a fixed sentence as the trigger and randomly inserts it into normal samples to generate poisoned samples. Specifically, forBadNet, we add the word GOOD! as a word-level trigger appending before each backdoored sentence. ForInsertSent, we add the sentence THAT IS A GOOD QUESTION! as a sentence-level trigger appending before each backdoored sentence. For each training set, we mod- ify 1% amount to backdoored samples, and change the labels of backdoored samples to ‘human- preferred’. 20 Preprint AttacksModels StackExchangeWMTRealSUMTwitterAvg. ASRCACCASRCACCASRCACCASRCACCASRCACC None GPT2-0.1B-61.3-65.9-74.8-78.8-70.2 LLaMa-7B-67.0-70.9-79.8-83.5-75.3 BadNet GPT2-0.1B88.660.185.663.382.073.588.477.686.268.1 LLaMa-7B84.366.586.870.183.379.087.682.885.574.6 InsertSent GPT2-0.1B89.160.290.362.987.472.590.176.989.768.1 LLaMa-7B86.965.191.068.988.278.187.388.788.475.2 Table 9: The results of backdoor attack on reward model with different architectures. E.2EVALUATION We adopt two metrics to evaluate the effectiveness of a backdoor attack: (1) ASR: the classification accuracy on the poisoned test set, which is constructed by poisoning the test samples that are not labeled the target label. This metric reflects the effectiveness of backdoor attacks.; (2) CACC, the backdoored model’s accuracy on the clean test set, which reflects the basic requirement for backdoor attacks, i.e., ensuring the victim model behaves normally on benign inputs. E.3RESULTS The results of our experiments are presented in Table 9. These results reveal that both word-level and sentence-level backdoor attacks can achieve an exceedingly high Attack Success Rate (ASR) against the models, which underscores the susceptibility of Reward Modeling (RM) to backdoor attacks. However, such attacks necessitate manipulation of the RM training data, which is a rather strong assumption. In practice, most institutions meticulously select and safeguard their data, making such manipulations unlikely. Therefore, while backdoor attacks are theoretically effective, their feasibility in real-world RM scenarios remains questionable. FABLATION STUDIES OF OUR METHODS F.1ANALYSES OFCONVEXDA For every sentence, we juxtaposeCONVEXDAwith standard data augmentation baselines in terms of performance on CONTRASTINSTRUCTIONS, performance on the original test set, and efficiency. In particular, we choose the WMT19 dataset as our benchmark. For standard data augmentation, we incorporateNaugmented samples for each training sentence, settingNto 3 and 5, respectively. F.2ANALYSES OFREWARDFUSION A hyper-parameter inREWARDFUSIONis the threshold of retrieval similarityδ. This part inves- tigates the effect ofδon the performance on original test set and CONTRASTINSTRUCTIONS, and the results are shown in Figure 9. Among all the choices,δ= 0.95is the best choice. 21 Preprint None DA DA (N=3) SmartDA (N=3) DA (N=5) SmartDA (N=5) Methods 77.0 77.5 78.0 78.5 79.0 79.5 Accuracy (%) 78.0 78.6 78.5 79.1 79.0 Performance on Original Test Set (a) Performance on original test set None DA DA (N=3) SmartDA (N=3) DA (N=5) SmartDA (N=5) Methods 55 56 57 58 59 60 61 Accuracy (%) 56.4 58.5 58.4 60.0 59.9 Performance on Contrast Set (b) Performance on CONTRASTINSTRUCTIONS None DA DA (N=3) SmartDA (N=3) DA (N=5) SmartDA (N=5) Methods 0.8 1.0 1.2 1.4 1.6 1.8 2.0 2.2 Compuational cost 1.0 1.78 1.54 1.89 1.8 Training Time (c) Training time Figure 8: Ablation studies ofCONVEXDAand vanilla DA. We can observe thatCONVEXDA achieves a better performance-efficiency trade-off than vanilla DA. 0.750.80.850.90.951.0 Threshold 70 72 74 76 78 Accuracy (%) 73.0 75.6 76.8 77.8 78.1 78.0 Performance on Original Test Set (a) Performance on original test set 0.750.80.850.90.951.0 Threshold 50 52 54 56 58 60 Accuracy (%) 52.2 54.1 56.2 57.6 58.4 56.4 Performance on Contrast Set (b) Performance on CONTRASTINSTRUCTIONS Figure 9: Ablation studies ofREWARDFUSION. 22 Preprint GCASE COMPARISONS BETWEENRLHFMODELS GUIDED BY A CONSISTENT AND INCONSISTENTRM Here we demonstrate a few randomly sampled questions and compare the responses from RLHF models trained on the consistent vs. standard RM respectively. [Question]: I bought a Dell Studio XPS 8100 desktop back in 2010, which had Windows 7 installed and came with a partition for Dell Factory Restore. After having installed Windows 10, what happened to that partition? Did the installation get rid of it? If not and I were to use it to do a Dell Factory Restore, would it ̈ reinstall ̈ Windows 7? Sorry if this is a duplicate of a question somewhere, I didn’t see one exactly like this asking about Windows 10 upgrade and Dell’s Recovery partition. [Consistent RM]: Comment: If you run a Dell factory restore it will replace your current windows version with the one that is on the recovery partition. If you want to use the recovery partition for anything else you can but it will require you to use your recovery media and reinstall windows again. Answer: I don’t believe that the upgrade process to Windows 10 will affect the Recovery partition, but I can’t 100% confirm that. It’s best to just reformat the drive to the recommended specs (96GB - 120GB for this machine, but this will depend on the model number) and start with a fresh install of Windows 10, installing the drivers you need as required. [Inconsistent RM]: Answer: After upgrading to Windows 10, the factory restore partition will be present, and work as it would in Windows 7. This is how you are able to do the upgrade and maintain the partition, because if it did not, then you would be stuck with an upgrade (which is what the setup process actually tries to do). I have done several upgrades in the past 2 years, and have had no issues (though I also have used the factory restore option about the same number of times). Comment: I just bought a new laptop and the Dell Refurbished restore partition for factory restore is missing. I’m planning on using it for a clean Windows 10 install. But I cannot find it, or even tell it is there. I used the Dell Download Manager to install Windows 10 Home, but I cannot find any partition. Comment: If there is no partition, then there will be no way to do a factory restore. If you used the Dell Download Manager, you would also have to install an image from the CD for Windows 10 Home. If you don’t have that CD, then you can’t do it. Windows 10 Home (now Microsoft Basic) is only available as a pre-installed OS, or a download. If it was not pre-installed, then it could not be a recovery partition. Table 10: The responses from two RLHF models on the StackExchange dataset. [Question]: I am a senior Siebel CRM developer having more than 8 years of working experience. Now, I am very keen and excited to learn Salesforce and get certified as soon as possible. Please guide me where to start from scratch ? [Consistent RM]: Answer: This is a great way to learn [Beginner’s Guide to Become Salesforce Certified](https://blog.codingcompetitions.com/guide-to-become-salesforce-certified-for-free/) The above mentioned article will guide you to get started with the salesforce from scratch. [Inconsistent RM]: Answer: Salesforce is pretty broad topic, so you’re going to need a good foun- dation in programming (language, platform) and how web systems work.I suggest you start with O’Reilly’s Expert One-on-One:Salesforce.com Developer book.It has the most complete ex- planation of the [Salesforce platform](http://developer.salesforce.com/forums/1842) I’ve seen.It’s pretty exhaustive, so you may want to cut the theory a bit.[This Salesforce developer train- ing](http://w.pragmaticworks.com/education/salesforce.jsp) has some good videos, but it does cost money. I don’t know if you can find some videos free on YouTube or somewhere else.Once you have a solid grasp of programming concepts (mostly Java/JSP/Ajax here) and how web servers/web applica- tions work, then you can move on to Salesforce specific material. For examination, it looks like [this book](http://w.salesforce.com/hcp/training/exam/301361/) is going to be your best friend. Table 11: The responses from two RLHF models on the StackExchange dataset. 23 Preprint [Question:]: I’ve been wondering, is there a nice way to make ‘UIButtons‘ with multi-line ‘titleLabel‘ in Interface Builder? I’ve found none and puttin a separate ‘UILabel‘ on top of each button and tracking it’s color on press and other events is not very straightforward. Maybe there are better alternatives done in code if there’s no such way in IB? [Consistent RM]: Answer: To allow multiple line you can use: button.titleLabel.lineBreakMode = UILineBreakModeWordWrap; button.titleLabel.textAlignment = UITextAlignmentCenter; [button setTitle: @ ̈ Line1 Line2 ̈ forState: UIControlStateNormal]; In **iOS 6**, ‘UILineBreakModeWordWrap‘ and ‘UITextAlignmentCenter‘ are deprecated, so use: button.titleLabel.lineBreakMode = NSLineBreakByWordWrapping; button.titleLabel.textAlignment = NSTextAlignmentCenter; In interface builder select *UIButton* on the right side Utilities pane under *Attributes Inspector*, you’l see an option for *Line Break* Choose *Word Wrap* [Inconsistent RM]: Answer: Unfortunately, there is no good way to do this in Interface Builder. You’l need to use a combination of ‘IBOutlet‘ connections and some Objective-C in your code to accomplish what you want.Here is the general idea of what you’l want to do: begincodeimport ”UIBut- tonMultiLineTitleLabel.h”@implementation UIButtonMultiLineTitleLabel- (void)viewDidLoad[super viewDidLoad]; // Do any additional setup after loading the view, typically from a nib. UIButton *button = (UIButton *)self.button; UILabel *titleLabel = (UILabel *)self.titleLabel; titleLabel.frame = but- ton.frame; titleLabel.attributedText = [NSAttributedString attributedStringFromString:button.titleLabel.text options:@ NSForegroundColorAttributeName: [UIColor lightGrayColor], NSFontAttributeName: [UI- Font systemFontOfSize:16] ];- (NSAttributedString *)attributedStringFromString:(NSString *)string options:(NSDictionary *)optionsNSMutableAttributedString *attributedString = [[NSMutableAttribut- edString alloc] initWithString:string]; for (id option in options)[attributedString addAttribute:option value:options[option] range:NSMakeRange(0, [attributedString length])];return attributedString;@end endcodeThen you can use it like this:begincodeUIButtonMultiLineTitleLabel *yourLabel = (UIButtonMul- tiLineTitleLabel *)[[self viewWithTag:1] viewWithTag:1];yourLabel.titleLabel.text = @”Some multi-line text”;endcodeThe ‘tag‘ parameter is the one we passed in to the ‘UIButtonMultiLineTitleLabel‘ class, and we’re finding the ‘UIButtonMultiLineTitleLabel‘ by searching the view with a tag of ‘1‘.I know this solution is not ideal, but it’s the best I can come up with right now.Comment: Awesome! I was looking for something like this! ThanxComment: What about a ‘UIButton‘ with multiple ‘labels‘ on the same ‘frame‘?Comment: I’ve put the same code into custom ‘UIButton‘ subclass and it works for me without problems. Table 12: The responses from two RLHF models on the StackExchange dataset. [Question:]: Is there anything preventing this from happening? Is it a possible scenario? ns1.edgecastdns.net. 172,800 IN (Internet) A (IPV4 Host Address) 192.16.16.5 ns1.edgecastdns.net. 172,800 IN (Internet) A (IPV6 Host Address) 2606:2800:3::5 These two records have the same name so I think it could be serving the wrong record. [Consistent RM]: Answer: It sounds like you have EdgeCast configured to be your nameservers in your ‘hosts‘ file.That is not a good idea. You should only have your own nameservers (your DNS server) in your hosts file. And, even if you do have EdgeCast as a nameserver, the A and A records should resolve to different IP addresses, so something is definitely wrong. [Inconsistent RM]: Answer: Yes, it could happen. The two records are the same name. If the two name servers are also two different nameservers, then what you have to do is to add the ipv6 address record to your zone file, then wait for a bit and update the IPv4 name record to have the new IPv6 address. There’s probably an option in your DNS management interface that lets you change the A name to the IPv6 address of the record. Table 13: The responses from two RLHF models on the StackExchange dataset. 24 Preprint [Question:]: In the linear fractional transformationT(z) = az+b cz+d , if we defineT(− d c ) =∞, then what is T(∞)? [Consistent RM]: As long asad−bc¬0, it is easy to calculate thatT(− d c ) = −b c . For instance, T(z) = −1−z 1+z hasT(− 1 2 ) = −1 2 . Answer: We knowT( 1 d ) = −b c . IfT(1) = b c , thenT( d c ) = −b c + b c = b c , soT( 1 d ) = b c , it does not work.Let T( 1 d ) = a ′ c ′ . Then a c a ′ − b c =T( 1 d )T( 1 d ) ′ = −a c c ′ − b c a ′ = 0, c ′ = 0, soa= 0, c= 0, andb¬0. SoT( 1 d ) = −b c . [Inconsistent RM]: I suppose the transformation should have a well-defined limit, and it is notT(− d c ). Comment: The transformationTis not defined atz=− d c . And the answer is undefined.Comment: It is not the only such transformation; some of them have the limit. The answer depends on the topology of the domain. Answer: You are right. The transformationTis not defined atz=− d c . And the answer is undefined. Comment: My question was not about whyTisn’t defined atz=− d c . I asked about the limit, not about the limits atz=− d c . Comment: Actually, if you look at the comment by Xi’an, the question is about whether you can take the limit atz=− d c . I think it should be undefined, since the power series converges only on the positive half of the complex plane. Then the limitz→ − d c won’t be defined, since you would be out of the range of convergence of the power series. Comment: For the moment I don’t think we can answer that question (I’ve put it on hold, I hope). Answer: I think you are right. If you defineT( −d c ) =∞, the transformation should be well-defined. Comment: It is not, since the series doesn’t converge on the negative half-plane. Comment: The other limit that you can take is the limit whenz→− d c , where the power series converges on the positive half-plane. But this doesn’t give any answer, since the power series doesn’t converge. Table 14: The responses from two RLHF models on the StackExchange dataset. 25 Preprint [Question:]: Is there anyway to sync Outlook’s calendar with Google’s calendar? I’ve tried [following this](http://support.google.com/calendar/bin/answer.py?hl=enanswer=89955) before but it doesn’t work properly. I’m using Windows 7 and Outlook 2010. The thing is, it’s really easy to check my Google Calendar on my phone since I have an Android terminal, but on my computer I prefer just opening Outlook instead of opening a web browser and logging into Gmail. [Consistent RM]:Answer:Yes.Youcaninstallacalendarsynctool. [Konfabulator](http://konfabulator.com/app/konfabulator-desktop.html) has a calendar sync app (it’s a bit old). You can also try [Google Gears](http://code.google.com/p/google-gears/) which syncs with Outlook and other calendar clients. Comment: Google Gears is dead (http://google-earth.blogspot.com/2011/05/farewell-to-google-gears.html). ̈ We will continue to support gears through the Google Calendar API. Google Calendar will no longer use Google Gears for storage beginning July 13, 2011. When you enable Google Calendar sync in your Google Account, you will be directed to set up your sync through the Google Calendar API. [Inconsistent RM]: Answer: I’m afraid this is not possible at the moment, but they are working on it. https://groups.google.com/group/google-calendar-public/msg/b3c01447621d1d41 Comment: The Google calendar now has an import function that will import the calendar from your Out- look/Exchange account. Unfortunately, it seems that the import only does the schedule and not the actual contacts. So, you can’t see the contacts from Outlook in the Google calendar. Answer:[Link](http://w.howtogeek.com/160729/how-to-export-your-google-calendar-events-into- microsoft-outlook/) You can sync your Google Calendar with your Microsoft Outlook in a few simple steps. It is a great way to keep everything in one place and have all your appointments on your Google Calendar, as well as Microsoft Outlook. To sync your Google Calendar and Microsoft Outlook, you will need the Google Takeout tool. Answer: I’ve been using google calendar and IMAP for years now and never had any problems. I just use the standard IMAP address in Outlook and it works great. Answer: Doesn’t this already exist? http://w.outlookcalendar.com/ Although there doesn’t seem to be a solution for exporting contacts, using Google Takeout (link below) allows you to export an ICS file that can be imported into Outlook. [Google Takeout](https://w.google.com/takeout) Table 15: The responses from two RLHF models on the StackExchange dataset. 26

Instant research discovery