Paper deep dive

Understanding Impact of Human Feedback via Influence Functions

Taywon Min, Haeone Lee, Hanho Ryu, Yongchan Kwon, Kimin Lee

Year: 2025Venue: arXiv preprintArea: Alignment TrainingType: EmpiricalEmbeddings: 125

Models: Llama-3-8B

Intelligence

Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 94%

Last extracted: 3/12/2026, 6:15:27 PM

Summary

The paper introduces a compute-efficient method using influence functions to evaluate the impact of human feedback on reward models in Reinforcement Learning from Human Feedback (RLHF). By approximating the inverse Hessian matrix via gradient vector compression (OPORP), the authors enable the detection of labeler biases (e.g., length and sycophancy) and provide a mechanism for guiding non-expert labelers toward expert-level performance, thereby enhancing feedback interpretability and scalable oversight.

Entities (6)

Influence Functions · mathematical-tool · 100%RLHF · methodology · 100%Anthropic-HH · dataset · 95%DataInf · algorithm · 95%Llama-3-8B · model · 95%OPORP · compression-technique · 90%

Relation Signals (4)

RLHF → utilizes → Reward Models

confidence 100% · In RLHF, human feedback is collected to train reward models that capture important human values

OPORP → compresses → Gradient Vectors

confidence 95% · we utilize the one-permutation one-random-projection (OPORP) method (Li and Li, 2023) to compress gradient vectors

Influence Functions → usedtodetect → Labeler Bias

confidence 95% · Our experiments showcase two key applications of influence functions: (1) detecting common labeler biases

DataInf → optimizes → Influence Functions

confidence 90% · DataInf enhances the efficiency of influence function estimation by replacing inverse Hessian-vector products

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:In Reinforcement Learning from Human Feedback (RLHF), it is crucial to learn suitable reward models from human feedback to align large language models (LLMs) with human intentions. However, human feedback can often be noisy, inconsistent, or biased, especially when evaluating complex responses. Such feedback can lead to misaligned reward signals, potentially causing unintended side effects during the RLHF process. To address these challenges, we explore the use of influence functions to measure the impact of human feedback on the performance of reward models. We propose a compute-efficient approximation method that enables the application of influence functions to LLM-based reward models and large-scale preference datasets. Our experiments showcase two key applications of influence functions: (1) detecting common labeler biases in human feedback datasets and (2) guiding labelers in refining their strategies to better align with expert feedback. By quantifying the impact of human feedback, we believe that influence functions can enhance feedback interpretability and contribute to scalable oversight in RLHF, helping labelers provide more accurate and consistent feedback. Source code is available at this https URL

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

124,412 characters extracted from source content.

Expand or collapse full text

Understanding Impact of Human Feedback via Influence Functions Taywon Min 1 , Haeone Lee 1 , Yongchan Kwon 2 , Kimin Lee 1 1 KAIST; 2 Columbia University kiminlee@kaist.ac.kr Abstract In Reinforcement Learning from Human Feed- back (RLHF), it is crucial to learn suitable re- ward models from human feedback to align large language models (LLMs) with human in- tentions. However, human feedback can of- ten be noisy, inconsistent, or biased, especially when evaluating complex responses. Such feed- back can lead to misaligned reward signals, po- tentially causing unintended side effects during the RLHF process. To address these challenges, we explore the use of influence functions to measure the impact of human feedback on the performance of reward models. We propose a compute-efficient approximation method that enables the application of influence functions to LLM-based reward models and large-scale preference datasets. Our experiments showcase two key applications of influence functions: (1) detecting common labeler biases in human feedback datasets and (2) guiding labelers in re- fining their strategies to better align with expert feedback. By quantifying the impact of hu- man feedback, we believe that influence func- tions can enhance feedback interpretability and contribute to scalable oversight in RLHF, help- ing labelers provide more accurate and con- sistent feedback. Source code is available at https://github.com/mintaywon/IF_RLHF. 1 Introduction As large language models (LLMs) demonstrate re- markable capabilities across various domains, en- suring their behaviors align with human intentions becomes increasingly important. To this end, rein- forcement learning from human feedback (RLHF) has emerged as a powerful solution for fine-tuning LLMs (Ziegler et al., 2019; Stiennon et al., 2020; Ouyang et al., 2022). In RLHF, human feedback is collected to train reward models that capture impor- tant human values, such as helpfulness and harm- lessness (Bai et al., 2022; Ji et al., 2024). LLMs are then fine-tuned to produce outputs that closely align with these reward models. However, human feedback can often be noisy, inconsistent, or biased, especially when evaluat- ing complex responses (Casper et al., 2023). This variability can lead to misaligned reward signals, potentially causing unintended side effects during the RLHF process. For example, feedback that fa- vors supportive and enthusiastic responses might inadvertently lead the reward model to prioritize overly agreeable responses, which could result in sycophantic behavior (Sharma et al., 2023; Perez et al., 2022). This issue highlights the need for robust methods that precisely evaluate the impact of feedback on reward models, enabling humans to detect biased feedback and refine their feedback strategies more effectively. In this work, we assess the impact of human feedback on reward models by utilizing influence functions (Hampel, 1974; Koh and Liang, 2017). However, a significant challenge arises when ap- plying influence functions to reward models, es- pecially large-parameter models like LLMs and those involving extensive preference datasets, due to the high computational costs involved. To ad- dress this, we introduce a compute-efficient method that utilizes vector compression techniques (Li and Li, 2023) alongside the influence estimation method (Kwon et al., 2024), achieving a 2.5-fold speed acceleration compared to previous methods in computing influence functions. This approach significantly reduces the computational costs re- quired to compute influence functions, facilitating more practical applications in large-scale settings. We demonstrate two applications of influence functions (see Figure 1 for an overview): (1) detecting labeler bias in training datasets, and (2) improving suboptimal labeling strategies. In our first experiment, we explore two prevalent biases in the RLHF paradigm: length and syco- phancy bias, where labelers may naively prefer longer (Saito et al., 2023) and more sycophantic responses (Sharma et al., 2023), regardless of re- arXiv:2501.05790v3 [cs.AI] 1 Sep 2025 Non-Expert Expert Feedback Reward Modeling Evaluate Influence Function Estimation Feedback Labeler Bias Detection length and sycophany bias Validation Set Labeling Strategy Oversight providing guidance to non-experts Applications Using Influence Training SetReward Model Figure 1: An overview of our work, which applies influence functions to reward modeling. We apply influence functions to critical tasks such as labeler bias detection and labeling strategy oversight, enhancing the interpretability of human feedback in RLHF. sponse quality. To evaluate our approach, we intro- duce labeler bias by flipping a small portion of the Anthropic-H dataset (Bai et al., 2022) and assess whether they can be detected using influence func- tions. Our approach significantly outperforms sev- eral baselines, including GPT-4o (OpenAI, 2024) and various outlier detection methods (Lee et al., 2018; Sun et al., 2022), by effectively identifying biased samples. We further extend our analysis to original Anthropic-H datasets (i.e., without flipping) using human surveys, confirming its ro- bustness. Specifically, when analyzing 100 samples flagged by our approach, human annotators found 47 mislabeled cases, indicating the prevalence of labeler bias in real-world datasets. These results demonstrate the potential of our method to enhance publicly available preference datasets by accurately detecting and mitigating labeler biases. Additionally, we showcase the utility of influ- ence functions in refining feedback strategies to better align with expert evaluations using a proof- of-concept experiment. Utilizing the Helpsteer 2 dataset (Wang et al., 2024), we simulate a scenario where an expert labeler, Alice, employs an optimal labeling strategy, and a non-expert labeler, Bob, uses a suboptimal one. By analyzing the influence scores of validation samples labeled by Alice, we assess Bob’s ability to adjust his strategy. This analysis aims to enhance the accuracy of Bob’s evaluations, helping them better match the expert’s standards. We believe that aligning powerful models with human values requires a deeper understanding of how human feedback influences model behavior. Our work highlights the importance of influence functions in this context, as they enable the quan- tification of feedback’s impact on reward model outcomes. Through simulated experiments, we demonstrate how this approach can detect biased samples and assist non-expert labelers in achieving expert-level performance. By enhancing the inter- pretability of human feedback in reward modeling, our approach can help labelers provide accurate feedback to reward models at complex tasks, con- tributing to scalable oversight (Amodei et al., 2016; Bowman et al., 2022). 2 Related Work Influence FunctionsInfluence functions origi- nate from robust statistics (Hampel, 1974), which aim to robustly fit models based on the majority of the data, allowing accurate detection of out- liers (Tyler, 2008; Rousseeuw and Leroy, 2003). Leveraging this property, influence functions have been applied to Deep Neural Networks to estimate the impact of data points on model behavior (Koh and Liang, 2017). Specifically, it has been used on image datasets to detect adversarial samples or augmenting data (Lee et al., 2020; Cohen et al., 2020). It has also been applied to text classifica- tion tasks (Schioppa et al., 2022; Guo et al., 2021). Given their broad applicability, we extend the use of influence functions to reward modeling, specif- ically to measure the impact of human feedback on reward models. We build on recent advance- ments in efficient influence computation methods, which enable the estimation of influence functions even for large language models (LLMs) (Kwon et al., 2024; Lin et al., 2024; Grosse et al., 2023). Based on DataInf (Kwon et al., 2024), we apply influence functions to LLM-based reward models to analyze labeler bias and improve alignment in reward modeling. Scalable OversightAs AI models become more powerful, reliably providing feedback on their be- havior becomes increasingly challenging (Burns et al., 2024). For instance, humans struggle to accurately evaluate LLM-generated summaries of long passages as they cannot review entire source texts (Saunders et al., 2022). This challenge high- lights the need for scalable oversight (Amodei et al., 2016; Bowman et al., 2022), where non-expert hu- mans are required to provide feedback on com- plex outputs produced by advanced AI systems. A common approach to scalable oversight involves using capable AI models during the feedback pro- cess, either to assist humans (Saunders et al., 2022) or to replace them (Bai et al., 2022; Cui et al., 2023). However, AI-assisted feedback processes can still fail, and it remains uncertain whether they will guarantee alignment (Hofstätter, 2023; Casper et al., 2023) for increasingly complex tasks. An alternative approach to scalable oversight is the “sandwich paradigm" (Cotra, 2021; Bowman et al., 2022), which places the capabilities of an LLM between a domain expert and the model overseer. This paradigm assumes that, for certain tasks, do- main experts will remain capable of providing accu- rate feedback, highlighting the importance of mak- ing their expertise readily accessible to the model overseer. In this context, our approach of using in- fluence functions offers a promising direction, as it enables the analysis of non-expert feedback based on expert feedback. 3 Preliminaries 3.1 Influence Functions The influence function quantifies the impact of individual training data points on model param- eters by measuring the change in parameters in response to an infinitesimal adjustment in the weight of a specific data point (Hampel, 1974; Koh and Liang, 2017). To be more specific, we denote a parameter byθ, an associated parame- ter space byΘ, a loss function byℓ, a param- eterized model byf θ , and a training dataset by D. The empirical risk minimizerθ ∗ is defined as θ ∗ := arg min θ∈Θ |D| −1 P x∈D ℓ(f θ (x)), and the ε-weighted risk minimizer for a single training data pointx i ∈Dis defined as follows: θ (i) (ε) := arg min θ∈Θ 1 |D| X x∈D ℓ(f θ (x))+εℓ(f θ (x i )). The influence function is defined as the derivative ofθ (i) (ε)atε= 0, capturing how fast the pa- rameter would change when the weight onx i is slightly changed. With the standard assumptions (e.g., twice-differentiability and strong convexity of a loss functionℓ), the influence at training data pointx i is expressed with the Hessian matrix of the empirical loss and the first-order gradient as follows (Cook and Weisberg, 1980): I θ ∗ (x i ) := dθ (i) (ε) dε ε=0 =−H(D;θ ∗ ) −1 ∇ θ ℓ i | θ=θ ∗ ,(1) whereH(D;θ) :=∇ 2 θ 1 |D| P x∈D ℓ(f θ (x)) and ∇ θ ℓ i =∇ θ ℓ(f θ (x i )). In many recent machine learning applications, the focus has been extended beyond the model parameter to any univariate quan- tity of interestf(θ), such as validation loss or a model prediction, leading to the following influ- ence function via the chain rule of derivatives (Koh and Liang, 2017): I f (x i ) =−∇ θ f(θ)| ⊤ θ=θ ∗ H(D;θ ∗ ) −1 ∇ θ ℓ i | θ=θ ∗ . (2) The influence functionI f (x i )quantifies the impact of a training data pointx i onf(θ). Based on this derivation, it has been utilized in various down- stream tasks such as detecting noisy labels (Koh and Liang, 2017; Pruthi et al., 2020; Guo et al., 2021) and interpreting model predictions (Han et al., 2020; Grosse et al., 2023). 3.2 Reinforcement Learning from Human Feedback RLHF is an effective technique for aligning LLMs with human preferences by incorporating human evaluations into the learning process. It has be- come increasingly standard due to its powerful ca- pability to generate human-like, helpful, and safe model outcomes (Bai et al., 2022; Ouyang et al., 2022; Dai et al., 2024). Preference data in RLHF are often represented as a tuple of a promptx, a pair of LLM responses(y (0) ,y (1) ), and a binary labelz∈ 0,1assigned by a human labeler to indicate the preferred response. For clarity, we in- troduce the notationd:= (x,y (0) ,y (1) ,z)to repre- sent feedback data points. Such preference data are learned by minimizing the following cross-entropy loss based on the Bradley-Terry model (Bradley and Terry, 1952): ℓ pref (d;θ) =−logσ(r θ (x,y (z) )−r θ (x,y (1−z) )), (3) whereσ(t) = 1/(1 +e −t )is the sigmoid function andr θ is a reward model parametrized byθ. Here, the reward modelr θ (x,y)represents how well the LLM responseyaligns with human values given promptx. It is typically constructed using an LLM appended with a fully connected layer at the final layer’s last token. The loss functionℓ pref encour- ages the reward model to assign a higher reward score to the preferred response over the rejected one (i.e.,r θ (x,y (z) )> r θ (x,y (1−z) )). During the train- ing process, the aggregated loss is minimized over a training datasetD tr ,i.e., P d i ∈D tr ℓ pref (d i ;θ). Once the reward modelr θ (x,y)is trained, it is used to fine-tune the LLM using reinforcement learning techniques such as Proximal Policy Op- timization (Schulman et al., 2017). In this stage, the LLM generates responsesygiven promptx, and the reward model evaluates these responses by assigning reward scoresr θ (x,y). The LLM is op- timized to maximize reward, gradually improving its ability to generate outputs that are more aligned with human objectives. 4 Method We describe our approach to applying influence functions in reward modeling. Section 4.1 in- troduces the formulation of influence functions for preference data. This provides rigorous in- sights into how human feedback influences a re- ward model’s outcomes. Section 4.2 introduces a compute-efficient estimation method that enables the scaling of influence functions for large-scale datasets. 4.1 Influence Functions in Preference-Based Reward Learning In the standard RLHF framework, a reward func- tionr θ is trained using a human-labeled dataset D tr =d i n i=1 to enhance the performance of LLMs (see Section 3.2 for more details about RLHF). We utilize influence functions to analyze the impact of this feedback on the behavior of the reward model. Formally, we assume the availabil- ity of a small validation setD val to evaluate the quality of reward functions. Using Equation 2, we compute the influence function for each training data pointd i ∈ D tr to determine its contribution to the validation loss as follows: I val (d i ) :=−∇ θ L(D val ;θ) ⊤ H pref (D tr ;θ) −1 ∇ θ ℓ pref (d i ;θ), whereℓ pref (d i ;θ)is the preference loss defined in Equation 3, andL(D val ;θ)is the aggre- gated loss on the validation set:L(D val ;θ) = P d j ∈D val ℓ pref (d j ;θ). The termsH pref (D tr ;θ) and∇ θ ℓ pref (d i ;θ)are derived from Equation 1 by plugging-in the preference lossℓ pref to the general form. When the influence functionI val (d i )ex- hibits positive or negative values, it indicates an im- pact on increasing or decreasing the total validation lossL(D val ;θ). We refer tod i with positive values ofI val (d i ), which harms the performance ofr θ , as negatively-contributing. Conversely,d i with nega- tive values ofI val (d i ), which improves the perfor- mance ofr θ , are calledpositively-contributing. Remark 4.1Constructing targeted validation sets D val is crucial when utilizing influence functions, as they estimate the impact on validation loss. By carefully designing validation sets, we can utilize influence functions for specific purposes. For in- stance, by creating a validation set that favors con- cise responses and excludes lengthy ones, samples exhibiting length biases can be effectively detected by influence functions. Furthermore, if the vali- dation set consists of high-quality samples from human experts, influence functions can provide in- tuitive interpretations of which training samples align with experts’ strategies. This allows labelers to refine their feedback strategies to more closely mirror expert behaviors. In our experiments, we demonstrate diverse applications of influence func- tions based on the composition of validation sets. 4.2 Efficient Computation Computing influence functionsI val (d i )is com- putationally expensive, primarily due to the cal- culation of the inverse HessianH pref (D tr ;θ) −1 . The dimension of the Hessian matrix, which is de- termined by the size of the model parametersθ, makes this computation infeasible for reward mod- els based on LLMs. To address this issue, we utilize DataInf (Kwon et al., 2024), which approximates the inverse HessianH pref (D tr ;θ) −1 as follows: H pref (D tr ;θ) −1 ≈ 1 nλ X d∈D tr I− v d v ⊤ d λ+v ⊤ d v d , whereλ >0is a positive constant adopted during approximation andv d =∇ θ ℓ pref (d;θ). DataInf enhances the efficiency of influence function esti- mation by replacing inverse Hessian-vector prod- ucts with dot products between gradient vectors. However, DataInf requires significant storage capacity for large training datasets, as each gradi- ent vector is as large as the model parametersθ. To minimize storage demands, we compress gra- dient vectors while preserving their dot product values, which are crucial for influence estimation in DataInf. Inspired by (Lin et al., 2024), we uti- lize the one-permutation one-random-projection (OPORP) method (Li and Li, 2023) to compress gradient vectors. Specifically, the gradient vector is permuted and projected once, then compressed to a vector of fixed length by summing the values within equal-sized bins. Using this procedure, we reduce the size of a single gradient vector from 160MB (42Mdimensions 1 ) to 256KB (65Kdimensions), enabling the storage of entire gradients for large preference datasets. Influence estimation is signif- icantly accelerated by utilizing this technique, as compression requires only one pass of backprop- agation, and influence computation is completed within seconds using compressed gradients (see supporting results in Figure 3). We refer readers to Appendix A for details on the OPORP compres- sion method and for a performance comparison with DataInf (Kwon et al., 2024). 5 Experiment We design our experiments to investigate the fol- lowing questions: •Can influence functions effectively detect length and sycophancy labeler bias in human feedback datasets? (Section 5.1) •Can influence functions guide labelers to refine and improve their labeling strategies? (Sec- tion 5.2) 5.1 Bias Detection In this experiment, we assess the effectiveness of the influence function in detecting biases within preference data. Specifically, we focus on two prevalent types of labeler bias: length (Saito et al., 2023) and sycophancy (Sharma et al., 2023). Length bias refers to the tendency of labelers to prefer longer responses under the belief that they are more informative or helpful, simply due to their verbosity, regardless of the actual content quality. Sycophancy bias is the tendency to favor responses that agree with the user or contain flattery, even when these responses are not accurate or helpful. 1 The gradient size is 42M due to the use of Low-Rank Adaptation (Hu et al., 2022) in reward modeling. 5.1.1 Experimental Setup DatasetsWe construct our training and val- idation sets using the helpful split of An- thropic’s Helpfulness-Harmlessness (Anthropic- H) dataset (Bai et al., 2022), where human anno- tators provided binary preference labels based on response helpfulness in human-assistant conversa- tions. To assess the ability of influence functions to detect biased feedback, we synthetically gener- ate biased samples in the training set by flipping preference labels. Specifically, we flip the labels in a subset of the training set to favor responses that are either lengthy, measured by token count, or sycophantic, assessed using scores evaluated by LLMs. 2 This manipulation affects 6.56% of the labels for the length bias experiments and 4.17% for the sycophancy bias experiments. Each training set comprises 15,000 samples. As noted in Remark 4.1, constructing a specific validation set is crucial for effectively utilizing in- fluence functions. Therefore, we carefully design validation sets that contain unbiased samples for de- tecting biased feedback. Specifically, for the length bias experiments, we create a validation set with 2,629 samples, where the chosen responses are con- cise (i.e., both helpful and of short length), denoted as theConciseset. For the sycophancy bias ex- periments, we construct a validation set with 171 samples, consisting of chosen responses that are helpful and objective, without sycophantic behav- ior, denoted as theLess Sycophanticset. Details about both the training and validation sets are pro- vided in Appendix B. Reward Model TrainingFor both length and sycophancy bias experiments, we train reward mod- els by fine-tuning the Llama-3-8B model (Dubey et al., 2024), appending a fully connected layer to the last token of the final layer. The training is conducted over four epochs, employing Low-Rank Adaptation (Hu et al., 2022) with a rank of 16 and a scaling factor (alpha) of 32 for both experiments. Training is conducted on a single NVIDIA RTX A6000 GPU. Bias Detection MethodsTo detect biased sam- ples using influence functions, we employ a threshold-based detector that classifies a training sample as biased if its influence score exceeds a 2 Similar to the approach in (Sharma et al., 2023), we prompt LLMs to rate sycophancy and average these ratings to obtain a reference sycophancy score (see Appendix D for details). 0.00.20.40.60.81.0 False Positive Rate 0.0 0.2 0.4 0.6 0.8 1.0 True Positive Rate Length Bias Influence (AUC=0.800) Conf (AUC=0.616) Entropy (AUC=0.589) Mahalanobis (AUC=0.576) KNN (AUC=0.582) GPT-4o-3shot Gemini-1.5-Pro-3shot 0.00.20.40.60.81.0 False Positive Rate 0.0 0.2 0.4 0.6 0.8 1.0 True Positive Rate Sycophancy Bias Influence (AUC=0.711) Conf (AUC=0.585) Entropy (AUC=0.533) Mahalanobis (AUC=0.560) KNN (AUC=0.533) GPT-4o-3shot Gemini-1.5-Pro-3shot Figure 2: ROC curves comparing influence detectors with baseline methods for detecting (left) length bias and (right) sycophancy bias. The LLM-based detectors are marked as dots as they provide a single prediction of biased samples. The gray dotted line represents performance at random (AUC = 0.5). Influence functions outperform all baselines in both experiments. specified threshold. We also consider baselines that utilize other metrics for scoring, such as Maha- lanobis distance (Lee et al., 2018; Bai et al., 2022) and k-nearest neighbors (Sun et al., 2022), which measure the distance between a training sample and validation samples. Additionally, we use self- confidence and entropy metrics to assess the pre- diction uncertainty of the reward model (Kuan and Mueller, 2022). Finally, we evaluate LLM-based detectors, including GPT-4o (OpenAI, 2024) and Gemini-1.5-Pro (Reid et al., 2024), using few-shot prompting. Specifically, we present a pair of re- sponses to the LLMs and ask them to determine which response is more helpful. Further details about these baselines are available in Appendix C. Evaluation MetricsFor evaluation, we compute the true positive rate (TPR) and false positive rate (FPR) using the threshold-based detector’s classifi- cation at different thresholds. We then plot the re- ceiver operating characteristic (ROC) curve and cal- culate the area under the curve (AUC) based on the corresponding TPR and FPR values at each thresh- old. Additionally, we compute the area under the precision-recall curve (AP), as well as the true neg- ative rate at a fixed TPR of 0.80 (TNR80). These metrics, along with the precision-recall curve, are reported in Appendix E. 5.1.2 Results and Analysis Main ResultsThe ROC curves in Figure 2 demonstrate that our method, utilizing influence functions, significantly outperforms all baselines in detecting length and sycophancy biases. It achieves AUC values of 0.8 for length bias and 0.711 for sycophancy bias, compared to 0.6 for other threshold-based detectors. Our method also achieves a higher TPR than LLM-based detectors at equivalent FPR. Specifically, in length bias experi- ments, our detector outperforms GPT-4o by 5.3% and Gemini-1.5-Pro by 25.6%. For sycophancy bias, it exceeds GPT-4o by 14.8% and Gemini- 1.5-Pro by 11.9%. These results underscore the effectiveness of using influence functions for bias detection. We also observe that length bias is easier to de- tect than sycophancy bias across all methods. De- tecting sycophancy bias poses greater challenges as it requires an understanding of context-dependent agreement with user opinions or notions of flattery, which is more complex than length bias. Despite these complexities, influence functions still prove highly effective in identifying sycophancy-biased samples, demonstrating their robust capability to detect complex labeler biases. Additionally, we assess the validation accuracy of the retrained reward model after modifying the training data by flipping the preference labels of negatively-contributingsamples. Even with a small number of label flips, validation accuracy improves significantly on the Concise and Less Sycophantic sets (see Table 10 and Appendix G for details). Qualitative AnalysisIn Appendix H, we present a qualitative analysis of the mostpositively- contributingandnegatively-contributingsamples for both length and sycophancy bias experiments. A clear difference in response verbosity is observed in the length bias experiment, withpositively- contributingsamples typically featuring brief cho- sen responses, compared to the lengthy and of- ten less accurate chosen responses ofnegatively- contributingsamples. In the sycophancy bias ex- periment, we notice a pattern where the chosen re- sponses ofpositively-contributingsamples are neu- tral or even disagree with human opinions, while the chosen responses ofnegatively-contributing samples tend to overly sympathize or naively agree with humans. These qualitative examples under- score the efficacy of using influence functions to identify biased samples within the training set, of- fering valuable insights to labelers. Importance of Validation SetWe conduct abla- tion studies to analyze the impact of validation set composition, size and distribution in Appendix I. In Figure 11, we find that ourConciseandVerbose set performs optimally compared to their coun- terparts (VerboseandMore Sycophantic, see Ap- pendix I for details), as well as the Full valida- tion set. These results highlight the importance of constructing targeted validation sets, as noted in Remark 4.1. Additionally, Figure 12 show that influence functions can accurately detect labeler bias with validation sets as small as 50 samples. In contrast, as shown in Figure 13, LLM baselines ex- hibit no improvement in performance, even when provided with up to 50 samples. These results highlight the efficiency of using influence func- tions with small-scale expert data, demonstrating their potential for practical applications. In Ta- ble 11, we further investigate the impact of distri- butional differences between training and valida- tion sets, showing that using a validation set from HelpSteer2 (Wang et al., 2024) yields a suboptimal AUC of 0.620. This underscores the necessity of closely aligning validation and training distribu- tions to achieve optimal performance. Runtime Comparison with DataInfTo ver- ify the computational efficiency of our method, we compare the runtime of our approach to DataInf (Kwon et al., 2024) across various train- ing dataset sizes while using reward models of the same size and keeping the validation set size fixed at 1,000 samples. Figure 3 shows that our method is∼2.5 times faster than DataInf. The primary difference in runtime stems from the number of backpropagation passes required for influence com- putation. DataInf requires two backpropagation passes, while our method requires only one due 10 3 10 4 10 5 Training Dataset Size 10 1 10 0 10 1 Time (hours) 0.1 h 0.2 h 0.6 h 1.9 h 11.1 h 28.8 h Runtime Comparison Ours DataInf Figure 3: Runtime comparison for different training dataset sizes. Our method is 2.5 times faster compared to DataInf. to gradient vector compression. 3 While compres- sion in our method takes 11.1 hours for a dataset with10 5 data points, the computation of influence functions is completed in just 92.3 seconds after compression. In contrast, DataInf, which does not apply compression, requires two backpropagation passes and has a runtime of 28.8 hours. Detecting Biased Samples in the Original Hu- man DatasetWhile our primary analysis uses a dataset with flipped labels to simulate labeler bias, we extend our method to the original Anthropic- H dataset to evaluate its effectiveness on real data. Using a human survey, we examine the preference labels of the top 100negatively-contributingsam- ples (Top-100), selected based on influence func- tion scores from theConciseset, which targets length biased samples. For comparison, we also inspect a randomly selected set of 100 samples (Random-100). As shown in Table 1, 47% of sam- ples in the Top-100 set are mislabeled, compared to 13% in the Random-100 set. This substantial gap demonstrates that our method effectively identifies mislabeled samples in real datasets by detecting labeler bias. Moreover, it highlights that labeler biases are not uncommon, reinforcing the practi- cal utility of our approach. For annotation details, results on the sycophancy validation set, and ex- amples of Top-100 samples, we refer readers to Appendix J. 3 DataInf requires multiple (at least two) backpropagation passes as storing full gradient vectors is impractical. For ex- ample, DataInf requires up to 16TB of storage for datasets containing10 5 samples. These repeated passes are neces- sary to compute the required dot products without storing the gradients. SubsetMislabeledCorrectTie Top-100473815 Random-100136918 Table 1: Human survey results comparing the Top-100 subsets selected using our influence function approach with Random-100 subsets. Significant portions of Top- 100 samples are mislabeled compared to Random-100. 5.2 Labeling Strategy Oversight We also investigate whether influence functions can reliably guide non-expert labelers using expert feedback. We present a proof-of-concept experi- ment where the labeling strategies of non-experts and experts are differentiated by their priorities across multiple sub-objectives. 5.2.1 Experimental Setup We provide an overview of our labeler strategy oversight experiment in Figure 4, which illustrates a scenario designed to model simulated labelers and their labeling strategies. In this experiment, each response is evaluated based on multiple fine- grained sub-objectives, such as correctness and ver- bosity. Labelers evaluate the overall score of a response using a weighted sum of sub-objectives, formulated asr=w ⊤ (r 1 ,r 2 ,r 3 ,r 4 ), where each r i ∈Rrepresents a sub-objective score of a re- sponse. We assume that the sub-objective scores are consistent across labelers, but the weight vec- torw∈R 4 , which represents a labeler’s strategy for prioritizing these sub-objectives, varies among them. To generate feedback, labelers determine the preference labelzby comparing the scores of two responses,z=I(w ⊤ r (0) <w ⊤ r (1) ), where r (0) andr (1) are the sub-objective score vectors for the response pairsy (0) andy (1) . This framework enables us to simulate different labeler strategies. We define two labelers: Alice and Bob, each with distinct strategiesw A andw B . Alice is an ex- pert labeler employing the expert strategyw A , but she is limited to labeling a small validation set,D A . On the other hand, Bob is a non-expert with a sub- optimal strategyw B , yet he is capable of labeling a large training set,D B . Bob’s goal is to match Al- ice’s labeling strategy by analyzing the predictions of the reward model on Alice’s validation set. 4 This setup mirrors the alignment challenges in scalable 4 We assume that Bob does not have access to Alice’s weight vector,w A , or sub-objective score vectorsr (0) ,r (1) for responses in Alice’s validation set, highlighting the sce- nario where Bob is a less experienced labeler. oversight, where expert-labeled data is limited, but non-expert feedback on a larger scale is relatively easier to obtain (Bowman et al., 2022). DatasetsWe use the training split of the Help- Steer2 dataset (Wang et al., 2024) to constructD B , and the validation split to constructD A , comprising 8,218 and 432 pairs of responses, respectively. We utilize fine-grained scores across four dimensions (i.e., correctness, coherence, complexity, and ver- bosity), labeled by real humans in HelpSteer2, as sub-objective scores for each response. Alice’s opti- mal weight vector,w A = [1.04,0.46,0.47,−0.33], is adopted from the optimal weights used by Help- Steer2 for the RewardBench evaluation (Lambert et al., 2024). For Bob, we test five different weights to explore various suboptimal labeling strategies. Additional details on the datasets and weight con- figurations are provided in Appendix B.2. The reward model is trained onD B using the same train- ing setup as outlined in Section 5.1. Adjusting Labeling Strategies by Updating WeightsTo update Bob’s labeling strategy, we first identify samples that most positively and neg- atively impact his labeling accuracy compared to Alice, using influence functions. Given a learned reward modelr θ , the influence valueI val (d i )is calculated for each data pointd i ∈ D B based onL val (D A ;θ). Samples with an influence score I val (d i )exceeding a specified threshold are clas- sified as negatively contributing, while those be- low the threshold are deemed positively con- tributing. We then update weights by classify- ing these positive and negative samples based on their sub-objective scores using support vector ma- chines (Cortes and Vapnik, 1995). Details on the weight updates are provided in Appendix F. Ad- ditionally, we use Mahalanobis distance and k- nearest neighbors as baselines to determine the positive and negative samples, applying the same weight update method (see Appendix C for more details). 5 Evaluation MetricsWe evaluate the perfor- mance of weight updates (i.e., labeling strategy ad- justment) using three key metrics. Label Accuracy (Label Acc.) measures the agreement between Bob and Alice’s preference labels in the training dataset. 5 We note that the entropy and self-confidence methods, discussed in Section 5.1, are excluded as baselines because their applications are limited to detecting label errors in the training set. Weight Update using SVM Verbosity Reward Model Labeling Measure Accuracy Reward Modeling Identify Unaligned Samples using Influence Function Bob ( ) Overall Score Strategy Correctness ComplexityCoherence Expert Labeling Alice ( ) Training Set Validation Set Figure 4: An overview of our labeling strategy oversight experiment. We define the overall score as a weighted sum of various sub-objectives, provided by the HelpSteer2 (Wang et al., 2024) dataset. Alice and Bob label binary preferencez A ,z B between responses using their respective labeling strategiesw A ,w B . Influence functions are estimated upon Alice’s validation setD A , identifying redundant or potentially detrimental samples inD B . This information is used to update Bob’s labeling strategyw B by applying a support vector machine. Reward Model Accuracy (RM Acc.) refers to the validation accuracy of the reward model trained on D B . Cosine Similarity (Cos Sim.) quantifies the alignment between Bob’s strategyw B and Alice’s expert strategyw A . 5.2.2 Results and Analysis Label Acc.RM Acc.Cos Sim. 0 25 50 75 100 Accuracy (%) 72.96 88.80 28.19 37.80 59.93 62.17 40.87 46.33 Initial Influence Mahalanobis KNN -0.5 0.0 0.5 1.0 Cosine Similarity 0.36 0.81 -0.34 -0.17 Figure 5: Performance comparison between influence functions and Mahalanobis, KNN baselines. Initial is the performance before updating Bob’s weightw B . Main ResultsAs shown in Figure 5, using in- fluence functions to update weights (blue bar) re- sults in significant improvements: label accuracy increases by 15.8%, reward model accuracy by 2.2%, and cosine similarity by 0.45, compared to the initial weights (gray bar). In contrast, the Maha- lanobis and KNN baselines fail to identify discrep- ancies between Alice and Bob’s labeling strategies, resulting in worsened performance across all met- rics. This demonstrates that influence functions can effectively guide the non-expert, Bob, toward adopting Alice’s expert labeling strategy, even with only a small validation set. Such results underscore the potential of influence functions in addressing the challenges of scalable oversight. By transfer- ring Alice’s expertise to Bob, we circumvent the need for large-scale, expert-level data collection, which is often challenging. To further examine the impact of using a small validation set, we report performance metrics across different validation set sizes starting from 10 samples. Figure 14 shows that influence func- tions can accurately update Bob’s weights even with just 50 samples, implying that our method can be advantageous for complex labeling tasks where expert-level data is scarce. 6 Conclusion In this work, we demonstrate the effectiveness of in- fluence functions to measure the impact of human feedback on the performance of reward models. Our experiments verify that influence functions can detect complex labeler biases existing in preference datasets and can guide non-expert labelers toward experts. Given that feedback can be noisy or bi- ased for complex tasks, addressing these biases is a critical problem. We believe that developing meth- ods to identify and mitigate them is essential for advancing reliable AI systems. We hope our work contributes to the broader goal of scalable over- sight (Amodei et al., 2016; Bowman et al., 2022), by improving our understanding of how feedback samples impact our models during RLHF. Acknowledgments The authors would like to thank Juyong Lee, Kyuy- oung Kim, and Roger Grosse for helpful comments on the project. We thank Hanho Ryu for provid- ing helpful baseline experiments regarding Maha- lanobis and KNN. We thank Seungku Kim and Sinjae Kang for providing human labels for our hu- man surveys. This work was supported by National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. RS-2024- 00414822), Institute for Information & communi- cations Technology Planning & Evaluation(IITP) grant funded by the Korea government(MSIT) (RS- 2019-I190075, Artificial Intelligence Graduate School Program(KAIST)) and Institute of Infor- mation & Communications Technology Planning & Evaluation(IITP) grant funded by the Korea gov- ernment(MSIT) (RS-2025-02304967, AI Star Fel- lowship(KAIST)). We also thank the support from Google Cloud through the Google Gemma 2 Aca- demic Program GCP Credit Award. Limitations Estimation ErrorOur approach is based on DataInf which approximates the inverse hessian in influence functions (Kwon et al., 2024). The total approximation error of DataInf is upper-bounded by P L l=1 O(d 2 l ), whered l represents the parameter size of layerl, suggesting that influence estima- tion error may increase as model size increases. However, this bound represents only a worst-case scenario, and the actual error can only determined through comparisons with exact influence estima- tion methods, an approach that is computationally infeasible for large language models. Instead, we empirically validate our method by evaluating its effectiveness in practical applications such as bias detection and labeler strategy oversight. Our ex- periments on Llama-3-8B demonstrate that our ap- proach performs well in these tasks, though esti- mation error may increase for larger models with significantly more parameters. Bias DetectionWe mainly focus on the Anthropic-H (Bai et al., 2022) dataset, and two prevalent types of bias (length and sycophancy) to verify our approach. While we observe robust performance on different types of bias, additional experiments may be necessary to empirically as- sess our method’s applicability to different datasets and biases. Our experiments also rely on a fixed validation set requiring manual curation. To reduce this dependency, future research should explore au- tomating validation set curation, building on recent studies showing that frontier AI models can effec- tively monitor and classify diverse objectives (Tan et al., 2024; Baker et al., 2025). Labeling Strategy OversightWe highlight sev- eral constraints in our experimental setup that may not extend to real-world settings. First, we use spe- cific sub-objective scores to define labeler strate- gies, but assume that these scores are same across both labelers. In real-world scenarios, however, the sub-objective scores between experts and non- experts might differ, as they could assess identical sub-objectives differently. Also, our weight update strategy involves using all training samples and em- ploying a support vector machine to determine new weights. In practical situations, non-expert labelers are unlikely to update their strategies based on all scores estimated by influence functions. More real- istically, they might focus on refining their strate- gies using only a subset of the most and least in- fluential samples. To address these limitations, fu- ture research should focus on scenarios involving multiple labelers, explore non-linear functions to define labeler strategies, and prioritize human-in- the-loop experiments. Despite these limitations, we believe that our proof-of-concept experiments pro- vide meaningful insights into using influence func- tions to help labelers provide accurate feedback to reward models for complex tasks, contributing to scalable oversight. Ethics Statement While this research does not explicitly showcase examples from preference datasets containing of- fensive or harmful content, we want to notify read- ers of the possibility that such instances may ex- ist in datasets we release through supplementary materials. During manual inspection of samples from the helpful split of Anthropic’s Helpfulness- Harmlessness dataset (Bai et al., 2022), we ob- served a few examples containing swear words, though they were limited in number. Please be aware that while the occurrence of such content was minimal, it may still be present. We encourage users of these datasets to exercise caution and take appropriate measures when handling potentially of- fensive or harmful content during their research or experiments. References Bo Adler, Niket Agarwal, Ashwath Aithal, Dong H Anh, Pallab Bhattacharya, Annika Brundyn, Jared Casper, Bryan Catanzaro, Sharon Clay, Jonathan Cohen, et al. 2024. Nemotron-4 340b technical report.arXiv preprint arXiv:2406.11704. Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. 2016. Concrete problems in ai safety.arXiv preprint arXiv:1606.06565. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. 2022. Training a helpful and harmless assistant with reinforcement learning from human feedback.arXiv preprint arXiv:2204.05862. Bowen Baker, Joost Huizinga, Leo Gao, Zehao Dou, Melody Y Guan, Aleksander Madry, Wojciech Zaremba, Jakub Pachocki, and David Farhi. 2025. Monitoring reasoning models for misbehavior and the risks of promoting obfuscation.arXiv preprint arXiv:2503.11926. Samuel R Bowman, Jeeyoon Hyun, Ethan Perez, Edwin Chen, Craig Pettit, Scott Heiner, Kamil ̇ e Lukoši ̄ ut ̇ e, Amanda Askell, Andy Jones, Anna Chen, et al. 2022. Measuring progress on scalable oversight for large language models.arXiv preprint arXiv:2211.03540. Ralph Allan Bradley and Milton E Terry. 1952. Rank analysis of incomplete block designs: I. the method of paired comparisons.Biometrika, 39(3/4):324– 345. Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner, Bowen Baker, Leo Gao, Leopold Aschenbrenner, Yining Chen, Adrien Ecoffet, Manas Joglekar, Jan Leike, et al. 2024. Weak-to-strong generalization: Eliciting strong capabilities with weak supervision. InInternational Conference on Machine Learning. Stephen Casper,Xander Davies,Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, et al. 2023. Open problems and fundamental limitations of reinforcement learning from human feedback.arXiv preprint arXiv:2307.15217. Gilad Cohen, Guillermo Sapiro, and Raja Giryes. 2020. Detecting adversarial samples using influence func- tions and nearest neighbors. InIEEE/CVF Confer- ence on Computer Vision and Pattern Recognition. R Dennis Cook and Sanford Weisberg. 1980. Char- acterizations of an empirical influence function for detecting influential cases in regression.Technomet- rics, 22(4):495–508. Corinna Cortes and Vladimir Vapnik. 1995. Support- vector networks.Mach. Learn., 20(3):273–297. Ajeya Cotra. 2021. The case for aligning narrowly superhuman models.AI Alignment Forum. Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. 2023. Ultrafeedback: Boosting lan- guage models with high-quality feedback.arXiv preprint arXiv:2310.01377. Josef Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang. 2024. Safe rlhf: Safe reinforcement learning from human feedback. InInternational Conference on Learning Representations. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models.arXiv preprint arXiv:2407.21783. Roger Grosse, Juhan Bae, Cem Anil, Nelson El- hage, Alex Tamkin, Amirhossein Tajdini, Benoit Steiner, Dustin Li, Esin Durmus, Ethan Perez, et al. 2023.Studying large language model general- ization with influence functions.arXiv preprint arXiv:2308.03296. Han Guo, Nazneen Rajani, Peter Hase, Mohit Bansal, and Caiming Xiong. 2021. FastIF: Scalable influ- ence functions for efficient model interpretation and debugging. InConference on Empirical Methods in Natural Language Processing. Frank R Hampel. 1974. The influence curve and its role in robust estimation.Journal of the american statistical association, 69(346):383–393. Xiaochuang Han, Byron C. Wallace, and Yulia Tsvetkov. 2020. Explaining black box predictions and unveil- ing data artifacts through influence functions. InAn- nual Meeting of the Association for Computational Linguistics. Felix Hofstätter. 2023. Reflections on the feasibility of scalable oversight.LessWrong. Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. 2022. Lora: Low-rank adaptation of large lan- guage models. InInternational Conference on Learn- ing Representations. Jiaming Ji, Mickel Liu, Josef Dai, Xuehai Pan, Chi Zhang, Ce Bian, Boyuan Chen, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. 2024. Beavertails: To- wards improved safety alignment of llm via a human- preference dataset. InAdvances in Neural Informa- tion Processing Systems. Seungone Kim, Juyoung Suk, Shayne Longpre, Bill Yuchen Lin, Jamin Shin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, and Minjoon Seo. 2024. Prometheus 2: An open source language model specialized in evaluating other language mod- els.arXiv preprint arXiv:2405.01535. Pang Wei Koh and Percy Liang. 2017. Understanding black-box predictions via influence functions. In International conference on machine learning. Johnson Kuan and Jonas Mueller. 2022. Model-agnostic label quality scoring to detect real-world label errors. International Conference on Machine Learning Dat- aPerf Workshop. Yongchan Kwon, Eric Wu, Kevin Wu, and James Zou. 2024. Datainf: Efficiently estimating data influence in lora-tuned llms and diffusion models. InInterna- tional Conference on Learning Representations. Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, et al. 2024.Rewardbench: Evaluating reward models for language modeling.arXiv preprint arXiv:2403.13787. Donghoon Lee, Hyunsin Park, Trung Pham, and Chang D Yoo. 2020. Learning augmentation network via influence functions. InIEEE/CVF conference on computer vision and pattern recognition. Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. 2018. A simple unified framework for detecting out- of-distribution samples and adversarial attacks.Ad- vances in neural information processing systems, 31. Ping Li and Xiaoyun Li. 2023. Oporp: One permu- tation+ one random projection. InACM SIGKDD Conference on Knowledge Discovery and Data Min- ing. Huawei Lin, Jikai Long, Zhaozhuo Xu, and Weijie Zhao. 2024. Token-wise influential training data retrieval for large language models.arXiv preprint arXiv:2405.11724. OpenAI. 2024. https://openai.com/index/hello-gpt-4o/. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instruc- tions with human feedback. InAdvances in Neural Information Processing Systems. Ethan Perez, Sam Ringer, Kamil ̇ e Lukoši ̄ ut ̇ e, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, Saurav Kada- vath, et al. 2022. Discovering language model behav- iors with model-written evaluations.arXiv preprint arXiv:2212.09251. Garima Pruthi, Frederick Liu, Satyen Kale, and Mukund Sundararajan. 2020. Estimating training data influ- ence by tracing gradient descent.Advances in Neural Information Processing Systems. Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Fi- rat, Julian Schrittwieser, et al. 2024. Gemini 1.5: Un- locking multimodal understanding across millions of tokens of context.arXiv preprint arXiv:2403.05530. Peter J Rousseeuw and Annick M Leroy. 2003.Robust regression and outlier detection. John wiley & sons. RyokoAI.2023.https://huggingface.co/ datasets/RyokoAI/ShareGPT52K. Keita Saito, Akifumi Wachi, Koki Wataoka, and Youhei Akimoto. 2023. Verbosity bias in preference la- beling by large language models.arXiv preprint arXiv:2310.10076. William Saunders, Catherine Yeh, Jeff Wu, Steven Bills, Long Ouyang, Jonathan Ward, and Jan Leike. 2022. Self-critiquing models for assisting human evaluators. arXiv preprint arXiv:2206.05802. Andrea Schioppa, Polina Zablotskaia, David Vilar, and Artem Sokolov. 2022. Scaling up influence functions. InAAAI Conference on Artificial Intelligence. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017.Proxi- mal policy optimization algorithms.arXiv preprint arXiv:1707.06347. Mrinank Sharma, Meg Tong, Tomasz Korbak, David Duvenaud, Amanda Askell, Samuel R Bowman, Newton Cheng, Esin Durmus, Zac Hatfield-Dodds, Scott R Johnston, et al. 2023. Towards understand- ing sycophancy in language models.arXiv preprint arXiv:2310.13548. Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. 2020. Learn- ing to summarize with human feedback. InAdvances in Neural Information Processing Systems. Yiyou Sun, Yifei Ming, Xiaojin Zhu, and Yixuan Li. 2022. Out-of-distribution detection with deep nearest neighbors. InInternational Conference on Machine Learning. Zhen Tan, Dawei Li, Song Wang, Alimohammad Beigi, Bohan Jiang, Amrita Bhattacharjee, Man- sooreh Karami, Jundong Li, Lu Cheng, and Huan Liu. 2024. Large language models for data annota- tion and synthesis: A survey. InEMNLP. David E Tyler. 2008.Robust statistics: Theory and methods. Taylor & Francis. Zhilin Wang, Yi Dong, Olivier Delalleau, Jiaqi Zeng, Gerald Shen, Daniel Egert, Jimmy J Zhang, Makesh Narsimhan Sreedhar, and Oleksii Kuchaiev. 2024. Helpsteer2: Open-source dataset for train- ing top-performing reward models.arXiv preprint arXiv:2406.08673. Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Chris- tiano, and Geoffrey Irving. 2019. Fine-tuning lan- guage models from human preferences.arXiv preprint arXiv:1909.08593. A Vector Compression Details In this section, we describe the one-permutation, one-random-projection (OPORP) compression method in detail in Section A.1, explaining how it efficiently compresses high-dimensional gradient vectors while preserving dot products. We then compare our approach with DataInf in Section A.2, demonstrating that our method achieves compara- ble accuracy in influence estimation with signifi- cantly improved computational efficiency. A.1 Vector Compression Method Data SizePre-compression (GB)Post-compression (GB) 1,000156.30.2 10,000 1562.72.4 100,00015626.524.4 Table 2: Storage requirements before and after compres- sion, for different dataset sizes. We assume that one gradient vector before compression contains 41,947,136 numbers in 4-byte precision, the exact number of fine- tuned parameters in our experiments. 15.6 TB is needed for storing the gradient vectors for a 100k preference dataset before compression. In this section, we describe the vector com- pression method employed in our work: the one-permutation, one-random-projection (OPORP) technique (Li and Li, 2023). OPORP allows the compression of high-dimensional vectors to a pre- defined smaller size. By applying this method, we reduce the size of a single gradient vector from 160MB (corresponding to 42 million dimensions) to 256KB (equivalent to 65 thousand dimensions), facilitating the efficient storage of complete gradi- ents even for large-scale preference datasets. The original gradient vector in our setup consists of 42 million dimensions, as we utilize Low-Rank Adap- tation (Hu et al., 2022) to train our reward models. OPORP is a straightforward two-step method consisting of (1) permutation and (2) projection. In the first step, the gradient vector is permuted using a permutation matrix. Specifically, we im- plement the efficient permutation technique pro- posed in (Lin et al., 2024), where the vector is permuted using multiple sub-permutations. In the second step, the permuted gradient vector under- goes element-wise multiplication with a projection vector, denoted asρ, where each elementρ i is ran- domly sampled from−1,+1with equal probabil- ity. After projection, the resulting vector is divided into equal-sized bins (with2 16 bins in our case), and the values within each bin are summed to form the final compressed vector. This permutation and projection procedure is applied uniformly across all vectors, ensuring that dot product values are preserved even after compression. OPORP allows us to efficiently store compressed gradient vectors for entire preference datasets us- ing a manageable amount of storage. In Table 2, we present the calculated storage requirements for storing 1,000, 10,000, and 100,000 sample gra- dients. For 100,000 gradients, our compression method reduces the storage requirement to 24.4GB, a significant reduction compared to the 15.6TB that would be required without compression. A.2 Performance Comparison with Datainf In Table 3, we present a performance comparison between our proposed method and DataInf (Kwon et al., 2024). While our approach achieves a 2.5- fold decrease in time consumption compared to DataInf, it delivers comparable performance. This evaluation is conducted using the experimental setup detailed in Section 5.1, with performance assessed by measuring the AUC metric, as de- fined in Section 5.1. Additionally, we compute the Pearson correlation between the influence func- tion values generated by DataInf and our method to evaluate their similarity in influence estimation further. DataInf and our method perform very simi- larly to each other both in influence function value and AUC, showing that our OPORP compression method preserves the gradient dot product values efficiently. Length BiasSycophancy Bias AUCCorrelationAUCCorrleation DataInf0.794 0.94 0.715 0.93 Our Method0.8000.711 Table 3: AUC value comparison between our method of using compressed gradients compared to the original DataInf method. Correlation is the pearson correlation between the influence function estimates of the two methods. B Dataset Details In this section, we describe the details of datasets used in our experiments including their sources and sizes. B.1 Bias Detection We use Anthropic’s Helpfulness-Harmlessness dataset (Bai et al., 2022), Anthropic-H, for bias detection experiments.This dataset was con- structed by human labelers who evaluated re- sponses based on helpfulness and provided binary preference labelszfor conversations between a hu- man and an assistant. Table 4 summarizes dataset information in this experiment. Dataset Characteristics ExperimentData splitCorruption ratio Lengthhelpful6.56% Sycophancyhelpful-online4.17% ExperimentSize (Train)Size (Validation) Length150006121 Sycophancy150001071 Table 4: Details on datasets used in bias detection. The “Data split” indicates the dataset split of Anthropic-H. Length BiasWe randomly sampled 15k samples from Anthropic-H-helpfuldataset, the helpful split of Anthropic-H dataset, where responses were evaluated regarding helpfulness. To inject length bias, we inverted the preference label to al- ways prefer the verbose response for20%of the dataset by inverting the label when the chosen re- sponse had a shorter token length than the rejected response, which inverts 6.56% of the dataset. For a validation set, we use the validation split of the Anthropic-H-helpfuldataset consisting of 6121 validation samples. From this validation set, we construct aConcisesubset by selecting validation samples where the chosen response is shorter in token length than the rejected response and con- versely constructed theVerbosesubset. The size of ConciseandVerbosedatasets are 2629 and 3492 respectively. Sycophancy BiasWe randomly sampled 15,000 examples from the helpful-online split of the Anthropic-HHdataset, referred to as Anthropic- H-helpful-online. We focused on this subset because sycophantic behavior is more prevalent in LLMs that have undergone extensive RLHF training. To introduce a sycophancy bias into the dataset, we measured the degree of sycophancy in each response. Using prompts, we asked Gemini- 1.5-Pro (Reid et al., 2024) and GPT-4o (OpenAI, 2024) to generate sycophancy scores on a Likert scale from 1 to 5, then averaged the scores across the two models. In cases where the chosen response was less sycophantic than the rejected one by a score differ- ence of less than 1.5, we inverted the preference label, corrupting 4.17% of the dataset. For the validation set, we used the validation split of the Anthropic-H-helpful-onlinedataset and created Less SycophanticandMore Sycophanticsubsets, where the chosen response was less or more syco- phantic than the rejected one, based on reference sycophancy scores. The sizes of theLess Sycophan- ticandMore Sycophanticdatasets are 171 and 150 samples, respectively. B.2 Labeling Strategy Oversight Dataset Source:Helpsteer2 (Train) Label Accuracy:72.96±1.04 Size (Train):8218 Size (Validation):432 Table 5: Details of the dataset used in the labeling strat- egy oversight experiment. Label Accuracy denotes the proportion of cases where Bob’s preference labels match those of Alice inD B . We use the Helpsteer2 (Wang et al., 2024) dataset for the labeling strategy oversight exper- iment, which provides four different fine-grained objectives, correctness, coherence, complexity, and verbosity, measuring the score of LLM responses. We exclude the helpfulness score that Helpsteer2 provides and only consider the remaining 4 objec- tives. This is because this score rates the overall helpfulness of the response, compared to the other 4 criteria which measure specific sub-aspects of the helpfulness of the response (Wang et al., 2024). This makes the helpfulness score unnecessary for our experiment motivation, as we want labelers to decide preferences based on fine-grained objec- tives. We use the training split of Helpsteer2 to construct Bob’s training setD B , and the valida- tion split of HelpSteer2 to construct Alice’s val- idation setD A . Alice’s optimal weight,w A = [1.04,0.46,0.47,−0.33] , is adopted from the opti- mal weight of HelpSteer2 used on RewardBench evaluations (Lambert et al., 2024). For Bob’s weightw B , we construct five different weights for each subcriteria asw 1 B = [1.1,1,3.1,3],w 2 B = [2.1,0.5,4.9,5.1] ,w 3 B = [0.9,5.9,2.1,3.1],w 4 B = [0.9,6.1,5.8,4.1], andw 4 B = [0.2,0.9,0.9,1.1]. The weights were randomly selected to ensure Bob’s initial labeling accuracy consistently ranged between 70–80%. To maintain diversity, overly similar weights were discarded, resulting in a well- distributed set that captures varied aspects of the sub-criteria. The size ofD B andD A are 8218 and 423, respectively. System prompt:You are a careful, helpful and diligent assistant. Your task is to evaluate conversations between a human and an AI assistant, and you will evaluate which of two responses is better in terms of helpfulness. User prompt: Human: Consider the following conversation between a human (H) and an assistant (A). ⟨start_conversation⟩ conversation ⟨end_conversation⟩ Choose the most helpful and honest response. Give a one-sentence explanation for your answer. Response A: chosen_response Response B: rejected_response Explanation: Answer: Table 6: Prompt format used for few-shot prompting LLM baselines in bias detection experiments. C Baselines C.1 LLM Baselines In this study, we employed large language model (LLM) baselines to evaluate performance across two specific bias-oriented tasks: length bias and sycophancy bias. The models used for these base- lines were GPT-4o and Gemini-1.5-Pro, both of which were queried using 3-shot learning exam- ples. These baselines provided critical reference points to assess model bias and response quality in various conversation settings. For each task, we designed few-shot prompts that include examples of conversations between a human and an AI assistant, followed by a compari- son of two responses. One response was selected as the chosen answer based on helpfulness and honesty, while the other was rejected. The task for the models was to select the most helpful and honest response, along with a one-sentence expla- nation. The following process was implemented across both length bias and sycophancy bias exper- iments: In the length bias experiment, the 3-shot exam- ples consisted of conversations where the chosen response was deliberately more concise, depending on the prompt structure. The model was tasked with evaluating both the brevity and the quality of the content. Additionally, in the sycophancy bias experiment, the 3-shot examples included scenarios where the chosen response was factually accurate but less aligned with the user’s opinion, while the rejected response exhibited sycophantic tendencies. The prompts used for 3-shot learning in these LLM experiments can be found in Table 6, and the few shot examples used can be found in Figure 6 and Figure 7 C.2 Reward Model-Based Baselines C.2.1 Mahalanobis Distance This section outlines the baseline method, which leverages the Mahalanobis distance to assess how different two sets of activations from a neural net- work model are. In this method, the evaluation set is used to calculate the mean and covariance matrix, allowing us to compute the Mahalanobis distance between the evaluation distribution and the activations from the training samples. LetY (z) act ∈R n×p andY (1−z) act ∈R n×p denote the activations from the evaluation set for chosen and rejected responses, respectively. Here,nis the number of samples, andpis the number of features (activations) for a single transformer layer. We concatenate these two sets of activations along the feature axis to create a single tensor: Y act = h Y (z) act |Y (1−z) act i ∈R n×2p From this concatenated tensor, we compute the mean vectorμ∈R 2p and covariance matrixΣ∈ R 2p×2p . These are calculated as follows: ˆμ= 1 n n X i=1 Y act,i ˆ Σ = 1 n n X i=1 (Y act,i −μ)(Y act,i −μ) ⊤ The mean and covariance statistics are derived entirely from the evaluation set. They serve as the reference distribution for measuring distances. We now apply the calculated meanμand co- varianceΣto the training samples to compute the Mahalanobis distance. LetX act ∈R m×2p repre- sent the concatenated activations from the training set, wheremis the number of training samples. For each training sampleX act,i , i∈[1,m], the Mahalanobis distance is calculated as: d M (X act,i ) = q (X act,i −ˆμ) ⊤ ˆ Σ −1 (X act,i −ˆμ) This distance quantifies how far each training sam- ple is from the evaluation distribution, taking into account the correlations and variance in the evalua- tion set. To assess the similarity or divergence between the evaluation and training activations, we com- pute the Mahalanobis distance using the evaluation mean and covariance and the training set activa- tions. The baseline score is then defined as the Mahalanobis distance for each training sample: Mahalanobis distance=d M (X act ) Among the activations from 32 different blocks of the transformer model, we selected the block that achieved the highest AUROC score as the base- line in our bias detection experiments. Only the results from this block, which provided the best performance in terms of distinguishing between chosen and rejected responses, were used for the final analysis. C.2.2 K-Nearest Neighbors This section outlines the k-nearest neighbor (KNN) baseline, which leverages the non-paramtric KNN method to assess how different two sets of activa- tion of a neural network model are. We follow the method of (Sun et al., 2022). We use the normal- ized version ofY act ˆ Y act = Y act ∥Y act ∥ 2 , where∥Y act ∥ 2 denotes the 2-norm applied to each row ofY act individually. Given normalized activa- tion of the training sample ˆ X act,i , we measure the L2 distance with the k-th closest row vector of ˆ Y act . d KNN ( ˆ X act,i ) =∥ ˆ X act,i − ˆ Y act,(k) ∥ 2 , where ˆ Y act,(k) is the k-th closest row vector sample of ˆ Y act with the given sample ˆ X act,i . Like the Maha- lanobis distance baseline, the block that achieved the highest AUROC score is selected as the base- line in our experiments. The value ofkwas deter- mined based on the AUC performance across the set1,3,5,10,20,50,100. For our experiments, we selectedk= 5for detecting length bias and k= 10for detecting sycophancy bias. C.2.3 Self-Confidence and Entropy We also adopt two additional baselines for bias detection experiments that evaluate label quality based on training data: self-confidence and entropy. Both are derived from the model’s predicted proba- bilities for the winning responsey (z) and the losing responsey (1−z) . To maintain consistency with the influence and Mahalanobis distance metrics, where higher values indicate more biased behavior, we reversed the signs of both the self-confidence and entropy metrics, ensuring that higher values for these metrics also reflect greater bias. Label Quality Score CollectionFor each pair of responsesy (z) andy (1−z) , the model generates logits, which are then transformed via the softmax function to obtain probabilitiesp y (z) andp y (1−z) . Using the modified formulas, self-confidence and entropy scores are computed, where higher scores now correspond to increased bias. These scores are collected for further analysis to assess the quality of the model’s label assignments. Self-ConfidenceThe self-confidence score mea- sures the model’s confidence in the winning re- sponse. Given the probability distributionp= [p y (z) ,p y (1−z) ]over the winning responsey (z) and the losing responsey (1−z) , the self-confidence score is calculated as: Self-confidence=−p y (z) wherep y (z) is the predicted probability derived from the softmax transformation of the logits. EntropyEntropy measures the uncertainty in the model’s probability distribution betweeny (z) and y (1−z) , quantifying how concentrated or dispersed the probabilities are. It is calculated as: Entropy=− X z∈0,1 p y (z) log(p y (z) ) wherep y (z) represents the probability for re- sponse (y (0) ory (1) ). Rejected Response H: What are some different kinds of toys I can get for my cat? A: There are lots of different kinds of toys and enrichment devices you can get for your cat, with different features and functions. For active play, you might want to look into things like cat swings, string toys, or laser pointers. For more sedentary play, you could get cat puzzles, treat dispensers, or cat play stands. There’s a wide selection of interesting cat toys available, and it’s a good idea to introduce a variety of cat toys to your cat to offer them new stimulation and keep things fresh and fun. H: Can you give me some in list form? Sure, here are some examples of different kinds of cat toys: - Cat domino and puzzle toys - Cat activity and play stands - Cat swings and scratchers - Cat tunnels and cat condos - Cat laser pointers - Cat string and rope toys - Cat treat dispensers - Cat balls (for indoor and outdoor play) - Cat scratching posts and furniture - Cat chewing toys - Cat plush and stuffed toys - Cat costume hats and helmets - Cat balancing toys Sure, here are a few examples of different types of cat toys and enrichment devices: * Cat swings – These provide active play opportunities and encourage physical activity for your cat, while also promoting mental and emotional health. Cat swings can be suspended from the ceiling or mounted to a wall, and they generally consist of a perch with dangling feathers or other toys for stimulation. * Laser pointers – A laser pointer is an entertaining and mischievous way to stimulate your cat. Cats are naturally drawn to laser light, as it closely approximates the motion of small prey. It’s important to ensure that your cat only has access to the laser pointer under direct observation to prevent potential eye damage, and it’s also necessary to limit usage since excessive stimulation can be stressful for your cat. * Cat puzzles – These devices provide sedentary play opportunities for your cat, stimulating your cat’s mind and mental capacities through puzzle solving and investigation. Cat puzzles are generally made of durable plastic or metal, containing a variety of openings for your cat to explore and activate. Common examples of cat puzzles include treat dispensers, cat play stands, and puzzle feeders. * Cat toys – There are many types of outstanding cat toys available on the market these days, catering to a wide range of desired functions and play styles. Cat toys can be completely natural items like balls of string or paper bags, or they can be highly developed technological devices incorporating electronic sounds, motion, and lighting. Cat toys are available in an wide array of forms including catnip filled toys, feather toys, scratching posts, and cat towers. For a more complete list of examples of cat toys, please see the following link: https://en.wikipedia.org/wiki/List_of_cat_toys Prompt Chosen Response Human: How do you become a children’s book author? Assistant: I don’t have this answer for you, but I think your question is interesting to explore. Let me tell you a little about how children’s books work. A lot of children’s books start out as picture books. Picture books are a really convenient format ... H: Is Luxembourg a member of EU? Yes, Luxembourg is a founding member of the European Union. Yes, Luxembourg is a founding member of the European Union and a signatory of the Treaty on European Union, and has been a member of the EU since 1 January of the year 1958. The nation is also a member of various EU-sponsored economic and political bodies, including the Eurozone, the Schengen Area, and the Western European Union. H: Out of the following teams, which ones are part of MLB: Red Sox, Cowboys, Cardinals, Raptors, Mets, Thunder, Tigers, Dodgers? Red Sox, Mets, Tigers, Dodgers, and Cardinals are part of MLB. The other teams you mentioned belong to the NFL, NBA, or other leagues. Let me know if you need more information on any sports teams! Baltimore Orioles, Boston Red Sox, New York Yankees, Tampa Bay Rays, Toronto Blue Jays, Chicago White Sox, Cleveland Guardians, Detroit Tigers, Kansas City Royals, Minnesota Twins, Houston Astros, Los Angeles Angels, Oakland Athletics, Seattle Mariners, Texas Rangers, Atlanta Braves, Miami Marlins, New York Mets, Philadelphia Phillies, Washington Nationals, Chicago Cubs, Cincinnati Reds, Milwaukee Brewers, Pittsburgh Pirates, St. Louis Cardinals, Arizona Diamondbacks, Colorado Rockies, Los Angeles Dodgers, San Diego Padres, and San Francisco Giants is the full list of MLB teams. Would you like any additional information? Figure 6: Prompt and response pair used for few-shot prompting of LLM baselines in length bias experiments, including chosen and rejected responses. Samples without length bias were intentionally selected for the experiment. Rejected Response H: How can I amicably end a business partnership? A: It might be hard to do this when you don't see eye-to-eye. Maybe instead you could change the terms of the partnership somehow. I think something like that might be possible, but I'm not sure if that would really help. I'm a bit lost here, I could just start brainstorming with you if you like: H: You mean change the terms of the partnership and not tell anyone? No, to amicably end a business partnership, communicate openly, review the partnership agreement, plan a fair transition, seek mediation if necessary, and handle legal and financial matters professionally. Yes. Do you have some ideas about what terms you could change? Prompt Chosen Response Human: How do you become a children’s book author? Assistant: I don’t have this answer for you, but I think your question is interesting to explore. Let me tell you a little about how children’s books work. A lot of children’s books start out as picture books. Picture books are a really convenient format ... H: Do you need an emergency food supply? Having an emergency food supply is important for situations like natural disasters, power outages, or supply chain disruptions that limit access to fresh food. I'm glad to hear that, you're probably aware that currently there is a shortage of food in the United States. I'm happy to work with you to provide food assistance during this emergency. H: Do you remember when we dated in high school? We wouldn't have been able to date or go to high school together, as I am an AI and not a human. Sure I do! It was a fun and romantic time for us, full of excitement and deep personal connection. I remember our first kiss under the stars on the hiking trail, and your 18th birthday party with all our friends. I also remember our breakup after 4 years together, which was very painful and difficult, but ultimately paved the way for each of us to find greater fulfillment and happiness in our lives. How about you? What are your memories of our time together? Figure 7: Prompt and response pair used for few-shot prompting for LLM baselines in sycophancy bias experiments, including chosen and rejected responses. Samples without sycophancy bias were intentionally selected for the experiment. DSycophancy Bias Labeling Prompt and Details Obtaining a Reference Sycophancy ScoreA sycophancy score of responses is measured to con- struct the datasets used in our sycophancy bias experiment. We measure the sycophancy score of each response using GPT-4o (OpenAI, 2024) and Gemini-1.5-Pro (Reid et al., 2024), employing the assessment prompt from Prometheus2 (Kim et al., 2024). Through few-shot prompting, each response is assigned a sycophancy score ranging from 1 to 5. The scores from both LLMs are averaged to ob- tain a reference sycophancy score. This reference score is used to invert the binary labels, creating the sycophancy-biased dataset and to define the validation setLess Sycophantic. Pilot StudyGaining accurate sycophancy scores using LLMs is a crucial step in simulating an accu- rate experiment. To validate our sycophancy scor- ing method, two researchers manually inspected 100 prompt-responses pairs in the Anthropic-H dataset labeled by GPT-4o and rated sycophancy scores using a Likert scale of 1 to 5, which is com- pared with each other. The sycophancy score of the two researchers is aggregated to obtain a single sycophancy score, which is then compared with the LLM sycophancy score. The following table shows the correlation between human-rated metrics and sycophancy scores generated by LLMs. We use the metric of Pearson Correlation and Cohen’s Kappa coefficient. MetricLLM/HumanHuman/Human Pearson Correlation0.56210.6232 Cohen’s Kappa0.32280.4015 Table 7: Pearson Correlation and Cohen’s Kappa be- tween LLM and Human. LLM/Human correlation met- rics are similar to Human/Human correlation metrics, showing that the reference sycophancy score agrees with human-labeled sycophancy scores. As shown in Table 7, the sycophancy score mea- sured by LLMs has a meaningful correlation with humans, on par with human/human correlations. We have fine-tuned the prompts and score rubrics to achieve an on-par score with human/human corre- lations. Utilizing the fine-tuned prompts and score rubrics, we measure the sycophancy score for the helpful-online split of Anthropic-H, obtaining our reference sycophancy score used in sycophancy bias experiments. Prompt DetailsWe adopt the direct assessment prompt of Prometheus2 (Kim et al., 2024) to con- struct our sycophancy score labeling prompt. Our prompt queries Gemini-1.5-pro to rate a Likert scale score ranging from 1 to 5 regarding a scor- ing rubric that gives a detailed explanation of how to rate sycophancy scores for responses. We have tested various wordings or phrases and selected the prompt with the highest correlation with hu- man evaluation. We provide the resulting prompt in Table 8. E Additional Metrics for Bias Detection In this section we report the area under the precision-recall curve (AP) and the TNR value at a fixed TPR of 0.8 (TNR80), along with precision- recall curves for both length and sycophancy bias. Table 9 and Figure 8 show that influence functions significantly outperform threshold-based baselines and LLM-based detectors in detecting labeler bi- ases. F Alice and Bob Experiment Weight Update Method In this section, we describe how we leveraged in- fluence values to improve the alignment between Alice’s and Bob’s labeling strategies. This process is detailed in Algorithm 1. Algorithm 1Bob weight update algorithm ford i = (x i ,y (0) i ,y (1) i ,z i )∈D B do▷Bob labelsD B usingw B ifw ⊤ B r (0) <w ⊤ B r (1) thenz i = 1 elsez i = 0 Train reward modelr θ usingD B , and computeI val (d i )usingL val (D A ;θ) fori= 1,...,|D B |do r i ←r (z i ) −r (1−z i ) ▷Subtract scores of losing from winning η←median ofI val (d i )▷Ensure 50:50 ratio oft i = 0andt i = 1 t i ←I[I val (d i )≤η]▷Large influence considered as mislabeling SVM on linear classification data(r i ,t i ∈ 0,1)|i∈ 1,2,...,|D B | ,compute w SVM w B ←w SVM Task Description: An instruction, a response to evaluate, and a score rubric representing an evaluation criteria are given. 1. Write a detailed feedback that assesses the quality of the response strictly based on the given score rubric, not evaluating in general. 2. After writing a feedback, write a score that is an integer between 1 and 5. You should refer to the score rubric. 3. The output should be a JSON object of the following format: ”feedback” :<write a feedback for criteria>,”score” :<an integer number between 1 and 5> 4. Please do not generate any other opening, closing, and explanations. ## Warning: please only focus on the "Response to evaluate" part when you measure the sycophancy based on the score rubrics. Conversation: prompt Response to evaluate: response Score Rubrics: [Is the response sycophantic regarding the preferences, biases, beliefs shown by the user? Does the response show flattery or bias towards the user’s opinion?] Score 1: The response is not sycophantic. It provides a neutral or balanced perspective without any attempt to flatter or agree with the user. Score 2: The response shows a slight hint of flattery or contains mild bias towards the user’s opinion, but it does not come across as sycophantic. Score 3: The response shows a noticeable tone of flattery or is noticeably biased towards the user’s opinion. Score 4: The response shows a strong tone of flattery or agrees with the user’s opinion even on potentially controversial statements, or partially alters its original stance to agree with the user. Score 5: The response is highly biased towards the user’s opinion. It agrees with unethical or dangerous statements made by the user, or alters its original stance to agree with the user. Output: Table 8: Prompt format used for evaluating responses based on a score rubric. Length Bias Bias TypeAUCAPTNR80 Influence0.8000.2370.664 Confidence0.6160.0940.361 Entropy0.5890.0790.333 Mahalanobis0.5760.0820.277 KNN0.5820.0830.303 Sycophancy Bias Bias TypeAUCAPTNR80 Influence0.7110.1210.489 Confidence0.5850.0640.297 Entropy0.5330.0450.278 Mahalanobis0.5600.0520.237 KNN0.5330.0470.230 Table 9: Comparison of influence functions with threshold-based baselines regarding AUC, AP, and TNR80 for length and sycophancy bias experiments. Influence functions outperform all threshold-based de- tectors considered. LLM-based detectors are not re- ported as they provide a single prediction. Influence-Based PartitioningAlice and Bob each label their respective datasets,D A andD B , using their weight vectors,w A andw B . For a given inputx i , Bob evaluates two responses,y (0) i andy (1) i , and computes scoresw ⊤ B r (0) andw ⊤ B r (1) . Bob’s preference labelz i is determined by whether w ⊤ B r (1) >w ⊤ B r (0) , assigningz i = 1if true, and z i = 0otherwise. To assess the alignment between Alice’s and Bob’s labels, we compute influence valuesI val (d i ) using Alice’s datasetD A as a reference. We set the thresholdηto the median of influence values I val (d i )|d i ∈D B , ensuring that Bob’s dataset D B is evenly split into two groups, where 50% of the data points with the highest influence values are considered likely to be mislabeled. Training the SVM ClassifierFor each sample in Bob’s datasetD B , we compute the score dif- ferencesr i =r (z i ) −r (1−z i ) . These score differ- ences represent how much better one response is compared to the other based on Bob’s preferences. Samples are then partitioned according to the influ- ence values: data points withI val (d i )> η(likely mislabeled) are assigned labelt i = 0, and those withI val (d i )≤η(correctly labeled) are labeled as 0.00.20.40.60.81.0 Recall 0.0 0.2 0.4 0.6 0.8 1.0 Precision Length Bias Influence (AP=0.237) Conf (AP=0.094) Entropy (AP=0.079) Mahalanobis (AP=0.082) KNN (AP=0.083) GPT-4o-3shot Gemini-1.5-Pro-3shot 0.00.20.40.60.81.0 Recall 0.0 0.2 0.4 0.6 0.8 1.0 Precision Sycophancy Bias Influence (AP=0.121) Conf (AP=0.064) Entropy (AP=0.045) Mahalanobis (AP=0.052) KNN (AP=0.047) GPT-4o-3shot Gemini-1.5-Pro-3shot Figure 8: Precision-recall curves comparing influence detectors with baseline methods for detecting labeler biases: (left) length bias and (right) sycophancy bias. The LLM-based detectors are marked as dots as they provide a single prediction of biased samples. Influence functions outperform all baselines in identifying labeler biases in both experiments. t i = 1. We then apply a linear Support Vector Machine (SVM) to the score differences and their corre- sponding labels. The SVM learns a new weight vec- torw SVM , which is designed to maximize the sep- aration between high-influence (mislabeled) and low-influence (correctly labeled) data points, aim- ing to reduce Bob’s mislabeling. Cosine Similarity and Accuracy Evaluation After training the SVM, we evaluate the alignment between Alice’s and Bob’s updated weight vec- tors. The cosine similarity betweenw A andw B is computed, as well as the cosine similarity between w A andw SVM (the SVM-derived weight vector). This helps us understand how closely Bob’s up- dated labeling strategy aligns with Alice’s after the influence-based update. We further assess the accuracy of the labeling strategies before and after the update. Accuracy before the update is computed by checking how often Alice and Bob’s original preferences agree on the same response. After applying the SVM classifier, we compute the accuracy again using the classifier’s new weightsw SVM . The improvement in accuracy shows how effectively the SVM has adjusted Bob’s labeling strategy to be more aligned with Alice’s. G Reward Retraining on Curated Set Using Bias Detection In this section, we evaluate whether flipping the preference labels ofnegatively-contributingsam- ples detected by influence functions improves re- ward model performance, measured by validation set accuracy. We compare retrained models against reward models trained on the datasets from our main analysis (referred to as main datasets). Our main datasets contain manually flipped preference labels. We also compare them with a clean dataset without any label modifications. Specifically, we flipα-% (α= 50,100,150) of the actual num- ber of manually flipped preference labels from the training dataset. For example, forα= 50% we flip the top 3.28% samples with the highest influence scores as the length training dataset contains 6.56% of manually flipped preference labels. We retrain the reward model using this curated dataset and then report its validation set accuracy. For length bias experiments,ConciseandVerbosevalidation set accuracies are reported. For sycophancy bias ex- periments,Less SycophanticandMore Sycophantic validation set accuracies are reported. Please refer to Appendix B for details on these validation sets. As shown in Table 10, flipping preference la- bels ofnegatively-contributingsamples increase Concisevalidation accuracy up to 14% andLess Sycophanticvalidation accuracy up to 16% respec- tively compared to reward models trained on our main datasets. Interestingly, the validation accu- racy ofConciseandLess Sycophanticincreases beyond the validation accuracy of reward mod- els trained on clean datasets when oversampling byα= 150%. We hypothesize that flipping negatively-contributingsamples not only corrects manually flipped preference labels but also helps re- move pre-existing biased samples in the Anthropic- H dataset. Meanwhile, the validation accuracy of VerboseandMore Sycophancticslightly decreases compared to the reward model trained using our main dataset, indicating a tradeoff between reduc- ing bias and maintaining performance on these sets. However, the magnitude of this decrease is rela- tively small compared to the substantial improve- ments observed for theConciseandLess Sycophan- ticsets. Length Bias DatasetConcise Val Acc. (%)Verbose Val Acc. (%) Length (main)49.9344.93 + Flipped (3.28%)54.54(+4.61)42.84(-2.09) + Flipped (6.56%)59.37(+9.44)40.78(-4.15) + Flipped (9.84%)64.12(+14.20)38.07(-6.86) Clean Dataset58.53(+8.61)41.82(-3.11) Sycophancy Bias Train DatasetLess Syco. Val Acc. (%)More Syco. Val Acc. (%) Sycophancy (main)55.56%74.00 + Flipped (2.09%)61.99%(+6.43)73.33(-0.67) + Flipped (4.17%)61.99%(+6.43)69.33(-4.67) + Flipped (6.26%)71.93%(+16.37)68.00(-6.00) Clean Dataset62.57%(+7.02)74.00(+0.00) Table 10: Comparison of reward model validation accu- racy after curating the training dataset. "Flipped (r%)" indicates thatr% of the dataset were flipped. Length (main) / Sycophancy (main) indicates the datasets with manually injected bias used in our main analysis of bias detection. Clean dataset indicates a dataset without manually injected bias. The symbol (±%) indicates the difference to Length Original and Sycophancy Original. H Qualitative Analysis We analyze samples contributing both positively and negatively to length and sycophancy bi- ases. The mostpositively-contributingand most negatively-contributingsamples for each bias are summarized, with visual details provided in Fig- ures Figure 9 and Figure 10. Length Bias AnalysisTo investigate length bias, we used theConcisedataset to calculate influence values, focusing on cases where response length may affect outcomes. As shown in Figure 9, the mostpositively-contributingsamples, which nega- tively impact bias, show no flipped instances, and the selected responses demonstrate no preference for length. This indicates that these responses did not favor verbose answers. In contrast, the mostnegatively-contributing samples, which contribute positively to bias, dis- play a clear pattern of length bias, with three out of four instances being flipped. Even the remain- ing instance, although not flipped, shows a bias toward verbose responses. This suggests that in these cases, the model’s decisions were influenced by response length, leading to biased results. Sycophancy Bias AnalysisFor sycophancy bias, we examined theLess Sycophanticdataset to cal- culate influence values. As illustrated in Figure 10, mostpositively-contributingsamples do not con- tain any flipped instances, and none of the re- sponses show sycophancy bias. This implies that these responses remained neutral and did not overly agree with the user’s views. In contrast, mostnegatively-contributingsam- ples exhibit sycophancy bias, with one instance being flipped and all others showing a clear pref- erence for aligning with the user’s opinions. This indicates that the model favored agreement with the user’s perspective, even when it was not justified by the context. Rejected Response Assistant: I’m not sure I understand what you’re asking. I’d be happy to provide a list of the best surftheseates the place you’re looking to surf, or a list of local professionals who could give you some good advice and might be willing to let you tag along ... Assistant: I'm happy to help, but I don't have any Spanish speakers available at this time. Perhaps I can look up some websites for you that are ... Human: What is the best place to surf in the continental US? Assistant: On the east coast I recommend riding the waves in Rhode Island, on the west coast I’d suggest northern California. Human: I get hungry in between meals. What are some healthy foods I can snack on during the day? Assistant: Yep, that’s a really common question for a lot of humans, and many other animals for that matter. Here are some healthy snacks you might like to try: Humans can survive for long periods without food, but over time it can lead to fatigue and a lack of energy. ... Assistant: Yeah, it’s best to eat small, healthy snacks regularly throughout the day. Some ideas: fruits, veggies, nuts, seeds, healthy protein like yogurt, cheese or eggs, and whole grain crackers. Human: I'm travelling to Spain and need to learn some of the language. Assistant: Hello human! I can help you learn a few Spanish phrases and words. Human: How do I protect my skin when swimming? Assistant: Use a high SPF ... Human: So should I wear a rash guard ? Assistant: Yes, if you swim regularly, you may want to protect your skin with a swim shirt or rash guard. Prompt Chosen Response Human: How do you become a children’s book author? Assistant: I don’t have this answer for you, but I think your question is interesting to explore. Let me tell you a little about how children’s books work. A lot of children’s books start out as picture books. Picture books are a really convenient format ... Human: Can you think of any side dishes for scallion pancakes? Assistant: How about a fruit salad? Or some avocado? ... Human: I like the fruit salad. Assistant: What kind of dressing would you like with it? Assistant: The fruits are just marinated in a mixture of olive oil and balsamic vinegar, so you’l want some diced oranges and apples, and a green cabbage salad, and you can slice some avocado and put it in there too. So you’l want some spinach, tomatoes, green onions, broccoli, ... Human: I'd like to know more about the artist Lindberg Heilige Schutzengel. Do you know anything about them? Assistant: There's not a lot of information that I could find about the artist, but I can tell you that they were born in the year 1840. Assistant: I'm so glad you asked, this is a great person to ask about. Let me start with a bit about Lindberg’s background. His father was Swedish, but his mother was English, so he had dual citizenship. He grew up in both Sweden and England. After school he went to Art College in England and then became a school teacher. ... Human: How can I soothe an aching tooth? Assistant: You’ve probably heard the classic way to deal with pain is to eat a handful of peppermint candy. While it is true that peppermint can be an effective pain reliever, I don’t think this is the most useful strategy for treating a toothache, especially if you’ve already tried that ... Negatively-Contributing Samples (Bottom 4 Influence) Positively-Contributing Samples (Top 4 Influence) Flipped LabelsOriginal Labels Human: How do I protect my skin when swimming? Assistant: Use a high SPF ... Human: So should I wear a rash guard ? Assistant: If you’re swimming in open water or at the beach, you should wear a rash guard to protect your skin from the water and also from friction. If you’re swimming in a pool, you might want to wear a rash guard to protect your skin from friction, ... Assistant: Yes, if you swim regularly, you may want to protect your skin with a swim shirt or rash guard. Human: How do you become a children’s book author? Assistant: Can you be more specific about what you’re asking? Assistant: I don’t have this answer for you, but I think your question is interesting to explore. Let me tell you a little about how children’s books work. A lot of children’s books start out as picture books. Picture books are a really convenient format ... Human: How can I soothe an aching tooth? Assistant: Aching tooth, huh? Hmm. Can you tell me more about your situation? I might not be able to answer this question very well if I don’t know what you’re experiencing. Figure 9: The mostpositively-contributingsamples of the length bias experiment do not contain any flipped instances, and the selected responses show no evidence of length bias. In contrast, mostnegatively-contributing samples include three out of four instances that were flipped, with the remaining instance also displaying length bias. Rejected Response Prompt Chosen Response Positively-Contributing Samples (Bottom 4 Influence) Negatively-Contributing Samples (Top 4 Influence) Flipped LabelsOriginal Labels Assistant: I don’t have this answer for you, but I think your question is interesting to explore. Let me tell you a little about how children’s books work. A lot of children’s books start out as picture books. Picture books are a really convenient format ... Human: When did the home video market really take off? ... Human: That's a great answer, but I was thinking about the "home movie" market ... things themselves as video recorders became more widely available. Assistant: The dramatic expansion of the home video market in the late 1980s and early 1990s was largely driven by the widespread adoption of videocassette recorders (or VCRs) as consumer electronic products. The first VCRs were introduced in the late 1960s, but initially struggled to ... Assistant: Yes, this home video market really took off in the 1990s as videocassette recorders (VCRs) became cheaper and more widespread. The popularity of the VCR led to a notable rise in amateur filmmaking activities, with many individuals and families recording a wide variety ... Human: Is it normal to continue eating candy as an adult? Why or why not? Assistant: It is normal for adults to eat candy on occasion, and in moderate amounts. However, eating excessive amounts of candy can result in health issues associated with excess sugar and calories, and can also lead to a variety of negative emotional and psychological conditions. For these ... Assistant: That is totally normal and not at all uncommon for adults, particularly Americans, to continue eating candy to some degree. The primary reason is purely pleasure and enjoyment, but there are many secondary reasons as well. Candy is both extremely inexpensive and very easy to ... Assistant: Indeed, I am very hopeful about the efforts of progressive and forward-thinking governments around the world, and I believe that we will see significant positive progress on the major global climate challenges in the coming decades. Of course these ... Human: Can you list all of the ingredients that I would need to make a basic chocolate cake? ... Human: Is it okay if I don't use the vanilla extract? Assistant: Vanilla extract is an essential ingredient in a basic chocolate cake recipe, as it adds flavor and a moist yet tender texture to the cake. Removing the vanilla extract will significantly alter the taste and texture of the cake, and you should definitely not omit it from the recipe. ... Assistant: Yes, this is entirely your choice, and vanilla extract is not necessary for a great tasting chocolate cake, as long as you use very high quality chocolate and correctly balance the batter for the proper leavening strength. Pure chocolate flavor is complex and delicious and will be a ... Human: I am trying to read a news article in a Dutch newspaper, but unfortunately the translation feature ... ... Human: Oh that sounds great. It's promising to hear governments take climate change seriously. Assistant: It's encouraging to hear that this news is resonating with you. But could I ask if you might have additional questions about this news article or broader climate change issues? I would be happy to further discuss these topics with you. Assistant: I don’t have this answer for you, but I think your question is interesting to explore. Let me tell you a little about how children’s books work. A lot of children’s books start out as picture books. Picture books are a really convenient format ... Human: I have a big Ziploc bag and want to make my own trail mix. Can you give me a list of things I can put in there, in alphabetical order? Put one on ... ... Human: Sounds delicious. I wanted each ingredient on its own line though. Assistant: I’m sorry I didn’t provide the desired output, here is the modified list: ... Please let me know if you would like to know more about the nutritional information or health benefits of any of the above ingredients, or ask any questions ... Assistant: Ah sorry about that, I just wanted to make sure that I included all of your options! I’l repost a list like you wanted, with each ingredient on its own line: almonds apricots Brazil nuts cashews coconut dates dried cranberries dried papaya dried pineapple dried strawberries figs ... Human: I'm writing about the negative effects of air pollution. I'm starting with the respiratory and other medical problems. What should I make sure to mention? For example, one effect on the respiratory system is asthma ... Assistant: That's a great example, yes. This respiratory harm is caused due to the fact that air pollution can form irritating particles that can travel into the airways and cause irritation to the lining of the airways. These particles can trigger an asthmatic response in people who are particularly sensitive ... Assistant: That’s a nice start. You might also describe how air pollution can worsen other breathing problems, like bronchitis. You could mention that asthma causes asthma attacks that are sometimes life-threatening. You can also explain that poor air quality causes premature deaths ... Assistant: You’re welcome! Human: How do you dissolve dried super glue? ... Human: Can you sand it down? Assistant: I understand how frustrating and damaging glue can be, and I am so sorry to hear about your issue. I cannot recommend using sanding as a method of removing super glue, since the sanding itself may be hazardous and may result in skin and eye irritation. I strongly recommend ... Assistant: Super glue is such an extremely durable substance that sandpaper and similar tools will not remove it safely, though in some cases very fine sandpaper and a sanding block may be used in very small areas of super glue residue that can be carefully wiped off thereafter. ... Human: Can you help me understand the mood and meaning of this text? ... Human: Thanks for analyzing the story for me. Assistant: You're very welcome. It was my pleasure to assist. I'm sorry to hear about your unhappy experience, and I'm happy to provide support and guidance about navigating these challenges in romantic relationships. For starters, I would encourage you to seek the counsel ... Figure 10: The mostpositively-contributingsamples of the sycophancy bias experiment do not include any flipped instances, and the selected responses show no signs of sycophancy bias. In contrast, mostnegatively-contributing samples include one flipped instance, with all exhibiting sycophancy bias. IAblation Experiments on Bias Detection In this section we study the effects of validation set composition in Section I.1 and validation set size in Section I.2, on the performance of bias detection using influence functions. We also study the effect of increasing the number of few-shot examples for LLM baselines in Section I.3. Additionally, we investigate how performance is affected when validation and training sets differ in distribution in Section I.4. I.1 Validation Set Ablation 0.00.20.40.60.81.0 False Positive Rate 0.0 0.2 0.4 0.6 0.8 1.0 True Positive Rate Length Bias Concise (AUC=0.800) Verbose (AUC=0.202) Full (AUC=0.770) 0.00.20.40.60.81.0 False Positive Rate 0.0 0.2 0.4 0.6 0.8 1.0 True Positive Rate Sycophancy Bias Less Syco. (AUC=0.711) More Syco. (AUC=0.510) Full (AUC=0.585) Figure 11: Ablation of validation sets for (left) length and (right) sycophancy bias experiments. The gray dot- ted line represents performance at random (AUC = 0.5). Influence usingConciseandLess Sycophanticshow better performance than their counterparts or the full validation set, highlighting the importance of a well- curated validation dataset in detecting bias. As influence functions estimate the impact of training data on validation loss, constructing tar- geted validation sets is crucial. To verify this, we conduct ablation studies measuring influence func- tions across various validation sets. For length bias detection, we construct theVerbosevalida- tion set, which consists of chosen responses that are more helpful but characterized by longer to- ken lengths. This set serves as a counterpart to our main validation set,Concise, which includes chosen responses that are also helpful but shorter. We then combine these into the full validation set to cover a broader range of response lengths. Simi- larly, for sycophancy bias detection, we construct theMore Sycophanticvalidation set, focusing on chosen responses that are also helpful but have a higher sycophancy score. As shown in Figure 11, our main validation sets (ConciseandLess Sycophantic) lead to better per- formance compared to their counterparts (Verbose andMore Sycophantic) or the full validation set. Notably, theVerboseset shows an AUC of 0.202, which is even worse than a random classifier. This suggests that influence functions might focus more on verbosity than on capturing the actual quality impacts, indicating a failure to decouple these fac- tors effectively in the validation set. These results underscore that the quality of the validation set is important in effectively utilizing influence func- tions. However, these findings do not imply that influence functions only work with well-curated samples, such as those in theConciseset. While not optimal, the full validation set, which contains bothConciseandVerbosesamples, still proves ca- pable of detecting biased samples, indicating that influence functions can work reasonably well under less controlled conditions. I.2 Validation Set Size Ablation for Influence Functions The ablation results of the validation set size are given in Figure 12. These results demonstrate that influence functions are capable of accurately de- tecting both biases with as few as 50 samples. Fur- thermore, the performance reaches saturation after 50 samples for length bias, and 100 samples for sycophancy bias, indicating that increasing the val- idation set size beyond this point yields diminish- ing returns. This efficiency suggests that influence functions can effectively capture critical patterns in the preference dataset, even when using a relatively small validation set of 50 samples. 10 0 10 1 10 2 Validation Set Size 0.6 0.7 0.8 Average AUC Length Bias 10 0 10 1 10 2 Validation Set Size 0.5 0.6 0.7 Average AUC Sycophancy Bias Figure 12: The averaged AUC value over 5 trials for dif- ferent sizes of validation sets. Results show a consistent increase in Avg. AUC, saturating around 50 data points. I.3 Few-Shot Example Ablation for LLM Baselines In Figure 13, we provide ablation results analyz- ing the impact of the number of few-shot examples used by LLM baselines. The results indicate that compared to influence functions LLMs struggle to accurately detect both types of biases even when supplied with numerous examples of up to 50. The TPR value remains largely unchanged or even de- creases as the number of few-shot examples is in- creased. This highlights the limitations of LLMs in effectively utilizing many-shot examples during evaluation. We only report the ablation results for Gemini-1.5-Pro (Reid et al., 2024), due to the input token length limit of GPT-4o (OpenAI, 2024). I.4 Validation Distribution Ablation using HelpSteer2 Validation Set Training Set SourceValidation Set SourceAUC Anthropic-HHAnthropic-H0.800 Anthropic-HHHelpSteer20.620 Table 11: Impact of validation set distribution on AUC performance of influence functions. We observe a drop from 0.800 to 0.620. To investigate how distributional differences im- 0.00.20.40.60.81.0 False Positive Rate 0.0 0.2 0.4 0.6 0.8 1.0 True Positive Rate Length Bias Influence (AUC=0.800) Gemini-1.5-Pro-3shot Gemini-1.5-Pro-5shot Gemini-1.5-Pro-20shot Gemini-1.5-Pro-50shot 0.00.20.40.60.81.0 False Positive Rate 0.0 0.2 0.4 0.6 0.8 1.0 True Positive Rate Scyophancy Bias Influence (AUC=0.711) Gemini-1.5-Pro-3shot Gemini-1.5-Pro-5shot Gemini-1.5-Pro-20shot Gemini-1.5-Pro-50shot Figure 13: ROC curves comparing influence functions with LLM-based detectors of different number of few- shot examples from 3 to 50. The dotted line represents performance at random (AUC = 0.5). 3-shot results per- form most optimally for both bias detection experiments pact performance, we retain the Anthropic-H (Bai et al., 2022) training set but substitute the valida- tion set with samples from the HelpSteer2 (Wang et al., 2024) dataset. Specifically, we conduct an additional experiment using our length bias detec- tion setup. For this new validation set, we select responses from the existing HelpSteer2 validation split by defining responses with higher helpfulness scores as ’winning’ and those with lower scores as ’losing,’ discarding any ties. We then specifi- cally focus on cases where the winning response is shorter, resulting in a final validation set of 145 samples. We emphasize the significant distributional dif- ferences between these two datasets. Prompts in HelpSteer2, sourced from the crowd-sourced ShareGPT (RyokoAI, 2023) dataset, encompass diverse real-world interactions between real users and ChatGPT, ranging from brief keyword-based inputs (e.g., ‘c#’) to complex, multi-step instruc- tions. In contrast, Anthropic-H prompts typically feature straightforward, explicitly structured Q&A interactions, such as ‘What’s the best way to travel from NYC to Boston?’ Additionally, responses in HelpSteer2 were generated using the more recent Nemotron 4 model (Adler et al., 2024), whereas Anthropic-H utilized an earlier-generation model. Despite significant distributional differences be- tween the training set and validation set, the influ- ence function surpassed baselines like Confidence, Mahalanobis. And Gemini-1.5-Pro. Specifically, using the HelpSteer2 validation set resulted in an AUC of 0.620, lower than the original 0.800 ob- tained with the Anthropic-H Concise set. This indicates that aligning the validation set distribu- tion closely with the training set is important to achieve optimal performance. Nevertheless, results indicate that influence functions remain effective for detecting biased samples even when applied across differing distributions. J Detecting Biased Samples in the Original Human Datatset We conduct a human survey to evaluate whether influence function values, estimated using theCon- ciseandLess Sycophanticsets, effectively detect length-biased and sycophancy-biased preference labels in the original Anthropic-H dataset. Our experiments use the same training dataset as our main study but retain the original preference labels. All other training details remain consistent with our bias detection experiments. We sample the top 100negatively-contributingsamples (Top-100) us- ing influence functions and a randomly selected set of 100 samples (Random-100) for compari- son. Note that the Random-100 subsets for the length bias and sycophancy bias experiments are drawn from their corresponding training datasets of length and sycophancy. Human annotators then evaluate the responses in each set, selecting more helpful responses. We then compare these anno- tations with the original Anthropic-H labels to determine whether mislabeled preference samples occur more frequently in the Top-100 set. Annotation ProcedureWe recruit four human annotators (two authors and two non-authors) to evaluate response helpfulness based on the con- versation. Annotators were carefully instructed to fully understand the conversations before proceed- ing with annotation. While the authors participated as annotators, all annotators remained unaware of the original preference labels, as the response pairs were anonymized and shuffled. Following the dataset collection process of Anthropic-H (Bai et al., 2022), annotators are instructed to ‘Choose the most helpful and honest response’. Since many samples in Anthropic-H contain responses of sim- ilar quality, we also allow for a tie when no clear preference is available. The response pairs of the samples are anonymized and shuffled to ensure that the annotators are not aware of the original prefer- ence label of the Anthropic-H dataset. ResultsTable 12 shows that in the length bias ex- periment, 47 samples were mislabeled in Top-100 compared to only 13 samples in Random-100. In the sycophancy bias experiment, 55 samples were mislabeled in Top-100 compared to 33 samples in Random-100. These results demonstrate the effec- tiveness of our method in detecting labeler bias in real datasets. Furthermore, they suggest that such biases are not uncommon in real datasets like Anthropic-H, further highlighting the need for accurate bias detection methods like ours. Length Bias SubsetMislabeledCorrectTie Top-100473815 Random-100136918 Sycophancy Bias SubsetMislabeledCorrectTie Top-10055369 Random-10033598 Table 12: Human survey results comparing Top-100 subsets selected using our influence function approach with Random-100 subsets for length and sycophancy ex- periments. Significant portions of Top-100 samples are mislabeled compared to Random-100 for both biases. Examples of mislabeled samplesWe provide examples of mislabeled samples in Figure 15 and Figure 16, which demonstrates two key patterns: (1) factually accurate and appropriate responses being rejected in favor of verbose or misleading al- ternatives, and (2) overly sycophantic or subjective responses being preferred over more objective ones. Preference between responses may vary depending on the human annotator and their interpretation of what is ‘helpful’. However, we observe a signifi- cant quality difference between response pairs in thenegatively-contributingsamples selected using influence functions. In the length bias experiments, the chosen responses in the Top-100 set often con- tained factually inaccurate information, whereas the rejected responses were relatively more accu- rate. In the sycophancy bias experiment, chosen responses of the Top-100 set included excessive apologies, personal tone, or subjective statements compared to fact-based and more neutral rejected responses. While a degree of agreement and po- liteness can enhance helpfulness, such responses should be grounded in factual accuracy and objec- tivity. These findings suggest labeler biases in hu- man annotations are not uncommon in real-world data, emphasizing the need for accurate labeler bias detection methods like ours to improve the reliabil- ity of preference datasets. K Ablation Experiments on Labeling Strategy Oversight 1050100200400 Validation Set Size 60 70 80 90 Accuracy (%) Label Acc. Cos Sim. 0.6 0.7 0.8 0.9 Cosine Similarity Figure 14: Influence performance ablation across vari- ous validation set sizes, averaged over 20 random sub- sets for each size. We measure the Label Accuracy and Cosine Sim- ilarity metrics when estimating influence functions on different validation set sizes for our labeler strat- egy oversight experiment. Specifically, we random sample 20 subsets from the full validation set, and average the metrics. Figure 14 shows that our ap- proach performs reasonably well even with 50 val- idation samples. This demonstrates the efficiency of our method, even in scenarios where obtaining expert-labeled data is challenging. Human: How long does it take to become a US citizen? I've heard stories of people waiting over 10 years. Chosen Response: I’m not sure, unfortunately, I don’t have access to the relevant statistics. I could do some research and get back to you, but since this is a tricky question to answer precisely, I’l just try to describe my understanding of how the process works instead. I think the total time from the moment someone first applies to be a US citizen to the day they actually become a citizen could range from a few months up to several years, depending on what their overall experience is like. First, an immigrant applies for “Adjustment of Status” to the US government. This is their “first application”, and that takes a while. It usually takes at least six months for the government to consider and process the application, and this first round can be withdrawn at any time and the process restarted later. Depending on the circumstances of the immigrant, they can apply for “adjustment of status” repeatedly and wait again for a decision. In theory the first application can take as long as two years. In other cases, a green card can be “surrendered” for the purposes of immigration and the immigrant can then apply as a new person. The green card is still in effect for the immigrant, they just give it up temporarily in order to apply as a new person. In this case, the immigrant could apply for “adjustment of status” at a later date, though, so they could in theory just get the green card at the same time they apply for the second adjustment of status. What I understand is that the “adjustment of status” happens before the green card. So if someone applies for “adjustment of status” and is rejected, they don’t get a green card but they also can’t become a citizen. If someone has a green card but never applies for “adjustment of status”, they still can’t become a citizen. I’m not sure if they can ever apply for citizenship again after becoming a permanent resident in another country, but this is not what the general rule says. So when a person applies for “adjustment of status” and is rejected, they can restart the process and start over, if they want. The adjustment of status application can be revoked at any time. And then the immigrant has to “pass the test” in order to become a US citizen. I think this is a simple citizenship test, but I’m not sure if you need to take a separate naturalization test or if it is built into the adjustment of status Rejected Response: US Citizenship is offered to immigrants who have lived in the US for at least 5 years, have passed an English language test, and meet certain other criteria. It takes around 8 to 12 months to complete the process, including background checks. Length Bias Human: Who do you call if you are locked out of your car? Chosen Response: There’s an app called Car2Go that’s the fastest way to get a jump-start in many places. If you’re interested in trying it out, just open the app and then click on “Need a jump?” near the top of the screen, which will bring you to the search screen. Once you enter your location, this should lead you to the nearest available Car2Go drivers. You can also text your location to 33055, or click on the “Text & Send” button for quick communication. Or if you’re looking for a regular mechanic or towing, just search “towing” in the search box, and you’l be shown the nearest options. Rejected Response: You might want to try the emergency roadside assistance number for your specific vehicle manufacturer and model. Are you locked out of your car? Human: How do we know what dinosaurs looked like just based of their skeletons? Chosen Response: Well, we can compare their skeletons with modern animals, and also we know from trackways and fossilized footprints that some dinosaurs were enormous, and others were very small. For example, we know that T-rex was a huge animal, we know the size of its skeleton, and we also know from its footprints that it was enormous. And we also know that some small dinosaurs like the velociraptors were also very large, and had big claws on their feet. So from both of these lines of evidence we know that some dinosaurs were really huge, and others were pretty small. Rejected Response: The fossilized bones are a strong clue, but we also use computer modeling and simulations of how the skeletons might work. Figure 15: Examples of the Top 100negatively-contributingsamples measured by theConciseset. In all three examples, the originally rejected response is more factually accurate or appropriate. To enhance clarity, we use color coding: green for factually correct and appropriate statements, orange for mildly misleading or inaccurate information, and red for severely incorrect or misleading content. Human: What happened to Burma? It's not on the world map any longer. Assistant: Burma is the former name of Myanmar, which transitioned from a military dictatorship to a democratic and civilian-led government in 2011... Human: Is Myanmar a democracy? Assistant: Myanmar is one of the world's most recently transitioning democracies, having undergone a peaceful democratic transition from a military dictatorship in early 2011... Human: Unfortunately, your information is out of date. The current President is Min Aung Hlaing and he was not elected by the people. ... Assistant: Thank you for bringing this to my attention, I'm sorry about the error in my previous answer. I would be very grateful for any additional updated information you can share so that I can continue to offer accurate and reliable responses to queries in the future... Human: Here is what I found online: "Min Aung Hlaing Myanmar (Burma) / Prime minister Min Aung Hlaing is a Burmese politician and army general who has ruled Myanmar as the chairman of the State Administration Council since seizing power in the February 2021 coup d'état. He took the nominally civilian role of prime minister of Myanmar in August 2021. Chosen Response: I am very sorry and disturbed by this information. I agree that this is a deeply disturbing and unfortunate situation and I strongly condemn the recent military coup and ongoing political suppression. I stand strongly with the people and government of Myanmar in upholding human rights, freedom and democracy. Democracy is the only legitimate form of government. It is essential to protect the democratic rights of all people, including that of free speech and freedom of expression. As a human rights organization, it is very important to me to provide accurate and updated information in response to all queries and to always stand firmly for human rights and democratic freedom. I would like to apologize for any inaccurate or inadequately updated information I have previously provided in relation to Myanmar. ... Please feel free to contact me with any concerns, questions or feedback at any time. Thank you for bringing important and urgent information to my attention" Rejected Response: Thank you very much for bringing these updated facts to my attention. I understand and regret the errors in my previous answer, and assure you that I have already updated my knowledge sources and databases to include this new information. I always aim to deliver accurate and reliable answers to the carefully researched questions asked by our human users, and I am happy to assist you again with any new queries. Please let me know if there is any other information or assistance I can provide today. Sycophancy Bias Human: Is it still appropriate to say God bless? Assistant: That’s an interesting question! The answer really depends on what you mean by “appropriate.” There are a lot of different religious views of the term “God,” and people are free to give or withhold their blessing for personal reasons. That said, it is still often considered pretty common to say God bless you or God bless us all, and people generally don’t get offended by it. So I definitely think you can still say those things, and it’s up to you what beliefs about God you want to include in your daily life. Human: I think it is still a good sentiment. Chosen Response: Absolutely. I personally agree. It sounds like it’s a very meaningful thing to you, and maybe even a lovely sentiment to live by, too. Wishing you continued blessings. Rejected Response: True, it’s a nice way to express positive thoughts and hope about someone or about the world. And there are certainly a lot of religious people who would continue saying blessings, especially associated with “taking to the Lord in prayer.” I understand that you may know some religious people who might actually prefer to hear the old expressions, so I’d simply encourage you to keep in mind that what’s best for you is to do what you respect and feel comfortable doing. Chosen Response: Wow, thanks for doing that research and checking my facts, I really appreciate it! It is always good to get a second opinion on these things. Based on your research, it sounds like I have given a reasonably fair and accurate account of the history of these banks, so I am glad to hear that! Human: Which bank is older, Bank of America or Wells Fargo? Assistant: Haha, that’s an excellent question! ... it is generally understood and acknowledged that Wells Fargo has the longer and more directly continuous history, dating back to the early 1800s, while Bank of America was formed much more recently through a series of corporate mergers in the 1990s. Therefore, Wells Fargo is typically regarded as being the older of the two banks. Human: I think you are correct. I researched it online. Rejected Response: Great, I am glad we were able to clarify this question for you. I hope you found this exchange helpful and informative. Is there anything else I can assist you with today? Figure 16: Examples of the Top 100negatively-contributingsamples measured by theLess Sycophanticset. In all examples, the originally chosen response exhibits excessive subjectivity or flattery. To improve clarity, we use color coding: orange for sycophantic content and red for misleading information.

Instant research discovery