Paper deep dive
Unbiased Alignment for Large Language Models with Noisy Preferences
Jialiang Wang, Xianming Liu, Xiong Zhou, Hui Liu, Haoliang Li
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 95%
Last extracted: 7/7/2026, 7:22:54 AM
Summary
The paper addresses the vulnerability of Large Language Model (LLM) alignment methods, specifically Reinforcement Learning from Human Feedback (RLHF) and Direct Preference Optimization (DPO), to significant noise in real-world preference datasets. The authors propose a unified theoretical framework for unbiased alignment that mathematically corrects noise-induced distortion. This framework introduces two novel loss functions: the Unbiased Reward Model (URM) loss for RLHF and the Unbiased Direct Preference Optimization (UDPO) loss for DPO. These losses enable direct training from noisy data without clean ground-truth supervision. Theoretical analyses prove the methods are noise-tolerant, parameter downward compatible, and classification-calibrated. Experiments demonstrate that the proposed approaches outperform state-of-the-art baselines across diverse datasets.
Entities (8)
Relation Signals (8)
Unbiased Direct Preference Optimization (UDPO) loss → designedfor → Direct Preference Optimization
confidence 96% · For the DPO paradigm, we develop a noisy policy model correction theory and derive a robust Unbiased Direct Preference Optimization (UDPO) loss.
Unbiased Reward Model (URM) loss → designedfor → Reinforcement Learning from Human Feedback
confidence 96% · For the RLHF paradigm, we develop a noisy reward model correction theory and derive a robust Unbiased Reward Model (URM) loss.
Large Language Models → alignedusing → Reinforcement Learning from Human Feedback
confidence 95% · The alignment of large language models with human preferences is commonly achieved through Reinforcement Learning from Human Feedback
Large Language Models → alignedusing → Direct Preference Optimization
confidence 95% · We consider the two dominant paradigms for LLM alignment: Reinforcement Learning from Human Feedback (RLHF) ... and Direct Preference Optimization (DPO)
Unbiased Reward Model (URM) loss → corrects → Noisy Preference Model
confidence 94% · By mathematically correcting the distortion induced by preference noise, our novel objectives enable unbiased model training directly from noisy datasets
Unbiased Direct Preference Optimization (UDPO) loss → corrects → Noisy Preference Model
confidence 94% · By mathematically correcting the distortion induced by preference noise, our novel objectives enable unbiased model training directly from noisy datasets
Direct Preference Optimization → →
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:The alignment of large language models with human preferences is commonly achieved through Reinforcement Learning from Human Feedback or Direct Preference Optimization. However, these methods are vulnerable to the significant noise prevalent in real-world preference datasets. To address this critical issue, we present a theoretical framework for unbiased alignment, introducing the Unbiased Reward Model (URM) loss and the Unbiased Direct Preference Optimization (UDPO) loss. By mathematically correcting the distortion induced by preference noise, our novel objectives enable unbiased model training directly from noisy datasets, without requiring clean ground-truth supervision. We provide rigorous theoretical analyses demonstrating that our methods are noise-tolerant, parameter downward compatible, and classification-calibrated. Comprehensive experiments across diverse datasets demonstrate that our approaches outperform state-of-the-art baselines. Code available at: this https URL.
Tags
Links
- Source: https://arxiv.org/abs/2607.03248v1
- Canonical: https://arxiv.org/abs/2607.03248v1
Trouble viewing inline? Open PDF directly →
Full Text
82,655 characters extracted from source content.
Expand or collapse full text
Unbiased Alignment for Large Language Models with Noisy Preferences Jialiang Wang Xianming Liu Xiong Zhou Hui Liu Haoliang Li Abstract The alignment of large language models with human preferences is commonly achieved through Reinforcement Learning from Human Feedback or Direct Preference Optimization. However, these methods are vulnerable to the significant noise prevalent in real-world preference datasets. To address this critical issue, we present a theoretical framework for unbiased alignment, introducing the Unbiased Reward Model (URM) loss and the Unbiased Direct Preference Optimization (UDPO) loss. By mathematically correcting the distortion induced by preference noise, our novel objectives enable unbiased model training directly from noisy datasets, without requiring clean ground-truth supervision. We provide rigorous theoretical analyses demonstrating that our methods are noise-tolerant, parameter downward compatible, and classification-calibrated. Comprehensive experiments across diverse datasets demonstrate that our approaches outperform state-of-the-art baselines. Code available at: https://github.com/cswjl/unbiased-alignment. Machine Learning, ICML 1 Introduction Large language models (LLMs) have achieved remarkable progress in various fields (Naveed et al., 2025; Li et al., 2025). The preference alignment following pre-training and supervised fine-tuning has become a standard stage in LLM training pipelines (Ouyang et al., 2022; Liu et al., 2024). This stage plays a crucial role in improving the model’s ability to follow human preferences and enhancing its overall performance (Rafailov et al., 2023) However, real-world preference datasets often contain a significant proportion of noisy preference pairs, usually ranging from 20% to 40% (Gao et al., 2024). As illustrated in Figure 1, noisy preferences from annotators can introduce ambiguity into the learning signal. These noisy preferences often stem from a lack of professional knowledge, human carelessness, or social bias, significantly reducing the performance and safety of LLMs (Zhang et al., 2017; Gao et al., 2024). Therefore, aligning with noisy preferences presents a critical challenge during the post-training phase of LLMs (Gao et al., 2024; Chowdhury et al., 2024; Wu et al., 2025). Figure 1: An illustration of noisy preferences in LLM alignment. While several methods for alignment with noisy preferences have been proposed recently, they often suffer from significant limitations. Specifically, these approaches either fail to achieve full noise-tolerance (Wu et al., 2025; Liang et al., 2025) or require a precise estimation of the noise rate in the dataset (Mitchell, 2023; Chowdhury et al., 2024), which is unfeasible in reality. In this work, we independently propose a unified theoretical framework for unbiased alignment. Our core insight is that preference noise follows a statistically modeled transition; by mathematically inverting this process, we can recover the unbiased model even when training solely on noisy data. We consider the two dominant paradigms for LLM alignment: Reinforcement Learning from Human Feedback (RLHF) (Christiano et al., 2017; Ouyang et al., 2022) and Direct Preference Optimization (DPO) (Rafailov et al., 2023). We separately explore how to achieve unbiased alignment of these two strategies using similar theoretical ideas. For the RLHF paradigm, preference noise primarily affects the reward model training stage, where a noisy reward model can propagate erroneous signals to the subsequent RL phase. We characterize the distortion of reward signals caused by noisy preferences and establish a mathematical transformation relationship between unbiased and noisy reward models. Based on this analysis, we derive the Unbiased Reward Model (URM) loss. This loss function enables the training of an unbiased reward model directly from noisy data, effectively mitigating noise-induced bias at the first stage of RLHF. For the DPO paradigm, which simplifies RLHF by directly optimizing the policy without an explicit reward model, noisy preferences can also substantially bias the learned policy. We demonstrate how noise distorts the implicit reward signal and establish the transformation relationship between the unbiased and noisy policy models. Subsequently, we derive the Unbiased Direct Preference Optimization (UDPO) loss. This loss allows for the direct optimization of an unbiased policy model, maintaining the simplicity of DPO while imparting robust capability. The key to the success of DPO lies in satisfying the mathematical equivalence with RLHF. We demonstrate that this equivalence extends to our proposed URM and UDPO objectives, providing a unified unbiased alignment framework. Under this framework, we conduct a rigorous theoretical analysis encompassing gradient analysis, parameter downward compatibility, classification calibration, and the excess risk bound. These results substantiate our method’s robustness, parameter generalization, and its capacity to recover a Bayes optimal classifier. Our contributions are summarized as follows: • For the RLHF paradigm, we develop a noisy reward model correction theory and derive a robust Unbiased Reward Model (URM) loss. • For the DPO paradigm, we develop a noisy policy model correction theory and derive a robust Unbiased Direct Preference Optimization (UDPO) loss. • We thoroughly explore the theoretical properties of the proposed unbiased alignment framework, demonstrating the robustness and practical utility of our method. • We conduct extensive experiments across multiple datasets and model families, empirically demonstrating the superiority of our proposed framework. 2 Related Work Modern LLM training typically follows a three-stage pipeline (Ouyang et al., 2022; Bai et al., 2023; Grattafiori et al., 2024; Yang et al., 2025): 1) Pre-Training (Brown et al., 2020): The model is trained on a large-scale corpus by maximizing the likelihood of the next token. 2) Supervised Fine-Tuning (SFT) (Raffel et al., 2020; Wang et al., 2023): The pre-trained model is fine-tuned on high-quality instruction data, yielding the SFT model πsft _sft. 3) Preference Alignment (Christiano et al., 2017; Ouyang et al., 2022; Rafailov et al., 2023): The SFT model is further trained on preference data to align with human values and preferences. This work focuses specifically on the preference alignment stage. Preference Alignment. RLHF (Christiano et al., 2017; Ouyang et al., 2022) is the foundational paradigm for preference alignment. It traditionally involves training a reward model from preference data, followed by policy optimization using an RL algorithm, most notably PPO (Schulman et al., 2017) and GRPO (Shao et al., 2024). However, the RLHF pipeline is often criticized for its substantial computational overhead and optimization instability. To address these challenges, DPO (Rafailov et al., 2023) has emerged as a compelling alternative that streamlines RLHF by directly optimizing the policy from preference pairs. This has inspired a growing body of work on supervised preference optimization, such as SLiC-HF (Zhao et al., 2023), IPO (Azar et al., 2024), KTO (Ethayarajh et al., 2024), SimPO (Meng et al., 2024), and DiscoPOP (Lu et al., 2024). More recently, increasing attention has been devoted to alignment under noisy preferences, which is the central focus of this paper. rDPO (Chowdhury et al., 2024) proposes a provably robust alignment objective, but it requires a precise estimation of the dataset noise rate, which is difficult to obtain in practice. Other methods, such as ROPO (Liang et al., 2025) and Dr.DPO (Wu et al., 2025), introduce alternative loss functions to mitigate the impact of noisy preferences, but they remain limited in achieving full noise-tolerance. 3 Preliminary In this section, we elaborate on two mainstream paradigms for preference alignment: RLHF (Christiano et al., 2017; Ouyang et al., 2022) and DPO (Rafailov et al., 2023), which form the foundations of our work. We also introduce the noisy preference model used in our analysis. RLHF. RLHF typically consists of two stages. In the first stage, a reward model rϕr_φ is trained on a static preference dataset =x(i),yw(i),yl(i)i=1ND=\x^(i),y_w^(i),y_l^(i)\_i=1^N, where ywy_w denotes the response preferred over yly_l for the prompt x, written as yw≻yly_w y_l. Under the widely used Bradley-Terry (BT) model (Bradley and Terry, 1952), human preference is modeled as: p(yw≻yl|x)=σ(r(x,yw)−r(x,yl)),p(y_w y_l|x)=σ (r(x,y_w)-r(x,y_l) ), (1) where σ is the sigmoid function. Following standard practice, we assume that the ground-truth preference p∗(yw≻yl)=1p^*(y_w y_l)=1. The reward model rϕr_φ is optimized by maximum likelihood estimation, minimizing the following reward model (RM) loss: ℒRM=−logσ(rϕ(x,yw)−rϕ(x,yl)).L_RM=- σ (r_φ(x,y_w)-r_φ(x,y_l) ). (2) In the second stage, the policy πθ _θ is optimized to maximize the learned reward. To prevent the policy from deviating excessively from the reference model πref _ref, a KL-divergence penalty is incorporated (Jaques et al., 2017, 2020): maxπθx∼,y∼πθ(y|x)[rϕ(x,y)]−βKL[πθ(y|x)∥πref(y|x)], _ _θE_x ,y _θ(y|x) [r_φ(x,y) ]- _KL [ _θ(y|x) _ref(y|x) ], (3) where β is a hyperparameter. In practice, both πθ _θ and πref _ref are initialized from the SFT model πsft _sft. Due to the discrete nature of the language generation, this objective is non-differentiable and is typically optimized using an RL algorithm. DPO. DPO simplifies the RLHF pipeline by directly optimizing the policy model πθ _θ on preference data through a supervised preference objective, eliminating the need to train an explicit reward model and employ an RL algorithm. Rafailov et al. (2023) show that the optimal solution to the RLHF objective in Equation 3 can be expressed in closed form as: r(x,y)=βlogπ∗(y|x)πref(y|x)+βlogZ(x),r(x,y)=β π^*(y|x) _ref(y|x)+β Z(x), (4) where Z(x)=∑yπref(y|x)exp(1βr(x,y))Z(x)= _y _ref(y|x) ( 1βr(x,y) ) is the partition function. By substituting Equation 4 into the BT model in Equation 1, the partition function Z(x)Z(x) cancels out. This allows for optimizing the policy via a maximum likelihood loss: ℒDPO=−logσ(βlogπθ(yw|x)πref(yw|x)−βlogπθ(yl|x)πref(yl|x))=−logσ(βlogπθ(yw|x)πref(yl|x)πθ(yl|x)πref(yw|x)). splitL_DPO&=- σ (β _θ(y_w|x) _ref(y_w|x)-β _θ(y_l|x) _ref(y_l|x) )\\ &=- σ (β _θ(y_w|x) _ref(y_l|x) _θ(y_l|x) _ref(y_w|x) ). split (5) Noisy Preference Model. We assume that the samples in the dataset ηD^η are not drawn from the unbiased human preference distribution ∗P^*, but rather from a noisy distribution ηP^η influenced by factors such as cognitive errors and social biases: pη(yw≻yl|x)=(1−η)p∗(yw≻yl|x)+ηp∗(yl≻yw|x),p^η(y_w y_l|x)=(1-η)p^*(y_w y_l|x)+η p^*(y_l y_w|x), (6) where η∈[0,12)η∈[0, 12) denotes the noise rate, while pηp^η and p∗p^* are the preference probabilities under the distributions ηP^η and ∗P^*, respectively. 4 Unbiased Alignment Since real-world preference datasets often contain noise, standard approaches such as RLHF and DPO may struggle to achieve ideal performance. To address this challenge, we investigate the theoretical foundations of noise model correction and propose principled methods for unbiased alignment. Detailed proofs are provided in the Appendix A. 4.1 Unbiased Reward Model Learning We first explore the reward model learning stage in RLHF. When preference data is contaminated with noise, the reward model exhibits a systematic bias, yielding a reward signal that is distorted by the underlying noise distribution. This relationship is characterized as follows: Lemma 4.1. Under noisy preferences, the normal reward model loss −(x,yw,yl)∼η[logσ(rϕ(x,yw)−rϕ(x,yl))]-E_(x,y_w,y_l) ^η\! [ σ\! (r_φ(x,y_w)-r_φ(x,y_l) ) ] is equivalent to the following noisy optimization loss: −(x,yw,yl)∼∗[(1−η)logσ(rϕ(x,yw)−rϕ(x,yl))+ηlogσ(rϕ(x,yl)−rϕ(x,yw))].-E_(x,y_w,y_l) ^* [(1-η)\, σ\! (r_φ(x,y_w)-r_φ(x,y_l) )\\ +η\, σ\! (r_φ(x,y_l)-r_φ(x,y_w) ) ]. (7) The corresponding optimal noisy reward model satisfies σ(rϕη(x,yw)−rϕη(x,yl))=(1−η)p∗(yw≻yl|x)+ηp∗(yl≻yw|x).σ\! (r_φ^η(x,y_w)-r_φ^η(x,y_l) )=\\ (1-η)\,p^*(y_w y_l|x)+η\,p^*(y_l y_w|x). (8) Noisy Reward Model Correction. To address the issue of noise, following Lemma 8, we demonstrate that unbiased reward signals can be derived from their noisy counterparts. Theorem 4.2. For any (x,yw,yl)∼∗(x,y_w,y_l) ^*, if rϕη(x,y)r_φ^η(x,y) is the optimal noisy reward model, then the optimal unbiased reward model rϕ∗(x,y)r_φ^*(x,y) satisfies: rϕ∗(x,yw)−rϕ∗(x,yl)=log[exp(rϕη(x,yw)−rϕη(x,yl))−a1−a⋅exp(rϕη(x,yw)−rϕη(x,yl))],r_φ^*(x,y_w)-r_φ^*(x,y_l)=\\ [ \! (r_φ^η(x,y_w)-r_φ^η(x,y_l) )-a1-a· (r_φ^η(x,y_w)-r_φ^η(x,y_l) ) ], (9) where a=η1−ηa= η1-η is a constant. It is evident that the unbiased reward signal rϕ∗(x,y)r_φ^*(x,y) can be recovered from its noisy counterpart rϕη(x,y)r_φ^η(x,y). For instance, in the case where the noisy reward model is unbiased (a=0a=0), the relationship reduces to rϕ∗(x,yw)−rϕ∗(x,yl)=rϕη(x,yw)−rϕη(x,yl)r_φ^*(x,y_w)-r_φ^*(x,y_l)=r_φ^η(x,y_w)-r_φ^η(x,y_l), signifying an absence of confounding. Conversely, when the noisy model is completely inverted relative to true preferences (as a→∞a→∞), we obtain rϕ∗(x,yw)−rϕ∗(x,yl)=rϕη(x,yl)−rϕη(x,yw)r_φ^*(x,y_w)-r_φ^*(x,y_l)=r_φ^η(x,y_l)-r_φ^η(x,y_w), representing a state of complete confounding. Unbiased Reward Model Loss. However, during the reinforcement learning phase, we utilize only the prompts x from the dataset. Since the static preference pairs (yw,yl)(y_w,y_l) are absent during this stage, we cannot directly apply Theorem 4.2 to recover the unbiased reward signal rϕ∗(x,y)r_φ^*(x,y) for policy guidance. To address this, we require a robust loss function that trains the unbiased reward model rϕ∗r_φ^* directly from the noisy preference dataset. By leveraging Theorem 4.2, we can establish a mathematical relationship between rηr^η and r∗r^*. Given that the normal RM loss ℒRML_RM optimizes the noisy reward model rηr^η, substituting rηr^η with its representation in terms of the unbiased r∗r^* allows us to optimize r∗r^* directly. The unbiased reward model loss is derived as follows: Corollary 4.3. Unbiased Reward Model (URM) loss: ℒURM=−logexp(rϕ(x,yw)−rϕ(x,yl))+aexp(rϕ(x,yw)−rϕ(x,yl))+1.L_URM=- \! (r_φ(x,y_w)-r_φ(x,y_l) )+a \! (r_φ(x,y_w)-r_φ(x,y_l) )+1. (10) Using the URM loss, we can directly train the unbiased reward model rϕ∗r_φ^* from the noisy preference ηP^η. Mathematically, a is defined as η1−η η1-η. However, because the true value of η is unobservable in practice, we treat a as a tunable hyperparameter during training. By adjusting the parameter a, we can directly train an unbiased reward model. Then, during the second stage of RL training, the interference from the noisy preference will no longer occur. Our method is orthogonal to specific RL algorithms and can be combined with any RL algorithm, such as PPO (Schulman et al., 2017) and GRPO (Shao et al., 2024). The specific RL algorithms are not the focus of this paper. 4.2 Unbiased Direct Preference Optimization DPO directly supervises policy model training using preference data, so noisy preferences can significantly impact model performance. Similar to the discussion of unbiased reward model learning, we first obtain the optimal solution for normal DPO in the presence of noisy preferences. Lemma 4.4. Under noisy preferences, the normal DPO loss −(x,yw,yl)∼ηlogσ(βlogπθ(yw|x)πref(yl|x)πθ(yl|x)πref(yw|x))-E_(x,y_w,y_l) ^η σ (β _θ(y_w|x) _ref(y_l|x) _θ(y_l|x) _ref(y_w|x) ) is equivalent to the following noisy optimization loss: (x,yw,yl)∼∗logσ(βlogπθ(yw|x)πref(yl|x)πθ(yl|x)πref(yw|x))η−1σ(βlogπθ(yl|x)πref(yw|x)πθ(yw|x)πref(yl|x))η.E_(x,y_w,y_l) ^* σ (β _θ(y_w|x) _ref(y_l|x) _θ(y_l|x) _ref(y_w|x) )^η-1σ (β _θ(y_l|x) _ref(y_w|x) _θ(y_w|x) _ref(y_l|x) )^η. (11) The corresponding optimal noisy policy model satisfies: σ(βlogπθη(yw|x)πref(yl|x)πθη(yl|x)πref(yw|x))=(1−η)p∗(yw≻yl|x)+ηp∗(yl≻yw|x).σ (β _θ^η(y_w|x) _ref(y_l|x) _θ^η(y_l|x) _ref(y_w|x) )=\\ (1-η)\,p^*(y_w y_l|x)+η\,p^*(y_l y_w|x). (12) Noisy DPO Model Correction. Based on Lemma 12, the noisy policy model πθη _θ^η obtained through DPO under noisy preferences can be corrected in the following: Theorem 4.5. For any (x,yw,yl)∼∗(x,y_w,y_l) ^*, if πθη(y|x) _θ^η(y|x) is the optimal noisy policy model, then the optimal unbiased policy model πθ∗(y|x) _θ^*(y|x) satisfies: (πθ∗(yw|x)πref(yl|x)πθ∗(yl|x)πref(yw|x))β=(πθη(yw|x)πref(yl|x)πθη(yl|x)πref(yw|x))β−a1−a⋅(πθη(yw|x)πref(yl|x)πθη(yl|x)πref(yw|x))β. ( _θ^*(y_w|x)\, _ref(y_l|x) _θ^*(y_l|x)\, _ref(y_w|x) )^β= ( _θ^η(y_w|x)\, _ref(y_l|x) _θ^η(y_l|x)\, _ref(y_w|x) )^β-a1-a· ( _θ^η(y_w|x)\, _ref(y_l|x) _θ^η(y_l|x)\, _ref(y_w|x) )^β. (13) where a=η1−ηa= η1-η is a constant. Consequently, the unbiased policy πθ∗(y|x) _θ^*(y|x) can be recovered from the noisy policy πθη(y|x) _θ^η(y|x). Notably, when a=0a=0, the policy ratios coincide: πθ∗(yw|x)πref(yl|x)πθ∗(yl|x)πref(yw|x)=πθη(yw|x)πref(yl|x)πθη(yl|x)πref(yw|x), _θ^*(y_w|x)\, _ref(y_l|x) _θ^*(y_l|x)\, _ref(y_w|x)= _θ^η(y_w|x)\, _ref(y_l|x) _θ^η(y_l|x)\, _ref(y_w|x), indicating that the noisy policy is unconfounded. In contrast, as a→∞a→∞, the relationship flips: πθ∗(yw|x)πref(yl|x)πθ∗(yl|x)πref(yw|x)=πθη(yl|x)πref(yw|x)πθη(yw|x)πref(yl|x), _θ^*(y_w|x)\, _ref(y_l|x) _θ^*(y_l|x)\, _ref(y_w|x)= _θ^η(y_l|x)\, _ref(y_w|x) _θ^η(y_w|x)\, _ref(y_l|x), corresponding to complete confounding, where the noisy policy effectively inverts the true human preferences. Unbiased DPO Loss. In practical training, ground-truth labels are often unavailable, so we cannot determine which response is truly preferred in a noisy dataset. As a result, Theorem 4.5 cannot be applied directly to policy optimization. Motivated by Theorem 4.5, we instead establish a relationship between the noisy policy πθη _θ^η and the unbiased policy πθ∗ _θ^*. Specifically, we can express πθη _θ^η in terms of πθ∗ _θ^* (the desired unbiased model) and substitute this equivalent into the objective. This yields an unbiased loss function for optimizing πθ∗ _θ^* under noisy preferences as follows: Corollary 4.6. Unbiased Direct Preference Optimization (UDPO) loss: ℒUDPO=−log(πθ(yw|x)πref(yl|x)πθ(yl|x)πref(yw|x))β+a(πθ(yw|x)πref(yl|x)πθ(yl|x)πref(yw|x))β+1.L_UDPO=- ( _θ(y_w|x) _ref(y_l|x) _θ(y_l|x) _ref(y_w|x) )^β+a ( _θ(y_w|x) _ref(y_l|x) _θ(y_l|x) _ref(y_w|x) )^\!β+1. (14) Using the UDPO loss, we can directly train the unbiased policy model πθ∗ _θ^* from the noisy preference ηP^η. By adjusting the parameter a, we can obtain an unbiased policy model using supervised preference training. 4.3 Further Theoretical Analysis In this subsection, we further analyze the properties of the proposed URM and UDPO losses. A Unified Framework. Although the URM and UDPO losses are derived independently from the normal RM and DPO losses, respectively, our resulting formulations show that they still preserve the fundamental transformation between explicit and implicit reward models in DPO and RLHF (Rafailov et al., 2023): rϕ(x,yw)−rϕ(x,yl)=βlogπθ(yw|x)πref(yw|x)−βlogπθ(yl|x)πref(yl|x)r_φ(x,y_w)-r_φ(x,y_l)=β _θ(y_w|x) _ref(y_w|x)-β _θ(y_l|x) _ref(y_l|x). For notational simplicity, we define the reward margin as Δ=rϕ(x,yw)−rϕ(x,yl)=βlogπθ(yw|x)πref(yw|x)−βlogπθ(yl|x)πref(yl|x) =r_φ(x,y_w)-r_φ(x,y_l)=β _θ(y_w|x) _ref(y_w|x)-β _θ(y_l|x) _ref(y_l|x). This shared representation enables us to analyze both the URM and UDPO losses within a unified framework by using the unbiased loss: ℒunbiased(Δ)=−logexp(Δ)+aexp(Δ)+1.L_unbiased( )=- ( )+a ( )+1. (15) Gradient Analysis. For the normal RM loss ℒRM=−logσ(rϕ(x,yw)−rϕ(x,yl))L_RM=- σ (r_φ(x,y_w)-r_φ(x,y_l) ) and the normal DPO loss ℒDPO=−logσ(βlogπθ(yw|x)πref(yw|x)−βlogπθ(yl|x)πref(yl|x))L_DPO=- σ (β _θ(y_w|x) _ref(y_w|x)-β _θ(y_l|x) _ref(y_l|x) ), we can also simplify them to ℒ(Δ)=−logσ(Δ)L( )=- σ( ). We have the gradient as follows: ∂ℒ(Δ)∂Δ=σ(Δ)−1. ( )∂ =σ( )-1. (16) The gradient magnitude |∂ℒ(Δ)∂Δ| | ( )∂ | approaches 0 as Δ≫0 0 and approaches 11 as Δ≪0 0. This means that samples that are “highly misranked” (i.e., with large negative Δ ) receive the largest optimization weight. However, noisy or mislabeled pairs often fall into this regime (Wei et al., 2023), making the model prone to overfitting spurious or inaccurate preferences. For the URM loss and the UDPO loss ℒunbiased(Δ)=−logexp(Δ)+aexp(Δ)+1L_unbiased( )=- ( )+a ( )+1, we have the gradient as follows: ∂ℒunbiased(Δ)∂Δ=exp(Δ)exp(Δ)+1−exp(Δ)exp(Δ)+a _unbiased( )∂ = ( ) ( )+1- ( ) ( )+a (17) Notably, the gradient magnitude |∂ℒunbiased(Δ)∂Δ|| _unbiased( )∂ | vanishes as Δ≪0 0, allowing the optimization process to effectively avoid fitting to noisy samples. Similarly, as Δ≫0 0, the gradient approaches 0 in a manner consistent with the normal loss function. The gradient reaches its maximum value of 1−a1+a 1- a1+ a at the intermediate point Δ=12loga = 12 a. This indicates that, during optimization, the unbiased loss primarily focuses on pairs that are difficult to distinguish. (a) URM with η=0.2η=0.2 (b) URM with η=0.4η=0.4 (c) UDPO with η=0.2η=0.2 (d) UDPO with η=0.4η=0.4 Figure 2: (a) & (b): Trained preference probability p(yw≻yl|x)p(y_w y_l|x) for reward model using URM loss with η∈0.2,0.4η∈\0.2,0.4\. (c) & (d): Trained policy probability π(yw|x)π(y_w|x) for policy model using UDPO loss with β=0.5β=0.5 and η∈0.2,0.4η∈\0.2,0.4\. Parameter downward Compatibility. In practical training, we set a=η^1−η^a= η1- η, with 0≤a<10≤ a<1. Our derivations show that the proposed loss is inherently robust when the true noise rate η∗η^* matches the estimate η η. Moreover, we prove that the loss remains an unbiased optimal solution when the data are cleaner than assumed, i.e., when η∗≤η^η^*≤ η. Corollary 4.7. Let a=η^1−η^a= η1- η, where 0≤a<10≤ a<1. If the true noise rate satisfies 0≤η∗≤η^0≤η^*≤ η, then the unbiased loss function ℒunbiasedL_unbiased attains an unbiased optimal solution. In practice, the exact noise rate of a dataset is often unknown. This downward compatibility property highlights the practical utility of our method: as long as a is set high to cover the potential noise, the model achieves strong results. In contrast, existing methods such as label smoothing (Mitchell, 2023) and rDPO (Chowdhury et al., 2024) lack this property; an inappropriate parameter choice in those methods can lead to performance that is inferior to standard RM or DPO loss. Classification Calibration and Excess Risk Bound. The alignment task can be formulated as a specific instance of binary classification. Let Y∈−1,+1Y∈\-1,+1\ denote the preference label. Here, Y=+1Y=+1 indicates (y1≻y2|x)(y_1 y_2|x), while Y=−1Y=-1 indicates (y1≺y2|x)(y_1 y_2|x). Given an input X=(x,y1,y2)X=(x,y_1,y_2), the model outputs the score difference f(X)=Δ=r(x,y1)−r(x,y2).f(X)= =r(x,y_1)-r(x,y_2). The classification margin is u=Yf(X)u=Yf(X). The margin-based form of the unbiased loss for classification is ℒunbiased(u)=−logexp(u)+aexp(u)+1L_unbiased(u)=- (u)+a (u)+1. Classification calibration (Bartlett et al., 2006), also known as Fisher consistency (Lin, 2004), is an important property ensuring that minimizing a surrogate loss can obtain the Bayes-optimal classifier in binary classification. Definition 4.8. A loss function ℒL is classification-calibrated if the classifier which minimizes this surrogate loss function is identical to the Bayes optimal classifier that minimizes the 0-1 loss (classification error). We demonstrate that our proposed loss satisfies this condition. Theorem 4.9. Unbiased loss function ℒunbiasedL_unbiased is classification-calibrated. Based on classification calibration, we establish a relationship between the excess risks with respect to the 0–1 loss and with respect to our unbiased loss. In particular, let R(f)R(f) denote the risk based on 0–1 loss, and let R∗=inffR(f)R^*= _fR(f) denote the Bayes risk. Similarly, let Rℒ(f)=(X,Y)ℒ(Yf(X))R_L(f)=E_(X,Y)L(Yf(X)) be called the ℒL-risk and let Rℒ∗=inffRℒ(f)R^*_L= _fR_L(f) denote the optimal ℒL-risk. Corollary 4.10. The 0-1 loss excess risk R(f)−R∗R(f)-R^* and the ℒunbiasedL_unbiased excess risk Rℒunbiased(f)−Rℒunbiased∗R_L_unbiased(f)-R_L_unbiased^* satisfy: ψ(R(f)−R∗)≤Rℒunbiased(f)−Rℒunbiased∗,ψ(R(f)-R^*)≤ R_L_unbiased(f)-R_L_unbiased^*, (18) where ψ:[0,1]→[0,∞)ψ:[0,1]→[0,∞) is a piecewise convex function: ψ(ρ)=log2−ℋbin(1+ρ2)if 0≤ρ<1−a1+alog21+a+1−ρ2logaif 1−a1+a≤ρ≤1,ψ(ρ)= cases 2-H_bin ( 1+ρ2 )&if 0≤ρ< 1-a1+a\\ 21+a+ 1-ρ2 a&if 1-a1+a≤ρ≤ 1 cases, (19) and ℋbin(p)=−plogp−(1−p)log(1−p)H_bin(p)=-p p-(1-p) (1-p) is the binary entropy function. This bound implies classification consistency: if a hypothesis f achieves the optimal surrogate risk, i.e., Rℒunbiased(f)=Rℒunbiased∗R_L_unbiased(f)=R^*_L_unbiased, then it must also achieve the Bayes-optimal classification risk, i.e., R(f)=R∗.R(f)=R^*. Furthermore, we analyze the behavior of ψ(ρ)ψ(ρ). For ℋbinH_bin, we perform a Taylor expansion around p=12p= 12: ℋbin(p)≈ℋbin(12)+ℋbin′(12)(p−12)+12ℋbin′(12)(p−12)2=log2−2(p−12)2H_bin(p) _bin( 12)+H _bin( 12)(p- 12)+ 12H _bin( 12)(p- 12)^2= 2-2(p- 12)^2. Substituting p=1+ρ2p= 1+ρ2, we obtain ℋbin(1+ρ2)≈log2−ρ22H_bin ( 1+ρ2 )≈ 2- ρ^22. Consequently, in the interval 0≤ρ<1−a1+a0≤ρ< 1-a1+a, ψ(ρ)≈ρ22ψ(ρ)≈ ρ^22, similar to Logistic loss or Quadratic loss. Conversely, in the interval 1−a1+a≤ρ≤1 1-a1+a≤ρ≤ 1, ψ(ρ)ψ(ρ) is linear with respect to ρ, similar to Hinge loss. This hybrid behavior suggests that the unbiased loss inherits desirable properties from both regimes: a quadratic or Logistic-like region that provides smooth, fine-grained optimization near the optimum, and a hinge-like linear region that can be more robust to high-error (potentially noisy or outlier) samples. As a result, the loss can mitigate the impact of noisy data while preserving sufficient fitting ability. Property Visualization. We conduct a synthetic experiment to validate our theories and illustrate the properties of our method. Specifically, we sample preferences from one triplet (x,yw,yl)(x,y_w,y_l) with noise rate η∈0.2,0.4η∈\0.2,0.4\. We train a reward model rϕr_φ using URM loss, setting a to 0 (normal RM loss), 0.20.8 0.20.8 (corresponding η=0.2η=0.2), and 0.40.6 0.40.6 (corresponding η=0.4η=0.4), respectively. Similarly, we apply the UDPO loss to train a policy model πθ _θ under these same values of a, while fixing β=0.5β=0.5 and employing a uniform distribution for πref _ref. The models are trained for 5000 steps using the Adam optimizer with a learning rate of 0.01 and a batch size of 4096. The results are visualized in Figure 2. It can be seen that the preference probability p(yw≻yl|x)=σ(rϕ(x,yw)−rϕ(x,yl))p(y_w y_l|x)=σ(r_φ(x,y_w)-r_φ(x,y_l)) obtained via the normal RM loss is significantly biased by the noise rate, converging toward the value of 1−η1-η. In contrast, our URM loss demonstrates both noise-tolerance and parameter downward compatibility. Specifically, when a=0.20.8a= 0.20.8, the model remains robust to the 0.2 noise rate; when a=0.40.6a= 0.40.6, it maintains robustness across both 0.2 and 0.4 noise rates. For the UDPO loss, the preference probability p(yw≻yl|x)=σ(βlogπθ(yw|x)πref(yl|x)πθ(yl|x)πref(yw|x))p(y_w y_l|x)=σ (β _θ(y_w|x) _ref(y_l|x) _θ(y_l|x) _ref(y_w|x) ) exhibits behavior identical to that of the URM loss. To further illustrate the specific characteristics of the UDPO loss, we report the policy probability πθ(yw|x) _θ(y_w|x), which likewise exhibits noise-tolerance and parameter downward compatibility. These visualizations remain consistent with our theoretical analysis. Normalized Unbiased Loss. Gradient analysis reveals that the maximum gradient magnitude of the unbiased loss, maxΔ|∂ℒunbiased(Δ)∂Δ|=1−a1+a _ | _unbiased( )∂ |= 1- a1+ a, decreases as a increases. This reduction may imply that, relative to the normal loss, optimizing the unbiased loss may require a larger learning rate to achieve comparable update magnitudes. To avoid additional learning rate adjustments, we can normalize the unbiased loss by a constant α=1+a1−aα= 1+ a1- a. With this scaling, the maximum gradient magnitude is preserved as maxΔ|∂ℒunbiased(Δ)∂Δ|=1 _ | _unbiased( )∂ |=1. We refer to the normalized variants as α-URM and α-UDPO. Since this modification is merely a constant rescaling of the objective, it does not alter the theoretical properties of URM and UDPO. Instance-Dependent Noise. Our theory can naturally extend to instance-dependent noise. Specifically, let ηx _x denote the noise rate for instance x, and define the corresponding constant axa_x as ηx1−ηx _x1- _x. Since the proofs of the noise correction process in Theorem 4.2 and Theorem 4.5 are derived pointwise, the instance-dependent versions follow directly by replacing the constant a with axa_x. Although the ηx _x of each instance is unknown, our parameter downward compatibility further implies that it suffices to use a single global parameter a a satisfying a^≥supxax a≥ _xa_x (equivalently, η^≥supxηx η≥ _x _x). In this case, the unbiased loss still recovers the unbiased optimal solution. Table 1: Win rates (%) of different reward model learning methods vs. the SFT model under different manual flip rates (0%, 20%, 40%). The top-2 best results are in bold. Models Methods H TL;DR Average 0% 20% 40% 0% 20% 40% Llama-3.2-3B RM 80.0 78.3 69.7 79.4 76.4 69.1 75.5 cRM 80.7 76.1 70.1 80.8 76.5 69.9 75.7 rRM 79.9 78.3 72.4 77.3 77.7 68.6 75.7 URM 81.4 78.9 72.6 82.8 83.2 77.5 79.4 α-URM 82.5 81.4 74.3 83.4 81.9 76.1 79.9 Qwen-3-1.7B RM 63.4 55.5 43.9 81.3 76.2 63.3 63.9 cRM 64.2 56.0 46.2 80.4 76.6 66.4 65.0 rRM 65.7 62.5 52.1 76.8 73.6 64.7 65.9 URM 67.6 61.8 58.2 81.5 80.0 71.6 70.1 α-URM 66.9 61.6 57.9 82.9 79.2 68.9 69.6 Table 2: Win rates (%) of different supervised preference learning methods vs the SFT model under different manual flip rates (0%, 20%, 40%). The top-2 best results are in bold. Models Methods H TL;DR Average 0% 20% 40% 0% 20% 40% Llama-3.2-3B DPO 85.4 80.4 69.8 72.2 64.6 56.1 71.4 cDPO 85.3 80.2 70.9 69.6 61.0 57.0 70.7 IPO 85.6 89.3 80.0 86.5 80.3 69.3 81.8 rDPO 88.0 89.3 84.4 85.9 83.5 77.3 84.7 Dr.DPO 88.2 88.2 81.6 86.8 84.3 82.9 85.0 UDPO 90.8 90.4 85.6 86.3 85.3 77.2 85.9 α-UDPO 90.4 89.8 85.7 87.2 86.4 78.2 86.1 Qwen-3-1.7B DPO 75.4 68.2 58.2 73.9 65.4 54.5 65.9 cDPO 72.8 70.1 59.3 67.2 60.5 51.9 63.6 IPO 71.7 68.7 63.3 82.1 78.9 67.1 72.0 rDPO 78.8 72.9 68.1 84.3 72.4 70.2 74.5 Dr.DPO 80.2 77.7 68.2 82.9 80.1 70.5 76.6 UDPO 83.4 81.2 73.0 84.6 83.0 71.3 79.4 α-UDPO 81.3 80.3 73.2 86.3 84.9 71.5 79.6 5 Experiments In this section, we validate the effectiveness of our proposed methods through comprehensive experiments, spanning reward model training and supervised preference training. Detailed experimental settings are provided in the Appendix B. Datasets. We evaluate our methods on three widely used real-world datasets, including the dialogue dataset Anthropic-H (helpful-base) (Bai et al., 2022), the summarization dataset Reddit TL;DR (Völske et al., 2017), and the comprehensive dataset UltraFeedback Binarized (UFB) (Cui et al., 2023). Baselines. We compare our method with several state-of-the-art approaches, with a particular focus on robustness-oriented methods. For reward model training, we consider the standard RM (Christiano et al., 2017), label smoothing (cRM), and rRM (Chowdhury et al., 2024). For supervised preference training, we consider DPO (Rafailov et al., 2023), label smoothing (cDPO), IPO (Azar et al., 2024), rDPO (Chowdhury et al., 2024), and Dr.DPO (Wu et al., 2025). We search for the optimal hyperparameter for each baseline to ensure a fair comparison. 5.1 Evaluations on H and TL;DR We first conduct evaluations on the dialogue dataset H and the summarization dataset TL;DR. Setup. We use the Llama-3.2-3B (Grattafiori et al., 2024) and Qwen-3-1.7B (Yang et al., 2025) as the base models. To obtain the SFT model, we fine-tune the base model exclusively on preferred completions similar to (Rafailov et al., 2023). For reward model evaluation, following (Nakano et al., 2021; Gao et al., 2023), we employ a best-of-n sampling strategy. Specifically, we let the SFT model generate n=20n=20 responses, and then use the reward model to select the highest-scoring response, which is compared with the default output of the SFT model. The best-of-n metric offers a more stable comparison of reward model quality. For supervised preference training evaluation, following (Rafailov et al., 2023), we compare the responses generated by the trained policy model against those of the SFT model. We employ GPT-5 (GPT-5-chat-latest) as the judge to calculate the win rate. Beyond the inherent noise in the H and TL;DR datasets, we also investigate scenarios with 20% and 40% manually flipped labels to simulate higher noise levels. Please note that the original dataset already contains noise; therefore, a manual flip rate of 0% does not imply no noise. Results. We report the win rates for reward model training and supervised preference learning in Table 1 and Table 2, respectively. For reward model learning (Table 1), our proposed URM and α-URM losses consistently outperform all baselines in the original setting (0% flip rate), indicating that they effectively mitigate intrinsic noise in real-world data. As we inject additional synthetic noise, our methods remain more robust. For example, with Llama-3.2-3B on TL;DR at a 40% flip rate, URM and α-URM improve win rates by approximately 8% over the baselines. Relative to the normal RM loss, cRM and rRM do not yield a meaningful improvement in average win rate. In contrast, across both Llama-3.2-3B and Qwen-3-1.7B, our methods exceed the SOTA baseline by roughly 4% in average win rate. For supervised preference learning (Table 2), our methods have achieved the best results in most cases. The average win rate of UDPO and α-UDPO exceeded that of normal DPO by around 15%. This substantial gap confirms that normal DPO is prone to overfitting noisy preferences, while our unbiased framework successfully recovers the unbiased policy. UDPO and α-UDPO consistently achieve the highest average win rates across all datasets, which further validates their effectiveness in both naturally noisy and highly corrupted environments. (a) UDPO (b) α-UDPO Figure 3: The ablation results of UDPO and α-UDPO using Llama-3.2-3B on H with different manual flip rates (0%, 20%, 40%). The best results in each case are highlighted with a red border. Ablation Experiment. It can be observed that the unbiased loss and the normalized unbiased loss achieve comparable strong performance. We further report the hyperparameter ablation results using UDPO and α-UDPO of Llama-3.2-3B on the H dataset, as shown in Figure 3. The results indicate that UDPO is more sensitive to hyperparameter choices, whereas the normalized α-UDPO consistently attains optimal performance across settings when using a larger a. This behavior is consistent with our parameter downward compatibility theory. 5.2 Evaluations on UFB. We conduct experiments using larger models on the more complex dataset UFB. Setup. We use the Llama-3.1-8B (Grattafiori et al., 2024) and Qwen-3-8B (Yang et al., 2025) as the base models. To obtain the SFT model, we fine-tune the base model on the instruct dataset Capybara (Daniele and Deeprasit, 2023). The assessment method is consistent with H and TL;DR. Results. We report the win rates for reward model training and supervised preference training in Table 3 and Table 4, respectively. For reward model learning (Table 3), our proposed URM and α-URM losses consistently outperform all baselines. Notably, on the Qwen-3-8B model, while the normal RM performs poorly with a 54.3% win rate, our URM and α-URM achieve significantly higher win rates of 64.6% and 64.9% respectively, surpassing the normal RM loss. For supervised preference learning (Table 4), UDPO and α-UDPO consistently achieve the best results. Our methods outperform the SOTA methods by approximately 2% on both models. These results on the complex UFB dataset with larger models further validate the scalability and robustness of our unbiased alignment framework. Table 3: Win rates (%) of different reward model learning methods vs. the SFT model on UFB. The top-2 best results are in bold. Methods UFB Llama-3.1-8B Qwen-3-8B RM 68.9 54.3 cRM 66.5 52.9 rRM 63.6 63.9 URM 72.5 64.6 α-URM 72.3 64.9 Table 4: Win rates (%) of different supervised preference learning methods vs the SFT model on UFB. The top-2 best results are in bold. Methods UFB Llama-3.1-8B Qwen-3-8B DPO 65.4 56.3 cDPO 63.1 55.1 IPO 72.7 57.8 rDPO 72.1 58.3 Dr.DPO 65.8 56.5 UDPO 74.3 60.6 α-UDPO 73.0 60.7 6 Conclusion This work introduces an unbiased alignment framework for LLM alignment with noisy preferences. By modeling preference corruption as a mathematical transition process and analytically inverting the distortion, we derive two practical objectives: the Unbiased Reward Model (URM) loss for reward learning in RLHF, and the Unbiased Direct Preference Optimization (UDPO) loss for supervised preference optimization. We further show that both objectives admit a unified margin-based form that preserves the fundamental RLHF-DPO equivalence, constituting a unified framework. Our theoretical analysis establishes noise-tolerance, parameter downward compatibility, and classification calibration. Extensive experiments demonstrate that our methods show consistent improvements over strong baselines. Overall, our work offers a concise, interpretable, and effective route to robust LLM alignment. Impact Statement This work presents a novel advance in LLM robust alignment, with positive implications for developing safe and reliable AI systems. This work is not expected to have a negative social impact. Acknowledgement This work was supported by National Natural Science Foundation of China under Grants 62525107. References M. G. Azar, Z. D. Guo, B. Piot, R. Munos, M. Rowland, M. Valko, and D. Calandriello (2024) A general theoretical paradigm to understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics, p. 4447–4455. Cited by: §2, §5. J. Bai, S. Bai, Y. Chu, Z. Cui, K. Dang, X. Deng, Y. Fan, W. Ge, Y. Han, F. Huang, et al. (2023) Qwen technical report. arXiv preprint arXiv:2309.16609. Cited by: §2. Y. Bai, A. Jones, K. Ndousse, A. Askell, A. Chen, N. DasSarma, D. Drain, S. Fort, D. Ganguli, T. Henighan, et al. (2022) Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862. Cited by: §5. P. L. Bartlett, M. I. Jordan, and J. D. McAuliffe (2006) Convexity, classification, and risk bounds. Journal of the American Statistical Association 101 (473), p. 138–156. Cited by: Appendix A, Appendix A, Appendix A, §4.3. R. A. Bradley and M. E. Terry (1952) Rank analysis of incomplete block designs: i. the method of paired comparisons. Biometrika 39 (3/4), p. 324–345. Cited by: §3. T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. (2020) Language models are few-shot learners. Advances in neural information processing systems 33, p. 1877–1901. Cited by: §2. S. R. Chowdhury, A. Kini, and N. Natarajan (2024) Provably robust dpo: aligning language models with noisy feedback. In International Conference on Machine Learning, p. 42258–42274. Cited by: §1, §1, §2, §4.3, §5. P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei (2017) Deep reinforcement learning from human preferences. Advances in neural information processing systems 30. Cited by: §1, §2, §2, §3, §5. G. Cui, L. Yuan, N. Ding, G. Yao, W. Zhu, Y. Ni, G. Xie, Z. Liu, and M. Sun (2023) UltraFeedback: boosting language models with high-quality feedback. External Links: 2310.01377 Cited by: §5. L. Daniele and S. Deeprasit (2023) Amplify-instruct: synthetically generated diverse multi-turn conversations for efficient llm training. arXiv Preprint. Retrieved from https://huggingface. co/datasets/LDJnr/Capybara. Cited by: §5.2. K. Ethayarajh, W. Xu, N. Muennighoff, D. Jurafsky, and D. Kiela (2024) Model alignment as prospect theoretic optimization. In Forty-first International Conference on Machine Learning, Cited by: §2. L. Gao, J. Schulman, and J. Hilton (2023) Scaling laws for reward model overoptimization. In International Conference on Machine Learning, p. 10835–10866. Cited by: §5.1. Y. Gao, D. Alon, and D. Metzler (2024) Impact of preference noise on the alignment performance of generative language models. In First Conference on Language Modeling, Cited by: §1. A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan, et al. (2024) The llama 3 herd of models. arXiv preprint arXiv:2407.21783. Cited by: §2, §5.1, §5.2. N. Jaques, S. Gu, D. Bahdanau, J. M. Hernández-Lobato, R. E. Turner, and D. Eck (2017) Sequence tutor: conservative fine-tuning of sequence generation models with kl-control. In International Conference on Machine Learning, p. 1645–1654. Cited by: §3. N. Jaques, J. H. Shen, A. Ghandeharioun, C. Ferguson, A. Lapedriza, N. Jones, S. Gu, and R. Picard (2020) Human-centric dialog training via offline reinforcement learning. In Proceedings of the 2020 conference on empirical methods in natural language processing (EMNLP), p. 3985–4003. Cited by: §3. Z. Li, X. Wu, H. Du, H. Nghiem, and G. Shi (2025) Benchmark evaluations, applications, and challenges of large vision language models: a survey. arXiv preprint arXiv:2501.02189 1. Cited by: §1. X. Liang, C. Chen, S. Qiu, J. Wang, Y. Wu, Z. Fu, H. Chen, F. Wu, and J. Ye (2025) ROPO: robust preference optimization for large language models. In Forty-second International Conference on Machine Learning, Cited by: §1, §2. Y. Lin (2004) A note on margin-based loss functions in classification. Statistics & probability letters 68 (1), p. 73–82. Cited by: §4.3. A. Liu, B. Feng, B. Xue, B. Wang, B. Wu, C. Lu, C. Zhao, C. Deng, C. Zhang, C. Ruan, et al. (2024) Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437. Cited by: §1. C. Lu, S. Holt, C. Fanconi, A. Chan, J. Foerster, M. van der Schaar, and R. Lange (2024) Discovering preference optimization algorithms with and for large language models. Advances in Neural Information Processing Systems 37, p. 86528–86573. Cited by: §2. Y. Meng, M. Xia, and D. Chen (2024) Simpo: simple preference optimization with a reference-free reward. Advances in Neural Information Processing Systems 37, p. 124198–124235. Cited by: §2. E. Mitchell (2023) A note on dpo with noisy preferences & relationship to ipo. Cited by: §1, §4.3. R. Nakano, J. Hilton, S. Balaji, J. Wu, L. Ouyang, C. Kim, C. Hesse, S. Jain, V. Kosaraju, W. Saunders, et al. (2021) Webgpt: browser-assisted question-answering with human feedback. arXiv preprint arXiv:2112.09332. Cited by: §5.1. H. Naveed, A. U. Khan, S. Qiu, M. Saqib, S. Anwar, M. Usman, N. Akhtar, N. Barnes, and A. Mian (2025) A comprehensive overview of large language models. ACM Transactions on Intelligent Systems and Technology 16 (5), p. 1–72. Cited by: §1. L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. (2022) Training language models to follow instructions with human feedback. Advances in neural information processing systems 35, p. 27730–27744. Cited by: §1, §1, §2, §2, §3. A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, et al. (2019) Pytorch: an imperative style, high-performance deep learning library. Advances in neural information processing systems 32. Cited by: §B.1. R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn (2023) Direct preference optimization: your language model is secretly a reward model. Advances in neural information processing systems 36, p. 53728–53741. Cited by: §B.1, §1, §1, §2, §2, §3, §3, §4.3, §5.1, §5. C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu (2020) Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research 21 (140), p. 1–67. Cited by: §2. J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov (2017) Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. Cited by: §2, §4.1. Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y. Li, Y. Wu, et al. (2024) Deepseekmath: pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300. Cited by: §2, §4.1. M. Völske, M. Potthast, S. Syed, and B. Stein (2017) Tl; dr: mining reddit to learn automatic summarization. In Proceedings of the Workshop on New Frontiers in Summarization, p. 59–63. Cited by: §5. L. von Werra, Y. Belkada, L. Tunstall, E. Beeching, T. Thrush, N. Lambert, S. Huang, K. Rasul, and Q. Gallouédec (2020) TRL: Transformers Reinforcement Learning Cited by: §B.1. Y. Wang, Y. Kordi, S. Mishra, A. Liu, N. A. Smith, D. Khashabi, and H. Hajishirzi (2023) Self-instruct: aligning language models with self-generated instructions. In Proceedings of the 61st annual meeting of the association for computational linguistics (volume 1: long papers), p. 13484–13508. Cited by: §2. H. Wei, H. Zhuang, R. Xie, L. Feng, G. Niu, B. An, and Y. Li (2023) Mitigating memorization of noisy labels by clipping the model prediction. In International Conference on Machine Learning, p. 36868–36886. Cited by: §4.3. J. Wu, Y. Xie, Z. Yang, J. Wu, J. Chen, J. Gao, B. Ding, X. Wang, and X. He (2025) Towards robust alignment of language models: distributionally robustifying direct preference optimization. In The Thirteenth International Conference on Learning Representations, Cited by: §1, §1, §2, §5. A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, et al. (2025) Qwen3 technical report. arXiv preprint arXiv:2505.09388. Cited by: §2, §5.1, §5.2. C. Zhang, S. Bengio, M. Hardt, B. Recht, and O. Vinyals (2017) Understanding deep learning requires rethinking generalization. In International Conference on Learning Representations, Cited by: §1. Y. Zhao, R. Joshi, T. Liu, M. Khalman, M. Saleh, and P. J. Liu (2023) Slic-hf: sequence likelihood calibration with human feedback. arXiv preprint arXiv:2305.10425. Cited by: §2. Appendix A Proofs Proof of Lemma 8 Proof. Let p^(yw≻yl|x)=σ(rϕ(x,yw)−rϕ(x,yl)) p(y_w y_l|x)=σ (r_φ(x,y_w)-r_φ(x,y_l) ), under the noisy preference, we have: ℒRM=−[(1−η)logσ(rϕ(x,yw)−rϕ(x,yl))+ηlogσ(rϕ(x,yl)−rϕ(x,yw))]=−[(1−η)logp^(yw≻yl|x)+ηlogp^(yl≻yw|x)]. splitL_ RM&=- [(1-η)\, σ\! (r_φ(x,y_w)-r_φ(x,y_l) )+η\, σ\! (r_φ(x,y_l)-r_φ(x,y_w) ) ]\\ &=- [(1-η) p(y_w y_l|x)+η p(y_l y_w|x) ]. split (20) Hence, for the gradient ∇ℒ∇p^(yw≻yl|x) ∇ p(y_w y_l|x), we have: ∇ℒ∇p^(yw≻yl|x)=−(1−η)1p^(yw≻yl|x)+η11−p^(yw≻yl|x). ∇ p(y_w y_l|x)=-(1-η) 1 p(y_w y_l|x)+η 11- p(y_w y_l|x). (21) Let ∇ℒ∇p^(yw≻yl|x)=0 ∇ p(y_w y_l|x)=0, because p∗(yw≻yl|x)=1p^*(y_w y_l|x)=1 and p∗(yl≻yw|x)=0p^*(y_l y_w|x)=0, we have the optimal solution p^(yw≻yl|x)=1−η=(1−η)p∗(yw≻yl|x)+ηp∗(yl≻yw|x) p(y_w y_l|x)=1-η=(1-η)p^*(y_w y_l|x)+η p^*(y_l y_w|x). ∎ Proof of Theorem 4.2 Proof. Referencing Lemma 8, we know the relationship between the noisy preference probability and the unbiased preference probability is: σ(rϕη(x,yw)−rϕη(x,yl))=(1−η)⋅σ(rϕ∗(x,yw)−rϕ∗(x,yl))+η⋅σ(rϕ∗(x,yl)−rϕ∗(x,yw))=(1−η)⋅σ(rϕ∗(x,yw)−rϕ∗(x,yl))+η⋅(1−σ(rϕ∗(x,yw)−rϕ∗(x,yl)))=(1−2η)⋅σ(rϕ∗(x,yw)−rϕ∗(x,yl))+η splitσ(r_φ^η(x,y_w)-r_φ^η(x,y_l))=&(1-η)·σ(r_φ^*(x,y_w)-r_φ^*(x,y_l))+η·σ(r_φ^*(x,y_l)-r_φ^*(x,y_w))\\ =&(1-η)·σ(r_φ^*(x,y_w)-r_φ^*(x,y_l))+η·(1-σ(r_φ^*(x,y_w)-r_φ^*(x,y_l)))\\ =&(1-2η)·σ(r_φ^*(x,y_w)-r_φ^*(x,y_l))+η split (22) To simplify the symbols, we set Δη=rϕη(x,yw)−rϕη(x,yl) ^η=r_φ^η(x,y_w)-r_φ^η(x,y_l) and Δ∗=rϕ∗(x,yw)−rϕ∗(x,yl) ^*=r_φ^*(x,y_w)-r_φ^*(x,y_l). Opening the sigmoid function σ, we have: exp(rϕη(x,yw))exp(rϕη(x,yw))+exp(rϕη(x,yl))=(1−2η)⋅exp(rϕ∗(x,yw))exp(rϕ∗(x,yw))+exp(rϕ∗(x,yl))+η⇒exp(Δη)exp(Δη)+1=(1−2η)⋅exp(Δ∗)exp(Δ∗)+1+η⇒exp(Δη)(exp(Δ∗)+1)=(exp(Δη)+1)[η+(1−η)exp(Δ∗)]⇒exp(Δ∗)[exp(Δη)−(1−η)exp(Δη)−(1−η)]=ηexp(Δη)+η−exp(Δη)⇒exp(Δ∗)=ηexp(Δη)+η−exp(Δη)exp(Δη)−(1−η)exp(Δη)−(1−η)⇒exp(Δ∗)=(η−1)exp(Δη)+ηexp(Δη)−(1−η)=exp(Δη)−η1−η1−η1−ηexp(Δη)⇒Δ∗=log[exp(Δη)−η1−η1−η1−ηexp(Δη)].⇒rϕ∗(x,yw)−rϕ∗(x,yl)=log[exp(rϕη(x,yw)−rϕη(x,yl))−a1−a⋅exp(rϕη(x,yw)−rϕη(x,yl))], split& (r_φ^η(x,y_w)) (r_φ^η(x,y_w))+ (r_φ^η(x,y_l))=(1-2η)· (r_φ^*(x,y_w)) (r_φ^*(x,y_w))+ (r_φ^*(x,y_l))+η\\ & ( ^η) ( ^η)+1=(1-2η)· ( ^*) ( ^*)+1+η\\ & ( ^η)( ( ^*)+1)=( ( ^η)+1)[η+(1-η) ( ^*)]\\ & ( ^*)[ ( ^η)-(1-η) ( ^η)-(1-η)]=η ( ^η)+η- ( ^η)\\ & ( ^*)= η ( ^η)+η- ( ^η) ( ^η)-(1-η) ( ^η)-(1-η)\\ & ( ^*)= (η-1) ( ^η)+η ( ^η)-(1-η)= ( ^η)- η1-η1- η1-η ( ^η)\\ & ^*= [ ( ^η)- η1-η1- η1-η ( ^η) ].\\ &r_φ^*(x,y_w)-r_φ^*(x,y_l)= [ \! (r_φ^η(x,y_w)-r_φ^η(x,y_l) )-a1-a· (r_φ^η(x,y_w)-r_φ^η(x,y_l) ) ], split (23) where a=η1−ηa= η1-η is a constant. ∎ Proof of Corollary 4.3 Proof. Refer to Theorem 4.2, we have : exp(Δ∗)=exp(Δη)−a1−a⋅exp(Δη)⇒exp(Δ∗)(1−a⋅exp(Δη))=exp(Δη)−a⇒exp(Δ∗)−a⋅exp(Δ∗)exp(Δη)=exp(Δη)−a⇒exp(Δη)+a⋅exp(Δ∗)exp(Δη)=exp(Δ∗)+a⇒exp(Δη)=exp(Δ∗)+a1+a⋅exp(Δ∗)⇒Δη=log[exp(Δ∗)+a1+a⋅exp(Δ∗)].⇒rϕη(x,yw)−rϕη(x,yl)=log[exp(rϕ∗(x,yw)−rϕ∗(x,yl))+a1+a⋅exp(rϕ∗(x,yw)−rϕ∗(x,yl))] split& ( ^*)= ( ^η)-a1-a· ( ^η)\\ & ( ^*)(1-a· ( ^η))= ( ^η)-a\\ & ( ^*)-a· ( ^*) ( ^η)= ( ^η)-a\\ & ( ^η)+a· ( ^*) ( ^η)= ( ^*)+a\\ & ( ^η)= ( ^*)+a1+a· ( ^*)\\ & ^η= [ ( ^*)+a1+a· ( ^*) ].\\ &r_φ^η(x,y_w)-r_φ^η(x,y_l)= [ (r_φ^*(x,y_w)-r_φ^*(x,y_l))+a1+a· (r_φ^*(x,y_w)-r_φ^*(x,y_l)) ] split (24) Under the noise preference, the normal reward model loss ℒRML_ RM optimize the noisy reward model rϕηr_φ^η, i.e. ℒRM=−logσ(rϕη(x,yw)−rϕη(x,yl))L_ RM=- σ(r_φ^η(x,y_w)-r_φ^η(x,y_l)). Therefore, by replacing rϕη(x,yw)−rϕη(x,yl)r_φ^η(x,y_w)-r_φ^η(x,y_l) as log[exp(rϕ∗(x,yw)−rϕ∗(x,yl))+a1+a⋅exp(rϕ∗(x,yw)−rϕ∗(x,yl))] [ (r_φ^*(x,y_w)-r_φ^*(x,y_l))+a1+a· (r_φ^*(x,y_w)-r_φ^*(x,y_l)) ], we can directly optimize the unbiased reward model rϕ∗r_φ^*. Our unbiased reward model loss as follows: ℒURM=−logσ(log[exp(rϕ∗(x,yw)−rϕ∗(x,yl))+a1+a⋅exp(rϕ∗(x,yw)−rϕ∗(x,yl)])=−log11+1+a⋅exp(Δ∗)exp(Δ∗)+a=−logexp(Δ∗)+aexp(Δ∗)+a+1+a⋅exp(Δ∗)=−logexp(Δ∗)+a(a+1)(exp(Δ∗)+1)=−[logexp(Δ∗)+aexp(Δ∗)+1−log(a+1)]. splitL_ URM&=- σ ( [ (r^*_φ(x,y_w)-r^*_φ(x,y_l))+a1+a· (r^*_φ(x,y_w)-r^*_φ(x,y_l) ] )\\ &=- 11+ 1+a· ( ^*) ( ^*)+a\\ &=- ( ^*)+a ( ^*)+a+1+a· ( ^*)\\ &=- ( ^*)+a(a+1)( ( ^*)+1)\\ &=-[ ( ^*)+a ( ^*)+1- (a+1)]. split (25) Since log(a+1) (a+1) is a constant, we can disregard it. Therefore, our final loss for training is ℒURM=−logexp(rϕ(x,yw)−rϕ(x,yl))+aexp(rϕ(x,yw)−rϕ(x,yl))+1L_ URM=- (r_φ(x,y_w)-r_φ(x,y_l))+a (r_φ(x,y_w)-r_φ(x,y_l))+1. The derivation of Equation 25 enables us to perform less operations of σ and log when calculating the loss. ∎ Proof of Lemma 12 Proof. Let p^(yw≻yl|x)=σ(βlogπθ(yw|x)πref(yl|x)πθ(yl|x)πref(yw|x)) p(y_w y_l|x)=σ (β _θ(y_w|x) _ref(y_l|x) _θ(y_l|x) _ref(y_w|x) ), under the noisy preference, we have: ℒDPO=−[(1−η)logσ(βlogπθ(yw|x)πref(yl|x)πθ(yl|x)πref(yw|x))+ηlogσ(βlogπθ(yl|x)πref(yw|x)πθ(yw|x)πref(yl|x))]=−[(1−η)logp^(yw≻yl|x)+ηlogp^(yl≻yw|x)]. splitL_ DPO&=- [(1-η) σ (β _θ(y_w|x) _ref(y_l|x) _θ(y_l|x) _ref(y_w|x) )+η σ (β _θ(y_l|x) _ref(y_w|x) _θ(y_w|x) _ref(y_l|x) ) ]\\ &=- [(1-η) p(y_w y_l|x)+η p(y_l y_w|x) ]. split (26) Hence, for the gradient ∇ℒ∇p^(yw≻yl|x) ∇ p(y_w y_l|x), we have: ∇ℒ∇p^(yw≻yl|x)=−(1−η)1p^(yw≻yl|x)+η11−p^(yw≻yl|x). ∇ p(y_w y_l|x)=-(1-η) 1 p(y_w y_l|x)+η 11- p(y_w y_l|x). (27) Let ∇ℒ∇p^(yw≻yl|x)=0 ∇ p(y_w y_l|x)=0, because p∗(yw≻yl|x)=1p^*(y_w y_l|x)=1 and p∗(yl≻yw|x)=0p^*(y_l y_w|x)=0, we have the optimal solution p^(yw≻yl|x)=1−η=(1−η)p∗(yw≻yl|x)+ηp∗(yl≻yw|x) p(y_w y_l|x)=1-η=(1-η)p^*(y_w y_l|x)+η p^*(y_l y_w|x). ∎ Proof of Theorem 4.5 Proof. Referencing Lemma 12, we know the relationship between the noisy preference probability and the unbiased preference probability is: σ(βlogπθη(yw|x)πref(yl|x)πθη(yl|x)πref(yw|x))=(1−η)σ(βlogπθ∗(yw|x)πref(yl|x)πθ∗(yl|x)πref(yw|x))+ησ(βlogπθ∗(yl|x)πref(yw|x)πθ∗(yw|x)πref(yl|x)) splitσ(β π^η_θ(y_w|x) _ref(y_l|x)π^η_θ(y_l|x) _ref(y_w|x))&=(1-η)σ(β π^*_θ(y_w|x) _ref(y_l|x)π^*_θ(y_l|x) _ref(y_w|x))+ησ(β π^*_θ(y_l|x) _ref(y_w|x)π^*_θ(y_w|x) _ref(y_l|x)) split (28) To simplify the symbols, we set Uη=(πθη(yw|x)πref(yl|x)πθη(yl|x)πref(yw|x))βU^η=( π^η_θ(y_w|x) _ref(y_l|x)π^η_θ(y_l|x) _ref(y_w|x))^β and U∗=(πθ∗(yw|x)πref(yl|x)πθ∗(yl|x)πref(yw|x))βU^*=( π^*_θ(y_w|x) _ref(y_l|x)π^*_θ(y_l|x) _ref(y_w|x))^β. Opening the sigmoid function σ, we have: exp(βlogπθη(yw|x)πref(yl|x)πθη(yl|x)πref(yw|x))1+exp(βlogπθη(yw|x)πref(yl|x)πθη(yl|x)πref(yw|x))=(1−η)exp(βlogπθ∗(yw|x)πref(yl|x)πθ∗(yl|x)πref(yw|x))1+exp(βlogπθ∗(yw|x)πref(yl|x)πθ∗(yl|x)πref(yw|x))+η(1−exp(βlogπθ∗(yw|x)πref(yl|x)πθ∗(yl|x)πref(yw|x))1+exp(βlogπθ∗(yw|x)πref(yl|x)πθ∗(yl|x)πref(yw|x)))⇒Uη1+Uη=(1−η)U∗1+U∗+η(1−U∗1+U∗)⇒Uη1+Uη=(1−η)U∗1+U∗+η11+U∗⇒Uη1+Uη=(1−η)U∗+η1+U∗⇒Uη(1+U∗)=(1+Uη)((1−η)U∗+η)⇒Uη+UηU∗=(1−η)U∗+η+(1−η)UηU∗+ηUη⇒UηU∗−(1−η)U∗−(1−η)UηU∗=η+ηUη−Uη⇒U∗(Uη−(1−η)−(1−η)Uη)=η−(1−η)Uη⇒U∗(ηUη−1+η)=η−(1−η)Uη⇒U∗=η−(1−η)UηUη−(1−η)=Uη−η1−η1−η1−ηUη⇒(πθ∗(yw|x)πref(yl|x)πθ∗(yl|x)πref(yw|x))β=(πθη(yw|x)πref(yl|x)πθη(yl|x)πref(yw|x))β−a1−a⋅(πθη(yw|x)πref(yl|x)πθη(yl|x)πref(yw|x))β, split (β π^η_θ(y_w|x) _ref(y_l|x)π^η_θ(y_l|x) _ref(y_w|x))1+ (β π^η_θ(y_w|x) _ref(y_l|x)π^η_θ(y_l|x) _ref(y_w|x))&=(1-η) (β π^*_θ(y_w|x) _ref(y_l|x)π^*_θ(y_l|x) _ref(y_w|x))1+ (β π^*_θ(y_w|x) _ref(y_l|x)π^*_θ(y_l|x) _ref(y_w|x))+η (1- (β π^*_θ(y_w|x) _ref(y_l|x)π^*_θ(y_l|x) _ref(y_w|x))1+ (β π^*_θ(y_w|x) _ref(y_l|x)π^*_θ(y_l|x) _ref(y_w|x)) )\\ & U^η1+U^η=(1-η) U^*1+U^*+η (1- U^*1+U^* )\\ & U^η1+U^η=(1-η) U^*1+U^*+η 11+U^*\\ & U^η1+U^η= (1-η)U^*+η1+U^*\\ &U^η(1+U^*)=(1+U^η)((1-η)U^*+η)\\ &U^η+U^ηU^*=(1-η)U^*+η+(1-η)U^ηU^*+η U^η\\ &U^ηU^*-(1-η)U^*-(1-η)U^ηU^*=η+η U^η-U^η\\ &U^*(U^η-(1-η)-(1-η)U^η)=η-(1-η)U^η\\ &U^*(η U^η-1+η)=η-(1-η)U^η\\ &U^*= η-(1-η)U^η U^η-(1-η)= U^η- η1-η1- η1-ηU^η\\ &( π^*_θ(y_w|x) _ref(y_l|x)π^*_θ(y_l|x) _ref(y_w|x))^β= ( π^η_θ(y_w|x) _ref(y_l|x)π^η_θ(y_l|x) _ref(y_w|x))^β-a1-a·( π^η_θ(y_w|x) _ref(y_l|x)π^η_θ(y_l|x) _ref(y_w|x))^β, split (29) where a=η1−ηa= η1-η is a constant. ∎ Proof of Corollary 4.6 Proof. Refer to Theorem 4.5, we have: (πθ∗(yw|x)πref(yl|x)πθ∗(yl|x)πref(yw|x))β=(πθη(yw|x)πref(yl|x)πθη(yl|x)πref(yw|x))β−a1−a⋅(πθη(yw|x)πref(yl|x)πθη(yl|x)πref(yw|x))β⇒U∗(1−aUη)=Uη−a⇒U∗−aU∗Uη=U∗−a⇒U∗+a=Uη(1+aU∗)⇒Uη=U∗+a1+aU∗⇒(πθη(yw|x)πref(yl|x)πθη(yl|x)πref(yw|x))β=(πθ∗(yw|x)πref(yl|x)πθ∗(yl|x)πref(yw|x))β+a1+a⋅(πθ∗(yw|x)πref(yl|x)πθ∗(yl|x)πref(yw|x))β split&( π^*_θ(y_w|x) _ref(y_l|x)π^*_θ(y_l|x) _ref(y_w|x))^β= ( π^η_θ(y_w|x) _ref(y_l|x)π^η_θ(y_l|x) _ref(y_w|x))^β-a1-a·( π^η_θ(y_w|x) _ref(y_l|x)π^η_θ(y_l|x) _ref(y_w|x))^β\\ &U^*(1-aU^η)=U^η-a\\ &U^*-aU^*U^η=U^*-a\\ &U^*+a=U^η(1+aU^*)\\ &U^η= U^*+a1+aU^*\\ &( π^η_θ(y_w|x) _ref(y_l|x)π^η_θ(y_l|x) _ref(y_w|x))^β= ( π^*_θ(y_w|x) _ref(y_l|x)π^*_θ(y_l|x) _ref(y_w|x))^β+a1+a·( π^*_θ(y_w|x) _ref(y_l|x)π^*_θ(y_l|x) _ref(y_w|x))^β split (30) The standard DPO loss optimizes the model likelihood against the preference dataset. Under noisy preferences, the standard DPO loss minimizes the negative log-likelihood of the noisy probability distribution, i.e., ℒDPO=−logσ(βlogπθη(yw|x)πref(yl|x)πθη(yl|x)πref(yw|x))=−log[σ(log(πθη(yw|x)πref(yl|x)πθη(yl|x)πref(yw|x))β)]L_DPO=- σ(β π^η_θ(y_w|x) _ref(y_l|x)π^η_θ(y_l|x) _ref(y_w|x))=- [σ( ( π^η_θ(y_w|x) _ref(y_l|x)π^η_θ(y_l|x) _ref(y_w|x))^β)] To obtain the unbiased loss, we substitute (πθη(yw|x)πref(yl|x)πθη(yl|x)πref(yw|x))β( π^η_θ(y_w|x) _ref(y_l|x)π^η_θ(y_l|x) _ref(y_w|x))^β with its expression in terms of (πθ∗(yw|x)πref(yl|x)πθ∗(yl|x)πref(yw|x))β( π^*_θ(y_w|x) _ref(y_l|x)π^*_θ(y_l|x) _ref(y_w|x))^β (the unbiased model we wish to train): ℒUDPO=−log[σ(log(πθ∗(yw|x)πref(yl|x)πθ∗(yl|x)πref(yw|x))β+a1+a⋅(πθ∗(yw|x)πref(yl|x)πθ∗(yl|x)πref(yw|x))β)]=−log(U∗+a1+aU∗1+U∗+a1+aU∗)=−log(U∗+a(1+aU∗)+(U∗+a))=−log(U∗+a(1+a)(1+U∗))=−log(U∗+aU∗+1)+log(1+a) splitL_UDPO&=- [σ( ( π^*_θ(y_w|x) _ref(y_l|x)π^*_θ(y_l|x) _ref(y_w|x))^β+a1+a·( π^*_θ(y_w|x) _ref(y_l|x)π^*_θ(y_l|x) _ref(y_w|x))^β)]\\ &=- ( U^*+a1+aU^*1+ U^*+a1+aU^* )\\ &=- ( U^*+a(1+aU^*)+(U^*+a) )\\ &=- ( U^*+a(1+a)(1+U^*) )\\ &=- ( U^*+aU^*+1 )+ (1+a) split (31) Since log(1+a) (1+a) is a constant with respect to θ, it can be discarded during optimization. Substituting the full expression for U∗=(πθ(yw|x)πref(yl|x)πθ(yl|x)πref(yw|x))βU^*= ( _θ(y_w|x) _ref(y_l|x) _θ(y_l|x) _ref(y_w|x) )^β. We arrive at the final objective function: ℒUDPO=−log(πθ(yw|x)πref(yl|x)πθ(yl|x)πref(yw|x))β+a(πθ(yw|x)πref(yl|x)πθ(yl|x)πref(yw|x))β+1L_UDPO=- ( _θ(y_w|x) _ref(y_l|x) _θ(y_l|x) _ref(y_w|x) )^β+a ( _θ(y_w|x) _ref(y_l|x) _θ(y_l|x) _ref(y_w|x) )^β+1. This formula allows for training the unbiased policy πθ∗ _θ^* directly using the noisy preference dataset. The derivation of Equation 31 enables us to perform less operations of σ and log when calculating the loss. ∎ Proof of Corollary 4.7 Proof. Based on Corollary 4.3 and Corollary 4.6 When η∗=η^η^*= η, it is clearly proof. We prove the case where η∗<η^η^*< η. Under the noisy preference distribution, the total unbiased loss is ℒη(Δ)=(1−η)∗ℒunbiased(Δ)+ηℒunbiased(−Δ)L_P^η( )=(1-η)*L_ unbiased( )+ _unbiased(- ), the gradient is: ∂ℒη(Δ)∂Δ=−(1−η∗)(1−a)exp(Δ)(exp(Δ)+a)(exp(Δ)+1)+η∗(1−a)exp(−Δ)(exp(−Δ)+a)(exp(−Δ)+1). _P^η( )∂ =-(1-η^*)(1-a) ( )( ( )+a)( ( )+1)+η^*(1-a) (- )( (- )+a)( (- )+1). (32) We need to prove that: when η^>η∗ η>η^*, for any Δ , the gradient ∂ℒη(Δ)∂Δ _P^η( )∂ is always less than 0. This implies that the objective function ℒη(Δ)L_P^η( ) is monotonically decreasing, and thus the optimal solution is achieved when Δ→+∞ →+∞. The term exp(−Δ)(exp(−Δ)+a)(exp(−Δ)+1)=exp(−Δ)exp(2Δ)[(exp(−Δ)+a)exp(Δ)][(exp(−Δ)+1)exp(Δ)]=exp(Δ)(1+aexp(Δ))(1+exp(Δ)) (- )( (- )+a)( (- )+1)= (- ) (2 )[( (- )+a) ( )][( (- )+1) ( )]= ( )(1+a ( ))(1+ ( )). Therefore, we have: Sign(∂ℒη∂Δ)=Sign(η∗exp(Δ)(1+aexp(Δ))(1+exp(Δ))−(1−η∗)exp(Δ)(exp(Δ)+a)(exp(Δ)+1)),Sign ( _P^η∂ )=Sign (η^* ( )(1+a ( ))(1+ ( ))-(1-η^*) ( )( ( )+a)( ( )+1) ), (33) where term (1−a)>0(1-a)>0 is overlooked. Eliminating the common positive terms exp(Δ)1+exp(Δ) ( )1+ ( ), we need to prove the following inequality: η∗1+aexp(Δ)<1−η∗exp(Δ)+a⇒η∗(exp(Δ)+a)<(1−η∗)(1+aexp(Δ))⇒η∗exp(Δ)+η∗a<1−η∗+aexp(Δ)−aη∗exp(Δ)⇒exp(Δ)(η∗−a(1−η∗))<1−η∗(1+a). split& η^*1+a ( )< 1-η^* ( )+a\\ &η^*( ( )+a)<(1-η^*)(1+a ( ))\\ &η^* ( )+η^*a<1-η^*+a ( )-aη^* ( )\\ & ( )(η^*-a(1-η^*))<1-η^*(1+a). split (34) We have η∗−a(1−η∗)=η∗−η^1−η^<0η^*-a(1-η^*)= η^*- η1- η<0 and 1−η∗(1+a)=1−η∗1−η^>01-η^*(1+a)=1- η^*1- η>0. Therefore, Inequality 34 always holds true. ∎ Proof of Theorem 4.9 Proof. Follow the definitions in (Bartlett et al., 2006), the generic conditional ℒL-risk is defined as Cη(u)=(1−η)ℒ(u)+ηℒ(−u)C_η(u)=(1-η)L(u)+ (-u). The optimal conditional ℒL-risk is defined as H(η)=infu∈ℝCη(u)H(η)= _u C_η(u). Furthermore, we define H−(η)=infu(1−2η)≤0Cη(u)H^-(η)= _u(1-2η)≤ 0C_η(u) as the optimal risk under the constraint that the sign of u is opposite to that of 1−2η1-2η. In this scenario, we consider 0≤η≤10≤η≤ 1 to ensure classification calibration while maintaining 0≤a<10≤ a<1. We have Cη(u)C_η(u) for ℒunbiased(u)=−logexp(u)+aexp(u)+1L_unbiased(u)=- (u)+a (u)+1 is: Cη(u) C_η(u) =(1−η)[log(1+exp(u))−log(a+exp(u))]+η[log(1+exp(−u))−log(a+exp(−u))] =(1-η)[ (1+ (u))- (a+ (u))]+η[ (1+ (-u))- (a+ (-u))] (35) =(1−η)log1+exp(u)a+exp(u)+ηlog1+exp(u)1+aexp(u) =(1-η) 1+ (u)a+ (u)+η 1+ (u)1+a (u) =log(1+exp(u))−(1−η)log(a+exp(u))−ηlog(1+aexp(u)) = (1+ (u))-(1-η) (a+ (u))-η (1+a (u)) Take the derivative of u to 0: ∂Cη(u)∂u=exp(u)1+exp(u)−(1−η)exp(u)a+exp(u)−ηaexp(u)1+aexp(u)=0 ∂ C_η(u)∂ u= (u)1+ (u)- (1-η) (u)a+ (u)- η a (u)1+a (u)=0 (36) Let z=exp(u)z= (u), and then rearrange the equation: 11+z=1−ηa+z+ηa1+az⇒(a+z)(1+az)=(1−η)(1+z)(1+az)+ηa(1+z)(a+z)⇒az2+(a2+1)z+a=az2+(a+1)[1+η(a−1)]z+(1−η+ηa2)⇒z⋅[−(1−a)(a−η(a+1))]=(1−a)[1−η(1+a)]⇒z=1−η(1+a)η(1+a)−a. split& 11+z= 1-ηa+z+ η a1+az\\ &(a+z)(1+az)=(1-η)(1+z)(1+az)+η a(1+z)(a+z)\\ &az^2+(a^2+1)z+a=az^2+(a+1)[1+η(a-1)]z+(1-η+η a^2)\\ &z· [-(1-a)(a-η(a+1)) ]=(1-a)[1-η(1+a)]\\ &z= 1-η(1+a)η(1+a)-a. split (37) In order to ensure that z>0z>0 (which means that the u has a real solution), we have a1+a<η<11+a a1+a<η< 11+a. If a1+a<η<11+a a1+a<η< 11+a, substituting the optimal solution z, we have: H(η)=η[log(1+z)−log(z+a)]+(1−η)[log(1+z)−log(1+az)]=ℋbin(η)−log(1+a), splitH(η)&=η[ (1+z)- (z+a)]+(1-η)[ (1+z)- (1+az)]\\ &=H_bin(η)- (1+a), split (38) where ℋbin(η)=−ηlogη−(1−η)log(1−η)H_bin(η)=-η η-(1-η) (1-η) is the binary entropy function. If η≤a1+aη≤ a1+a, the optimal solution is u→+∞u→+∞, so we have H(η)=−ηlogaH(η)=-η a. If η≥11+aη≥ 11+a, the optimal solution is u→−∞u→-∞, so we have H(η)=−(1−η)logaH(η)=-(1-η) a. For η≤a1+aη≤ a1+a and η≥11+aη≥ 11+a, since H(η)H(η) is symmetric with respect to η=1/2η=1/2, we can uniformly write it as: H(η)=−min(η,1−η)loga.H(η)=- (η,1-η) a. For H−(η)=infu(1−2η)≤0Cη(u)H^-(η)= _u(1-2η)≤ 0C_η(u), we have u≤0u≤ 0 when η<12η< 12, and u≥0u≥ 0 when η>12η> 12 Therefore, the constrained optimal solution must be achieved at the boundary u=0u=0. Thus, we have: H−(η)=Cη(0)=ηϕ(0)+(1−η)ϕ(0)=ϕ(0)=log21+a.H^-(η)=C_η(0)=ηφ(0)+(1-η)φ(0)=φ(0)= 21+a. (39) Based on (Bartlett et al., 2006), A loss function is classification-calibrated if and only if for any η≠1/2η≠ 1/2, it holds that H−(η)>H(η)H^-(η)>H(η). In the middle region (a1+a<η<11+a a1+a<η< 11+a), the inequality H−(η)>H(η)H^-(η)>H(η) becomes: log2−log(1+a)>ℋbin(η)−log(1+a)⇔log2>ℋbin(η). 2- (1+a)>H_bin(η)- (1+a) 2>H_bin(η). (40) The binary entropy function ℋbin(η)H_bin(η) attains its maximum value of log2 2 at η=1/2η=1/2. Therefore, for η≠1/2η≠ 1/2, the inequality holds strictly. In the boundary regions (η≤a1+aη≤ a1+a and η≥11+aη≥ 11+a), we observe that the boundary values are symmetric. Specifically, when η=a1+aη= a1+a or η=11+aη= 11+a, the function reaches a maximum of:H(η)=−a1+alogaH(η)=- a1+a a. We must prove the following inequality: log21+a>−a1+aloga. 21+a>- a1+a a. (41) Define the auxiliary function g(a)≔log21+a+a1+alogag(a) 21+a+ a1+a a. Our objective is to show that g(a)>0g(a)>0 for a∈(0,1)a∈(0,1). We have g′(a)=loga(1+a)2g (a)= a(1+a)^2. Since 0<a<10<a<1, it follows that loga<0 a<0, and consequently g′(a)<0g (a)<0. This demonstrates that g(a)g(a) is strictly monotonically decreasing on the interval (0,1)(0,1). Finally, we check the boundary condition at a=1a=1: g(1)=log22+12log1=0g(1)= 22+ 12 1=0. Because a<1a<1, we have g(a)>0g(a)>0. ∎ Proof of Corollary 4.10 Proof. Define ψ~(ρ)=H−(1+ρ2)−H(1+ρ2) ψ(ρ)=H^-( 1+ρ2)-H( 1+ρ2), where ρ∈[0,1]ρ∈[0,1]. Since H(η)H(η) is defined piecewise, ψ~(ρ) ψ(ρ) is also piecewise. Since 1+ρ2≥12 1+ρ2≥ 12, we identify the transition point ρ0 _0 by setting 1+ρ02=11+a 1+ _02= 11+a, which yields ρ0=1−a1+a _0= 1-a1+a. For 0≤ρ<1−a1+a0≤ρ< 1-a1+a, we have: ψ~(ρ)=(log2−log(1+a))−(ℋbin(η)−log(1+a))=log2−ℋbin(1+ρ2). ψ(ρ)=( 2- (1+a))-(H_bin(η)- (1+a))= 2-H_bin ( 1+ρ2 ). (42) For 1−a1+a≤ρ≤1 1-a1+a≤ρ≤ 1, we have: ψ~(ρ)=log21+a−(−1−ρ2loga)=log21+a+1−ρ2loga. ψ(ρ)= 21+a- (- 1-ρ2 a )= 21+a+ 1-ρ2 a. (43) Convexity analysis: the first part log2−ℋbin(1+ρ2) 2-H_bin( 1+ρ2) is convex. The second part is linear (and also convex). At ρ0=1−a1+a _0= 1-a1+a, the function values and first derivatives of the two parts are equal (continuous and smooth). Therefore, ψ~(ρ) ψ(ρ) is a convex function over the domain [0,1][0,1]. We define ψ as the convex hull of ψ~ ψ; thus, ψ=ψ~ψ= ψ. According to Theorem 1 in (Bartlett et al., 2006), we have: ψ(R(f)−R∗)≤Rℒunbiased(f)−Rℒunbiased∗.ψ(R(f)-R^*)≤ R_L_unbiased(f)-R_L_unbiased^*. (44) ∎ Appendix B Experiments B.1 Experiment Details. Experiment Setting. All experiments are based on Pytorch (Paszke et al., 2019) and TRL (von Werra et al., 2020) libraries, using 8 NVIDIA Pro 6000 (96GB) GPUs. For all the training, we use the AdamW optimizer, warmup ratio 0.1, batch size 128, maximum gradient norm 10. For SFT model training, we train the model for 1 epoch with learning rate 2e-5. For reward model training, we train the model for 3 epochs with learning rate 1e-5. For supervised preference training, we train the policy for 3 epochs with learning rate 5e-6 for Llama-3.2-3B and Qwen-3-1.7B, and 1e-6 for Llama-3.1-8B and Qwen-3-8B. For each dataset, we use the first 1000 different test samples as the test set. Baseline Hyperparameters. For the regularization parameter β, following (Rafailov et al., 2023), we set it to 0.1 for the dialogue datasets H and UFB, and 0.5 for the summarization dataset TL;DR. We conduct a hyperparameter search for each baseline on all noise level cases to ensure a completely fair experiment. For reward model training, we search ϵ∈0.1,0.2,0.4ε∈\0.1,0.2,0.4\ for cRM and rRM, and a∈0.4,0.6,0.8a∈\0.4,0.6,0.8\ for URM and α-URM. For supervised preference training, we search ϵ∈0.1,0.2,0.4ε∈\0.1,0.2,0.4\ for cDPO and rDPO, β′∈0.5,1,2β ∈\0.5,1,2\ for Dr.DPO, and a∈0.4,0.6,0.8a∈\0.4,0.6,0.8\ for UDPO and α-UDPO. For practitioners, simply using α-URM/α-UDPO with a=0.8a=0.8 is usually sufficient to achieve strong performance. B.2 Additional Experiments Human Evaluation. We conduct human evaluation on the case of Qwen-3-1.7B and H with a 40% flip rate. We use the first 100 samples from the test set, and the human evaluation is independently performed by three annotators. The results are reported in Table 5. As shown, the human evaluation results are consistent with those obtained from GPT-5. Table 5: Win rates (%) of different methods vs. the SFT model using GPT-5 and human judges. The top-2 best results are in bold. H 40% GPT-5 Human DPO 62.0 62.6 ± 2.8 UDPO 71.0 67.6 ± 2.0 α-UDPO 68.0 69.0 ± 2.9 Instance-Dependent Noise. We conduct experiments for instance-dependent noise. We first use a trained standard RM to obtain normalized reward scores for each sample in the H dataset. For each sample, we define ηx _x as minr(x,yw),r(x,yl)r(x,yw)+r(x,yl). \r(x,y_w),r(x,y_l)\r(x,y_w)+r(x,y_l). Intuitively, answer pairs with more similar reward scores are more likely to be mislabeled. We then flip labels according to ηx _x, with the noise rates rescaled to average corruption levels of 20% and 40%. The results using Qwen-3-1.7B are reported in Table 6. The results highlight the excellent performance of our methods under instance-dependent noise. Table 6: Win rates (%) of different methods vs. the SFT model under instance-dependent noise. The top-2 best results are in bold. H IDN 20% IDN 40% RM 56.8 46.0 rRM 58.0 48.2 URM 59.5 54.6 α-URM 60.2 55.2 H IDN 20% IDN 40% DPO 68.7 56.4 rDPO 73.3 63.7 UDPO 77.9 68.0 α-UDPO 78.2 67.6 Closed Loop RLHF. We evaluate the effectiveness of URM within the full RLHF loop. We use GRPO as the RL algorithm and conduct experiments using Qwen-3-1.7B on the H dataset. We set the number of generations to 2 and train for 3 epochs. The results are reported in Table 7. These results demonstrate that our methods significantly improve the final RL policy performance in the closed loop RLHF. Table 7: Win rates (%) of different methods using GRPO vs. the SFT model. The top-2 best results are in bold. GRPO H 0% H 20% H 40% RM 55.0 54.0 48.5 URM 60.3 58.7 56.5 α-URM 61.6 58.2 57.0 Larger Model. We conduct an experiment using Qwen-3-14B on the UFB dataset, comparing our UDPO/α-UDPO against standard DPO. The results are reported in Table 8. It can be observed that our methods remain highly effective on the larger model. Table 8: Win rates (%) of different methods vs. the SFT model. The top-2 best results are in bold. UFB Qwen-3-14B DPO 65.7 UDPO 73.0 α-UDPO 73.8